Compare commits

..

134 Commits

Author SHA1 Message Date
Patrick Buckley c930078f3d chore: bump version to 1.5.9 2026-05-07 22:56:06 -07:00
Patrick Buckley f148c4b423 fix: apply repair=False to all display-read load_messages call sites 2026-05-07 22:48:13 -07:00
Patrick Buckley 8ce4c8e737 chore: bump version to 1.5.8 2026-05-07 17:36:12 -07:00
Patrick Buckley 95e67dc768 fix(replay): apply PR #488 review findings
Four Copilot findings on c6041c6 — all confirmed valid, all bounded
to authenticated-user prompt-injection scenarios but worth closing
before merge.

Wrapper-detect bypass (string + list branches of
``_apply_reminders_for_provider``):

The round-2 fix used ``content.startswith("<tool_output>\\n")`` to
detect already-wrapped content and skip ``escape_wrapper_tags``.  A
tool whose RAW output starts with that prefix (e.g. ``echo
'<tool_output>'``) would match and have its escape skipped, letting
literal ``<tool_output>`` / ``<system-reminder>`` tags reach the model
and impersonate a system envelope.  Replace the prefix check with
``extract_advisories_from_tool_envelope(content) is not None`` —
parsing requires the open AND matching close tags AND a structurally
valid envelope, raising the bypass bar significantly.

Mirror fix in the list-content branch so a tool emitting an unmatched
envelope as a text part can't bypass the per-text-part escape.

``_build_history`` legitimate-envelope drop:

The list-content drop path previously removed any text part starting
with ``<tool_output>\\n``.  A tool that legitimately outputs a
well-formed envelope (documentation viewer, code analyzer demoing the
wrapper, an echo tool) would have that part silently disappear on
replay.  Tighten the drop heuristic to require BOTH ``cleaned_text ==
""`` AND at least one extracted advisory — the structural signature of
the injected ``wrap_tool_result("", advisories)`` carrier we produce
in ``session.py`` for list-typed tool output.  A legitimate envelope
has non-empty inner body or no advisory blocks and survives the
projection.

Empty advisory body:

``queue_message`` accepts any non-None text including ``""`` and
whitespace-only strings.  ``_classify_advisory`` would return a
``user_interjection`` advisory with empty / whitespace body, which
``replayAdvisoriesAfterTool`` then renders as a featureless empty user
bubble.  Filter empty / whitespace-only bodies at classification time
so the wire-shape contract is uniform: no empty advisories ever ride
the wire.

Tests:

* ``test_apply_reminders_escapes_tool_output_starting_with_envelope_prefix``
  pins the structural-parser bypass close: a string starting with the
  envelope prefix but lacking a close tag still gets escaped.
* ``test_apply_reminders_escapes_list_text_part_with_unmatched_envelope_prefix``
  mirrors for the list-content branch.
* ``test_build_history_keeps_legitimate_envelope_text_part_with_body``
  pins that legitimate envelope output stays in the projected list.
* ``test_decorate_suppresses_empty_advisory_body`` and
  ``test_decorate_suppresses_whitespace_only_advisory_body`` pin the
  empty-body filter in ``_classify_advisory``.

Tests: 5923 passed, 3 deselected.  Lint + format + mypy clean.
(cherry picked from commit c2cb6a7ea5)
2026-05-07 17:35:23 -07:00
Patrick Buckley dc35cbc7bf fix(replay): seam 1 splice + storage symmetry for queued user messages
Reverses the seam-2-only design from the prior commits on this branch.
Queued user messages arriving DURING a tool batch (Seam 1) splice into
the last tool result's envelope as ``UserInterjection`` advisories via
``wrap_tool_result``.  Messages arriving BETWEEN turns (Seam 2) drain
as a single trailing user row via ``_flush_queued_messages`` with
``user_feedback`` (operator text alongside an approval, e.g. "y, use
full path") folded in as a prefix.  Cancel/exception drains (Seam 3)
keep the existing ``_flush_queued_messages()`` call unchanged.

Why all three seams:

* Strict-template providers (Mistral, Llama via vLLM with stock chat
  templates) reject role-alternation violations.  A literal ``user``
  row mid-tool-batch breaks ``assistant(tool_calls) → tool → ... →
  assistant``; back-to-back ``user → user`` rows on the wire also fail.
* The seam-2-only design produced back-to-back ``user`` whenever
  ``user_feedback`` and queued items both fired — bug-1 from the round-1
  review.  Folding ``user_feedback`` as a prefix to the queue-drain
  collapses the two into one row.
* During-batch arrivals couldn't ride seam 2 — the splice was the only
  way to deliver same-turn without violating role alternation.

Storage symmetry:

Tool DB rows now store the wrapped ``output`` (envelope + advisories)
unconditionally — ``self.messages[i]['content']`` and
``conversations.content`` match exactly.  List-typed output (image /
structured MCP results) uses ``wrap_tool_result(raw_joined_text,
advisories)`` at save time so the persisted string is anchored on
``<tool_output>\n`` for the replay parser.  ``TOOL_RESULT_STORAGE_CAP``
is removed entirely; tools are responsible for bounding their own
output, storage faithfully represents in-memory.  Removing the cap
also simplifies the parser — no truncated-envelope edge case.

Replay extraction:

``decorate_history_messages`` (REST ``/history``) and ``_build_history``
(SSE replay, resume, rewind, retry, post-load, rename re-replay) both
call the public ``extract_advisories_from_tool_envelope`` helper to
pull the envelope back into structured ``advisories`` for JS replay.
Both string content and list-typed content (image+queued-message
combo) covered.  JS renders extracted advisories as normal user
bubbles after the tool block via the shared ``replayAdvisoriesAfterTool``
helper in ``shared_static/utils.js``.

Wrapper-tag escape and provider splice:

``escape_wrapper_tags`` now encodes pre-existing ``&`` first using an
``&amp;`` sentinel so tool output containing literal entity strings
(documentation viewers, code analyzers, web scrapers returning entity-
encoded markup) round-trips correctly.  Both encode and decode helpers
short-circuit on absence of ``<`` / ``&``.

``_apply_reminders_for_provider`` detects already-wrapped content
(string body and list text-part) by ``startswith("<tool_output>\n")``
and skips re-escape so existing envelopes survive intact when a tool
message also carries ``_reminders`` (the queued-message + tool-error
co-occurrence case is now common).

``decorate_history_messages`` runs in ``asyncio.to_thread`` to keep
MB-scale string work off the event loop.

Other cleanup:

* ``_collect_advisories`` delegates the queue drain to a named helper
  ``_drain_queued_messages_to_advisories`` so the swap-and-clear pattern
  lives next to ``_flush_queued_messages``'s identical pattern and the
  side-effect is documented at the call site.
* Preamble strings + body marker for ``UserInterjection`` round-trip
  detection moved to module-level constants in ``tool_advisory.py``;
  imported by ``history_decoration.py`` so a producer-side rephrase
  can't silently desync the parser.
* ``_send_with_mocks`` ctxmgr extracted in ``test_session.py`` — the
  six new send-driven tests share an 8-deep ``patch.object`` block.
* ``replayAdvisoriesAfterTool`` shared helper in
  ``shared_static/utils.js``; ``app.js`` and ``coordinator.js`` both
  invoke it.
* Dead truncation-pill CSS removed (``.tool-output-truncated`` and
  ``.coord-tool-truncated``); the JS that added these elements went
  away with ``TOOL_RESULT_STORAGE_CAP``.
* Tautological tests (``TestBuildHistoryAdvisoryPropagation``)
  replaced with production-realistic round-trip tests built from
  ``wrap_tool_result(...)`` envelopes — REST and SSE-replay surfaces
  pinned to the same wire shape; full DB round-trip pinned end-to-end.

Negative-tested:

* Reverting the prefix-merge in ``_flush_queued_messages`` produces
  back-to-back ``user`` rows, breaking
  ``test_user_feedback_and_queued_coexistence_single_row_with_prefix``.
* Reverting the ``extract_advisories_from_tool_envelope`` call in
  ``_build_history``'s tool branch leaves the envelope verbatim in
  wire content, breaking the round-trip tests.
* Reverting the wrapper-detection in ``_apply_reminders_for_provider``
  entity-encodes the existing envelope's literal tags, breaking both
  the string-content and list-content envelope-preservation tests.
* Reverting the ``wrap_tool_result(raw_text, advisories)`` projection
  at the DB save site produces a string starting with the original
  raw text, breaking
  ``test_tool_db_row_round_trips_list_output_with_advisories``.

Tests: 5918 passed, 3 deselected.  Lint + format + mypy clean on
touched files.

(cherry picked from commit eca4bb79e4)
2026-05-07 17:35:23 -07:00
Patrick Buckley 79f4d0030d fix(replay): apply review findings q-2 through q-7
Round-1 ``/review`` apply-pass.  Drops stale ``UserInterjection``
references from comments and docstrings that no longer describe the
post-PR drain shape, asserts the two-stream invariant in the new
queued-message persistence test, and pins the ``content.trim()`` +
``renderAssistantToolBatch`` invariants on coord-side so a future
refactor can't silently regress the Qwen3 phantom-card fix or the
chronological-order render fix.

Deferred:

* **bug-1** (back-to-back ``user`` row when ``user_feedback`` from the
  approval-prompt UI callback coexists with a queued-message drain).
  Reachable on strict OpenAI-compatible local templates (Anthropic and
  Anthropic-via-merge-consecutive collapse fine; vLLM-hosted Mistral /
  Llama enforcing role alternation can reject).  The pre-PR splice
  guarded against this case by riding queued items inside the tool
  result envelope; that guard is what motivated the original
  UserInterjection design, so the fix lane needs a deliberate decision
  rather than a quick patch.  Sleeping on it.

* **q-1** (delete dead ``UserInterjection`` class + tests).  Held for
  the bug-1 decision — if the chosen fix is to resume the splice for
  the ``user_feedback``+queue coexistence case, the advisory shape
  stays load-bearing.  Class now carries a docstring note marking it
  retained-pending-decision so a passing reader doesn't grep for
  producers and assume it's actually dead.

Apply-pass content:

* ``q-2``: drop "queued user interjections" from the persistent-
  advisory parenthetical in ``send``'s tool-result loop comment;
  rewrite to point at ``_flush_queued_messages`` for the queue path.
* ``q-3``: ``__init__`` channel-routing comment loses "and
  ``UserInterjection``" — only ``GuardAdvisory`` remains.
* ``q-4``: ``_queue_tool_advisory`` docstring + the tool-error nudge
  comment lose the user-interjection mentions; the docstring also now
  describes the side-channel + ``_apply_reminders_for_provider``
  splice path (the actual mechanism).
* ``q-5``: ``AttachmentsNotQueueableError`` docstring rewritten to
  describe the post-PR ``_flush_queued_messages`` flow — the
  single-combined-turn ``\n\n``-join shape can't carry image / file
  blocks, and per-item separate user turns would expand the strict-
  template role-ordering surface that the post-batch drain already
  balances.
* ``q-6``: the new ``test_queued_message_persists_as_user_row_after_tool_batch``
  in ``test_session.py`` now asserts ``stream_idx == 2`` so a future
  regression where the post-batch flush runs but the send-loop short-
  circuits before the next iteration surfaces in CI rather than
  manual repro.
* ``q-7``: ``test_coordinator_page.py`` gets two new string-grep pins
  mirroring the existing ``test_app_js.py`` shape — ``content.trim()``
  on coord's assistant-replay branch and ``renderAssistantToolBatch``
  for the hoisted helper that orders content card before tool batch.

## Test plan

- [x] ``ruff check`` clean
- [x] ``mypy turnstone/`` clean (189 source files)
- [x] Affected test surface (``test_session.py`` +
  ``test_tool_advisory.py`` + ``test_app_js.py`` +
  ``test_coordinator_page.py``) — 240 passed

(cherry picked from commit a032e71ff3)
2026-05-07 17:35:23 -07:00
Patrick Buckley 14e504db1f fix(replay): coord render order + blank assistant cards + queued message persistence
Three independent rehydrate / replay regressions reported on long
multi-turn conversations after the pull-model wake stack landed.

**1. coord history replay rendered tool_calls above the assistant
narration that announced them.**

In ``coordinator.js``'s loadHistory loop, the ``role === "assistant"``
``tool_calls`` branch sat above the role switch — every assistant turn
with both narration AND tool dispatch produced ``[tool batch][content
card]`` in the DOM, even though chronological order is content first.
On a parallel fan-out (e.g. four ``close_workstream`` calls in one
turn) operators saw the assistant text "Let me close them out and
summarize" with NO tool batch between it and the next assistant
message — the four-row batch had been rendered above the announcing
text and was scrolled out of view.

Hoisted the ``tool_calls`` synthesis into a local
``renderAssistantToolBatch(m)``, called from inside the assistant
branch AFTER the content card.  Live SSE order (text → dispatch →
results) now matches replay order.

**2. Whitespace-only assistant content rendered as a blank card on
replay.**

Models with vLLM's ``--reasoning-parser`` (Qwen3 in production)
strip ``<think>…</think>`` and emit only the trailing ``"\n\n"`` as
``content`` before a tool call.  ``content_parts = ["\n\n"]`` saves
``content = "\n\n"`` to the conversations row.  Live the user only
sees ``.msg.reasoning`` (the thinking content) — the empty
``.msg.assistant`` card lives next to it but reads as a thin
divider.  On rehydrate the reasoning bubble is gone (not persisted)
and the empty assistant card is the only thing left, surfacing as
"blank cards where the assistant message was."

Both UIs now check ``content && content.trim()`` before rendering
the body — whitespace-only content skips the card entirely instead
of showing a phantom row.  Live render unchanged.

**3. Queued user messages disappeared on reconnect.**

PR #474 routed queued user messages into the tool-result envelope
via ``UserInterjection`` advisories — same-turn delivery, but no
persisted user row.  On page reload / cross-tab replay the
optimistic ``.msg-queued`` bubble vanished: there was no DB row to
rehydrate it.

Dropped the ``UserInterjection`` splice in ``_collect_advisories``;
the queue drains through ``_flush_queued_messages`` AFTER the tool
batch completes instead.  Sequence becomes
``assistant(tool_calls) → tool … tool → user(drained)``, which is
valid for Mistral and Anthropic strict role validators (the only
forbidden shape was user injected mid-batch BEFORE the tool result,
which this still avoids).  Persists a real user row → bubble survives
reconnect, and stays in the session's wire-side context window on
the next turn.

## Test plan

- [x] ``ruff check`` clean
- [x] ``mypy turnstone/`` clean (189 source files)
- [x] ``pytest -m "not live"`` — 5798 passed, 3 deselected
- [x] Updated ``test_collect_advisories_does_not_drain_queued_messages``
  (was pinning the old UserInterjection shape)
- [x] Added ``test_queued_message_persists_as_user_row_after_tool_batch``
  (drives ``send`` end-to-end with a queued message arriving during
  the tool batch; asserts the user row lands in self.messages AND
  hits ``save_message``)
- [x] Updated ``test_replay_history_renders_content_before_tool_block``
  to tolerate the new ``msg.content && msg.content.trim()`` guard
- [ ] Live browser pass on coord (close_workstream parallel fan-out
  rehydrates with the 4-row batch BETWEEN the announcing assistant
  text and the summary) and interactive (Qwen3 ``"\n\n"`` rows no
  longer paint blank cards on reload; queued bubble survives a tab
  refresh)

(cherry picked from commit c11692b327)
2026-05-07 17:35:23 -07:00
Patrick Buckley 0abe0cb77d fix(mcp): apply PR #489 review feedback + de-flake pool reuse 401 retry
PR #489 review feedback (Copilot + github-code-quality):
- closeSettingsPanel now closes nested revoke modal first on close-button
  path (Escape was already handled by the parent keydown trap deferring
  to the inner trap; missing-modal-on-close-button was an orphan-modal
  hazard).
- _refreshConsentBadge now updates the settings button's aria-label +
  title dynamically with the pending-consent count for screen readers
  (badge stays aria-hidden — the count is in the label).
- _MAX_INSUFFICIENT_SCOPE_REPORTED promoted to public
  MAX_INSUFFICIENT_SCOPE_REPORTED in mcp_http_parsers; drops cross-module
  private import in mcp_oauth's /start handler.
- Stale test comment in test_session_mcp_dispatch_error.py corrected:
  _exec_read_resource does not log with exc_info=True (bearer-leak
  invariant).
- Rejected the protocol-method ellipsis warning: rest of _protocol.py
  uses ... consistently per Protocol convention.

Lint:
- ruff format applied to test_mcp_pool_auth_integration.py and
  test_mcp_pool_auth_resource_integration.py (combined `with` grammar —
  pure formatting).

Flake fix — test_integration_pool_reuse_401_refresh_and_retry_succeeds
on Python 3.11 / resource-constrained CI:

Same cross-task scope hazard f6a3b66 fixed at the close side, surfacing
at the connect side. asyncio.wait_for at mcp_client.py:1206 wraps
streamablehttp_client.__aenter__ in a fresh asyncio.Task. That fresh
task enters anyio cancel scopes, completes, and dies. The eventual
stack.aclose() during eviction or auth_401 retry runs from a different
task and tries to exit scopes whose entering task is dead — anyio
raises RuntimeError, the wedged anyio state blocks the retry's stack
teardown + reconnect, and the call exceeds the 15s budget on slow
workers.

Fix: replace asyncio.wait_for with `async with asyncio.timeout(...)` so
the streamablehttp_client.__aenter__ runs in the dispatch task itself,
no fresh-task scope ownership. Aligns with invariant 18 (asyncio.timeout
not asyncio.wait_for for any SDK / AS / pool-loop await crossing anyio
scopes).

Static path (_connect_one) at lines 905 and 1000 deliberately retains
asyncio.wait_for — auth_type ∈ {none, static} is byte-identical
(invariant 1) and the narrow connect-once / no-eviction-then-reuse
pattern doesn't trigger the cross-task hazard. Anchor comments pin
both directions: a future migration there would break invariant 1; a
future revert at 1206 would re-introduce the flake.

The cited test is the symptom (non-deterministically times out under
load), not a structural gate (no deterministic asyncio.timeout
assertion exists). The comment block at line 1206 records this so a
maintainer who reverts and finds green on a fast machine doesn't
conclude the fix is unneeded.

Verified on Python 3.11.14 (/tmp/venv311) and 3.13.7 (.venv): ruff
format clean, ruff check clean, mypy clean. 368 unit tests + 30 pool
integration tests pass on both interpreters; the previously-flaky test
passed 20× in isolation on 3.11.

Multi-stage /review (4 finders × verify × dedupe): bug/security/perf
returned zero findings; quality returned 3 confirmed minor/nit items
all of which are applied here (q-1 anchor comments at 905+1000, q-2
symptom-vs-gate clarification at 1206, q-3 module-docstring sentence
in mcp_http_parsers).

(cherry picked from commit 4a3e3607be)
2026-05-07 17:35:23 -07:00
Patrick Buckley 610513398b feat(mcp): per-user MCP server consent UX (Phase 8)
Wires the structured-error envelopes produced by Phase 7b's pool
dispatcher (mcp_consent_required / mcp_insufficient_scope /
mcp_*_forbidden / mcp_token_undecryptable_key_unknown /
mcp_oauth_url_insecure) through to the user-facing dashboard, and
adds a per-user settings panel for managing MCP server consents.

Changes
- ``_dispatch_pool_sync`` and ``_dispatch_pool_resource_sync`` wrap
  structured-error string returns as ``RuntimeError(json_str)`` via
  ``_is_structured_error()`` so the session-layer ``except Exception``
  branch fires uniformly across tool / resource / prompt dispatchers
  (the prompt path's ``isinstance(result, str)`` shortcut works only
  because prompts return ``list[dict]`` on success). Without this,
  the consent UX silently does not render for tool / resource calls.
- ``_structured_error`` extended with an optional ``consent_url``
  field; ``_build_consent_url`` produces ``/v1/api/mcp/oauth/start``
  query strings (path-relative; the dashboard appends ``return_url``
  at click time). Wired to all 12 ``mcp_consent_required`` and the
  ``mcp_insufficient_scope`` emit sites.
- New endpoints ``GET /v1/api/mcp/oauth/connections`` and
  ``DELETE /v1/api/mcp/oauth/connections/{server_name}`` registered
  on both ``turnstone-server`` and ``turnstone-console``. The DELETE
  handler runs local delete + audit + 204 first, then schedules the
  RFC 7009 upstream revoke as a fire-and-forget ``asyncio.create_task``
  with strong-ref tracking via ``_revoke_upstream_tasks`` (mirrors
  the ``_pg_refresh_drain_tasks`` pattern). Soft cap of 256 concurrent
  in-flight revokes prevents pile-up under coordinated mass-revoke;
  the audit detail records ``upstream_revoke_outcome`` as
  ``scheduled | no_refresh_token | no_http_client | shed_by_cap``.
- ``ASMetadata`` extended with ``revocation_endpoint`` parsed from
  RFC 8414 metadata. ``revoke_token_at_as`` helper posts the form
  body under ``asyncio.timeout`` (not ``asyncio.wait_for``) and
  never raises; ``_attempt_upstream_revoke`` is wrapped in an outer
  ``try/except Exception`` so unhandled exceptions don't surface as
  ``Task exception was never retrieved``.
- ``/v1/api/mcp/oauth/start`` accepts an optional ``scopes=`` query
  param; tokens are validated against RFC 6749 §3.3 grammar via
  ``is_valid_scope_token`` (promoted to ``mcp_http_parsers``),
  capped at ``_MAX_INSUFFICIENT_SCOPE_REPORTED`` (32), and unioned
  with the configured server scopes for the step-up consent flow.
- Storage primitive ``list_mcp_user_token_metadata_by_user`` projects
  the metadata columns at the SQL boundary so ciphertext blobs never
  cross the wire on the settings-list path. New
  ``MCPUserTokenMetadataRow`` TypedDict in ``_protocol.py``;
  ``MCPTokenStore.list_user_token_metadata`` re-types to the existing
  ``MCPUserTokenMetadata`` shape.
- Dashboard renderer (``app.js``): ``tryParseMcpError`` detects the
  envelope shape on ``tool_result`` SSE events with ``is_error=True``
  and ``buildMcpErrorEmbed`` renders an action card mirroring the
  existing ``buildMediaEmbed`` pattern. Three categories: actionable
  (consent_required / insufficient_scope) with a ``Connect`` button
  that opens ``/v1/api/mcp/oauth/start`` in a popup with a scheme
  guard, forbidden (mcp_*_forbidden) with a static notice, operator
  (key-mismatch / url-insecure) with an operator-action notice.
- New gear button in the appbar opens an MCP-connections settings
  modal driven by ``loadMcpConnections`` / ``confirmRevokeMcp``
  (two-step revoke confirmation matching the existing delete-ws
  pattern). Pending-consent badge tracks unresolved consent prompts
  in this tab; cleared after the connections list returns. Console
  proxy collision-checked: the IIFE only prepends a node-id pill to
  ``header.firstChild``, so the right-anchored gear button is safe.

Bearer-leak invariant
- No ``exc_info=True`` on any new path that can carry a chained
  ``httpx.Request`` (revoke handler, dispatch sites, exec sites).
  The two pre-existing ``exc_info=True`` calls in
  ``_exec_read_resource`` / ``_exec_use_prompt`` were replaced with
  structured-field logs as a Phase 8 sibling fix.

Tests
- 440 pytest passes on both Python 3.13 (.venv) and 3.11
  (/tmp/venv311); ruff + mypy clean.
- 5 new test files: ``test_mcp_consent_url_sibling_audit`` (structural
  gate that every ``code="mcp_consent_required"`` / ``mcp_insufficient_scope``
  site carries ``consent_url=``), ``test_mcp_oauth_connections``,
  ``test_mcp_oauth_revoke``, ``test_mcp_token_store_metadata``,
  ``test_session_mcp_dispatch_error``.
- End-to-end regression coverage for the bug-1 sibling pattern:
  ``test_call_tool_sync_raises_on_structured_error_envelope``,
  ``test_read_resource_sync_raises_on_structured_error_envelope``,
  ``test_get_prompt_sync_raises_on_structured_error_envelope``, plus
  ``test_call_tool_sync_does_not_wrap_non_structured_string`` as the
  defensive gate (only ``mcp_*`` envelopes are wrapped).

Hard invariants honored
- Static path byte-identical for ``auth_type ∈ {none, static}``: the
  wrap fires only when the dispatcher returns a structured-mcp-error
  string, which only happens on the oauth_user pool path.
- ``asyncio.timeout`` (not ``asyncio.wait_for``) on every new
  AS / SDK / pool-loop await per Python 3.11 anyio cancel-scope
  hazard.
- Scope cap ``_MAX_INSUFFICIENT_SCOPE_REPORTED = 32`` enforced at
  every output / merge site.
- Cross-user isolation on the revoke endpoint: a non-owner DELETE
  returns 404 with the same body shape as a never-existed row;
  ``http_client_mock.post.assert_not_called()`` pins this in 3 tests.

Deferred (not Phase 8 blockers)
- perf-2 (``asyncio.gather`` parallelisation in revoke handler) —
  superseded by perf-1's fire-and-forget pattern.
- q-4 (prompt-path ``isinstance(str)`` vs sibling ``_is_structured_error``
  asymmetry) — already documented in the function docstring.
- q-9 (``_pendingConsentServers`` → ``_serversNeedingConsent``
  rename) — pure naming taste.

(cherry picked from commit 5a3f46a1fa)
2026-05-07 17:35:23 -07:00
Patrick Buckley 2d6519f9a8 fix(storage): sanitize NUL bytes on _source + _reminders columns
Apply sanitize_text() to the new _source and _reminders columns in
both save_message and save_messages_bulk on SQLite + PostgreSQL,
mirroring the existing pattern used for content and provider_data.

Producers (sanitize_payload on the watch dispatch path,
format_nudge constants on the standard nudge path) already strip
NUL bytes today so nothing in production reaches this clamp — but
the storage layer is opaque to those invariants, and PostgreSQL
TEXT columns reject NUL outright.  Without this clamp, a future
producer that forgets sanitize_payload (or hand-builds the column
string) hard-fails the chat-loop persist path on PostgreSQL.

Cost is negligible — sanitize_text early-exits on the common
no-NUL case via 'if value and "\x00" in value'.

Surfaced by Copilot's PR #486 review.

(cherry picked from commit fc8bd6ca33)
2026-05-07 17:35:23 -07:00
Patrick Buckley a99ce49311 revert(memory): drop dormant limit kwarg from load_messages
Closes round-2 review finding q-7 (nit).

The kwarg was added to close round-1 perf-2 cosmetically — the
storage backend's signature already accepted ``limit``, but the
single in-tree caller (``ChatSession.resume``) doesn't pass it and
other tail-load consumers go direct to ``storage.load_messages``.
Adding signature surface to mark a perf finding closed without an
actual consumer is API-surface bloat.

When a tail-load consumer is written (e.g. a heuristic in
``session.resume`` to skip ancient wake rows), the kwarg can come
back — at that point with a real caller driving the contract.

(cherry picked from commit 14af6f464e)
2026-05-07 17:35:22 -07:00
Patrick Buckley 46f3571c93 refactor(watch): rename _WATCH_REMINDER_OPTIONAL_KEYS public + hoist import
Closes round-2 review findings q-6 (nit) and perf-1 (nit).

* **q-6:** ``_WATCH_REMINDER_OPTIONAL_KEYS`` carried a leading
  underscore (Python's module-private convention) but was imported
  from two other modules — clearly a public contract between
  ``build_watch_reminder`` and its consumers
  (``ChatSession._dispatch`` + ``server._build_history``).  Drop the
  underscore so the import sites match the constant's documented
  cross-module role.

* **perf-1:** The dispatch closure imported the constant inside its
  body, paying ``IMPORT_NAME`` + ``IMPORT_FROM`` bytecode on every
  watch fire.  ``server.py`` already imports at module scope; hoist
  the same way in ``session.py``.  Microsecond savings per dispatch,
  but the in-closure form was just an oversight from the apply-pass.

(cherry picked from commit 668da26dce)
2026-05-07 17:35:22 -07:00
Patrick Buckley 53cabe7e20 fix(session): trim tombstone refs + WHAT-narration in apply-pass comments
Closes round-2 review findings q-1 (minor), q-3 (nit), q-4 (nit), q-5
(nit).

* **q-1:** Drop the ``post-migration 050`` clause from the fork-block
  comment — the apply-pass relocated rather than removed the
  tombstone-style temporal reference round-1 q-2 was supposed to fix.
  The bulk-row dict shape and ``_encode_reminders`` are
  self-explanatory; the WHY is pinned by
  ``test_fork_preserves_source_and_reminders``.

* **q-3:** Replace ``DOES persist now`` framing on the wake-row save
  comment with a present-tense invariant.  The ``now`` implies the
  reader knows the prior state, same family as the temporal
  tombstones.

* **q-4:** Trim the 12-line WHAT-narration block above the
  resume-time ``_reminders_delivered = True`` loop to two lines
  stating the WHY only.  The new regression test pins the contract.

* **q-5:** Reframe ``test_fork_preserves_source_and_reminders``
  docstring as a forward-looking invariant; drop the
  ``Dropping them was the original bug`` and ``post-migration 050``
  fix-narration.

Project convention: invariant statements, present tense; don't
reference the current task / fix / migration number.

(cherry picked from commit b120ee2fd7)
2026-05-07 17:35:22 -07:00
Patrick Buckley 1d9fd94e23 fix(session): byte-clamp REMINDER_TEXT_STORAGE_CAP + drop local-only doc citation
Closes round-2 review findings bug-1 (minor) and q-2 (minor).

* **bug-1:** ``_encode_reminders`` clamped each entry's ``text`` field
  with Python ``str`` slicing, which counts codepoints.  Multi-byte
  UTF-8 input (CJK, emoji) could land 4 bytes per character past the
  cap, defeating the row-width / FTS5-index protection by up to 4x.
  Switch to UTF-8 byte clamping with ``errors="ignore"`` on the
  decode boundary so a slice mid-codepoint drops the partial
  character cleanly.

* **q-2:** Both the constant block-comment and the ``_encode_reminders``
  docstring referenced ``docs/design/watch-card-ux-briefing.md`` —
  local-only per project convention (``feedback_no_design_doc_commits``)
  so the canonical repo reads as a dead reference.  The cap value
  stands by itself; the row-width / FTS5 WHY is enough.

(cherry picked from commit 779ec638a5)
2026-05-07 17:35:22 -07:00
Patrick Buckley eb89ddab1e fix(metacog): cleanup batch — share watch-key constant, sanitize metadata, drop tombstones
Closes round-1 review findings q-2 (minor), q-5 (minor), q-6 (nit), q-7
(nit), sec-1 (nit), perf-4 (nit).

* **q-5:** Export ``_WATCH_REMINDER_OPTIONAL_KEYS`` from
  ``turnstone/core/watch.py`` and import in the dispatch closure
  (session.py) and the replay filter (server.py:_build_history).  The
  three-place duplication of the literal tuple
  ``("watch_name", "command", "poll_count", "max_polls", "is_final")``
  is gone; future field adds touch one constant.

* **sec-1:** Run ``sanitize_payload`` over string-typed metadata fields
  (``watch_name`` / ``command``) before they enter the queue.  Today's
  consumers all use ``textContent``, but the asymmetry — sanitised
  ``text`` alongside unsanitised metadata — would survive forever in
  DB rows and resurface if a future consumer used a non-textContent
  sink (aria-label, copy-to-clipboard, markdown render).

* **q-7:** Drop the per-iteration ``isinstance(reminder, dict)`` from
  the dispatch closure's metadata comprehension.  By the time the
  block runs, ``text = reminder.get("text", "") if isinstance(...)``
  + the ``if not sanitized: return`` guard above already established
  ``reminder`` is a non-empty dict.

* **q-2:** Strip tombstone-style references — "post-#482", "post-#484",
  "Step 7 of the watch-card UX plan", "Post-Step-7 dispatch surface",
  and the brittle line-anchor "session.py:2685-2686" — across
  ``session.py``, ``test_session.py``, ``test_watch.py``,
  ``test_watch_dispatch.py``, ``test_watch_integration.py``.  Comment
  intent preserved; historical anchors gone.

* **q-6:** Drop the ``del source`` line in ``cli.py``'s
  ``on_user_reminder``; the parallel ``on_tool_reminder`` ignores
  ``tool_call_id`` without ``del`` and the comment alone is enough.

* **perf-4:** Document the SQLite ``render_as_batch=True`` recreate
  cost in migration 050's docstring — first deployment after upgrade
  copies the conversations table twice (one per ``add_column``).
  PostgreSQL is unaffected.

5734 non-live tests pass; ruff + mypy clean.

(cherry picked from commit 7e35050b68)
2026-05-07 17:35:22 -07:00
Patrick Buckley eb92e61755 fix(ui): wrap interactive reminder spans in .msg-body + exclude system-nudge from anchor lookup
Closes round-1 review findings q-3 + q-4 (minor, merged) and bug-3 + bug-4
(nit, merged).

* **q-3 + q-4:** The new ``.msg.user-reminder .msg-body { white-space:
  pre-wrap }`` rule was a no-op on the interactive UI because that
  frontend's ``_buildDefaultReminderBubble`` appended label + text spans
  directly to the outer ``.msg.user-reminder`` element with no
  ``.msg-body`` wrapper.  Coord rendered the same shape with a wrapper.
  The two implementations diverging on DOM structure also meant a
  shared-helper extraction was harder than necessary.  Reconciled by
  wrapping interactive's spans in ``.msg-body`` to match coord; the CSS
  rule now applies to both UIs and the shared-extraction follow-up to
  ``shared_static/cards.js`` is mechanical (deferred per the review
  report — out of scope for this commit).

* **bug-3 + bug-4:** The reminder anchor lookup ``.msg.user`` also
  matched ``.msg.user.system-nudge`` markers because the marker carries
  both classes.  A non-wake reminder fired between a wake marker and
  the next real user message would anchor below the wake marker rather
  than the previous real user message.  Edge case (``/history`` reload
  corrects), but the fix is mechanical: change the selector to
  ``.msg.user:not(.system-nudge)`` in both files.

(cherry picked from commit 869135d97a)
2026-05-07 17:35:22 -07:00
Patrick Buckley 0e2ea122eb fix(memory): wire limit kwarg through load_messages
Closes round-1 review finding perf-2 (minor).

Storage backends accept ``*, limit: int | None = None`` (see
:meth:`StorageBackend.load_messages` at storage/_protocol.py:146) but
the in-memory wrapper at memory.py:82-85 dropped the kwarg, so
callers that wanted to tail-load (e.g. ``session.resume`` against a
long-running coord with hundreds of wake rows + persisted reminder
JSON) were forced to pull every row through the wrapper anyway.

Wraparound is mechanical: signature widens, default leaves existing
callers unaffected.

(cherry picked from commit 885f6a9185)
2026-05-07 17:35:22 -07:00
Patrick Buckley eb9dd2402a fix(session): delete stale 'reminders stay in-memory' comment
Closes round-1 review finding q-1 (major).

The comment block above ``self._attach_pending_user_reminders(user_msg)``
asserted that reminders "stay in-memory only and don't persist across
reloads" — directly contradicted by the comment block immediately below
(at the save_message call site) that explains the new persistence
semantics, plus the actual code that now writes ``_source`` and
``_reminders`` to the conversations row.  Future readers hitting both
blocks would lose trust in the surrounding comments.

The lower block already documents the persistence contract, so the
upper block is just deleted rather than rewritten.

(cherry picked from commit 81502c962f)
2026-05-07 17:35:22 -07:00
Patrick Buckley 0c58910c4b fix(session): preserve _source/_reminders on fork + cap persisted reminder text
Closes round-1 review findings bug-2 (major), perf-1 (minor), perf-6 (nit).

* **bug-2:** ``ChatSession.resume(..., fork=True)``'s bulk-row builder
  silently dropped the ``_source`` and ``_reminders`` side-channel
  data the source workstream had persisted via ``_append_user_turn``.
  Both backends' ``save_messages_bulk`` already accept these keys
  (the columns exist post-migration 050) — the bulk builder just
  didn't supply them.  The fork's resumed transcript would then look
  like the assistant turn answered out of nowhere: every wake marker
  and every reminder bubble that survived to disk on the source got
  dropped on the fork.  New regression test
  ``test_fork_preserves_source_and_reminders`` pins the contract.

* **perf-6:** Extracts ``_encode_reminders(reminders) -> str | None``
  near ``_apply_reminders_for_provider`` so the user-turn save path,
  the tool-turn save path, and the new fork bulk builder share one
  encoder.  Eliminates the drift risk between three near-identical
  ``json.dumps(..., separators=(",", ":")) if X else None`` patterns.

* **perf-1:** The new helper clamps each entry's ``text`` field at
  ``REMINDER_TEXT_STORAGE_CAP = 8192`` characters before encoding so
  a single rogue producer (a watch streaming unbounded shell output,
  a corruption-class steering payload) can't blow the conversations
  row width or the FTS5 index.  The in-memory side-channel keeps the
  full body — only the persisted JSON is clamped.  Mirrors
  ``TOOL_RESULT_STORAGE_CAP`` on tool result rows.

5734 non-live tests pass; ruff + mypy clean.

(cherry picked from commit 91e7f2daca)
2026-05-07 17:35:22 -07:00
Patrick Buckley 2e393d76b4 fix(session): flag persisted reminders delivered on resume
Persisted ``_reminders`` survive ``load_messages`` but the in-memory
``_reminders_delivered`` flag does not (it's session-scoped — set by
``_mark_reminders_delivered`` after each successful provider stream,
never persisted alongside the JSON column).  Without a re-splice
guard at resume time, ``_apply_reminders_for_provider`` would walk
every loaded message, see ``_reminders`` set + the flag falsy, and
splice every historical ``<system-reminder>`` envelope onto the wire
on the very next user turn — leaking each reminder a second time, the
turn after it had already advised.

Mirror the post-stream hook in ``resume()``: every loaded message
that carries reminders has already been delivered (it survived to
disk), so flag it accordingly so ``_apply_reminders_for_provider``
short-circuits on the pass-through path.

Test pins the contract end-to-end — stage a workstream with a
persisted reminder, resume into a fresh session, append a live user
turn, run the wire transform, and assert the historical reminder
body does NOT land in the rendered output.

(cherry picked from commit f1466ca7e3)
2026-05-07 17:35:22 -07:00
Patrick Buckley dec175f176 feat(ui): structured watch-result card + system-nudge marker on replay
User-visible slice of the watch-card UX workstream — combines the
replay-path widening, both frontend renderers, the CSS, and the
cross-cutting Python tests.

server._build_history widens the reminder filter from {type, text} to
project on a known set of optional fields (watch_name, command,
poll_count, max_polls, is_final) and surfaces _source as
entry["source"] when set.  The known-key filter narrows the blast
radius if a future producer accidentally stuffs sensitive fields
into the dict.

SessionUIBase.on_user_reminder takes a new source: str | None kwarg
that rides on the SSE event when set.  _attach_pending_user_reminders
forwards user_msg["_source"] so non-originating tabs see the wake's
"system_nudge" tag and render the thin marker.  Protocol + cli + eval
implementations widen accordingly.

Frontend (coordinator.js + app.js — touched in lockstep per project
memory's "logic that lands in BOTH UIs must touch both files"):
* Branch on r.type === "watch_triggered" for a structured
  .msg.watch-result card with header / $ command / <pre> body /
  poll N/M [· final] footer.
* New addSystemNudgeMarker (interactive) + appendSystemNudgeMarker
  (coord) renders a thin .msg.user.system-nudge anchor for
  wake-driven reminders, both live (source === "system_nudge" on the
  SSE event) and replay (msg.source === "system_nudge").
* Default .msg.user-reminder rendering preserved for every other
  metacog nudge type.

CSS (shared_static/chat.css):
* New .msg.watch-result rules — full-width treatment, cyan accent,
  monospace body with word-break: break-word for mobile.
* New .msg.user.system-nudge rule — thin yellow marker.
* Bonus newline-collapse fix: .msg.user-reminder .msg-body now sets
  white-space: pre-wrap so multi-line shell output / bulleted lists
  stay readable inside the advisory bubble.

Plan reference: docs/design/watch-card-ux.md §4 Steps 9-12 + bonus
CSS §11 (Commit 4).

(cherry picked from commit 6ae6877acc)
2026-05-07 17:35:22 -07:00
Patrick Buckley 592433b46d feat(metacog): structured watch reminders carry watch metadata onto NudgeQueue
WatchRunner._dispatch_result now takes a structured reminder dict
produced by build_watch_reminder() — text matches format_watch_message
verbatim (so compaction / channel adapters / wire splice keep their
behaviour), and watch_name / command / poll_count / max_polls /
is_final ride alongside as queue-entry metadata.

The dispatch closure registered in ChatSession.set_watch_runner pulls
the optional fields out of the dict and passes them to enqueue via
the new metadata kwarg.  Drain seams already merge metadata into the
rendered reminder dict (Commit 2), so the SSE event for a watch fire
now carries the structured fields without further plumbing.

* turnstone/core/watch.py — new build_watch_reminder() helper, _poll_watch
  switches from format_watch_message + dispatch(str) to build_watch_reminder
  + dispatch(dict).  set_dispatch_fn / get_dispatch_fn / restore_fn
  signatures widen from Callable[[str, str], None] to
  Callable[[dict[str, Any], str], None].
* turnstone/core/session.py — dispatch closure builds the metadata dict
  via {k: reminder[k] for k in ("watch_name", "command", ...) if k in reminder}
  and passes it to nudge_queue.enqueue.
* tests/test_watch.py — new TestBuildWatchReminder class pinning the
  builder shape; existing dispatch_fn_registry / restore_fn tests
  updated to dict shape.
* tests/test_watch_dispatch.py — every dispatch(...) call updated to
  pass a structured reminder dict via _reminder() helper; new
  TestMetadataPropagation class pins the metadata-on-enqueue contract.
* tests/test_watch_integration.py — _dispatch_result calls updated to
  dict shape.

Plan reference: docs/design/watch-card-ux.md §4 Step 7 + Step 8 watch-test
subset (Commit 3).

(cherry picked from commit 13db19905a)
2026-05-07 17:35:22 -07:00
Patrick Buckley da5321eb88 refactor(metacog): widen NudgeQueue._Entry with optional metadata field
Producers (today only watch_triggered) can now attach a metadata dict
to a queued nudge so the rendered reminder dict on the user/tool side
carries fields beyond {type, text}.  Wire shape stays additive: the
SSE event picks up the optional fields when present, and producers
without metadata leave it None.

* _Entry grows from 4 fields to 5 — metadata: dict[str, Any] | None.
* enqueue accepts metadata=... as a kwarg.
* drain returns list[tuple[str, str, dict | None]] (was 2-tuples).
* pending stays narrow at (type, text) for legacy callers; new
  pending_with_metadata projects the third slot for tests that need
  to assert producer-specific fields.
* Three drain consumers in session.py — _collect_advisories,
  _attach_pending_user_reminders, deliver_wake_nudge_from_queue —
  unpack the new 3-tuple shape and merge metadata into each
  reminder dict.
* on_user_reminder / on_tool_reminder protocol signatures widen
  from list[dict[str, str]] to list[dict[str, Any]] across
  ChatSession.UI, SessionUIBase, CLI, eval harness.

Plan reference: docs/design/watch-card-ux.md §4 Step 6 + Step 8 _Entry
subset (Commit 2).

(cherry picked from commit 30b7e4dd24)
2026-05-07 17:35:22 -07:00
Patrick Buckley baa2214f96 feat(storage): persist _source + _reminders side-channels on conversations
Adds two TEXT-NULL columns to the conversations table so multi-tab /
multi-device replay sees the same metacognitive bubble shape the
originating tab saw live.  Until now, reminders lived only on the
in-memory ChatSession.messages dict, and the wake-driven empty user
turn was not persisted at all (skip at session.py:2685-2686) — a
second tab connecting via /history saw the assistant turn with no
preceding wake context, and missed every other tab's reminder
bubbles besides.

Single Alembic revision 050 (head was 049) adds:
  * conversations._source — today only "system_nudge" for wake rows
  * conversations._reminders — JSON-encoded reminder list

Both backends (sqlite + postgresql) thread the columns through
save_message / save_messages_bulk / load_messages.  reconstruct_messages
unpacks the row tuple as 9 elements (was 7), JSON-decoding _reminders
on the user AND tool branches with the same contextlib.suppress guard
the existing provider_data / tool_calls decode uses.  Tool-row
reminders ride the same column so tool_error / repeat replay shape
matches user-channel parity.

session.py:2685-2686 wake-row persist skip is dropped; _append_user_turn
JSON-encodes user_msg["_reminders"] and passes both source + reminders
to save_message.  The tool-message save site at session.py:3014-3020
mirrors with metacog_reminders.

Plan reference: docs/design/watch-card-ux.md §4 Steps 1-5 (Commit 1).

(cherry picked from commit f64c3e7b10)
2026-05-07 17:35:22 -07:00
Patrick Buckley 3b60a69e4f fix(console): atomic coord-subsystem commit + offload startup teardown
Address Copilot review feedback on PR #487:

1. **Atomic commit invariant**: ``_bootstrap_coord_subsystem`` previously
   stamped ``coord_mgr`` ~50 lines before the final ``coord_registry``
   commit, and started threads + subscriptions in between.  A concurrent
   dashboard request running through ``_require_coord_mgr`` during the
   runtime-bootstrap window could observe ``coord_mgr`` set with
   ``coord_registry`` still ``None`` and surface the misleading
   "Restart the console after adding a model definition" 503.

   Refactored to two phases: (a) build everything as locals, (b) start
   side-effects (StateWriter / observer / nudge watcher / child fan-out
   / cleanup thread), then atomic commit at the end with ``coord_mgr``
   stamped LAST.  The build-phase ``try/except`` rolls back any started
   side-effects from local handles before re-raising — no daemon thread
   or subscription leaks across retries, and ``app.state`` is never
   stamped on a partial failure.

2. **Class-attr cleanup symmetry**: ``_teardown_partial_coord_subsystem``
   now also clears ``ConsoleCoordinatorUI._coord_mgr`` /
   ``_collector`` / ``_console_metrics`` to match the lifespan shutdown
   path (server.py ~line 4629).  A failed bootstrap (or test teardown
   reuse) no longer leaks process-global pointers at a half-built
   subsystem.

3. **Lifespan startup offload**: the lifespan startup error path used
   to call ``_teardown_partial_coord_subsystem`` synchronously, which
   in turn calls ``StateWriter.shutdown(timeout=2.0)`` — a thread-join
   + sync DB writes that could block the event loop for up to 2s
   while the console is still coming up.  Wrapped the whole
   load-and-bootstrap in ``asyncio.to_thread`` via the new
   ``_load_and_bootstrap_coord_subsystem`` synchronous helper, so all
   blocking work (including any rollback) runs on a worker thread.
   Mirrors the pattern the regular lifespan shutdown (line ~4620) and
   the runtime CRUD-triggered path already use.

Tests:
- ``test_bootstrap_atomic_commit_no_partial_visibility``: a polling
  thread in tight loop watches ``coord_mgr`` / ``coord_registry``
  during a real bootstrap and asserts no observation has ``coord_mgr``
  set with ``coord_registry`` still ``None``.
- ``test_real_bootstrap_rolls_back_partial_state_on_side_effect_failure``:
  monkeypatches ``install_idle_nudge_watcher`` to raise mid-build,
  asserts ``app.state`` shows the clean fresh-install state and the
  builder-failure error string surfaces ``RuntimeError`` (not the
  stale "no models" boot-time message).

(cherry picked from commit c6b4dc26be)
2026-05-07 17:35:22 -07:00
Patrick Buckley 5d1213d3dc fix(console): bootstrap coord subsystem on first model add
A freshly-installed console with no model rows in the DB at boot
caught the ``ValueError`` from ``load_model_registry()`` in the
lifespan and skipped the entire coord subsystem build, leaving
``coord_mgr`` ``None``.  ``_refresh_coord_registry`` then bailed
out at ``existing is None`` rather than building the subsystem on
first model add — operators had to restart the console after
configuring their first model in the admin panel for the
"Coordinator subsystem not initialized" banner to clear.

Extract the lifespan's coord build into a reusable
``_bootstrap_coord_subsystem`` and add ``_maybe_bootstrap_coord_subsystem``
that runs as an ``asyncio.to_thread`` follow-on after every admin
model-CRUD endpoint (create/update/delete/reload).  The helper:

- fast-paths to a no-op when ``coord_mgr`` is already set;
- guards concurrent first-install attempts with
  ``_COORD_BOOTSTRAP_LOCK`` + double-checked re-test inside the lock;
- pre-computes config-derived integers BEFORE any thread starts so
  ``int(config_store.get(...))`` failures don't strand a started
  ``StateWriter`` daemon;
- stamps ``coord_state_writer`` to ``app.state`` immediately after
  ``.start()`` so the new ``_teardown_partial_coord_subsystem`` can
  shut it down on a partial failure (no thread leaks across retries);
- atomically commits ``coord_registry`` + clears
  ``coord_registry_error`` as the final step so callers can rely on
  the invariant ``coord_registry`` is set iff ``coord_mgr`` is set;
- replaces the stale boot-time "no model definitions" message with
  a builder-failure-specific diagnosis (carrying ``type(exc).__name__``)
  on construction failure so the dashboard's 503 banner reflects the
  actual cause.

Both the lifespan path and the runtime-bootstrap path now route
through the same helper and the same teardown on failure.

Tests: 12 new tests covering the helper-level wiring (idempotent
fast-path, missing-prereq parametrised over ``config_store`` /
``collector`` / ``console_metrics``, no-rows error recording, builder
failure error replacement, partial-state teardown), the endpoint
integration, the deterministic concurrent-call lock test (uses an
instrumented lock wrapper that signals when a second acquirer arrives,
so the test fails fast on slow CI rather than depending on a
wall-clock sleep), and a real-builder end-to-end case constructing a
working ``SessionManager`` against a real ``ConfigStore`` + real
``ClusterCollector``.

(cherry picked from commit 3143965e00)
2026-05-07 17:35:22 -07:00
Patrick Buckley 9ae2b376c7 fix(mcp): apply Phase 7b PR #485 review feedback
Two of five Copilot comments on PR #485 were valid; this commit applies
both. The other three (one duplicate of comment 1, plus the INFO-logging
and `_pending`-naming nits) get rationale on-thread and resolution.

1. emit_oauth_failure_audit action now derived from `code` (#485 bug-1)

The Phase 7b refactor generalized `emit_insufficient_scope_audit` →
`emit_oauth_failure_audit`, routing both `mcp_insufficient_scope` AND
generic-403 (`mcp_*_forbidden`) through the same helper. The audit
`action` field stayed hardcoded as
`"mcp_server.oauth.insufficient_scope_emitted"`, mislabeling generic
forbidden events under the insufficient_scope bucket — downstream
alerting / analytics filtering on `action` would silently fold both
categories together.

The action is now selected from `code`:
  * `mcp_insufficient_scope` →
    `mcp_server.oauth.insufficient_scope_emitted` (preserves existing
    alerting consumers)
  * `mcp_tool_call_forbidden` / `mcp_resource_read_forbidden` /
    `mcp_prompt_get_forbidden` →
    `mcp_server.oauth.forbidden_emitted` (new, distinct label)

Detail row continues to carry both `code` and `kind` so operators get
sub-bucket distinction within either action.

2. Resource-listener docstrings cite RFC §3.2 (#485 doc-1)

Per the codebase convention established in Phase 7b round-1 q-1
(`_rebuild_user_prompt_map` corrected §3.2 → §3.3 because prompts are
§3.3 in the MCP spec), resource-related docstrings should cite §3.2.
The three resource-listener docstrings were citing §3.3, and the
"Mirrors `_notify_listeners` for tools (RFC §3.3)" parenthetical in
both `_notify_resource_listeners` and `_notify_prompt_listeners` read
as "tools are at §3.3" — confusing twice over. All four sites now
carry the correct catalog-kind citation explicitly:
  * resource-listener docstrings → "RFC §3.2 (resources)"
  * prompt-listener docstrings → "RFC §3.3 (prompts)"

Tests / lint:
  * 119 passed on 3.13 + 3.11 (targeted MCP OAuth pool tests)
  * ruff + mypy clean on both files

(cherry picked from commit 12cc052bca)
2026-05-07 17:35:22 -07:00
Patrick Buckley b368bdeecc feat(mcp): per-user resource + prompt pool dispatch (Phase 7b)
Extends the Phase 7 per-(user, server) ClientSession pool to cover
RFC §3.2 (resources/read) and §3.3 (prompts/get) on the same shape
already proven for tools/call. Pool discovery is capability-gated so
servers without resources/ or prompts/ stay free of extra round-trips.

API additions / widenings (MCPClientManager):
- ``read_resource_sync(uri, *, user_id=None, timeout=120)`` —
  per-user-first dispatch; falls through to the byte-identical static
  path when ``user_id`` is None or the URI doesn't resolve to an
  ``oauth_user`` pool entry.
- ``get_prompt_sync(prefixed_name, arguments=None, *, user_id=None,
  timeout=30)`` — same dispatch shape; structured-error responses
  surface via ``RuntimeError`` so the agent-loop's ``except Exception``
  block renders the JSON without polluting the prompt-protocol return
  shape.
- ``get_resources(user_id=None)`` / ``get_prompts(user_id=None)`` —
  per-user merged catalogs (admin/global call still passes None).
- ``add_{resource,prompt}_listener`` /
  ``remove_{resource,prompt}_listener`` —  ``user_id`` keyword scopes
  the listener so a pool-only catalog change for one user does not
  wake another user's session.
- ``resource_count_for_user(user_id=None)`` /
  ``prompt_count_for_user(user_id=None)`` — method-form variants used
  by ChatSession's ``read_resource`` / ``use_prompt`` tool gating; the
  legacy ``resource_count`` / ``prompt_count`` properties remain
  static-only for admin paths.
- ``_dispatch_pool_resource`` / ``_dispatch_pool_prompt`` async coros
  — mirror ``_dispatch_pool`` for the new SDK calls; share the
  carrier-race-and-cancel core via ``_dispatch_pool_with_entry_call``.
- ``_handle_auth_403`` extended with ``kind=Literal["tool",
  "resource", "prompt"]`` so the per-operation ``mcp_*_forbidden``
  code surfaces (kind="tool" remains the default for back-compat).
- Pool notification handler now refreshes resources / prompts on
  ``ResourceListChangedNotification`` / ``PromptListChangedNotification``
  via ``_refresh_pool_server_resources`` / ``_refresh_pool_server_prompts``.

ChatSession (``turnstone/core/session.py``) call-site updates:
- 12 sites threaded the session-bound ``user_id`` through
  ``add_*_listener`` / ``remove_*_listener``, ``get_resources`` /
  ``get_prompts``, gating, ``read_resource_sync`` /
  ``get_prompt_sync``, and ``is_mcp_prompt`` so the per-user merged
  catalog drives both the visible-tool set and dispatch.
- ``/mcp`` slash command now lists this user's pool resources and
  prompts alongside tools (Phase 7 already scoped tools).

Scope decisions:
- Per-user-first URI ordering (decision 0.1): the dispatcher attempts
  the user's pool catalog first, falling back to the static catalog
  only when no pool entry resolves the URI / prefixed name. Pool-only
  users never see the static catalog leak into their resolution.
- Method-form ``*_count_for_user`` (vs property) keeps the legacy
  ``resource_count`` / ``prompt_count`` properties intact for admin
  endpoints whose contract is "static catalog size only".
- Shared ``_dispatch_pool_with_entry_call`` helper accepts an
  ``sdk_call: Callable[[ClientSession], Awaitable[Any]]`` closure,
  keeping the entry-locked carrier-race / classification / retry
  plumbing single-source instead of a 3x copy across tool / resource
  / prompt paths.

R6 (anyio uniformity): every pool-side list / read / get path uses
``async with asyncio.timeout(...)`` — ``asyncio.wait_for`` is
forbidden in those paths because it wraps the inner awaitable in a
fresh task and surfaces ``CancelledError`` from inside
``streamablehttp_client``'s anyio TaskGroup on Python 3.11
(per ``feedback_asyncio_timeout_vs_wait_for.md``).

Tests:
- ``test_mcp_pool_auth_resource_integration.py`` — 9 real-transport
  resource tests (FastMCP upstream + ``BehaviorMiddleware``):
  401-refresh-retry success, persistent 401 -> consent_required,
  403+insufficient_scope, 403 generic -> mcp_resource_read_forbidden,
  breaker-isolation under repeated auth failures, missing-token,
  decrypt-failure, http:// URL guard, unknown-URI ValueError.
- ``test_mcp_pool_auth_prompt_integration.py`` — 9 mirror tests for
  the prompt path; structured-error responses verified via
  ``RuntimeError`` payload shape.
- ``test_mcp_user_catalog.py`` — extended unit coverage for per-user
  resource / prompt rebuild + collision policy + symmetric eviction.
- ``test_sessions.py::TestMCPToolGating`` — pool-only-user canary
  asserts ``read_resource`` / ``use_prompt`` stay visible when the
  static catalog is empty but the user has pool entries.

Round-1 review fixes (4-finder review applied, no push yet):
- bug-1: ``_exec_use_prompt`` was hardcoding ``"MCP prompt error: failed
  to invoke prompt"`` — discarding the structured-error JSON that
  ``_dispatch_pool_prompt_sync`` raises via ``RuntimeError``. Now uses
  ``f"MCP prompt error: {e}"`` mirroring ``_exec_mcp_tool``; pool-prompt
  consent_required / insufficient_scope / forbidden errors now reach
  the LLM as intended.
- bug-2 + bug-3: resource template discovery was uncapped —
  ``_cap_server_resources`` covered ``res_result.resources`` but the
  separate ``tmpl_result.resourceTemplates`` loop appended every
  template a server returned. Added ``_MAX_RESOURCE_TEMPLATES_PER_SERVER``
  (1000) + ``_cap_server_resource_templates`` helper, applied at both
  the initial discovery site (``_connect_one_pool``) and the refresh
  site (``_refresh_pool_server_resources``). Mirrors the existing
  ``_MAX_TOOLS_PER_SERVER`` / ``_MAX_PROMPTS_PER_SERVER`` defensive
  ceilings.
- sec-1 + sec-2: ``emit_insufficient_scope_audit`` generalized to
  ``emit_oauth_failure_audit(kind, code, ...)``, called from both the
  insufficient_scope branch AND the previously-silent generic 403
  branch. Audit detail now records ``{"kind": kind, "code": code,
  "scopes_required": [...]}`` so operators can distinguish tool-call
  vs resource-read vs prompt-get 403s in audit logs and so cross-
  tenant probing on the generic 403 path leaves a trail. The Phase 7
  inherited gap (``mcp_tool_call_forbidden`` had the same silence) is
  closed in the same refactor.
- perf-1: pool resource discovery now uses ``asyncio.gather(
  list_resources, list_resource_templates)`` inside the existing
  ``async with asyncio.timeout(...)`` budget — disjoint catalogs, no
  ordering dependency. Typical-case 2-RTT cold-connect resource block
  collapses to 1-RTT. Same change applied at ``_refresh_pool_server_resources``.
- q-1: ``_rebuild_user_prompt_map`` docstring corrected RFC §3.2 →
  §3.3 (resources are §3.2; prompts are §3.3).
- q-2: ``_refresh_pool_server_prompts`` docstring now carries the
  R6 / mcp-loop note that the resource sibling already had — both
  refresh paths now declare the asyncio.timeout invariant explicitly.
- q-5: added the ``_user_resource_map`` / DB-mismatch guard to
  ``read_resource_sync`` for parity with ``get_prompt_sync``. A stale
  per-user map entry with no matching oauth_user row now raises a
  specific ValueError instead of silently falling through to a
  generic ``Unknown MCP resource``.
- q-6: ``_dispatch_pool_with_entry`` (now a single-caller wrapper
  after the ``_dispatch_pool_with_entry_call`` extraction) gains a
  one-line docstring explaining why the wrapper is preserved
  (tool-decode localization + stack-trace identity for debugging).
- q-7: added 1 resource + 1 prompt end-to-end integration test that
  drive REAL discovery + dispatch in the same connect (no
  ``_seed_pool_*_map`` shortcuts), mirroring the tool path's
  ``test_integration_pool_reuse_401_refresh_and_retry_succeeds``.
  The seeded-map tests stay (faster, focused on dispatch); the new
  e2e tests cover the connect-discover-dispatch composition that
  caught Phase 6's carrier-on-entry bug.

Pre-push round-1 review fixes (3-finder review on the final state —
the lesson from Phase 7 round-3's q-1 regression: round-2 catches
what the round-1 apply pass missed):
- q-1 (MAJOR): the bug-1 sibling that round-1 missed —
  ``_exec_read_resource`` was hardcoding ``"MCP resource error: failed
  to read resource"`` while ``_exec_use_prompt`` (post-bug-1) preserved
  the structured-error JSON via ``f"... error: {e}"``. The round-1
  apply pass patched the prompt side but not the resource side. q-5's
  per-user-map / DB-mismatch ValueError was being swallowed at the
  agent loop boundary, defeating the operator-diagnostic intent. Now
  ``_exec_read_resource`` mirrors ``_exec_mcp_tool`` and ``_exec_use_prompt``.
- q-6 (nit): defensive-cap comment block at module-level cited
  "(RFC §3.2)" while covering both resource and prompt list paths;
  prompts are §3.3. Now reads "(RFC §3.2 for resources, §3.3 for
  prompts)" matching the convention the q-1 apply established.
- q-5 (rejected with better justification): the reviewer flagged
  ``_dispatch_pool_with_entry`` as a single-caller wrapper that should
  be inlined. After examination — the autouse fixture
  ``tests/test_mcp_pool_auth_introspection.py::_install_capture_intercept``
  monkeypatches this method to stash ``entry.auth_capture`` for the
  fake call_tool stubs in dispatcher-asserting tests. Inlining would
  redirect the patch to ``_dispatch_pool_with_entry_call`` (different
  kwargs shape) and require re-validating every test that depends on
  the interception. The wrapper IS load-bearing; q-6 docstring updated
  to cite the test-fixture rationale instead of the thin "stack-trace
  identity" claim.

Deferred to follow-up (documented rationale):
- perf-2: single-pass partition for system-message resource list
  (concrete vs templates). Sub-microsecond at expected scale;
  opportunistic-only.
- q-2 (pre-push): ~200 lines of fixture infrastructure
  (``BehaviorMiddleware``, ``_build_server``, ``_seed_oauth_server``,
  ``running_loop_mgr``, etc.) duplicated across three pool-integration
  test files. Real maintenance cost, but a 200-line conftest extraction
  is a focused refactor that earns its own commit / PR. Tracking as
  follow-up rather than balloon Phase 7b's diff further.
- q-3 / q-4 (refactor): extract shared dispatcher / scheduler
  helpers to compress three near-identical 90-line bodies (round-1
  q-3 was the same root cause; the pre-push q-3/q-4 reviewer
  reaffirmed it concretely). Three named methods preserve readability
  for the codebase's hottest correctness path; follow-up if
  duplication grows further or if a per-path divergence ships.
- q-4 (round-1, distinct from pre-push q-4): split pool concerns
  into ``mcp_pool.py``. Out-of-scope per finder; future refactor as
  the file approaches the navigation/merge-conflict threshold.

3.13: 5590 passed (5541 baseline -> +49 net; pre-review +47, q-7
e2e tests added +2). Existing audit-detail tests updated in-place
to expect the new ``kind`` and ``code`` fields.
3.11: 5590 passed (parity gate per ``feedback_pytest_env_parity.md``).

(cherry picked from commit 124615cce0)
2026-05-07 17:35:22 -07:00
Patrick Buckley d767aca784 fix(metacog): atomic cap-and-drop helper for soft-cap producers
Closes PR #484 review findings (Copilot): the soft-cap pattern in
``ChatSession.set_watch_runner``'s dispatch closure was a non-atomic
two-call pair (``count_by_type`` then ``drop_oldest_by_type``) with
two separate lock acquisitions.  A concurrent drain on the worker
thread (``USER_DRAIN`` / ``TOOL_DRAIN`` consuming ``"watch_triggered"``
entries via the ``"any"`` channel) could slip between the two calls,
making the drop a no-op.  The dispatch closure also discarded
``drop_oldest_by_type``'s return value and unconditionally logged
``dropped_oldest=True``, so a no-op drop got reported as a successful
drop.

* New ``NudgeQueue.cap_at_or_drop_oldest(nudge_type, max_depth,
  channel=None) -> bool`` does the count+drop in a single critical
  section.  Returns the actual outcome.

* Dispatch closure (``session.py:1410-1416``) now calls the helper and
  uses its return value to gate the WARNING log line, so the log is
  accurate when a drop did NOT happen.

* ``drop_oldest_by_type``'s docstring no longer overstates the
  per-call lock as covering a count+drop pair — it points readers
  to ``cap_at_or_drop_oldest`` for that contract.

7 new tests in ``TestCapAtOrDropOldest`` cover: below-cap no-op,
at-cap drop-oldest, above-cap drop-only-one (per-call), channel
filter, other-type isolation, ``max_depth <= 0`` defensive no-op,
no-match.

5708 non-live tests pass; ruff + mypy clean.

The github-code-quality bot finding ("Statement has no effect" on
``_protocol.py:939``'s ``...`` body) is a false positive — every
Protocol method in ``_protocol.py`` uses ``...`` as its body, which
is the canonical Python Protocol pattern.  Replacing with ``pass``
would diverge from the file's existing style.  No code change.

(cherry picked from commit c757c22f55)
2026-05-07 17:35:22 -07:00
Patrick Buckley 12bc580dee fix(metacog): factor sanitiser regex tail + trim docstrings + drop tombstone
Closes round-2 review findings q-3, q-4, q-5, q-7.

* **q-4:** ``_NAME_CONTROL_CHARS`` and ``_PAYLOAD_CONTROL_CHARS`` shared
  7 lines of Unicode-steering character classes (zero-width / bidi /
  separators / BOM / tag chars above BMP).  Factored into a single
  ``_CONTROL_CHARS_TAIL`` constant; each regex now differs only in its
  leading ASCII range.  Future bidi or zero-width additions edit one
  place.

  Side effect: this corrects a latent bug where ``_NAME_CONTROL_CHARS``
  had two literal ASCII spaces in place of U+2028 / U+2029 (line and
  paragraph separators) — visible as ``r"  "`` in source but rendered
  as the actual codepoints in ``_PAYLOAD_CONTROL_CHARS``.  After the
  factoring both regexes correctly include U+2028 / U+2029, closing
  the gap that would have let a workstream name with embedded line
  separators forge a sibling bullet (the same vector ``\n`` was
  blocked for in the original bug-1 fix).

  Switched to ``\u`` escapes for readability (and to keep future Edit
  tool runs against this block reliable).

* **q-3:** Tombstone clause "standing in for the deleted
  ``_watch_pending`` maxsize bound" survived in
  ``ChatSession.set_watch_runner``'s docstring after the apply-pass
  trim cleaned the inline soft-cap comment.  Dropped.

* **q-5:** ``test_newline_in_name_does_not_forge_extra_bullet`` carried
  five WHAT-narration comments restating what the immediately-following
  asserts already say.  Dropped — the docstring carries the security
  invariant; the assertions speak for themselves.

* **q-7:** ``patch_session_storage`` had a 14-line docstring including
  fallback-guidance and self-justification ("accumulated 7 near-duplicate
  sites").  Trimmed to a 3-line contract.

(cherry picked from commit 39e0f930c1)
2026-05-07 17:35:22 -07:00
Patrick Buckley aa1446364b test(metacog): drop redundant valid_until test + tighten concurrency bound + cover is_watch_active
Closes round-2 review findings q-1, q-2, q-6.

* **q-1:** ``test_valid_until_drops_when_watch_missing`` collapsed to the
  same code path as ``test_valid_until_drops_when_watch_inactive`` after
  the apply-pass switched the predicate from ``get_watch[active]`` to
  ``is_watch_active`` (both stubbed via ``patch_session_storage(active=False)``).
  The "missing" case has no distinguishable branch at the dispatch
  layer, so dropping it removes a tautological duplicate.  The
  missing-row mapping moves to the storage layer (q-2 below) where it
  IS distinguishable.

* **q-2:** ``is_watch_active`` was a new public storage primitive with
  zero direct backend coverage — only via-session-via-stub coverage.
  New ``TestIsWatchActive`` in ``tests/test_watch_storage.py`` covers
  active row → True, inactive row → False, missing row → False.
  Pinned at the storage boundary so future backend changes fail loudly
  there instead of in the dispatch tests.

* **q-6:** Concurrency test had ``n_threads = 2`` alongside two literal
  Thread objects and a tautological ``assert len(threads) == n_threads``.
  Threads are now built from a labels tuple, so ``len(threads)`` drives
  the slack bound; the redundant assertion is gone.

(cherry picked from commit 751ed9c85f)
2026-05-07 17:35:22 -07:00
Patrick Buckley 21507dc02e fix(metacog): tighten concurrency bound + lift storage-patch helper
Closes review findings bug-4 and q-6.

bug-4 — the watch dispatch concurrency test bounded depth at
``_WATCH_QUEUE_SOFT_CAP + 2 * per_thread`` (= 250) which is
tautologically true: two threads × 100 fires can append at most 200
entries above the cap, so the bound asserted nothing more than what
``depth <= 2 * per_thread`` already says.  Tighten to
``_WATCH_QUEUE_SOFT_CAP + N_THREADS`` (= 52): the count-then-drop window
admits at most one slip per concurrent thread.

q-6 — 7 near-duplicate ``monkeypatch.setattr(session_mod, "get_storage",
lambda: _StubStorage())`` sites across ``test_watch_dispatch.py`` +
``test_watch_integration.py`` (4 different stub shapes, mostly trivial
variations on the active flag).  Lift a ``patch_session_storage``
helper into the existing ``tests/_helpers.py`` with kwargs for the
common cases (``active``, ``raise_on_is_active``), returns the call list
so call-shape assertions still work.  Tests collapse from ~10-line
inline-class blocks to one-line helper calls.

(cherry picked from commit 20c4dfaca6)
2026-05-07 17:35:22 -07:00
Patrick Buckley a0ed4b9897 fix(metacog): drop watch_id rebind + trim soft-cap inline comment
Closes review findings q-2 and q-5.

q-2 — ``bound_watch_id = watch_id`` rebind was unnecessary.  ``_dispatch``
is constructed fresh per fire (not in a loop), so ``_still_active``
closes over the function parameter directly without any
loop-variable-capture risk.  Drop the rebind.

q-5 — the inline soft-cap comment restated rationale already covered by
the ``_WATCH_QUEUE_SOFT_CAP`` block-comment at module scope and dragged
in a tombstone reference to the deleted ``_watch_pending`` path.  Trim
to one line stating only the WHY (drop-oldest because latest output is
most useful).  Leave the ``set_watch_runner`` docstring's operational
detail at lines 1356-1378 alone — trimming further risks losing the
``valid_until`` predicate semantics.

(cherry picked from commit 28d9bb4802)
2026-05-07 17:35:22 -07:00
Patrick Buckley ab8ee0d759 test(metacog): integration coverage for _watch_restore_fn closure
Closes review finding q-4.

The closure built inside ``server.py``'s ``_watch_restore_fn`` is the
new contract surface introduced by the switchover — it constructs a
fresh ChatSession, calls ``session.resume(ws_id)`` to adopt the
original ws_id, re-registers the dispatch closure via
``set_watch_runner``, and returns ``WatchRunner.get_dispatch_fn`` for
the runner to invoke directly.  No automated coverage exists today;
a future refactor (e.g. swapping ``manager.create + session.resume``
for ``manager.open``) could silently break the watch-restore pipeline.

Adds ``test_watch_dispatch_through_restore_fn_lands_on_rehydrated_session``
to ``tests/test_watch_integration.py`` — drives the full restore path:
persists a kickoff message for the original ws_id, fires
``_dispatch_result`` against a runner with no registered dispatch fn,
asserts the restore_fn ran exactly once, the rehydrated session is a
distinct object that adopted the original ws_id, and the watch payload
landed on the rehydrated session's NudgeQueue (not on the original).

(cherry picked from commit ed1eaee216)
2026-05-07 17:35:22 -07:00
Patrick Buckley d34f6cd0b1 fix(metacog): is_watch_active storage primitive for hot-path valid_until
Closes review finding perf-1.

The watch dispatch closure's ``valid_until`` predicate fires once per
watch entry at every drain seam — on the chat-loop hot path.  It only
needs the ``active`` flag, but ``storage.get_watch`` runs a full-row
``SELECT *`` and marshals the result into a dict.  At the typical drain
depth (cap-50 + a busy chat loop) that's ~50 throwaway dict allocations
per drain pass for one boolean.

Adds ``StorageProtocol.is_watch_active(watch_id) -> bool`` plus
SQLite + Postgres implementations doing a single-column
``SELECT active FROM watches WHERE watch_id = ?`` (returns False on
missing row).  ``_still_active`` in ``ChatSession.set_watch_runner``
now calls that instead of indexing into the full row.

Test stubs that mocked ``get_watch`` for the predicate are converted
to mock ``is_watch_active`` directly.  Bulk variant deferred — single-row
fix is sufficient at typical drain depths.

(cherry picked from commit 3b495eba15)
2026-05-07 17:35:22 -07:00
Patrick Buckley b219c47ba8 fix(metacog): NudgeQueue.count_by_type primitive + channel-aligned soft cap
Closes review findings perf-2, q-3, bug-3.

The watch dispatch closure's soft-cap pre-check materialised the whole
queue snapshot via ``pending(channel="any")`` only to throw away the
text and count the type — wasteful at typical drain depths (cap-50 +
mixed producers means a 50-tuple allocation per fire just to read a
length).  The other half of the cap pair (``drop_oldest_by_type``)
walked the *whole* queue regardless of channel, so a future producer
that enqueued ``"watch_triggered"`` on a different channel could be
dropped by the watch cap, and vice versa — silently surprising once
that producer existed.

Adds ``NudgeQueue.count_by_type(nudge_type, channel=None) -> int`` that
walks ``_items`` once under the queue lock without materialising
tuples; extends ``drop_oldest_by_type`` to take an optional ``channel``
filter so both halves can agree on the entry set being capped.  The
watch dispatch closure now passes ``channel="any"`` to both —
consistent with where the closure enqueues — so a future channel split
can't bleed across producers.

Adds ``TestCountByType`` mirroring the existing ``TestDropOldestByType``
shape, plus a ``test_drop_oldest_by_type_channel_filter`` case pinning
the new optional argument's behaviour.

(cherry picked from commit e5e6e13307)
2026-05-07 17:35:21 -07:00
Patrick Buckley d770a811a8 fix(metacog): drop test_watch_live.py — defer R9 to operator-driven verification
Closes review finding q-1.

The live-marker scaffold in ``tests/test_watch_live.py`` couldn't actually
run as written: the ``live_client`` / ``live_model_id`` fixtures it
referenced live in ``tests/test_server_live.py`` at ``scope="module"``,
not on a shared ``conftest.py``, so the file would have ImportError'd
at collection if anyone ever tried ``pytest -m live`` against it.

Lifting the fixtures into a shared conftest is a larger refactor
than R9 justifies — the deterministic envelope-arrival contract is
already pinned end-to-end by ``test_watch_fires_then_user_send_drains_envelope``
and ``test_three_back_to_back_watch_fires_drain_into_one_turn`` in
``test_watch_integration.py`` (real ChatSession + real WatchRunner +
real chat-loop drain).  The model-quality-of-response leg is genuinely
manual; the plan doc's R9 entry is updated locally to reflect that
deferral.

(cherry picked from commit 68a44cc7e2)
2026-05-07 17:35:21 -07:00
Patrick Buckley 912e9c57b0 fix(metacog): split sanitiser regex — strict for names, permissive for payloads
Closes review finding bug-1.

The shared ``sanitize_payload`` regex preserved TAB/LF/CR so multi-line
watch shell output kept its layout — necessary for the watch path, but a
correctness gap for the idle_children formatter, which renders the
user-controlled ``name`` field as a single bullet item.  A child name
with an embedded ``\n`` would split the bullet across two rendered rows
and let a hostile name forge a fake sibling entry in the listing.

Splits the regex in two: ``_NAME_CONTROL_CHARS`` strips TAB/LF/CR
(used by the new ``sanitize_name`` helper for single-line name fields),
``_PAYLOAD_CONTROL_CHARS`` keeps the existing permissive shape (used by
``sanitize_payload`` for multi-line watch payloads).
``format_idle_children_nudge`` now calls ``sanitize_name``.

Adds ``test_newline_in_name_does_not_forge_extra_bullet`` — feeds a
hostile name with embedded ``\n`` + bullet-shaped continuation, asserts
the rendered listing still has exactly N bullet rows for N children
(no forged sibling), and the hostile newline got flattened to an inline
space.  Adds a ``TestSanitizeName`` class mirroring the existing
``TestSanitizePayload`` shape for the new strict variant.

(cherry picked from commit e596650a5c)
2026-05-07 17:35:21 -07:00
Patrick Buckley d7c6053441 fix(metacog): drop misleading _watch_restore_fn comment
The deleted comment claimed the closure may be registered "under the
rehydrated workstream's id, which may differ from the original ws_id we
restored against" — but ``ChatSession.resume(ws_id, fork=False)`` adopts
the parameter as the session's id at session.py:1682, so they match
exactly post-resume.  The lookup works because the ids are equal, not
because they may differ.

The accessor name ``get_dispatch_fn`` is self-explanatory; no replacement
comment is needed (per the project's "default to no comments" rule).

(cherry picked from commit d2028aa4f7)
2026-05-07 17:35:21 -07:00
Patrick Buckley e7a17a20b0 test(metacog): watch switchover boundary integration + live scaffold
Adds two boundary-crossing integration tests and one live-marker
scaffold for the watch switchover landed in the previous commits:

tests/test_watch_integration.py — drives a real ChatSession + real
WatchRunner end-to-end (LLM stubbed) through the unified pull-model
chat-loop drain seam.  Pins:

- test_watch_fires_then_user_send_drains_envelope: a synchronous
  WatchRunner.dispatch fire enqueues "watch_triggered" on "any";
  session.send drains the entry into the user message's _reminders
  side-channel — confirms the envelope splice path.
- test_three_back_to_back_watch_fires_drain_into_one_turn: pins the
  intentional behavioural delta from the plan section 3.4 / risk
  register R3 — N back-to-back fires now produce ONE assistant turn
  with N _reminders entries, not N successive turns.

tests/test_watch_live.py (new file, single test, marked @pytest.mark.live):
risk register R9 verification recipe — confirm a real LLM handles a
<system-reminder>-framed watch payload sensibly.  Collects under the
regular -m "not live" run; the user runs it on demand against an
Anthropic-backed config.

Implements watch-switchover plan section 5.2 (integration) and step 11
(live scaffold).

(cherry picked from commit 17c62f7ef3)
2026-05-07 17:35:21 -07:00
Patrick Buckley 931a1eca9d test(metacog): NudgeQueue-based dispatch tests for watch closure
Replaces the deleted tests/test_watch_dispatch.py with a focused
14-test suite exercising the closure that ChatSession.set_watch_runner
now constructs (per the previous commit's switchover).  Each test
pins one assertion:

- enqueue shape: ("watch_triggered", text, "any") on the per-session
  NudgeQueue; not on user / tool channels
- producer-side sanitisation strips control / bidi / zero-width chars
  and angle-bracket tag breakers; preserves TAB/LF/CR so multi-line
  shell output keeps its layout (R8); empty-after-strip → no enqueue
- soft-cap drop-oldest at _WATCH_QUEUE_SOFT_CAP with a queue_full
  WARNING log; non-watch entries on the same queue are not collateral
  damage
- valid_until predicate drops on inactive / missing / storage-raises;
  delivers when active (counter-test)
- concurrent enqueues across two threads stay bounded under the
  3-acquisition count-then-drop window

Implements watch-switchover plan section 5.1 / step 9.  No production
changes — pure test rewrite.

(cherry picked from commit 7ca00b564c)
2026-05-07 17:35:21 -07:00
Patrick Buckley 048285a423 feat(metacog): switchover — watches enqueue onto NudgeQueue not _watch_pending
Replaces the bespoke _make_watch_dispatch / _watch_pending /
_dispatch_pending_watch / _MAX_WATCH_CHAIN machinery with a single
NudgeQueue.enqueue("watch_triggered", ...) call inside
ChatSession.set_watch_runner.  Watch results now drain at the same
<system-reminder> envelope seams as every other metacog nudge
(USER_DRAIN, TOOL_DRAIN, IdleNudgeWatcher IDLE wake) — no separate
worker-spawn, no recursive watch chain, no per-session queue.Queue.

The dispatch closure built inside set_watch_runner carries:
- producer-side sanitize_payload over the whole formatted message
  before enqueue, so steering-vector / control-char shell output
  can't tamper with the envelope at interpolation time
- a soft cap of 50 entries on per-session "watch_triggered" depth
  via the new NudgeQueue.drop_oldest_by_type, replacing the prior
  _watch_pending maxsize=20 + _MAX_WATCH_CHAIN=5 bounds; drop policy
  is drop-oldest (latest output most useful), logged at WARNING
- a valid_until predicate that re-checks
  storage.get_watch(watch_id)["active"] at drain time so a cancelled
  watch's last splat doesn't ride out a future wake

Behavioural delta documented in the plan section 3.4: N back-to-back
watch fires now drain into ONE assistant turn responding to all N
(via the envelope splice) instead of N separate send turns.  This is
intentional — fewer model invocations for noisy watches, and uniform
with the rest of the metacog pull-model surface introduced by #482.

Implements watch-switchover plan steps 5-8.  Server-side simplifications
let the previously-load-bearing _make_watch_dispatch (47 lines), its
session_worker.send import, and the chat-loop _dispatch_pending_watch
seam at the no-tools IDLE branch all disappear.  The obsolete
tests/test_watch_dispatch.py and the wake-tag test in test_session.py
(both pinning contracts that no longer exist) are removed; the
NudgeQueue-based replacement plus an integration test land in the
following commit.

(cherry picked from commit 94ed79d488)
2026-05-07 17:35:21 -07:00
Patrick Buckley 481347eb17 refactor(metacog): widen WatchRunner dispatch_fn signature to (msg, watch_id)
Widens the per-workstream dispatch fn signature from ``(message,)``
to ``(message, watch_id)``.  The runner now passes the originating
``watch_id`` through ``_dispatch_result`` so dispatch closures can
capture per-watch metadata at fire time — the upcoming switchover
needs this for the ``valid_until`` predicate that re-checks
``storage.get_watch(watch_id)["active"]`` before a stale entry rides
out a wake.

Also adds ``WatchRunner.get_dispatch_fn(ws_id)`` as the public
accessor used by the server-side restore path to retrieve the
closure that ``set_watch_runner`` constructed during workstream
rehydrate (avoiding private-attr access into ``_dispatch_fns``).

Implements watch-switchover plan step 4 plus risk register R4.
The pre-existing single-arg callers (``_make_watch_dispatch`` and
``set_watch_runner``'s ``dispatch_fn=`` fallback) get replaced
in the next commit; their mypy types are ``Any`` today so the
type mismatch isn't caught at this step.

(cherry picked from commit 195ff985cc)
2026-05-07 17:35:21 -07:00
Patrick Buckley 31a554a4bd refactor(metacog): shared sanitize_payload + watch_triggered nudge type
Renames _sanitize_child_name to sanitize_payload and widens it to be
the shared producer-side sanitiser for both idle_children and the
incoming watch_triggered nudges.  The regex now skips TAB / LF / CR
so multi-line shell output rendered into a watch payload keeps its
line structure when sanitised as a whole formatted message — the
pre-switchover code path collapsed multi-line output to one line.

Adds the watch_triggered entry to _NUDGE_MAP alongside idle_children
so ``_NUDGE_MAP``-as-registry consumers (should_nudge gating, future
audit / UI tagging) recognise the type.  Body is empty — payload
comes from the producer (the watch dispatch closure), same shape as
idle_children.

Implements watch-switchover plan section 3.2 plus risk register R8
(TAB/LF/CR exclusion) and step 3 (_NUDGE_MAP registration).

(cherry picked from commit 78ae7ae6b5)
2026-05-07 17:35:21 -07:00
Patrick Buckley af2c0ae13a feat(metacog): NudgeQueue.drop_oldest_by_type helper for soft-cap producers
Adds an atomic drop-oldest-by-type operation to NudgeQueue used by
producers that need a per-type soft cap on their own queue depth.
The watch dispatcher (next commit in this stack) is the first user:
when "watch_triggered" saturates, the dispatch closure drops its
oldest entry under the queue lock so the count snapshot and drop
can't interleave with a concurrent enqueue from the same producer.

Implements watch-switchover plan section 3.1 — the producer-side soft
cap takes the place of the deleted _watch_pending maxsize=20 bound.
Other producers (idle_children, advisories) have natural rate limiters
already, so the helper is opt-in per producer rather than a global cap
in enqueue itself.

(cherry picked from commit 74f1958e47)
2026-05-07 17:35:21 -07:00
Patrick Buckley 0808dc0af0 fix(mcp): apply Phase 7 PR review feedback
Three Copilot findings on PR #483 (commit dad98c0); one rejected as a
false positive.

- mcp_client.py:1189 — pool notification handler's exception path
  used ``log.warning(..., exc_info=True)`` which serializes the
  chained ``httpx.Request.headers`` carrying ``Authorization: Bearer
  <token>`` into Sentry / faulthandler frame captures. Same threat
  model as the round-1 sec-1 dispatch-path fix, applied to a site
  the original review missed. Now logs structured fields only
  (server, user, exc type) without ``exc_info``.

- mcp_client.py:1202 — ``_connect_one_pool``'s handshake step used
  ``asyncio.wait_for(session.initialize(), ...)``, the same Python
  3.11 + anyio cross-task-cancel-scope anti-pattern that the
  Phase 7 round-3 q-1 fix removed from the discovery step (and that
  f6a3b66 originally addressed for ``_safe_close_stack``). Pre-
  existing Phase 5 code, but the same latent bug class — a 401
  during initialize() under 3.11 would surface ``RuntimeError:
  Attempted to exit cancel scope in a different task`` as the
  SDK's TaskGroup unwinds. Switched to ``async with asyncio.timeout(...)``
  matching the discovery step's pattern.

- mcp_client.py:1522 — renamed loop tuple-unpack variable
  ``_server_name`` → ``server_name`` in ``_rebuild_user_tool_map``.
  The leading underscore conventionally signals "intentionally
  unused", but the variable is read at the assignment a few lines
  below. Two other ``_server_name`` unpacks in this file (1410,
  3111) genuinely don't use the value and keep the underscore.

Rejected as false positive:
- test_mcp_user_catalog.py:58 (github-code-quality bot, "Statement
  has no effect"): ``await task`` inside ``contextlib.suppress(
  BaseException)`` is the standard pattern for cleanly draining a
  cancelled task. The bot's static analysis treats ``await`` of a
  result that's discarded as a no-op statement, but ``await`` here
  triggers cancellation propagation and waits for the task to
  finish — load-bearing in the fixture's teardown. No change.

Verified on Python 3.11 (``/tmp/venv311``) and 3.13 (``.venv``):
ruff + mypy clean, full test suite green.

(cherry picked from commit 62909d402c)
2026-05-07 17:35:21 -07:00
Patrick Buckley cfc8a6c8c0 feat(mcp): per-user catalog scoping (Phase 7 — tools)
Light up production reachability of pool dispatch (RFC §3, invariant 8)
by widening the public catalog API to optionally take a ``user_id``:

- ``MCPClientManager.get_tools(user_id=None)`` returns the merged
  static + per-user pool view when ``user_id`` is supplied; the default
  preserves the legacy global-only contract.
- ``is_mcp_tool(name, *, user_id=None)`` extends the lookup to the
  per-user ``_user_tool_map``. Pool tools become reachable from
  ``ChatSession._prepare_tool`` only when the session-bound user_id
  flows through — flipping invariant 8 from "must hold" to "satisfied".
- Listener identity becomes ``(user_id, callback)``. Static-path
  changes fire ALL listeners (admin + every user); pool-entry
  changes fire only matching-user + admin (``None``) listeners.
  RFC §3.3.
- Pool sessions discover their tool list on first connect
  (``_connect_one_pool`` → ``await session.list_tools()``); the
  notification closure binds to ``(user_id, server_name)`` so
  push-driven ``list_changed`` updates target the correct user's
  catalog. R6 verified empirically: ``list_tools()`` 401 propagates
  through anyio TaskGroup unwinding, no hang — plain ``await`` is
  fine, no carrier-race shape needed for discovery.
- ``_evict_session`` drops ``entry.tools`` and rebuilds the user's
  index so an evicted-then-reconnected session doesn't carry
  stale catalog state.
- ``web_search.resolve_web_search_client`` refuses
  ``auth_type=oauth_user`` backends (per-node web search can't
  carry per-user tokens).

Resources / prompts pool dispatch deferred to Phase 7b — invariant 8
is satisfied by the tool path alone, and the resource/prompt path
needs sibling ``_dispatch_pool_resource_sync`` /
``_dispatch_pool_prompt_sync`` helpers each with their own
carrier-race plumbing (~400 LOC). Phase 7b will follow the patterns
established here.

CLI sessions default ``user_id=""`` and so cannot use oauth_user
MCP servers — documented limitation; users must use the web UI.

Round-1 review fixes (4-finder review applied, no push yet):
- bug-1: get_tools(user_id) was iterating _user_pool_entries from sync
  threads while the mcp-loop concurrently mutated it (RuntimeError:
  dictionary changed size during iteration). Now reads from a sibling
  _user_tools dict updated atomically by _rebuild_user_tool_map.
- bug-2: _close_pool_entry_if_idle (LRU/TTL eviction) skipped the
  catalog cleanup that _evict_session does — stale tools persisted
  in _user_tool_map and ChatSession's tool list never rebuilt. Now
  mirrors _evict_session.
- perf-1: _last_pool_notification_refresh debounce dict was never
  pruned in either eviction path. Now popped alongside the entry.
- perf-3: web_search resolver was issuing a sync SQL query per LLM
  turn to gate oauth_user backends. Now reads from the cached
  in-memory config.
- sec-1: bearer token could leak into exc_info-rendered tracebacks
  via Sentry/faulthandler. log.debug now uses structured fields,
  not exc_info.
- sec-2: tools-per-server response now capped at 1000 (defensive,
  mirrors _MAX_ERROR_LEN / _MAX_INSUFFICIENT_SCOPE_REPORTED).
- Test cleanup: dropped two listener fan-out tests duplicating
  test_mcp_client.py coverage; renamed test_pool_session_notification_handler
  to match its actual scope (_refresh_pool_server_tools); removed
  stale comments referencing /tmp/r6-spike*.py scratchpads and a
  misleading "copy-on-write" comment.

Round-2 pre-push review fixes (focused single-pass review applied):
- round2-1: bug-2's catalog-cleanup block in _close_pool_entry_if_idle
  had no integration test (exactly the failure mode flagged in
  feedback_tests_through_boundaries.md). Added
  test_close_pool_entry_if_idle_clears_catalog_and_fires_listener
  driving the LRU/TTL eviction path through real streamablehttp_client +
  MockTransport. Negative-test verified: reverting the
  _rebuild_user_tool_map / _notify_user_tool_listeners calls makes
  the new test fail.
- round2-3: documented the _oauth_user_server_names cache invariant
  in add_server_sync / remove_server_sync docstrings. Cache is
  reconcile_sync's sole owner — direct callers leave it stale, but
  _db_servers_to_config strips oauth_user rows so production paths
  are unaffected. Static→oauth_user transitions correctly leave the
  name in the cache because remove_server_sync drops the static
  connection, not the cache identity.
- round2-6: strengthened test_rebuild_user_tool_map_populates and
  test_rebuild_user_tool_map_drops_empty_user to assert on the
  _user_tools sibling cache (bug-1 fix). Without this, a future
  revert dropping the sibling write would still pass the unit
  tests because get_tools coverage lives in separate tests.

Round-3 full-stack review fixes (multi-stage review on the final
state caught what the layered apply passes missed):
- q-1 REGRESSION: pool tool-discovery used asyncio.wait_for around
  session.list_tools(), the exact pattern the f6a3b66 fix (and
  feedback_asyncio_timeout_vs_wait_for.md) put in place to avoid.
  Python 3.11's asyncio.wait_for wraps the inner coroutine in a
  fresh task → cross-task scope-exit when the SDK's anyio TaskGroup
  unwinds on a 401. Switched to `async with asyncio.timeout(...):`
  pattern used by _safe_close_stack.
- sec-2: TOCTOU in _connect_one_pool — entry.tools was published
  (via _rebuild_user_tool_map + listener fan-out) BEFORE entry.session
  was assigned. A sync-thread reader could observe a tool whose
  backing entry has session=None. Defence-in-depth — dispatch
  re-fetches its own token and lazy-reconnects on session=None — but
  reordering catches the race at the source. entry.session now
  publishes BEFORE catalog visibility.
- bug-1: _close_pool_entry_if_idle's _user_pool_locks.pop ran
  unconditionally after the try/finally, but the early-return
  branches (entry None on re-check, in_flight > 0 under lock) skip
  it via Python's return-through-finally semantics. The lock was
  never popped on those paths. Now gated behind an `evicted` flag
  set only on the success path; in_flight > 0 leaves the lock for
  the active dispatcher to reuse, entry-None races leave the lock
  for re-allocation by _ensure_pool_entry. Comment now describes
  the actual semantics, not the original promise.
- bug-2: softened the _rebuild_user_tool_map docstring's atomicity
  claim. The two-dict write is technically non-atomic across Python
  statements; in practice the window is sub-microsecond on the
  mcp-loop with no awaits between writes, and the listener fan-out
  fires AFTER both writes complete. Docstring now says "back-to-back
  on the mcp-loop" instead of "atomically alongside".
- q-3: dropped `hasattr(mcp_client, "server_auth_type")` defensive
  check in web_search.py. The method ships in this commit; the
  hasattr created a silent fallthrough that would let a future
  rename silently re-enable oauth_user backends.
- q-4: surfaced the CLI / empty-user_id limitation in a docstring
  comment at ChatSession.__init__'s self._user_id assignment. The
  note previously lived only inside is_mcp_tool's docstring — a
  future maintainer wiring CLI features against MCP pool servers
  wouldn't think to read is_mcp_tool to find the constraint.
- q-2 + q-5: deleted a tautological duplicate test in
  test_mcp_user_catalog.py whose docstring claimed to test
  ChatSession.close but never instantiated a ChatSession (the
  manager-level identity semantics are already covered by
  test_listener_identity_includes_user_id in the same file and by
  test_session_close_removes_listener_with_same_user_id in
  test_mcp_client.py which DOES drive a ChatSession). Reworded a
  misleading "fixture provides only 5s" comment to point at the
  actual `_run_on_loop(..., timeout=5)` site.
- q-6: the `self._user_id or None` collapse repeated at 8 sites
  across session.py. Cached once at __init__ as
  ``self._mcp_user_id`` (since ``_user_id`` is set once and never
  mutated); 8 call sites now read the cached value. The empty-
  string-is-CLI-sentinel invariant is documented at the assignment
  site, not re-asserted at each consumer.

Deferred to follow-up:
- sec-1: a hostile MCP server bound to user-A could craft a
  tool.name containing `__` to synthesize a prefixed-name collision
  in user-A's own catalog. Bounded impact: cross-tenant dispatch is
  prevented by the per-tenant token gate in _dispatch_pool, and
  user-B's get_tools(user_id="B") never includes user-A's pool
  entries. The fix needs policy decisions (reject vs. sanitize)
  and touches _mcp_to_openai which is shared between static and
  pool paths; better discussed in its own follow-up where the
  policy applies uniformly to static-path servers too. The threat
  model already requires user-A to have consented to a malicious
  server, who has many more dangerous vectors than tool-name
  shenanigans.

Test count delta: +31 tests (5435 → 5466, ``-m "not live"``; one
test deleted in round-3 apply per q-2):
- ``tests/test_mcp_client.py`` +20 (per-user catalog state, listener
  identity, session thread-through)
- ``tests/test_mcp_user_catalog.py`` +9 NEW (integration tests
  driving real ``streamablehttp_client`` + ``httpx.MockTransport`` per
  invariant 14: discovery on connect, user isolation, eviction +
  reconnect, LRU/TTL eviction (round2-1), R6 401-propagation
  regression, static byte-identical canonical regression; review
  passes dropped duplicate listener fan-out tests from earlier
  drafts whose coverage lived in test_mcp_client.py)
- ``tests/test_web_search.py`` +2 (oauth_user backend rejection +
  static backend acceptance regression; updated to use the new
  ``server_auth_type`` in-memory accessor)

(cherry picked from commit a8b34bfe54)
2026-05-07 17:35:21 -07:00
Patrick Buckley 266e3536aa fix(metacog): bot-review fixes — watcher gate + two stale docstrings
Three confirmed findings from the PR #482 bot review pass.

* **Copilot (idle_nudge_watcher.py)**: ``IdleNudgeWatcher`` was gating
  wake dispatch on ``len(_nudge_queue) == 0`` (any channel), but
  ``deliver_wake_nudge_from_queue`` only drains ``USER_DRAIN``.  A
  ``"tool"``-channel entry queued by ``_queue_tool_advisory`` would
  pass the gate, spawn a wake daemon, and immediately no-op at the
  drain guard — repeating on every IDLE event for as long as the
  tool entry sat unconsumed.  No correctness bug (the no-op return
  prevents bad state) but a wasted thread spawn per IDLE.  Fixed by
  gating on ``has_pending(USER_DRAIN)``; tool-only queues no longer
  trigger the wake path.

* **Copilot (coordinator_idle_observer.py)**: docstring referenced
  the old module path ``turnstone.core.metacognition.IdleNudgeWatcher``;
  the class moved to ``turnstone.core.idle_nudge_watcher`` in q-3 of
  the apply-pass.

* **Copilot (nudge_queue.py)**: ``has_pending`` docstring cited
  ``ChatSession.deliver_wake_nudge_from_queue`` as its caller, but
  that method calls ``drain(USER_DRAIN)`` directly — no production
  caller used ``has_pending`` until this commit.  Updated to point
  at the now-actual caller (``IdleNudgeWatcher``).

* **github-code-quality (test_nudge_queue.py)**: false positive on
  ``test_channel_is_required`` — the no-channel ``q.enqueue("a", "1")``
  call is wrapped in ``pytest.raises(TypeError)`` to verify the
  validation contract.  No code change.

5571 non-live tests pass; ruff + mypy clean.

(cherry picked from commit 0fbf31e713)
2026-05-07 17:35:21 -07:00
Patrick Buckley 42bf9aecaf fix(metacog): apply-pass fixes from pre-push full-stack review
Round-2 review caught 11 confirmed findings on the 3-commit metacog stack;
this commit applies them.

* **bug-1 (major)**: Wake source tag was leaking onto real user messages
  flushed during a wake send.  ``_append_user_turn`` and ``send`` now
  take an explicit ``from_wake: bool`` parameter — only the wake's
  synthesized first turn passes True, so ``_flush_queued_messages``'s
  real user input no longer inherits the audit tag.  Regression test
  pins the contract.

* **perf-1 (major)**: ``CoordinatorIdleObserver._maybe_enqueue`` was
  issuing list_workstreams + visible_memory_count storage queries
  before the cheap cooldown gate could short-circuit.  New
  ``_cooldown_allows`` read-only peek runs first; storage queries only
  fire when cooldown actually allows the nudge.

* **q-1 (major)**: Added the missing coord-side integration test that
  exercises ``CoordinatorIdleObserver`` + ``IdleNudgeWatcher`` together
  in the production install order against a real ``SessionManager``,
  protecting the subscription-order contract from silent regression.

* **perf-2/3 (minor)**: Cap check moved above ``_last_assistant_used_wait``;
  ``_fire_counts`` restructured as ``dict[str, dict[str, int]]`` keyed by
  ws_id so the leave-IDLE existence check is O(1).

* **perf-4 (minor)**: ``NudgeQueue.drain`` fast-paths the all-match
  case (the common one for chat-loop drain seams) by swapping
  ``self._items`` directly instead of allocating a fresh ``kept``
  deque + per-entry append.

* **perf-5 (minor)**: Wake's synthesized empty user turn no longer
  writes a content-empty row to the conversations table — the
  ``_source`` audit tag isn't column-backed and the side-channel
  reminder is stripped before persist, so the row would carry nothing.

* **q-3 (minor)**: Split ``IdleNudgeWatcher`` + ``install_*`` /
  ``shutdown_*`` helpers out of ``metacognition.py`` into the new
  ``turnstone/core/idle_nudge_watcher.py``; metacog stays a
  static-template module.

* **sec-1 (nit)**: Widened ``_sanitize_child_name``'s control-char
  regex to cover Unicode bidi-overrides, zero-width chars,
  line/paragraph separators, BOM, and tag chars.

* **q-4/q-5 (nits)**: Docstring referenced the wrong peek primitive
  (``has_pending`` → ``len()``); ``_last_assistant_used_wait``'s
  ``session`` parameter now typed ``ChatSession``.

5571 non-live tests pass; ruff + mypy clean.

(cherry picked from commit 3f106f98b2)
2026-05-07 17:35:21 -07:00
Patrick Buckley 191775dd7e feat(metacog): coord idle-children nudge — observer + valid_until predicates
Adds the first concrete consumer of the wake trigger: when a coordinator
goes IDLE while interactive children are still running, a
``CoordinatorIdleObserver`` enqueues an ``idle_children`` nudge that the
``IdleNudgeWatcher`` then dispatches as a synthetic empty-user-turn
``send``.  The model receives a system-reminder body listing the active
children (capped at 6 inline + 32 in the suggested ``wait_for_workstream``
call) and a nudge to block on them rather than reply prematurely.

Observer gates (in order): coord-only filter, skip if last assistant
turn used ``wait_for_workstream``, per-(ws, nudge_type) hard cap (3)
that resets only on non-wake leave-IDLE, active-children query,
``should_nudge`` cooldown.  Console lifespan registers the observer
BEFORE the watcher so subscriber-fire order has the observer
enqueueing first on the same IDLE event.

Adds an opt-in ``valid_until`` predicate on ``NudgeQueue.enqueue``
(R9 from the design risk register) — drain re-checks the predicate
outside the queue lock; falsy / raising drops the entry without
delivering it.  ``deliver_wake_nudge_from_queue`` now drains inline
before synthesizing the empty user turn so a stale predicate-drop
doesn't leave the wake send with empty content; ``_attach_pending_user_reminders``
consumes the pre-drained reminders via ``_wake_drained_reminders``.

The observer's ``valid_until`` uses ``count_workstreams_by_state``
(boolean check, no row fetch) instead of full ``list_workstreams``,
keeping the chat-loop user-attach path off the heavy query.

User-controlled child workstream names are sanitized
(``_sanitize_child_name``) before interpolation so a name like
``</thinking>...`` can't steer the model's reasoning channels through
the rendered body — the wire-boundary ``escape_wrapper_tags`` only
covers ``<system-reminder>`` / ``<tool_output>`` envelopes.

(cherry picked from commit 908e67fe4f)
2026-05-07 17:35:21 -07:00
Patrick Buckley c41fd2be2e feat(metacog): wake trigger — IdleNudgeWatcher + ChatSession.deliver_wake_nudge_from_queue
Adds the third metacog channel: an out-of-band wake that converts a
workstream's IDLE transition into a synthetic empty-user-turn ``send``
when the session has any-channel nudges queued.  The ``IdleNudgeWatcher``
subscribes to ``SessionManager.subscribe_to_state``; on IDLE it dispatches
via ``session_worker.send`` with a no-op ``enqueue`` callback so a
busy-worker race silently drops without spawning a competing worker.

Wake-source-tag plumbing on ``ChatSession`` short-circuits metacog
detection on the synthetic empty input, suppresses queue producers
during the wake's own tool dispatch, and stamps ``_source = "system_nudge"``
on the synthetic user-message for audit / replay distinction.  The tag
is saved / restored across ``_dispatch_pending_watch`` so watch chains
recursing off the wake are processed as normal user turns rather than
inheriting the wake's guards.

Generic ``install_idle_nudge_watcher`` / ``shutdown_idle_nudge_watchers``
helpers wire the watcher into both the interactive and coord lifespans
via a single ``app.state`` registry so both surfaces share the same
teardown contract.

Foundation for PR 3 (CoordinatorIdleObserver + idle_children formatter)
and PR 4 (watch dispatcher switchover).

(cherry picked from commit f0e7fea549)
2026-05-07 17:35:21 -07:00
Patrick Buckley 1787fb5c11 refactor(metacog): unify advisory channels into pull-model NudgeQueue
Replaces the dual `_pending_user_advisories` / `_pending_tool_advisories`
list pair with a single channel-tagged `NudgeQueue` per session.
Producers tag entries with a channel ("user", "tool", or "any");
consumers drain by channel filter at their existing seams. Foundation
for the wake trigger (PR 2) and coordinator idle-children nudge (PR 3).

Existing nudges (start, correction, completion, denial, resume,
tool_error, repeat) keep their wire shape and drain timing — zero
behavior change. Cancel paths now `clear()` the unified queue.

(cherry picked from commit 94b3720916)
2026-05-07 17:35:21 -07:00
Patrick Buckley 814c42763d fix(mcp): asyncio.timeout (not wait_for) for safe-close-stack on Python 3.11
Python 3.11's ``asyncio.wait_for`` wraps its inner coroutine in a fresh
``asyncio.Task`` via ``ensure_future``. When the inner is
``stack.aclose()`` on an ``AsyncExitStack`` containing
``streamablehttp_client(...)`` (anyio cancel scopes entered in the
calling task), the fresh task's attempt to exit those scopes raises
``RuntimeError('Attempted to exit cancel scope in a different task
than it was entered in')``. Python 3.12+ rewrote ``wait_for`` to use
``asyncio.timeout`` internally — runs in the current task — so 3.13
ran the same code path successfully.

Symptom on 3.11: integration tests where ``session.initialize()``
returns 4xx (e.g., 403 insufficient_scope tests) hit
``_connect_one_pool``'s ``except Exception:`` handler →
``_safe_teardown_on_connect_failure`` → ``_safe_close_stack`` → cross-
task RuntimeError. The ``concurrent.futures._base.CancelledError``
that surfaces in ``future.result(timeout=...)`` is the cascade
fallout from the asyncio loop's exception handler reacting to the
unretrieved-task-exception.

Fix: use ``asyncio.timeout`` instead of ``asyncio.wait_for`` for the
5s aclose bound. Equivalent semantics, current-task execution, works
on 3.11+. The 5s guard against ``aclose()`` hanging on a broken stack
is preserved.

Verified on Python 3.11.14 (full suite 5427 passed) and 3.13.7 (full
suite 5427 passed); all 9 integration tests pass on both.

Pre-existing bug — surfaced only after the marker fix in 5c9850c
let CI's test (3.11) actually run the 4xx tests.

(cherry picked from commit f6a3b66ea4)
2026-05-07 17:35:21 -07:00
Patrick Buckley 242596ced3 fix(mcp): pool-reuse 401 — entry-owned carrier + race-and-cancel
Two pre-existing defects in the Phase 6 pool dispatch path that only
manifest when a pooled session is reused for a second dispatch:

1. The per-dispatch _AuthCapture allocated in _dispatch_pool was wired
   into the httpx response hook only at first connect (via
   _connect_one_pool). On a reused session no fresh connect runs, so
   the hook continues writing to the original-connect's carrier while
   the new dispatch inspects an empty carrier — auth_401/403 silently
   misclassified to "other", refresh-and-retry never fires.

2. Even with the carrier on the entry (so the hook writes to a stable
   reachable object), session.call_tool itself hangs forever on
   upstream 4xx for reused sessions. Trace: SDK's spawned
   handle_request_async raises HTTPStatusError, the outer
   streamablehttp_client TaskGroup cancels post_writer, post_writer's
   finally aclose's read_stream_writer, BaseSession's _receive_loop
   exits and enters its CONNECTION_CLOSED-fanout finally. anyio's
   send_nowait skips waiting receivers with pending_cancellation; the
   dispatch task (created by run_coroutine_threadsafe for the reuse
   case) is NOT in any cancel-scope chain, so the send "delivers" but
   the receiver's Event is set on stale state — receive() never
   wakes. Test 21 doesn't hit this because its 401 happens during
   initialize, in the same task that opens streamablehttp_client, so
   the cancel scope DOES propagate.

Fix:
- Move _AuthCapture ownership to PoolEntryState (and asyncio.Event
  alongside, allocated lazily on the mcp-loop). The hook closes over
  entry.auth_capture at first connect and stays valid across
  dispatches; reset under open_lock before each call_tool.
- Race session.call_tool against the carrier's fired_event in
  _dispatch_pool_with_entry. If the event wins (hook captured 4xx
  before SDK propagated), cancel call_tool and raise an internal
  _CarrierAuthSignal — _classify_failure resolves to auth_401/403
  via the carrier's status, the dispatcher evicts the broken
  session, and the cross-task retry handshake reconnects on a fresh
  bearer.

Adds tests/test_mcp_pool_auth_integration.py::test_integration_pool_reuse_401_refresh_and_retry_succeeds
which drives the reuse path through real upstream + real SDK and is
the structural gate against this class regressing. Negative-tested
twice: revert PoolEntryState.auth_capture → test fails (carrier
empty); revert the race → test times out (SDK hang).

Also drops the @pytest.mark.asyncio decorator (replaced with
@pytest.mark.anyio) on four tests in test_mcp_pool_auth_introspection.py.
The project depends on anyio's pytest plugin (anyio is in deps);
pytest-asyncio is NOT a project dep and CI's test (3.13) failed on
those four. Local pytest happened to pick it up via system Python.

Found via Copilot review on PR #481.

(cherry picked from commit 97086fc617)
2026-05-07 17:35:21 -07:00
Patrick Buckley bde0913442 feat(mcp): SDK 401/403 introspection via httpx response hook
Phase 6 of OAuth-MCP. Recovers upstream 401/403 from MCP servers via a
capturing httpx_client_factory: an async response hook records 4xx
status + WWW-Authenticate header into a per-dispatch carrier before
the SDK's post_writer swallows the underlying httpx.HTTPStatusError.

Splits _classify_failure into auth_401 (refresh-and-retry once) vs
auth_403 (parse insufficient_scope, emit mcp_insufficient_scope with
parsed scope set). The 401 retry runs on a fresh asyncio.Task via
run_coroutine_threadsafe in _dispatch_pool_sync, escaping the anyio
cancel-scope state of the prior dispatch's TaskGroup.

WWW-Authenticate parsing extracted to a new mcp_http_parsers module
with an RFC 7235 challenge tokenizer (replaces hand-rolled substring
scanners). Two-layer defense against multi-Bearer-challenge injection:
the hook uses get_list("www-authenticate")[0] to drop attacker's
second challenge, the parser truncates at challenge boundary as
belt-and-braces. Scope set capped at 32 entries before hitting the
audit row or the LLM-visible structured-error JSON.

Auth failures (401/403) never trip the per-server circuit breaker
(server-only breaker invariant). Static path remains byte-identical.
_PgRefreshLock untouched. Pool dispatch still reachable from the
agent loop only via Phase 7 catalog scoping; Phase 6 behaviour is
testable via direct call_tool_sync.

5557 tests pass. 33 tokenizer unit tests in tests/test_mcp_http_parsers
cover the RFC 7235 grammar + the scope/error wrappers + the 4 KB input
cap. 7 integration tests in tests/test_mcp_pool_auth_integration drive
real upstream 401/403 through streamablehttp_client + a FastMCP
subprocess fixture — the structural exit gate that makes
HTTPStatusError-injection-only unit tests insufficient.

(cherry picked from commit db9260d8c4)
2026-05-07 17:35:21 -07:00
Patrick Buckley 570b198f1b fix(man): accept canonical name(section) page notation
Models often emit page references in the standard man-page form
(``printf(3)``, ``open(2)``, ``perlfunc(3pm)``) rather than splitting
them into ``page`` + ``section`` args. The page-name sanitizer was
rejecting the parens as invalid input, killing the call. Parse the
section out of the page string before sanitization (explicit
``section`` arg still wins) and widen the section validator to accept
multi-letter suffixes like ``3pm`` / ``3perl`` that already appear on
real systems.

(cherry picked from commit 39a6b7b447)
2026-05-07 17:35:21 -07:00
Patrick Buckley 96d935f1f7 fix(mcp): cancellation-safe orphan-lock drain + lock-reorder + test integrity
Phase 5 PR #479 review fix-up. Three review rounds (bot + two internal
multi-stage /review) caught:

- _PgRefreshLock now allocates a per-instance ThreadPoolExecutor instead of
  a module-global single-worker one. The global shape preserved psycopg2
  thread-affinity but serialized every advisory-lock acquire on the node
  behind one thread, even for unrelated (user, server) keys.
- get_user_access_token_classified flips to `async with lock, pg_lock:` so
  concurrent same-key callers serialize on the in-process asyncio.Lock
  before allocating the pg_lock's per-instance executor + spin loop. N
  concurrent same-key callers collapse to one executor allocation.
- _drain_orphan_pg_lock no longer re-awaits the cancelled asyncio Future
  from `__aenter__`. It receives the underlying concurrent.futures.Future
  and re-wraps it via asyncio.wrap_future, getting an independent asyncio
  Future tied to the worker outcome. This way cancellation of the awaiter
  doesn't poison the drain's wait, and the drain genuinely waits for the
  worker to settle before deciding whether to call cm.__exit__.
- Module-level _pg_refresh_drain_tasks set holds strong refs to in-flight
  drains (asyncio's task set is weak — fire-and-forget tasks could be GC'd
  mid-cleanup; RUF006 hazard).
- Drain narrows except clauses to Exception so a drain-task cancellation
  records as cancelled instead of being silently logged as 'completed
  normally with no acquire'.

Test integrity (was a major finding in round 2 — old generator-based cm
let the test pass via GC finalization timing rather than drain logic):

- New _ObservableLockCm class-based context manager whose __exit__ is a real
  observable method (records call args + thread). Distinguishable from
  GeneratorExit thrown by GC of a generator-based cm.
- Strong external ref to the cm via created_cms list — keeps cm alive past
  the test's awaits, so a no-op drain genuinely fails the assertion rather
  than papering over via GC timing.
- Deterministic drain wait via _pg_refresh_drain_tasks gather — no
  fixed-duration sleeps.
- _run_cancel_scenario helper drops the duplicated setup between the two
  cancellation tests.

Negative-test verified: replacing _drain_orphan_pg_lock body with `return`
makes test_pg_refresh_lock_cancellation_releases_on_same_thread fail with
'drain did NOT call cm.__exit__ — orphan Postgres lock + open transaction'.

Other fixes: protocol docstring corrected to describe pg_try_advisory_xact_lock
spin + retry (was claiming pg_advisory_xact_lock blocking acquire);
get_user_access_token_classified docstring rewritten for new lock order;
narrow `except BaseException` -> `except Exception` in
test_mcp_user_pool.py concurrent-dispatch helper.

882 tests pass (MCP + auth + storage). ruff + mypy clean.

(cherry picked from commit 3eb9d22ad5)
2026-05-07 17:35:21 -07:00
Patrick Buckley 1a1043c4df feat(mcp): per-(user, server) ClientSession pool with OAuth dispatch
Phase 5 of OAuth-MCP — adds a per-(user, MCP-server) ClientSession
pool to MCPClientManager alongside the existing static-server path,
gated entirely on the per-server `auth_type='oauth_user'` config.

Pool architecture:
- `_user_pool_entries: dict[(user_id, server_name), PoolEntryState]`
  with lazy connect on first dispatch, per-key asyncio.Lock allocated
  on the mcp-loop, idle eviction coroutine (default 600s TTL, LRU cap
  200), and an `in_flight` counter as the eviction interlock so live
  calls can never be torn down mid-flight.
- `_dispatch_pool` runs the token-state machine: missing token →
  `mcp_consent_required`; key-rotation decrypt failure →
  `mcp_token_undecryptable_key_unknown` with NO consent prompt and NO
  auto-delete; expired token → silent refresh under per-(user, server)
  advisory lock; refresh failure → revoke + consent.
- `_classify_failure` separates transport (trips breaker) from auth
  401/403 (does NOT trip breaker — server-only invariant) from
  protocol (no breaker change).
- `entry.open_lock` held only across connect-or-reuse and released
  before the `await session.call_tool` so concurrent calls from one
  user against one server overlap (validated by Spike 1 scenario 2).

Auth-class failures are fail-soft in Phase 5: any 401/403 surfaced by
the SDK propagates to the agent as a tool error and the next dispatch
reconnects on a fresh refresh. Real introspection of upstream 401/403
is a Phase 6 concern — the MCP SDK's `streamable_http` post_writer
swallows `httpx.HTTPStatusError` upstream, so detecting status from
the response chain requires `McpError(CONNECTION_CLOSED)` payload
parsing or a custom httpx middleware around `streamablehttp_client`.
The mid-flight 401 refresh-retry path and the `mcp_insufficient_scope`
structured error for 403 step-up land together in Phase 6, gated by
an integration test that drives a real upstream 401/403 (the unit-
test injection of `HTTPStatusError` is what masked the production gap
on the first apply-findings pass — the integration test is the
structural gate so the gap can't reopen). RFC §1.5 steps 4-5 and the
phase table in §Implementation phases reflect this scope split.

Multi-node refresh contention:
- New `StorageBackend.acquire_advisory_lock_sync` Protocol method.
  SQLite returns nullcontext (single-node, in-process asyncio.Lock
  is sufficient). Postgres uses `pg_try_advisory_xact_lock` with
  retry on a fresh per-attempt connection, so waiters don't pin pool
  connections during the AS roundtrip. Inner try/except + nested
  finally ensures conn is always returned to the pool, even when
  begin / execute / yield / commit raises mid-body.
- Lock ordering: pg_advisory outer, asyncio.Lock inner. Re-read after
  lock collapses cluster-wide contention to one HTTP roundtrip per
  (user, server) per refresh window.
- `_PgRefreshLock` enter/exit pinned to a single-worker
  ThreadPoolExecutor so SQLAlchemy connection state stays
  thread-affine across cancellations.

Token storage refactor:
- `get_user_access_token_classified` returns a tagged TokenLookupResult
  (Token / MissingToken / DecryptFailure / RefreshFailed) so the
  dispatcher maps each state to the right user-facing error.
- `get_user_access_token` is now a thin wrapper around the classified
  variant; the previous duplicated state machine is gone.

Security:
- Pool dispatch + admin endpoints reject `http://` URLs for
  `auth_type='oauth_user'` servers (only exact loopback hostnames are
  exempt — `*.localhost` is intentionally NOT honored because RFC 6761
  localhost-zone resolution is configuration-dependent and could route
  bearers to non-loopback IPs via custom resolvers / hosts file /
  Docker overlays). Validated at three layers:
  `_dispatch_pool` (structured `mcp_oauth_url_insecure` error),
  `_connect_one_pool` (defensive ValueError), and
  `admin_create_mcp_server` / `admin_update_mcp_server` (400 before
  storage write).
- Admin URL change on an oauth_user row purges per-user OAuth tokens
  bound to the old URL: bearers are bound (via OAuth resource /
  audience) to the URL active at consent time, so silently rebinding
  them to a new URL is a token-binding violation. Re-consent forces
  fresh issuance for the new resource.
- Encryption-key fingerprints stay in audit logs only; no longer
  surfaced in agent-facing error payloads.

User_id thread-through:
- `MCPClientManager.call_tool_sync(..., user_id=None)` (additive;
  default None preserves the static path byte-identically).
- `ChatSession._exec_mcp_tool` passes `self._user_id or None`.
- `set_app_state(app_state)` setter wires OAuth state at lifespan
  startup, called from both turnstone-server and turnstone-console.

Performance:
- LRU cap eviction iterates `_user_pool_entries` (not
  `_user_pool_last_used`) so pre-dispatch entries are eligible.
- Eviction batch closes via `asyncio.gather` instead of serial await.
- `_resolve_pool_target` returns the resolved server row to
  `_dispatch_pool` to eliminate the second DB lookup.
- Production reachability of pool dispatch is gated on Phase 7
  (catalog scoping) wiring pool tools into `_tool_map`; until then
  pool dispatch is reachable only via direct `call_tool_sync` with a
  prefixed name (the path the new pool tests exercise).

Hardening parity preserved:
- Static path (auth_type ∈ {none, static}) byte-identical; PR #296
  hardening (SDK #2147 mitigations, anyio cancel-scope, stale-session-
  and-stack guard, server-only circuit breaker) intact.
- `test_reconnect_preserves_static_state_identity` unchanged + green.
- `MCPTokenStore.get_user_token` does not auto-delete on
  MCPTokenDecryptError (key-rotation safety).
- Notification debounce stays manager-level.
- Connect-failure cleanup factored into
  `_safe_teardown_on_connect_failure` shared by both connect paths.

Tests: 5475 → 5493 (+18). New file `tests/test_mcp_user_pool.py`
plus additions to test_mcp_oauth_refresh.py, test_mcp_admin_api.py,
and test_mcp_client.py covering: pool data structures, lazy connect,
eviction TTL + LRU + lock interlock, dispatch state machine (token
states), failure classification, http-rejection at dispatch and
admin layers, URL-change-purges-tokens (sec), concurrent dispatch on
one (user, server), pg_advisory lock parity, and user_id threading.

Phase exit criterion (synthetic load test 50 users × 3 servers × LRU
30 × 1000 calls × 200 evictions) deferred to a post-Phase-5 fitness
spike that runs against a staging deployment with real FDs and real
network behaviour, not a CI mock — same shape as Spike 1's
pre-Phase-0 SDK validation.

Out-of-scope for Phase 5 (Phase 6+): SDK-level 401 refresh-retry +
403 `mcp_insufficient_scope` (Phase 6), per-user catalog scoping
(Phase 7), consent UX SSE event + dashboard renderer (Phase 8),
admin UI status indicators (Phase 9).

(cherry picked from commit 4db7d9c6cf)
2026-05-07 17:35:21 -07:00
Patrick Buckley 55aab54774 test(mcp): SDK 1.27 concurrency spike for per-(user, server) pool
Spike artifact validating MCP SDK behavior before Phase 5 builds the
per-(user, MCP-server) ClientSession pool. Three scenarios, all pass:

1. N=20 concurrent ClientSession instances against the same URL — no
   FD blow-up, no shared transport state, each session's tools/list
   returns independently.

2. Two concurrent tools/call on a shared ClientSession with
   interleaving payloads — request_id demux works under contention.

3. Per-session Authorization header isolation across 5 sessions —
   httpx connection pooling does not cross headers between sessions,
   so per-session bearer tokens reach the server unmixed.

Outcome gates the Phase 5 architecture (lazy dict[(user_id,
server_name), ClientSession] + per-key asyncio.Lock + LRU eviction).
Had any scenario failed, the fallback was per-call header injection
(Alternative F in the OAuth-MCP RFC).

Spike-only — not collected by pytest. Run manually:

  uv run python tests/spike_sdk_concurrency.py

(cherry picked from commit e695a98c54)
2026-05-07 17:35:20 -07:00
Patrick Buckley 0f8c8b38a3 fix(mcp): pin OAuth return_url + sanitise read-scope status
Addresses ten findings on the Phase 4 OAuth-MCP commit: four from the
PR #478 review surface, plus six surfaced by a follow-up multi-stage
review of the first round of fixes. Two of the latter were genuine
security regressions in the very code that claimed to close those
holes.

Security
--------

- _validate_return_url now pins return_url same-origin against the
  configured oidc_config.redirect_base instead of request.url. Behind
  a permissive front proxy that did not normalise Host, an attacker
  could spoof Host and provide a matching absolute return_url to mint
  an open redirect off /api/mcp/oauth/start. Same fix pattern as
  PR #476 OIDC.
- Reject return_url values containing literal backslashes or starting
  with `//` up front. urlparse leaves backslashes inside `path`, so a
  value like `/\evil.example/foo` slipped through the path-only branch
  and became the protocol-relative `//evil.example/foo` after WHATWG-
  conformant browsers normalised the backslash — re-introducing the
  open redirect the same-origin pin was meant to close.
- internal_mcp_status (read-scoped) projects through a new
  _strip_server_status_for_read helper that drops the verbose `error`
  text and replaces it with a coarse `has_error` boolean. The error
  string is built as `f"{type(exc).__name__}: {exc}"` and so carries
  stdio binary paths (FileNotFoundError) or internal MCP URLs
  (httpx.ConnectError) — equivalent to leaking command/url, which
  this same patch deliberately strips. Approve-scoped refresh and
  reconnect callers continue to receive the full `error` text via
  the existing _strip_server_status helper.
- internal_mcp_status now returns the projected (sanitised) entries
  for every server in mcp_mgr.get_all_server_status() instead of
  emitting the un-sanitised dict that included `command` (stdio argv)
  and `url` (remote MCP endpoint). Sibling refresh/reconnect endpoints
  already used _public_server_status to strip these.
- internal_mcp_status docstring documents the trust boundary — server
  enumeration to read scope is intentional so dashboards can render
  per-server indicators; verbose error detail and command/url remain
  approve-scoped.

Correctness / UX
----------------

- _validate_return_url comparison normalises (scheme, host, port)
  before equality. Lowercases hostname and collapses the scheme's
  default port, so `https://App.Example.COM/x` and
  `https://app.example.com:443/x` are recognised as same-origin
  with `redirect_base = https://app.example.com` instead of being
  silently downgraded to the `/` fallback.
- mcp_crypto startup-gate error message now names both
  `mcp_token_encryption_keys` (rotation list) and
  `mcp_token_encryption_key` (single) so an operator using rotation
  isn't misled into thinking only the singular form is valid.

Cleanup
-------

- Delete the unused _KNOWN_TRUSTED_ENDPOINT_HOSTS legacy re-export
  shim in oidc.py (zero callers — a no-op that survived the Phase 4
  oauth_ssrf extraction). Sphinx :data: docstring reference at
  validate_discovered_endpoint updated to point at
  turnstone.core.oauth_ssrf.KNOWN_TRUSTED_OAUTH_ENDPOINT_HOSTS
  directly. The Google multi-origin allowlist is unaffected — it
  lives at the canonical name and is read from oauth_ssrf.py:164.
- test_mcp_oauth_handlers TestValidateReturnUrl imports
  _validate_return_url at module level instead of repeating the
  import inside each test method.
- test_server_lifespan_mcp_crypto replaces a fragile
  `messages.count("mcp_token_encryption_key") >= 2` substring trick
  with `re.search(r"mcp_token_encryption_key(?!s)", messages)` —
  asserts the singular form directly via negative lookahead.

Tests
-----

5448 pass (+13 vs the prior tip):

- TestValidateReturnUrl gains backslash-bypass, protocol-relative,
  default-port, uppercase-host, and explicit-port-mismatch cases
  alongside the original same-origin / cross-origin / scheme-
  mismatch / path-only cases.
- TestInternalMcpStatusEndpoint asserts the `error` text never
  reaches the read-scope wire (binary-path FileNotFoundError no
  longer appears anywhere in the rendered response) and that the
  coarse `has_error` boolean lights up correctly on the failed
  server.
- TestInternalMcpStatusEndpoint also pins the no-mcp-client path to
  `{"servers": {}}`.
- _routes_with_internal extended to include the
  /api/_internal/mcp-status route so the new tests can exercise it
  through TestClient.
- Existing test_startup_aborts_with_oauth_user_row_and_no_key
  strengthened to require both singular and plural key names appear
  in the error log.

(cherry picked from commit 62bbc332af)
2026-05-07 17:35:20 -07:00
Patrick Buckley b0f7029ff1 feat(mcp): per-(user, server) OAuth 2.1 + PKCE flow
Lands the OAuth flow that uses the token-at-rest store from the prior
commit: discovery (RFC 9728 PRM + RFC 8414 AS metadata with operator-
override precedence), PKCE S256 (mandatory — refuse AS without it),
RFC 8707 resource indicator on every authorize and token request,
RFC 7591 minimal one-shot dynamic client registration, authorization-
code exchange, refresh-token grant with re-read-after-acquire single-
flight lock, and the /v1/api/mcp/oauth/{start,callback} endpoints
mounted on both server and console.

Refactored:
- validate_url_no_ssrf, validate_discovered_endpoint, is_localhost,
  effective_port, sanitize_log_text moved out of oidc.py into a shared
  oauth_ssrf module; oidc.py re-exports for compatibility. The shared
  helpers also expose async wrappers (validate_url_no_ssrf_async,
  validate_discovered_endpoint_async) so OAuth-MCP discovery — invoked
  from async handlers — does not block the event loop on the
  synchronous socket.getaddrinfo call.
- MCPTokenStore.get_oauth_client_secret reader path added (the prior
  commit was write-only)
- Storage protocol gains create/pop/cleanup_*_mcp_oauth_pending_state
  and get_mcp_oauth_client_secret_ct (mirror OIDC pending-state
  pattern: SQLite BEGIN IMMEDIATE select-then-delete, Postgres atomic
  DELETE...RETURNING)

Refresh-grant correctness:
- When the AS omits refresh_token (RFC 6749 §6 — MAY rotate), the
  existing refresh value is preserved at the OAuth-flow layer rather
  than cleared, so production ASes (Google, Auth0 default, Okta) don't
  force re-consent every hour
- expires_in accepts int, float, str-with-decimal — earlier int-coerce
  through str() failed on float and silently dropped expiry tracking
- The refresh-grant `resource=` parameter (RFC 8707) is the canonical
  MCP server URL, not the audience. Audience and resource are distinct
  concepts; using audience as resource would mismatch the AS RS
  allowlist.

Audience handling:
- _validate_token_audience accepts str or tuple; the callback resolves
  accepted_audiences = {server_url, oauth_audience} and validates
  against the set, so Auth0-style ASes that honor `audience=` (not
  RFC 8707 `resource=`) issue tokens that pass audience-bound
  validation
- build_authorize_url emits both `resource=` (RFC 8707) and
  `audience=` (Auth0-style) per server config; comment documents which
  AS implementations need which form

Security hardening:
- redirect_uri pinned to oidc_config.redirect_base instead of the
  request Host header — closes the same Host-header injection PR #476
  fixed for OIDC. Both /start and /callback return 503 with operator-
  actionable hint when redirect_base is unset
- DCR registration runs under per-server asyncio.Lock with re-fetch
  inside the lock, so concurrent /start callers don't both register
  and overwrite each other's client_id (the second user's code is no
  longer rejected on callback)
- /callback error branch pops the pending state row before redirecting
  so a leaked state can't be replayed against a separately-obtained
  code in the 60s cleanup window
- WWW-Authenticate Bearer parser handles RFC 7235 quoted-string
  escapes (\" and \\) instead of the naive [^"]+ regex
- AS-controlled response bodies and error_description query params go
  through sanitize_log_text before reaching exception messages or
  audit details. AS error responses are parsed for the standard
  RFC 6749 fields (error, error_description, error_uri), each
  capped at 80 chars and run through redact_credentials to defend
  against ASes that echo the request body back into their error
  payload.
- oauth_as_issuer_cached is re-validated against the SSRF guard on
  read; on rejection the column is cleared and PRM rediscovery runs
- DCR / token-endpoint / refresh-endpoint response bodies cap at 64
  KiB (PRM/AS metadata cap stays at 256 KiB) so a hostile or
  malfunctioning AS can't exhaust client memory.
- oauth_client_secret operator input capped at 1024 chars at the
  admin-form boundary; longer plaintext rejected with 400.
- /start and /callback responses stamp `X-Frame-Options: DENY` so the
  redirected pages can't be framed by attacker sites.
- delete_user cascades to mcp_user_tokens and mcp_oauth_pending so
  user deletion no longer leaves dangling per-user OAuth state.
- Renaming or deleting an oauth_user MCP server purges per-user
  tokens and pending OAuth state for the previous server name
  (delete_mcp_oauth_rows_by_server_name). The OAuth tables key on the
  mutable server_name; without this purge, a future server with the
  same name (and an attacker-controlled URL) would silently rebind
  prior user tokens. A future schema migration will replace the
  server_name key with a server_id FK + ON DELETE CASCADE.
- get_user_access_token catches MCPTokenDecryptError (raised when no
  installed key can decrypt the row, e.g. after key rotation) and
  falls through to None so dispatch surfaces a re-consent rather than
  crashing.
- oauth_user MCP server rows are skipped in the static auto-connect
  path. Auto-connecting them at startup with empty headers fails the
  AS check and trips the circuit breaker; per-user tokens come online
  lazily once the user has consented.

Audit (mcp_server.oauth.* prefix):
- consent_started, consent_completed, consent_failed, token_refreshed,
  token_revoked, dcr_registered. _audit_event is async and wraps
  record_audit in asyncio.to_thread so the audit write doesn't block
  the event loop. resource_id on the audit row is the immutable
  server_id (PK UUID) so admin-driven server renames don't break
  event correlation; server_name is exposed in detail for cross-
  reference. dcr_registered detail.has_secret reflects whether the
  DCR-issued secret was actually persisted (the prior code reported
  has_secret=true even on persistence failure).
- _admin_mcp_action audits the immutable server_id, not the mutable
  server_name (which is what the column is — the table's PK was
  always server_id).
- All OAuth-flow log keys use the mcp_server.oauth.* prefix to match
  the audit-action taxonomy.

Lifespan close-order in turnstone.server and turnstone.console.server
is reversed (LIFO) — mcp_oauth → mcp_crypto → oidc — to match init
order.

Deferred until the upcoming per-user pool integration:
- Multi-node refresh-lock contention via pg_advisory_lock
- DCR re-register on token-endpoint 401 (the dispatch path surfaces
  those 401s)
- TTL-LRU caching of decrypted plaintext access tokens
- DNS-rebinding hardening (httpx Transport pin) — documented as
  limitation in oauth_ssrf module docstring

Tests: 7 new test files / ~85 new tests covering discovery precedence
+ PRM quoted-string parsing, PKCE round-trip, SSRF helper extraction,
authorize/callback handlers including 503-on-no-redirect-base + DCR
concurrency + JWT audience polymorphism + callback-error-pops-pending,
refresh single-flight lock, refresh resource-vs-audience regression,
decrypt-error fallthrough, _db_servers_to_config skipping oauth_user,
pending-state CRUD round-trip.

(cherry picked from commit 29c42c1427)
2026-05-07 17:35:20 -07:00
Patrick Buckley a4c335d7bf feat(mcp): token-at-rest encryption layer for OAuth-MCP
Phase 3 of docs/design/oauth-mcp.md. Adds the Fernet/MultiFernet wrapper,
[security] config loader with rotation support, MCPTokenStore CRUD facade,
typed MCPTokenDecryptError that maps to the RFC's mcp_token_undecryptable_
key_unknown class, and a startup gate that fails loud when auth_type=
'oauth_user' rows exist without a configured encryption key.

Crypto module (turnstone/core/mcp_crypto.py):
- MCPTokenCipher wraps cryptography.fernet.Fernet + MultiFernet for
  rotation; encrypt with first key, decrypt by trying each in order
- load_mcp_token_cipher_config reads [security] mcp_token_encryption_keys
  (plural list) or mcp_token_encryption_key (singular), validates each
  key is base64-decodable to exactly 32 bytes
- MCPTokenCipherConfig is repr=False with custom __repr__ that redacts
  raw key bytes (defense in depth against accidental log/traceback leak)
- _key_fingerprint produces an 8-hex-char SHA-256 prefix for audit
  attribution without exposing the key
- MCPTokenStore handles encrypt-on-write / decrypt-on-read for
  mcp_user_tokens and mcp_servers.oauth_client_secret_ct
- get_user_token MUST NOT auto-delete the row on MCPTokenDecryptError
  (test_get_user_token_with_wrong_key_raises_decrypt_error verifies
  the row stays intact across a key-mismatch read)
- initialize_mcp_crypto_state / close_mcp_crypto_state lifespan helpers
  shared between server and console

Storage protocol (5 new ciphertext-only methods):
- set_mcp_oauth_client_secret_ct (dedicated writer; deliberately NOT
  added to MCP_SERVER_MUTABLE so generic update_mcp_server cannot write
  the secret column)
- create_mcp_user_token, get_mcp_user_token,
  update_mcp_user_token_after_refresh, delete_mcp_user_token

Server + console lifespans (turnstone/server.py + console/server.py):
- after OIDC init, count auth_type='oauth_user' rows; if any exist and
  no encryption key is configured, log an actionable error and
  raise SystemExit(1)
- without oauth_user rows, missing key is fine (lazy validation; admin
  flip without restart returns 503 from the admin handler)
- app.state.mcp_token_cipher / .mcp_token_store populated when key
  configured; None otherwise

Admin handlers:
- _require_token_store_for_oauth_secret pre-mutation gate validates
  token_store availability and oauth_client_secret type BEFORE
  storage.create_mcp_server / update_mcp_server runs, so a 503 from a
  missing key never leaves an orphan row or partial-update state
- _apply_oauth_client_secret encapsulates the encrypt + audit write
  used after the storage mutation; rolled out across both create and
  update handlers
- 503 message references both mcp_token_encryption_key (singular) and
  mcp_token_encryption_keys (plural for rotation)
- non-string oauth_client_secret payloads (false / 0 / lists / dicts)
  are rejected with 400 instead of being str()-coerced
- when auth_type transitions away from oauth_user, the encrypted
  secret column is cleared in the same admin call (with audit), so
  flipping back doesn't silently resurrect a stale credential

Audit events (mcp_server.oauth.* per audit.py taxonomy; RFC's
mcp.oauth.* renamed for consistency):
- mcp_server.oauth.client_secret_set fired from admin handlers with
  cleared:bool and key_fingerprint
- mcp_server.oauth.token_decrypt_failure fired from MCPTokenStore
  .get_user_token when no installed key can decrypt; carries
  key_fingerprints_attempted

Tests: 35 new tests across test_mcp_crypto, test_mcp_token_store,
test_server_lifespan_mcp_crypto, plus 6 admin-API tests covering the
no-orphan-row, no-partial-update, secret-clear-on-transition, and
non-string-secret-rejection invariants. Suite at 5337 (Phase 3 added
~50 tests including the rebase-imported skill suite).

cryptography>=42 promoted from transitive (lacme[tls]) to direct dep
since the encryption layer is now core, not optional.

Phase 4 (OAuth flow) wires the actual callers; Phase 3 adds only the
crypto layer and is exercised entirely by tests.

(cherry picked from commit 7f132e7230)
2026-05-07 17:35:20 -07:00
Patrick Buckley 21663d1567 feat(mcp): oauth schema + minimum admin form
Adds the data model and admin UI surface required by the OAuth-MCP flow.
Phase 2 of the per-user delegation initiative.

Schema:
- migration 049 creates mcp_user_tokens (PK user_id, server_name) and
  mcp_oauth_pending (PK state, indexed by created_at)
- eight new columns on mcp_servers: auth_type ('none' / 'static' /
  'oauth_user', NOT NULL DEFAULT 'static') plus six oauth_* config
  fields and oauth_as_issuer_cached
- post-upgrade UPDATE normalises auth_type to 'none' for streamable-http
  rows whose headers are NULL/empty/'{}'; stdio rows are left at the
  'static' default (auth_type is HTTP-auth-only)
- _schema.py kept in lockstep with the migration so metadata.create_all
  and alembic upgrade produce identical shapes
- mcp_user_tokens / mcp_oauth_pending TypedDicts in _protocol.py for
  Phase 3/4 use (no CRUD methods yet)

Storage / API:
- create_mcp_server gains the eight kwargs across protocol + sqlite +
  postgresql
- MCP_SERVER_MUTABLE picks up auth_type and the six text oauth_* fields;
  oauth_client_secret_ct is intentionally NOT in the whitelist — Phase 3
  will own ciphertext writes via a dedicated method
- McpServerInfo + Create/Update Pydantic schemas extended; oauth_client_secret
  accepted as plaintext input but discarded (Phase 3 wires encryption)

Admin handlers:
- _parse_auth_type validates against {'none', 'static', 'oauth_user'} and
  rejects empty / unknown values; shared between create and update
- when auth_type changes away from 'oauth_user', the oauth_* config
  columns are explicitly nulled in the same UPDATE so the row stays
  consistent
- _clean_oauth_text caps text fields at 512 chars (URLs at 2048) to bound
  admin write surface
- _mask_mcp_secrets now masks oauth_client_secret_ct to '***' regardless
  of reveal=true (write-only field)
- audit detail dict redacts oauth_client_secret if present

Frontend:
- new "Multitenant Authorization" fieldset on the MCP-server modal with
  three radio buttons (None / Shared / Per-user OAuth 2.1)
- conditional OAuth subform: AS URL, registration mode (preregistered /
  dcr; cimd is future), client ID, client secret, scopes, audience
- secret input is autocomplete=off and never round-trips on edit
- audience auto-populates from the MCP server URL on blur
- headers textarea hidden and submitted as {} when auth_type is 'none' or
  'oauth_user' so flipping the radio cleans up server-side state

Tests: storage round-trip for the new columns, oauth_pending table smoke,
migration 049 upgrade/downgrade with stdio-vs-http normalisation, four
admin-API tests for auth_type validation and oauth_*-clear-on-flip-away.
Suite passes 5284 (matched pre-Phase-2 baseline 5267 + 17 new).

Stacks on Phase 0; no behavioural change for existing rows.

(cherry picked from commit d675b237a3)
2026-05-07 17:35:20 -07:00
Patrick Buckley c823156af5 refactor(mcp): consolidate per-server state into StaticServerState dataclass
Phase 0 of the OAuth-MCP RFC: prepare MCPClientManager for the per-(user,
server) session pool that lands in Phase 5, without changing static-path
behavior.

Two changes:

1. Hardening helpers _pre_close_streams and _tcp_probe rename their first
   parameter from `name` to `key`.  Type stays `str` for now; widening to
   `str | tuple[str, str]` happens in Phase 5 when callers actually pass
   tuples.  _safe_close_stack takes the stack directly and is unchanged.

2. The eleven parallel name-keyed dicts (_sessions, _per_server_stacks,
   _per_server_tools, _per_server_resources, _per_server_prompts,
   _supports_list_changed, _supports_resources, _supports_resource_list_changed,
   _supports_prompts, _supports_prompt_list_changed, _server_streams) are
   consolidated into _static_servers: dict[str, StaticServerState].  Server-
   level state (circuit breaker, notification debounce, last-error,
   db-managed, merged catalog maps, listener lists) stays on the manager,
   unchanged.

PoolEntryState is defined for Phase 5 use but no code instantiates it.  The
typed map declarations (dict[str, StaticServerState] vs dict[tuple[str, str],
PoolEntryState]) make accidental cross-keying lookups easier to catch.

PR #296 hardening preserved exactly:
- pre-close-streams atomic take-and-clear before stack teardown
- stale-session-and-stack guard at _connect_one top: both state.session and
  state.stack checked, cleared independently, entry preserved (not popped)
- transport-error session-eviction in dispatch sets state.session=None only,
  leaving stack/streams for the next connect-time guard sweep
- _safe_close_stack CancelledError suppression unchanged
- TCP probe before streamablehttp_client unchanged
- future.cancel() after TimeoutError in all sync bridges unchanged
- notification debounce stays manager-level (not migrated into the dataclass)

Refresh helpers (_refresh_server_tools/_resources/_prompts) snapshot
state.session into a local immediately after the None guard so concurrent
transport-error eviction during await cannot null the session reference
mid-call.

Tests: shared _seed_static_state helper in tests/conftest.py replaces eleven
direct dict mutations; new test_reconnect_preserves_static_state_identity
guards the entry-preservation invariant.  Pass count rises 5266 → 5267.

(cherry picked from commit be0950bb98)
2026-05-07 17:35:20 -07:00
Patrick Buckley bace928477 refactor(mcp): remove periodic refresh, add manual refresh/reconnect controls
Deletes the _periodic_refresh task and its supporting state
(_refresh_task, _refresh_failures, _refresh_backoff_until,
_REFRESH_BACKOFF_BASE/MAX, _DEFAULT_REFRESH_INTERVAL, refresh_interval
kwarg) from MCPClientManager. Push notifications and operator-driven
manual refresh now cover all catalog-update needs; the long-running
4-hour timer was dead complexity that obscured the per-user pool
work to come.

Catalog freshness on auto-reconnect is preserved by scheduling an
unblocking _refresh_server task on the mcp-loop after _connect_one
succeeds; the calling thread returns immediately so half-open
recovery latency does not double. Adds MCPClientManager.reconnect_sync
(clears the circuit, closes any existing session, calls _connect_one,
clears stale catalog on failure).

Wires a new pair of operator endpoints —
POST /v1/api/admin/mcp-servers/{name}/refresh and
/v1/api/admin/mcp-servers/{name}/reconnect — that fan out to all
nodes through the existing _internal route family, with per-row
"Refresh" and "Reconnect" buttons in the MCP Servers admin tab.
The new node-internal paths /api/_internal/mcp-{refresh,reconnect}/
are gated to the approve scope to prevent direct unprivileged
reconnects bypassing the console's admin.mcp gate. Internal
endpoints return generic error messages and a filtered status
payload (no command/url) to keep transport details admin-gated.

Drops the [mcp] refresh_interval setting, the
--mcp-refresh-interval CLI flag, and the matching config-mapping
entry; updates docs/architecture.md, docs/tools.md,
docs/settings.md, and the three PlantUML diagrams that referenced
the periodic loop.

Tradeoffs (intentional):
- Idle nodes will not auto-rejoin a recovered MCP server until
  traffic arrives or an operator clicks Reconnect. The previous
  background reconnection loop is gone by design — push
  notifications + operator controls replace it.
- Console fan-out blocks on the slowest node (existing pattern);
  not changed here.

This is Phase 1 of the OAuth-MCP series — feature subtraction
ahead of per-user state.

(cherry picked from commit eb2a119da9)
2026-05-07 17:35:20 -07:00
Patrick Buckley d16c911750 feat(skills): paste SKILL.md to auto-fill the Create Skill modal (#477)
* feat(skills): paste SKILL.md to auto-fill the Create Skill modal

When a user pastes an Anthropic-style SKILL.md (YAML frontmatter +
markdown body) into the Create Skill content textarea, the frontend
sniffs the leading ``---``, posts the raw text to a new backend parse
endpoint, and populates name / description / tags / author / version /
license / compatibility / allowed_tools from the parsed fields.  The
textarea is left with the body only (frontmatter stripped), and a toast
reports how many fields were set vs. kept (already-typed values are
preserved).

Backend
- ``POST /v1/api/admin/skills/parse`` (admin.skills permission) wraps
  the existing ``turnstone.core.skill_parser.parse_skill_md`` so admin
  imports and external installs share one parser.  ``ParseSkillRequest``
  / ``ParseSkillResponse`` schemas added; OpenAPI spec + sync/async
  console SDK methods updated.
- Hardening: 32 KiB cap on ``raw`` (Pydantic ``max_length`` + handler
  enforcement); ``Content-Length`` pre-check returns 413 before any body
  buffering; parse offloaded via ``asyncio.to_thread`` so deeply-nested
  YAML cannot stall the event loop.

Frontend (turnstone/console/static)
- New paste handler with optimistic paint (raw text shown immediately,
  textarea disabled + ``aria-busy`` flipped, hint switches to
  "Parsing...") so the round-trip is visible on slow networks.
- ``AbortController`` + generation guard (``_ctmPasteController``) so a
  fresh paste or modal close cancels a stale fetch — the previous
  handler's callbacks see the controller has been replaced and bail
  before touching the DOM.
- Non-destructive overwrite: ``_setSkillFormField`` returns "filled" /
  "skipped" / "absent" and refuses to clobber non-empty values.  Toast
  reports counts.
- Bumps ``#toast`` z-index above modal overlays (was 200 vs. modal 600
  — toasts fired while a modal was open were invisible).  Console-wide
  fix exposed by this being the first feature to fire toasts mid-modal.

HTML / CSS
- New ``.skill-paste-hint`` line above the textarea announcing the
  affordance, sized to match surrounding ``.label-hint`` text.
- ``aria-describedby`` ties the hint to the textarea; ``aria-live=
  "polite"`` announces the busy-state transition to screen readers.
- "Skill Content" heading hint reworded "system message — ..." →
  "available: ..." and the variables row label "Variables" → "Used"
  to disambiguate available vs. in-use template variables.

Tests
- 11 new cases in ``tests/test_skill_parse_api.py``: happy paths
  (full / minimal / nested-metadata / unquoted-colon recovery),
  malformed YAML 400, missing/blank/missing-name 400, RBAC 403, raw
  body 32 KiB cap (Content-Length pre-check), chunked-encoding bypass
  forces the application-layer cap.  Test pins ``raw_frontmatter``
  omission so a future ``dataclasses.asdict`` refactor can't silently
  leak the full YAML dict back to clients.

Validation
- 5146 / 5146 ``pytest -k "not live"`` pass.
- ``ruff`` + ``mypy`` clean on changed sources.
- ``node -c`` clean on governance.js.
- Two-stage code review (full pipeline + bug+quality re-review of the
  fix patches) applied; all confirmed findings addressed.

* fix(skills): Copilot PR #477 review fixes (cumulative bug-1, bug-2, q-1)

bug-1 (server.py): Content-Length pre-check was clamped to 32 KiB —
the same number as the per-string char cap on ``raw``.  A legitimate
``raw`` of exactly 32 KiB produces a JSON body well above 32 KiB once
the ``{"raw":"..."}`` wrapper and any escaping is added, so valid
near-max requests were 413'd.  New constant
``_PARSE_SKILL_MAX_BODY_BYTES = _PARSE_SKILL_MAX_CHARS * 4`` admits the
wrapper + multibyte expansion while still refusing obviously oversized
payloads early; the per-string ``len(raw)`` check stays authoritative.

bug-2 (governance.js): hideCreateTemplateModal aborted the inflight
paste controller and nulled the global, but the handler's ``.catch``
and ``.finally`` guard each DOM mutation behind ``_isCurrent()`` —
both bail when the controller has been nulled, leaving the textarea
``disabled`` + ``aria-busy`` and the hint stuck on "Parsing…".
Reopening the modal landed on a poisoned state.  The second-pass
review's q-2 cleanup that dropped the show-side defensive reset
missed this scenario — the verifier's reachability argument confused
"controller is null" with "UI state is reset"; the two are
independent.  Hide now resets the paste-induced visible state
alongside the abort.

q-1 (console_spec.py): error_codes for the parse endpoint listed only
400; handler also returns 413 for oversized bodies.  Added 413; kept
403 implicit per the convention sibling admin endpoints follow.

Test fixup: bumped the Content-Length test payload to 200 KB so it
clearly exceeds the new 128 KB pre-check threshold; otherwise it was
falling through to the per-string check and duplicating
test_oversized_raw_chunked_returns_413's coverage.

(cherry picked from commit 0a8083e6d5)
2026-05-07 17:35:20 -07:00
Patrick Buckley b8fadad94f fix(oidc): close transient client on disable paths + correct docstring
PR #476 review feedback (Copilot, oidc.py:584,616):

1. initialize_oidc_state's docstring claimed "on any failure
   enabled is False" but the JWKS-prefetch failure branch
   intentionally keeps enabled=True so the callback's lazy-fetch
   retry can recover from a transient IdP issue at startup.
   Docstring rewritten to spell out the three post-conditions:
   disable, JWKS-failure-keeps-enabled, success.

2. The long-lived httpx.AsyncClient was created up front, then
   three disable branches (discovery exception, discovery-returned-
   disabled, missing redirect_base) returned without closing it,
   leaving sockets held until shutdown.

   Restructured: discovery now uses a transient AsyncClient inside
   a context manager (closed at exit). The long-lived client is
   only created after the disable checks pass. The JWKS-failure
   branch still legitimately keeps the client open because the
   lazy-retry path needs it.

   The pre-existing single-client-passthrough test was replaced
   with three more specific tests: long-lived client only goes to
   fetch_jwks (not discover_oidc); discovery-exception path leaves
   http_client=None; missing-redirect_base path leaves
   http_client=None.

(cherry picked from commit b2153d907f)
2026-05-07 17:35:20 -07:00
Patrick Buckley 63aecdf2fa chore(oidc): consolidate test OIDCConfig helper + fix exceptions banner (cumulative q-4, q-5)
q-4: tests/test_oidc.py's _make_config and tests/test_oidc_handlers.py's
_make_oidc_config built the same OIDCConfig with sensible defaults but
had drifted — only the handlers helper set redirect_base. After b3
made redirect_base operationally required, every test_oidc.py test
that exercised redirect_base had to override it explicitly. A future
test could omit redirect_base and silently exercise the wrong
production path.

Moves make_oidc_test_config to tests/conftest.py with the more
complete handler-version defaults (including redirect_base). Both
test files import it under their existing local alias
(_make_config / _make_oidc_config) so the 60+ call sites in
test_oidc.py and the handler tests don't have to change.

q-5: section banner '# Exception' (singular) at oidc.py:79 became
inconsistent after b5 (callback robustness) added OIDCKeyNotFoundError.
Renamed to '# Exceptions'.

(cherry picked from commit 5d4a50d2cd)
2026-05-07 17:35:20 -07:00
Patrick Buckley cbe8940b30 perf(auth): migrate handle_auth_status to count_users (cumulative q-3)
The OIDC perf batch added storage.count_users() and migrated the two
OIDC handlers (handle_oidc_authorize, handle_oidc_callback) but missed
handle_auth_status — which still ran storage.list_users() then
len(users) > 0 for the same has-any-users gate.

count_users() is one COUNT(*) round-trip vs list_users() rehydrating
every row dict. Wrapped in asyncio.to_thread to match the OIDC handler
pattern; the async handler no longer blocks the event loop on storage
I/O for what's effectively an existence probe.

(cherry picked from commit 7c6bc22d02)
2026-05-07 17:35:20 -07:00
Patrick Buckley 1dcd1e2ec4 fix(oidc): serialise role-mapping concurrency + skip no-op write lock (cumulative bug-2, perf-1)
bug-2 (Postgres) — replace_oidc_roles read existing rows under default
READ COMMITTED with no row lock. Two concurrent OIDC callbacks for the
same user_id (racing token refreshes with differing claim sets) could
both observe the same baseline and produce a final role state matching
neither caller's intent. Adds .with_for_update() to the SELECT so the
existing rows for this user are locked for the duration of the
transaction.

The lock is per-user_id, not table-wide; unrelated user writes are
unaffected. Empty result sets acquire no locks, so a brand-new user
with no rows yet still allows two callers to proceed and merge via
ON CONFLICT DO NOTHING — that's a permissive race that self-heals on
the next reconciliation cycle, documented in code.

perf-1 (SQLite) — replace_oidc_roles took the SQLite global write
lock unconditionally via BEGIN IMMEDIATE before reading. Steady-state
re-logins (claims unchanged, no INSERT/DELETE needed) paid the lock
cost for nothing and serialised against unrelated writers.

Replaces with a double-check pattern: phase 1 reads under the default
deferred transaction (no write lock), computes the diff, and returns
(set(), set()) on no-op. Phase 2, only when mutation is needed,
commits the read txn, escalates to BEGIN IMMEDIATE, RE-READS, and
re-computes the diff under the lock before writing. The returned
(added, removed) reflects what was actually written, so caller logging
in apply_role_mapping stays truthful even when concurrent writers
shifted state between the two reads.

The OR IGNORE on insert is now defense-in-depth (the lock makes it
unnecessary) but kept as a safety net.

(cherry picked from commit d5087ef3b9)
2026-05-07 17:35:20 -07:00
Patrick Buckley 32e29ff255 docs(oidc): document TRUSTED_ENDPOINT_HOSTS + fix three-vs-four required drift (cumulative q-1, q-2)
The 8-commit OIDC stack added TURNSTONE_OIDC_TRUSTED_ENDPOINT_HOSTS
(operator allow-list for cross-host IdP discovery endpoints) and
promoted TURNSTONE_OIDC_REDIRECT_BASE to required, but the docs drifted
in two places:

q-1 — Troubleshooting > "OIDC not configured" still listed three
required env vars. An operator hitting the missing-redirect-base
startup error landed on a debugging entry that didn't mention the
variable they were missing. Fixed; added a separate troubleshooting
entry naming the exact log message produced by initialize_oidc_state
when redirect_base is unset.

q-2 — TURNSTONE_OIDC_TRUSTED_ENDPOINT_HOSTS was undocumented entirely.
Added a row to the env-var table and a new "Cross-host endpoints"
section explaining when the knob is needed (Google is the canonical
multi-origin IdP, but it's auto-handled; the env var is for any other
IdP whose discovery doc legitimately references hosts beyond the
issuer's origin). Added a troubleshooting entry pointing at the new
section.

(cherry picked from commit 3cf87628d2)
2026-05-07 17:35:20 -07:00
Patrick Buckley 3e2fe0bc9d fix(oidc): self-heal stranded user when role mapping fails post-create (cumulative bug-1)
If apply_role_mapping raised after create_oidc_user committed (transient
storage failure, race with role deletion, etc.), provision_oidc_user's
inline safety-net was skipped — and on retry the existing-identity
branch never reached the safety-net code, leaving the user permanently
stranded with zero roles.

Extracts _ensure_default_role(storage, user_id, desired_role_ids=None)
helper. Calls it on BOTH the new-user and existing-identity paths so a
user stranded by a transient failure recovers on next login.
desired_role_ids is a hint that lets the helper skip list_user_roles
when claim-driven mapping populated at least one role; the new-user
path was already paying that query, the existing-identity path now
pays it only when claim mapping returned an empty desired set.

Documents the admin-strip behavior in the helper docstring: stripping
all roles from an OIDC user no longer locks them out, since the next
login will re-grant builtin-viewer (assigned_by='oidc-default'). The
documented way to deny an OIDC user is to unlink their OIDC identity
via the admin endpoint, not to strip roles. The pre-fix behavior
(stripped user actually locked out) was the bug.

The 'oidc-default' vs 'oidc' assigned_by distinction is preserved:
apply_role_mapping's revocation lane only touches 'oidc' rows, so the
safety-net role survives every subsequent login regardless of claims.

Six new tests cover both paths, the hint short-circuit, the
list_user_roles fallback, the missing-builtin-viewer no-op, and the
self-heal regression case for already-stranded users.

(cherry picked from commit 1c41212f15)
2026-05-07 17:35:20 -07:00
Patrick Buckley 5f5eee4aab test(oidc): close coverage gaps + tighten fetch_jwks shape check (q-5, q-8)
q-5: _derive_username's UUID-retry tier (oidc.py:923-933) was untested.
  After perf-6 collapsed tier-2 to a single find_existing_usernames call,
  the only remaining tail was the 3-attempt UUID-retry loop and the final
  raise. New TestDeriveUsername class covers:
  - falls into UUID retry when all 10 suffix candidates are taken
  - UUID retry succeeds on the second attempt after one collision
  - UUID retry exhausted -> raises OIDCError

q-8: filled the unit-level coverage holes the multi-stage review flagged:
  - test_validate_id_token_retry_after_kid_rotation — direct unit test of
    the OIDCKeyNotFoundError path with real RS256 keys + JWKS rotation
    (previously only exercised end-to-end through the handler).
  - test_callback_uses_pending_audience_not_handler_audience — pins down
    the bug-3 fix by decoding the issued JWT cookie and asserting aud
    matches the audience stored at /authorize time, not the handler param.
  - test_apply_role_mapping_int_claim / _dict_claim — exercises the
    else: values = [str(claim_value)] branch for non-string non-list
    claim shapes.
  - TestFetchJWKS — non-200 status, non-dict body, dict-missing-keys,
    keys-not-list, transport network error.
  - TestExchangeCode network/4xx/5xx error tests (the non-dict-body case
    already shipped in batch 5).

Also a small production hardening that fell out of writing the
TestFetchJWKS::test_fetch_jwks_non_dict_body_raises test: fetch_jwks now
guards isinstance(result, dict) before result.get("keys"), matching the
shape-check pattern that discover_oidc and exchange_code already use.
A list/null body now surfaces as OIDCError("...not a JSON object") rather
than AttributeError leaking up to the lifespan.

(cherry picked from commit 5c11ab985f)
2026-05-07 17:35:20 -07:00
Patrick Buckley 6d532ed776 refactor(oidc): quality cleanup (bug-3, q-1/3/4/6/7/9/10/11/12/13)
Eleven small maintenance fixes; no behavior change beyond bug-3.

bug-3: pending.get('audience', audience) couldn't fall back because
  pop_oidc_pending_state always returns a dict with the audience key
  set verbatim from a non-null TEXT column. Replaced with
  pending.get('audience') or audience to cover the empty-string case
  defensively. Comment explains the security rationale.

q-1: extract _env_or_cfg_str / _env_or_cfg_bool helpers in oidc.py;
  load_oidc_config's six near-identical env-or-config blocks collapse
  to one-liners. role_map / trusted_endpoint_hosts / redirect_base
  retain bespoke parsing.

q-3: discover_oidc narrows except (httpx.HTTPError, ValueError, KeyError)
  with exc_info=True.

q-4: OIDC_STATE_TTL_SECONDS = 300 constant in oidc.py; auth.py imports
  and passes it explicitly. Storage signatures keep the literal default
  (storage layer doesn't know OIDC TTL semantics).

q-6: hoist runtime imports (OIDCError, OIDCKeyNotFoundError, exchange_code,
  fetch_jwks, provision_oidc_user, validate_id_token, build_authorize_url,
  generate_pkce_verifier) to module scope in auth.py. The genuine cycle
  is only oidc._derive_username -> auth.is_valid_username, kept
  function-scoped. test_oidc_handlers.py mock targets repointed to
  turnstone.core.auth.X to match the new binding.

q-7: comment + docs explain the 'oidc' vs 'oidc-default' assigned_by
  marker distinction.

q-9: OIDCIdentity / OIDCPendingState TypedDicts in storage protocol.
  Implementations construct via TypedDict syntax so mypy structurally
  verifies all required fields.

q-10: fetch_jwks narrows except (httpx.HTTPError, ValueError); docstring
  matches.

q-11: rename generate_pkce_pair -> generate_pkce_verifier; return only
  the verifier (build_authorize_url already recomputes the challenge).

q-12: extract _buildOidcRow helper in admin.js so future field additions
  go in one place.

q-13: OIDCConfig docstring lists startup-config vs discovery-derived
  field groups.
(cherry picked from commit bae4adca12)
2026-05-07 17:35:20 -07:00
Patrick Buckley c3d9cdae82 perf(oidc): batch perf hardening (perf-1..8)
Eight independent perf wins on the OIDC hot path:

perf-1: list_users() full-scan setup-gate replaced with new count_users()
  on both authorize and callback. Saves a full users-table fetch per login.

perf-2: handle_oidc_callback's sync DB chain wrapped in asyncio.to_thread
  for cleanup, pop_oidc_pending_state, count_users, and provision_oidc_user.
  handle_oidc_authorize gets the same treatment for count_users and
  create_oidc_pending_state. Event loop no longer blocks for the full
  callback duration on Postgres deployments.

perf-3: apply_role_mapping N+1 collapsed via new replace_oidc_roles
  storage method. One transaction handles the diff + insert + delete
  instead of 2N+1 commits per login. Returns (added, removed) so the
  caller can still emit per-role audit logs.

  The diff respects the documented invariant "manually-assigned roles
  are never touched" — desired_role_ids is filtered against rows where
  assigned_by != 'oidc' before computing added/removed. This prevents a
  PK conflict (Postgres lockout) or silent OR-IGNORE no-op (SQLite lying
  return) when admin-ui or oidc-default already holds the same role_id.

perf-4: provision_oidc_user no longer re-queries list_user_roles after
  apply_role_mapping. The new-user builtin-viewer fallback is gated on
  desired_role_ids being empty, which is information apply_role_mapping
  already returned.

perf-5: JWKS refetch dedup via asyncio.Lock on app.state. Both lazy-fetch
  (cold-start recovery) and rotation paths share the same lock with a
  double-check pattern: re-resolve kid against the current cache before
  issuing a new GET. N concurrent callbacks during rotation now produce
  at most 1 fetch.

perf-6: _derive_username's 9-suffix loop collapsed via new
  find_existing_usernames(candidates) -> set query. Worst case drops
  from 13 sequential queries to 1 + up-to-3 UUID-retry queries.

perf-7: cleanup_expired_oidc_states gated to once-per-60s per process
  via app.state.oidc_last_cleanup_monotonic. The pop already deletes
  the consumed row; the bulk cleanup is only relevant for abandoned
  authorize flows, so frequency was overkill.

perf-8: Long-lived httpx.AsyncClient stashed on app.state.oidc_http_client
  by initialize_oidc_state. discover_oidc/fetch_jwks/exchange_code accept
  an optional client= kwarg; when set, skip the per-call AsyncClient
  context-manager. New close_oidc_state lifespan teardown closes it.
  Tests pass client=None to keep the transient-client legacy path.

New storage methods (sqlite + postgresql):
- count_users() -> int
- find_existing_usernames(candidates) -> set[str]
- replace_oidc_roles(user_id, desired) -> (added, removed)

(cherry picked from commit 39a647f39c)
2026-05-07 17:35:20 -07:00
Patrick Buckley 366d316941 fix(oidc): callback robustness — typed exceptions, shape checks, log sanitize, JS race (bug-4, bug-5, bug-6, sec-4)
Four small hardening fixes on the OIDC callback hot path:

bug-4: JWKS rotation retry was matching the substring 'not found in JWKS'
  inside an OIDCError message. A future rephrasing would silently break
  key rotation. Adds OIDCKeyNotFoundError(OIDCError); validate_id_token
  raises the subclass at the kid-not-found site; handle_oidc_callback
  catches it explicitly. Other 'not found' errors in validate_id_token
  remain as plain OIDCError.

bug-5: tokens['id_token'] raised KeyError if the IdP returned 200 without
  id_token. exchange_code now rejects non-dict response bodies; the
  callback validates id_token shape (must be non-empty str) before
  passing to validate_id_token. Both raise OIDCError, surfaced as the
  standard 'Authentication failed' redirect.

bug-6: shared_static/auth.js — the OIDC error display raced showLogin's
  /v1/api/auth/status fetch via a 300ms setTimeout. showLogin now takes
  an optional oidcError parameter and paints it after _switchMode clears
  the error, in both the success and catch branches of the fetch.

sec-4: oidc.py exchange_code's non-200 OIDCError interpolated up to 500
  bytes of attacker-controlled IdP body, which then went to log.warning
  via 'OIDC callback failed: %s'. CRLF in resp.text could forge log
  lines. New _sanitize_log_text helper escapes control chars via
  unicode_escape and caps at the rendered length.
(cherry picked from commit 0af3adae1d)
2026-05-07 17:35:20 -07:00
Patrick Buckley 3e87f4262e fix(oidc): atomic user + identity provisioning to prevent orphan rows (bug-1)
provision_oidc_user previously called create_user (INSERT OR IGNORE
on SQLite — silent no-op on UNIQUE conflict), then create_oidc_identity
(also INSERT OR IGNORE), then apply_role_mapping which writes user_role
rows for the supposedly-new user_id. On a username TOCTOU race or
concurrent (issuer, sub) double-create, both inserts no-opped but
user_role rows were already written — leaving orphan rows pointing
at a user_id that doesn't exist.

PostgreSQL's create_user raised IntegrityError instead of silently
no-opping so it produced a misleading 'Authentication failed' error
without orphans, but the user-facing UX was equally poor.

Adds StorageConflictError to the storage protocol and create_oidc_user
that does both inserts in one transaction. Username collision and
(issuer, subject) collision both raise StorageConflictError, mapped
to OIDCError by provision_oidc_user. Crucially the new code does not
silently bind a colliding-username new identity to the existing user
— that would be an account-takeover vector. It raises.

SQLite uses BEGIN IMMEDIATE inside the try block so lock-contention
errors surface as StorageConflictError instead of leaking the raw
sqlalchemy OperationalError.

PostgreSQL relies on SQLAlchemy 2.x begin-on-demand semantics; the
explicit conn.commit()/rollback() in the catch block is the only
materialization path. Discrimination on PG uses
exc.orig.diag.constraint_name with message-substring fallback.

(cherry picked from commit 11618bb1d7)
2026-05-07 17:35:20 -07:00
Patrick Buckley f50b559792 fix(oidc): require TURNSTONE_OIDC_REDIRECT_BASE; drop Host-header fallback (sec-2)
_build_oidc_redirect_uri previously fell back to the request Host
header when redirect_base was unset. With a permissive reverse proxy
or direct backend access, a spoofed Host minted an authorize URL
pointing to attacker-controlled host — combined with a permissive
IdP redirect_uri allowlist this enables auth-code interception.

There is no production scenario where a Host-derived redirect_uri is
correct, so this fails closed:

- initialize_oidc_state checks redirect_base after discovery succeeds
  and disables OIDC (with an explicit error log naming the env var)
  if it's empty. Runs before fetch_jwks so a misconfigured deploy
  doesn't make a wasted JWKS call.
- _build_oidc_redirect_uri simplifies to f"{redirect_base}/v1/api/auth/oidc/callback".
  request parameter dropped; both call sites (handle_oidc_authorize,
  handle_oidc_callback) updated.
- docs/oidc.md promotes TURNSTONE_OIDC_REDIRECT_BASE from "Recommended"
  to "Required" with the security rationale.

(cherry picked from commit 52aba17740)
2026-05-07 17:35:20 -07:00
Patrick Buckley c6b3c0bc5f refactor(oidc): unify server+console lifespan via initialize_oidc_state (q-2, bug-2)
The OIDC discovery + JWKS prefetch block was duplicated byte-for-byte
between turnstone/server.py and turnstone/console/server.py. The bare
except branch in that block also left app.state.oidc_config unchanged
on unexpected exceptions — leaving the runtime with enabled=True and
empty endpoints, producing malformed authorize URLs.

Extracts initialize_oidc_state(app_state) into turnstone/core/oidc.py
which guarantees a coherent post-condition on every code path:
- discovery exception -> oidc_config replaced with enabled=False, jwks_data=None
- discovery returns enabled=False -> jwks_data=None
- JWKS prefetch fails -> jwks_data=None but enabled=True preserved (the
  callback's lazy-fetch retry path remains the recovery)
- success -> oidc_config + jwks_data both populated

Also hardens discover_oidc against non-dict discovery responses
(list/null/string/int) — previously these raised AttributeError out
of doc.get and propagated past the lifespan's bare except.

server.py and console/server.py lifespan blocks collapse to a single
await initialize_oidc_state(app.state) call.

(cherry picked from commit 6f9e140a41)
2026-05-07 17:35:20 -07:00
Patrick Buckley cefb74a226 fix(oidc): SSRF + plaintext credential exfil via discovery doc (sec-1, sec-3)
OIDC discovery-document endpoints (token_endpoint, jwks_uri,
userinfo_endpoint) were stored verbatim in OIDCConfig and later passed
to httpx without revalidation. Only the issuer URL was checked. A
hostile or compromised IdP could return token_endpoint pointing to an
internal IP (169.254.169.254, 10.0.0.0/8, etc.) and Turnstone would
POST the client_secret there.

Extracts the existing scheme/userinfo/SSRF check into
_validate_url_no_ssrf, adds validate_discovered_endpoint that runs the
same checks plus an issuer-binding check, and wires it into
discover_oidc for authorization_endpoint, token_endpoint, jwks_uri,
and userinfo_endpoint (when present).

Issuer binding accepts:
- Same (scheme, hostname, effective port) as the issuer.
- A hostname in _KNOWN_TRUSTED_ENDPOINT_HOSTS for the issuer (Google's
  multi-origin discovery is in the allow-map by default).
- A hostname in OIDCConfig.trusted_endpoint_hosts, settable via
  TURNSTONE_OIDC_TRUSTED_ENDPOINT_HOSTS env var or config.toml, for
  IdPs not in the static map.

Effective port comparison treats https://host and https://host:443 as
the same origin (urllib.parse.urlparse leaves the explicit form's port
as 443 and the implicit form's as None).

24 new tests cover the validator, the Google known-hosts path, the
operator allow-list, default-port equivalence, foreign-host
rejection, private-IP rejection, embedded credentials, and DNS
rotation between issuer check and endpoint use.

(cherry picked from commit 0df7dc026b)
2026-05-07 17:35:20 -07:00
Patrick Buckley 273d547f4e chore: bump version to 1.5.7 2026-05-04 03:04:09 -07:00
Patrick Buckley bbb404c363 feat(console): inline node picker replaces back-to-console banner (#475)
* feat(console): inline node picker replaces back-to-console banner

Drops the 32px banner the console proxy used to inject above proxied
server-UI pages and replaces it with an inline node-id pill in the
existing #ui-header.  Click the pill to open a dropdown that lists
healthy nodes (health dot, ws count, reachable/degraded/unreachable
text) plus a top-row link back to the console.

Reuses the .ws-tab-dropdown shell from ui/static/style.css for
animation, shadow, theme override, and item layout, so the picker
visually matches the workstream-tab chevron menu it sits next to.
Keyboard nav (ArrowDown/Up/Home/End/Tab/Escape) mirrors the chevron
menu's handler with cross-reference comments at both sites.
Lazy-fetches /v1/api/cluster/nodes against the console origin
(bypassing the prefix shim) on first open.

Reclaims 32px of vertical space, consolidates three separate
"you're on node X via console" indicators into one, and turns the
wayfinding chrome into a real cluster-nav primitive.

* fix(console): address Copilot review on node picker

- Request /v1/api/cluster/nodes?limit=1000 (collector's hard cap)
  instead of relying on the default 100 — clusters with more than
  100 nodes were silently dropping rows from the picker.
- Hand off focus to the first menu item after the async fetch
  resolves: openMenu()'s deferred focus hook ran while only the
  skeleton was in the DOM, so first-open keyboard users were
  stranded on the trigger until they pressed an arrow key.
- Tab now closes the menu without preventDefault, so focus moves
  to the next focusable element on the first press (ARIA APG menu
  pattern).  Escape still preventDefault + returns to the pill.
- Cap pill max-width at 240px and ellipsize the id span; node ids
  are accepted up to 256 chars upstream and could otherwise push
  the title and right-side controls off the appbar.  Pill carries
  a title attribute so the full id is still legible on hover.
2026-05-04 02:59:35 -07:00
Patrick Buckley 072113f7ca fix(session): properly inject queued user messages mid-loop (#474)
* fix(session): properly inject queued user messages mid-loop

Two queued-user-message bugs in ``ChatSession.send()``.

**Mid-tool-call: ``Unexpected role 'tool' after role 'user'`` on Mistral.**
The ``supports_tool_advisories`` capability flag (default False for
unknown openai-compatible models) routed cap-off providers down a
short-circuit branch in ``_collect_advisories`` that called
``_flush_queued_messages`` directly. That appended a ``user`` turn
between ``assistant(tool_calls)`` and ``tool``, which mistral-common's
``_validate_message_order`` rejects with a 400.

Drop the flag. All providers now run the unified path: queued user
messages become ``UserInterjection`` advisories that ride inside the
tool result envelope via ``wrap_tool_result``, splicing
``<system-reminder>`` text into the tool message's content. Role
sequence stays ``assistant → tool``. Live-confirmed on Mistral
medium and Qwen3 — both correctly distinguish system-reminder from
tool stdout in their reasoning.

**Mid-stream: queued message orphaned until next user send.**
After a no-tool assistant turn, ``_flush_queued_messages`` would
append the queued user message to history and the loop would
``break``, leaving the message at the tail of history with no
model response. Visible as "two sends to get one reply".

``_flush_queued_messages`` now returns ``bool``. The no-tool branch
``continue``s on drain instead of ``break``ing, so the model gets a
turn over the extended history.

Tests:
- ``test_collect_advisories_drains_text_queued_messages_to_persistent``
  pins the unified-path drain (text-only queue → ``UserInterjection``,
  no separate user turn appended to ``self.messages``).
- ``test_send_continues_when_messages_queued_during_streaming`` pins
  the loop-continue behavior (fails with 1 stream call pre-fix,
  passes with 2 post-fix).

* fix(session,ui): reject queued attachments + paperclip busy state

Copilot pointed out that the attachment-bearing branch in
``_collect_advisories`` had the same role-ordering bug as the
text-only path that 802658f fixed: an attachment-bearing queued
item would still call ``_append_user_turn`` mid-tool-call,
injecting ``user`` between ``assistant(tool_calls)`` and ``tool``.

Pragmatic fix: don't allow attachments to be queued at all.

**Backend.** ``ChatSession.queue_message`` raises a new
``AttachmentsNotQueueableError`` when called with non-empty
``attachment_ids``. The interactive ``/send`` route catches it,
releases reservations via the existing ``_release_reservation_on_fail``
hook, and surfaces ``status: "attachments_busy"`` to the caller
with the IDs in ``dropped_attachment_ids``. The coord adapter
mirrors the cleanup (releases the soft-locked reservation taken
for ``_send_id``) so the create-with-attachments path can't leak.

Now that the queue can never carry attachments, the per-item
``att_ids`` slot is gone:

- Queue tuple slimmed ``(cleaned, priority, att_ids)`` →
  ``(cleaned, priority)``.
- ``_flush_queued_messages`` collapses to a single combined-text
  user turn (no attachment branch).
- ``_collect_advisories`` queue-drain pushes ``UserInterjection``
  advisories only (no ``attachment_items`` list).
- ``dequeue_message`` no longer unreserves (queue can't reserve).
- ``_resolve_attachment_ids`` had no remaining production callers
  and is deleted along with the tests that exercised it in
  isolation.

**Frontend.** ``Composer.setBusy`` disables the paperclip whenever
busy (regardless of ``queueWhileBusy``) — text still queues,
attachments don't. ``chat.css`` gains a ``.composer-attach:disabled``
rule (mirrors the existing ``.composer-send:disabled`` treatment)
so the affordance actually looks unclickable instead of falling
through to the UA default. ``title`` and ``aria-label`` are kept in
sync for AT users (WCAG 4.1.2).

Both interactive and coordinator UIs handle the new
``attachments_busy`` response with a chat-surface error bubble:

> Attachments can't be sent while the assistant is working.
> Send a text-only message now, or wait and resend with attachments.

Chips stay in the composer so the user can retry once idle.

**Tests.** Replaced the now-impossible ``TestQueuedWithAttachments``
class with a rejection-coverage class. Rewrote the
``_queue_with_attachment`` route-test fixture to reserve directly
via ``reserve_attachments`` (the queue path no longer reaches the
reserved state). Added a route-level test for the new
``attachments_busy`` contract.
2026-05-04 02:59:35 -07:00
Patrick Buckley 7f1b0acf7a Bound search tool output against pathological inputs (#473)
* Bound search tool output against pathological inputs

Replaces the per-line truncation with a fully bounded pipeline so the
search tool can no longer overflow the LLM context — or OOM the parent —
on minified bundles, multi-GB JSONL records, or huge result sets.

Backend:
- Prefer ripgrep when on PATH; grep is the fallback. Detection is
  cached via functools.cache.
- ripgrep flags do most of the bounding natively: --max-columns 1024
  + --max-columns-preview, --max-filesize 10M, --max-count 100,
  --no-config, --no-messages, plus negative globs for the same
  noisy directories grep has been excluding.
- ripgrep added to the Dockerfile.

Streaming subprocess (_search_capture):
- subprocess.Popen with a streaming, byte-capped stdout read (4 MB).
  Defends against single-line files (training data, minified bundles)
  that would have OOM'd the previous subprocess.run capture.
- threading.Timer watchdog enforces tool_timeout even when the
  pipe read is blocked in the kernel — proc.wait(timeout=…) alone
  was insufficient because the read sat ahead of it.
- Stderr drained in a daemon thread to avoid pipe-deadlock when the
  child writes to stderr while we're still reading stdout. Cap on
  captured stderr keeps a hostile child from growing the buffer.

Tier-based formatter (_format_search_results):
- Tier 1: full path:line:content output, stream-emitted with a
  running-cost short-circuit so we never materialize past the budget.
- Tier 2: K samples per file with overflow notes; K is computed
  analytically from budget / file_count / avg-line-length so we hit
  the right ladder rung in a single pass.
- Tier 3: per-file counts only, also budget-bounded with a tail line
  reporting the omitted files. Sorted by descending count.
- Total output budget (32 KB) is well under tool_truncation, so the
  head+tail _truncate_output strategy never silently drops middle
  files in a search result.

Argument injection fix:
- The ripgrep arg list was missing the `--` separator that the grep
  branch already had. With auto_approve on the search tool, that was
  exploitable: path='--pre=COMMAND' would have made ripgrep run the
  script as a per-file preprocessor and surface its stdout. Added
  `--` and a regression test.

State-machine cleanup in _exec_search:
- rc < 0 (signal-killed by something other than us) now surfaces a
  dedicated 'killed by signal N' message instead of being parsed as
  success.
- capped + zero parsed records (e.g. one multi-MB line with no \n)
  now returns a dedicated byte-cap message instead of the malformed-
  output message that previously masked the real cause.
- _report_tool_result descriptions now match the returned payload
  (no more 'no matches' tag on a 'malformed' payload).

Defence-in-depth on env scrub:
- RIPGREP_CONFIG_PATH, GIT_CONFIG, GIT_CONFIG_GLOBAL, GIT_CONFIG_SYSTEM
  added to _EXPLICIT_SCRUB. We pass --no-config on the rg CLI today,
  but if a future caller forgets the flag, an attacker who can set
  one of these env vars could plant a config containing --pre=… and
  recreate the same RCE shape.

Tests:
- TestSearchLineTruncation rewritten to mock _search_capture instead
  of subprocess.run (the previous tests passed ChatSession kwargs
  that no longer satisfy the constructor).
- TestSearchBackendSelection covers rg/grep detection and arg
  construction, including the --pre flag-injection regression.
- TestSearchOutputBudget exercises Tier 1/2/3 directly.
- TestSearchCaptureStreaming spawns real Python subprocess writers
  to exercise the byte-cap trim, mega-line-no-newline edge case, the
  watchdog timeout when the child writes nothing, and the stderr
  drain under load.
- test_env_scrub picks up the new tool-config keys.

* Address Copilot review on #473

- Budget the Tier 2/3 header up front so the formatter's emission stays
  strictly within _SEARCH_OUTPUT_BUDGET. Previously the fit checks only
  counted body bytes, letting the final string overflow by ~120 chars
  (header + separator) and triggering _truncate_output's head+tail
  dropout — exactly the shape this code was trying to avoid.
- Restore the (5, 3, 1) ladder in Tier 2: the analytical K from perf-2
  is kept as a starting estimate, but if that K's actual emission
  doesn't fit (the estimate ignores the header and overweights shared-
  path compression) we step down through the ladder before falling
  through to Tier 3. The previous one-shot K could collapse to counts-
  only when 3/file or 1/file would have fit.
- Only normalise rc to 0 in the capped-output path when rc < 0 (our
  SIGKILL). There's a narrow race where the child can exit naturally
  between our read and our kill; preserving a non-negative rc means
  rg's rc=2 ('matches found but some files had errors') no longer
  silently turns into a clean success when the byte cap also fires.
- Clarify _MAX_SEARCH_LINE_LENGTH doc: the cap applies to the content
  portion (after path:lineno:), not the whole emitted line.
- Add explanatory comments on the two intentional `except Exception:
  pass` blocks in _search_capture (stderr drain, pipe close in the
  cleanup finally) so static analysis and future readers can see the
  silence is deliberate.
- Tighten the budget tests: now assert strict `<= _SEARCH_OUTPUT_BUDGET`
  instead of the +512-char slack that was masking the header overflow.
- New regression tests:
  - Tier 2 ladder step-down (K=5 over budget, K=3 fits, no Tier 3 fall-through)
  - capped + rc=2 surfaces stderr instead of being normalised to success
  - capped + rc<0 (our SIGKILL) flows through as a partial-result success

* chore(search): post-review cleanup

Follow-up to the Copilot-review fixes in 39d2aa2 — these are all small
quality items (no behaviour change, no new tests).

- q-1: collapse the Tier 2 candidates filter to a single expression.
  Drops the redundant inner ``max(estimated_k, 1)`` and the unreachable
  ``if not candidates`` branch (the ladder ends in 1 and ``estimated_k``
  is already floored at 1, so the comprehension always yields ≥ ``[1]``).
  ``or [...]`` is kept as defence against future ladder changes.
- q-2: update _format_search_results docstring to match the new ladder
  semantics (analytical seed → step down through (5, 3, 1) from the
  highest rung ≤ the estimate). The previous wording suggested every
  Tier 2 attempt started at 5.
- q-3: combine the two ``from turnstone.core.session import ...``
  statements in test_tier2_steps_down_ladder_before_falling_to_tier3
  into a single top-of-function import (matches the surrounding tests).
- q-4: shorten the explanatory comments on the two best-effort cleanup
  paths in _search_capture to one line each. Both sites now read with
  the same shape ("# best-effort: pipe may be torn down by ...").
- q-5: trim the _MAX_SEARCH_LINE_LENGTH comment from 7 lines back to 3.
  Keeps the load-bearing semantic (cap is on the content portion only)
  and the pathological-line defence; drops the paths-aren't-bounded
  parenthetical, which was background reading rather than WHY.
2026-05-04 02:59:35 -07:00
renovate[bot] 6904bd8f39 chore(deps): lock file maintenance 2026-05-04 02:59:35 -07:00
renovate[bot] 4d677d1ebf chore(deps): update github actions 2026-05-04 02:59:35 -07:00
Patrick Buckley 35f462a46d chore: bump version to 1.5.6 2026-05-03 13:44:50 -07:00
Patrick Buckley ec74334e74 feat(providers): api_surface toggle + mistral medium reasoning fix (#469)
* feat(providers): api_surface toggle + mistral medium reasoning fix

Mistral medium open-weights served by vLLM expects reasoning_effort via
the Responses API (`reasoning.effort`), not as a `chat_template_kwargs`
entry on Chat Completions.  The session was unconditionally injecting
`{"reasoning_effort": ...}` into `chat_template_kwargs` for every
openai-compatible request, which corrupted the prompt rendering for any
backend whose chat template didn't consume that key (Mistral medium,
Mistral cloud, Groq, OpenRouter).

Changes:
- Add `api_surface` ("chat" | "responses") to `ModelConfig.server_compat`
  and thread it through `create_provider` / `model_registry.get_provider`.
  `openai-compatible` defaults to Chat Completions; operators can flip
  individual aliases to Responses for endpoints that support it.
- New `vllm-mistral-medium` profile that pre-fills api_surface=responses
  on Detect for known Mistral medium model ids.
- Drop the unconditional `reasoning_effort` injection into
  `chat_template_kwargs`.  Operators running gpt-oss-style local
  templates that consume `reasoning_effort` from the chat template now
  opt in via `server_compat.extra_body.chat_template_kwargs`.
- New "API Surface" select in the Models admin tab; allowlist-validated
  server-side at create/update time; pre-filled by Detect via the
  profile suggestion.
- Evict the cached provider singleton in `ModelRegistry.reload()` when
  api_surface changes (previously only cfg.provider triggered eviction).
- Fix `_run_agent` fallback path to inherit the session's primary alias
  for capability and server_compat resolution; previously the fallback
  passed `alias=None`, which silently dropped per-model caps on the
  agent path.

Tests: 5117 passed (-m "not live"); ruff + mypy clean.

* fix(providers): don't auto-suggest Responses for Mistral medium

vLLM's Responses API surface for Mistral medium open-weights doesn't
wire up the Mistral tool-call parser as of vLLM 0.x — tool calls leak
into the response as ``[TOOL_CALLS]<name>{...}`` text instead of
structured tool_calls.  Chat Completions on the same engine handles
tools cleanly via ``--tool-call-parser mistral``, and reasoning can be
turned on via the vLLM CLI ``--reasoning-parser`` flag.

Drop the auto-suggest mapping so Detect falls back to the generic
``vllm`` profile.  Keep the ``vllm-mistral-medium`` profile definition
in place so an operator who specifically wants per-request effort and
accepts the tool-calling limitation can still pick "Responses API"
manually in the admin UI.

* fix(providers): address Copilot review on PR #469

- providers/__init__.py: drop the redundant *_responses_provider /
  *_chat_provider names; have create_provider use _openai_provider and
  _openai_compat_provider directly so they're not flagged as unused
  globals.
- console/server.py: tighten _validate_api_surface to a strict equality
  match against the canonical {"chat", "responses"} set.  The previous
  strip().lower() membership check accepted ' Responses '/'CHAT' but
  stored the raw string verbatim, which then failed to round-trip
  through the admin <select>.
- console/static/admin.js: gate the entire server_compat block (server
  type, api_surface, extra_body) on provider == "openai-compatible" at
  save time so toggling provider away can't leave a stale hidden surface
  selection in the persisted capabilities JSON.
- tests/test_session.py: splat the bad kwarg via **dict so CodeQL no
  longer flags the call as a wrong-name keyword (the point of the test
  is the runtime contract, not the static type).
- tests/test_admin_model_registry_refresh.py: add endpoint-level tests
  for the api_surface validation on both create and update — covers the
  bogus-value rejection, non-canonical-string rejection, and the happy
  path persisting through to the refreshed registry.
2026-05-03 13:40:30 -07:00
Patrick Buckley 2fd0c29a92 fix(memory): query-aware candidate selection + OR-of-terms search (#468)
* fix(memory): query-aware candidate selection + OR-of-terms search

The system-message memory composition path used a recency-ordered
candidate set (`_list_visible_memories(limit=fetch_limit)`).  On
deployments with more than `fetch_limit` (default 50) visible
memories, BM25 only ever ranked the 50 most-recently-touched memories
— a relevant memory written months ago was silently invisible
regardless of how well it matched the recent context.  Multi-word
search at the SQL layer used AND-of-terms, killing recall on any
multi-word query without an exact field overlap.

## Functional changes

- `_init_system_messages` (`turnstone/core/session.py`): extract
  recent context first, then `_search_visible_memories(context)` to
  pull query-aware candidates.  Search hits below `fetch_limit` union
  with the recency list (deduped by memory_id) so the BM25 candidate
  pool is always a SUPERSET of the prior recency-only pool — even on
  noisy queries where the cap fills with stopwords, the recency-50
  the original bug surfaced still reaches BM25.  Empty context skips
  search entirely.  Candidate-selection logic extracted into
  `_select_memory_candidates`.

- `search_structured_memories` (PostgreSQL + SQLite): per-term
  clauses join with OR instead of AND.  A row matches if ANY term
  matches ANY of name/description/content.  Downstream BM25 narrows
  back down by relevance.

## Perf hardening

- Collapse the 1-3 fanned scope queries into a single SQL.  New
  backend methods `list_visible_structured_memories` /
  `search_visible_structured_memories` union the visibility scopes
  into one WHERE OR-group, so a composition rebuild now hits the DB
  at most twice (search + recency) instead of up to six times.

- Cap and normalize search terms.  Composition can hand a multi-KB
  pasted message to ILIKE-based search; without a cap, every distinct
  token would emit one unindexable predicate per scope-fanned query.
  `normalize_search_terms` (`storage/_utils.py`) de-dupes
  case-insensitively, drops <2-char tokens, and hard-caps at 16.

- Per-turn search cache.  `_init_system_messages` fires from many
  call sites within one turn (state transitions, MCP refresh, tool
  results) and the recent-context query is identical across them.
  Session-instance cache keyed by (query, mem_type, limit) absorbs
  the duplicates; invalidated in `_append_user_turn` and after
  memory save/delete tool actions.

- Stable secondary sort by `memory_id`.  `updated` is second-precision
  and `touch_structured_memories` can land a batch on identical
  timestamps; without a tie-breaker SQL returns rows in
  implementation-defined order, BM25 input shuffles, and the
  LLM-side prompt cache misses across calls.  All four backend ORDER
  BYs now break ties on `memory_id ASC`.

## Quality cleanups

- Coalesce `memory.search.term_count` + `memory.search.zero_results`
  into a single `memory.search` log carrying both `term_count` and
  `result_count`.
- New `memory.composition` log: source / candidates / injected.
- Promote a shared `make_chat_session` factory to `tests/_helpers.py`.
- Rename SQL builder local `extra` -> `scope_filters` for clarity.
- Add docstrings on `search_structured_memories` so the AND->OR flip
  survives future readers.

## Tests

Adds 20 tests across `tests/test_structured_memory.py`,
`tests/test_structured_memory_storage.py`, and
`tests/test_memory_relevance.py`: recency-ceiling regression,
empty-query fallback, sparse-match union, recency-preserved-when-
search-returns-noise (locks in the pool-superset invariant),
OR-of-terms on both backends, scope filtering preserved,
search-facade multi-word behavior, term-cap normalization, the new
visible-scope helpers (list + search + empty-scopes guard),
coord-scope composition isolation, end-to-end
`memory(action='search')` tool execution, per-turn cache hit +
invalidation, and stable ordering under tied `updated` timestamps.

Memory test sweep: 102/102.  Broader regression
(session, storage, coordinator, load_skill): 411/411.

* fix(memory): address Copilot review on PR #468

Three follow-ups from Copilot's inline review:

1. SUPERSET invariant violation (Copilot, session.py:5510).
   `(search_hits + extra)[:fetch_limit]` capped the union back down to
   fetch_limit, evicting the recency tail when search added distinct
   hits.  Recency tail is exactly where ancient-but-recently-touched
   memories live — the recall this PR is supposed to improve — so
   tail eviction recreated the bug for the narrow case where a query
   term fell off the 16-cap and the matching memory sat in
   recency[40-49].  Drop the cap; both halves are already SQL-capped
   at fetch_limit, so the union is at most 2 × fetch_limit (~100 with
   defaults).  BM25 over 100 candidates in pure Python is sub-ms;
   irrelevant recency fillers get score=0 and don't pollute ranking.
   Updates the docstring to actually be honest about the invariant.
   Adds `test_recency_tail_preserved_when_search_adds_distinct_hits`
   that locks the behavior in: 5 search hits + 10 recency = 15-item
   pool, every recency item present, source="union".

2. Unbounded `query.split()` in normalize_search_terms (Copilot,
   _utils.py:74).  `str.split()` allocates the full token list before
   the cap-after-16 break, so a 100KB pasted query did MB of throwaway
   work even though only 16 tokens entered SQL.  Switch to
   `re.finditer(r'\S+', query)` — streaming iterator, stops scanning
   at the first 16 normalized terms regardless of input size.

3. Misleading + unbounded log term_count (Copilot, session.py:8571).
   `len(item["query"].split())` had two problems: same unbounded
   split as #2, and the value reported the raw input token count
   rather than the normalized term count that actually hit the SQL
   WHERE clause — misleading metric for an operator trying to
   understand storage-side behavior.  Switch to
   `len(normalize_search_terms(item["query"]))` — accurate count, and
   bounded for free via #2.

Refuted: github-code-quality flagged `...` bodies in the new Protocol
methods as "statement has no effect."  False positive — `...` is the
canonical Protocol body convention, used 213 other times in the same
file.

Memory test sweep: 103/103.  Broader regression: 411/411.
2026-05-03 13:40:30 -07:00
Patrick Buckley 1207d27363 fix(tests): isolate metrics-singleton swaps so they don't leak across files
CI failure on main: test_publish_records_metric_outcome saw an empty
calls list — its monkeypatch was patching a different metrics
instance from the one `_publish_models_metadata` reads.

Two changes:

- test_close_reason_persistence.py: replace the bare
  `srv_mod._metrics = MetricsCollector()` assignment in `_make_app`
  with an autouse `monkeypatch.setattr(srv_mod, "_metrics", ...)`
  fixture so the test's metrics swap auto-restores. Other test
  files (test_auth.py, test_server_attachments_endpoints.py) carry
  the same anti-pattern; left for a follow-up since they're not on
  the critical path here.

- test_server_node_models_metadata.py: switch the publish-helper
  metric test to a string-form `monkeypatch.setattr("turnstone.
  server._metrics", FakeMetrics())` so it replaces whatever binding
  the live module currently holds, regardless of what other tests
  did to it. Robust against future leaks of the same shape.
2026-05-03 13:40:29 -07:00
Patrick Buckley 733c9818d4 feat(coord): expose healthy model aliases per node on list_nodes (#466)
* feat(coord): expose healthy model aliases per node on list_nodes

Surfaces a `model_aliases` field on each `list_nodes` row so a
coordinator can discover which model aliases each cluster node will
accept on `spawn_workstream(model=...)` without an HTTP fan-out.

Each server projects its registry into a `models` entry on
`node_metadata` (`{alias, provider, healthy}` per alias) at lifespan
startup, on every 30s heartbeat tick, and after `internal_model_reload`.
The publish helper short-circuits on a payload-equality cache so a
stable cluster doesn't pay UPSERT churn — exposed via the new
`turnstone_node_models_publish_total{outcome="written|skipped"}`
Prometheus counter so operators can graph cache hit-rate.

Coord client filters the per-alias rows to healthy aliases only and
drops the provider-side model identifier (`cfg.model`) — coords kept
reaching for it when they should pass the local alias.

* fix(coord): address Copilot+CodeQL feedback on list_nodes models work

- internal_model_reload: reuse a single get_storage() local across the
  registry load and the metadata publish (Copilot:3047)
- _collect_node_models_metadata: iterate sorted aliases so two
  structurally identical registries built in different insertion orders
  serialize to the same JSON — directly improves the publish-cache hit
  rate exposed via turnstone_node_models_publish_total (Copilot:3105)
- tests: drop mixed turnstone.server import style flagged by CodeQL —
  hoist _metrics into the from-import block, and use sys.modules in
  the shutdown-race regression test instead of `import as srv`
2026-05-03 13:40:29 -07:00
Patrick Buckley 28a2779c10 fix(core): scope rehydrate fallback to manager, fix resume orphan
Address Copilot feedback on PR #465:

1. The has_alias fallback in both session_factories silently rewrote
   any unknown caller-supplied alias to the default, including on the
   fresh-create path where the create handler maps the factory's
   ValueError to a 503 with operator-friendly text. A typo in
   body.model would now silently start a workstream on the default
   instead of telling the caller their requested model could not be
   resolved. Move the fallback out of the factories: each factory
   raises again on unknown aliases, and SessionManager filters stale
   aliases out of the rehydrate path via a new ``model_validator``
   constructor kwarg (production wiring passes ``registry.has_alias``
   on both interactive and coordinator).

2. ChatSession.resume()'s elif branch flipped self.model to the
   persisted model name even when the alias was unresolvable, leaving
   the session paired with the constructor's default provider/client
   but a removed model name — a broken state whose next API call
   fails. Drop the model copy: keep the constructor's coherent
   default (provider + model + capabilities) and just log the
   unreachable saved values so the missing alias is auditable.

Tests:
- Move stale-alias coverage from the factory level into
  SessionManager (tests/test_session_manager.py): validator drops
  stale aliases before reaching build_session; live aliases pass
  through unchanged.
- tests/test_sessions.py renamed test_resume_restores_model →
  test_resume_keeps_defaults_when_alias_unresolvable to match the new
  contract.
2026-05-03 13:40:29 -07:00
Patrick Buckley 56364b0b5b fix(core): preserve workstream model + config on rehydrate
SessionManager.open() was calling build_session(ws) without a model
arg on the rehydrate path. The session_factory then resolved the
*current* default alias, ChatSession.__init__'s _save_config() (INSERT
OR REPLACE per-key) clobbered the persisted workstream_config with
those defaults, and the subsequent resume() "restored" what was now
the default — silently resetting model_alias, model, temperature,
reasoning_effort, max_tokens, skill, creative_mode, instructions,
token_budget, and notify_on_complete on every reopen and every
service restart, for both interactive and coordinator workstreams.

Three layers:

1. SessionManager.open() now reads workstream_config via
   self._storage.load_workstream_config(ws_id) and threads the saved
   model_alias into build_session(ws, model=saved_alias).

2. ChatSession.__init__ now skips its initial _save_config() when a
   workstream_config row already exists for self._ws_id — protects
   every other persisted knob without having to plumb each one
   through the adapter signature, and catches any future construction
   path that forgets to thread model through build_session.

3. Both session_factories (server.py interactive, console
   session_factory.py coordinator) now treat an unknown caller-
   supplied alias the same as an unset alias: fall back to the
   runtime default rather than raising. Without this, a workstream
   pinned to an alias an operator has since removed from the registry
   would 500 on every reopen — defeating the "best effort restore,
   default if the original is gone" contract this fix is meant to
   deliver. Mirrors _effective_default_alias's existing has_alias
   guard against a stale ConfigStore default.
2026-05-03 13:40:29 -07:00
Patrick Buckley afb5804a7c fix(console): address Copilot feedback on Models → Roles sub-tab
Three changes from PR review:

- Permission gating: hide the Roles sub-tab button when the user
  lacks ``admin.settings``.  The sub-tab loads/saves through
  ``/v1/api/admin/settings``, so an admin with ``admin.models`` but
  no ``admin.settings`` would otherwise see a perpetual 403 loader.
  When Roles is the active sub-tab and the permission check fails,
  snap the panel back to Definitions so the user lands somewhere
  usable.

- Drop the redundant ``/v1/api/admin/model-definitions`` fetch from
  ``loadAdminModelRoles``.  Both entry points (initial Models-tab
  open + ``models_changed`` SSE refresh) flow through
  ``loadAdminModels`` first, which already populates ``_modelDefs``
  + ``_modelDefaultAlias``; ``_saveModelRole`` doesn't touch model
  definitions, so the cached snapshot stays accurate when the save
  chains back here.  Halves the per-render request count and
  removes a wasted round-trip on every cluster-wide model edit.

- Add ``test_models_changed_event.py`` covering the SSE fanout the
  prior commit introduced: each model-definition CRUD endpoint
  emits exactly one ``models_changed``, settings PUT/DELETE only
  emit for keys in ``_MODEL_AFFECTING_SETTING_KEYS`` (parametrised
  over all eight), and unrelated settings (e.g.
  ``session.retention_days``) don't trigger spurious refreshes.
  The expected key set is pinned in the test so a stray addition
  to the allowlist doesn't silently bypass coverage.
2026-05-03 13:40:29 -07:00
Patrick Buckley 4b508a1319 feat(console): add plan_agent + task_agent to Models → Roles
Same shape as the coordinator/judge rows already there: alias dropdown
+ reasoning_effort dropdown sourced from the existing
``model.plan_alias`` / ``model.plan_effort`` and
``model.task_alias`` / ``model.task_effort`` settings.  Adds the four
keys to the SSE ``models_changed`` allowlist so changes from the
Settings API also trigger a live dropdown refresh, and filters them
out of the Settings tab so they only render in one place.
2026-05-03 13:40:29 -07:00
Patrick Buckley 9c2cb185e1 feat(console): consolidate role-model settings + live-refresh dropdowns
Lifts judge and coordinator model assignments out of their respective
admin tabs and into a new Models → Roles sub-tab so role overrides live
next to the model definitions they reference. Forward-looking shape for
the upcoming perception.{audio,image,video} model settings — adding a
new role is one entry in the declarative MODEL_ROLES array.

Also drops the misleading "Coordinator subsystem not configured" home
banner. The session factory already falls back to the registry's
default model when coordinator.model_alias is unset, so the banner was
nagging on fresh installs where the system was actually working. The
related _probeCoordSubsystem / _homeCoordReady plumbing went with it.

Wires SSE-driven live refresh: the console now emits a models_changed
event when a model definition is created/updated/deleted/reloaded, or
when a model-affecting setting (model.default_alias, judge.model,
coordinator.model_alias, coordinator.reasoning_effort) changes.
Connected browsers refetch /v1/api/models on receipt so the home
composer's model dropdown and the Roles sub-tab stay accurate without
a manual reload — fixes the case where editing the underlying model
for an existing alias left the dropdown showing the old model id.

Companion cleanups:
- Renamed .judge-section-* CSS classes to .admin-subtab-* and shared
  them with the Models sub-tab switcher (same a11y attrs, arrow-key
  nav). Old names had no other callers.
- Filtered judge.model out of the Judge Settings sub-tab and
  coordinator.model_alias / coordinator.reasoning_effort out of the
  Settings tab — they live exclusively under Models → Roles now.
- Reworded the _require_coord_mgr 503 messages to point operators at
  the Models tab instead of suggesting they set coordinator.model_alias.
2026-05-03 13:40:29 -07:00
Patrick Buckley 0519b847bd docs(skills): add import-conversation-history SKILL.md
Source-agnostic guide that teaches an agent Turnstone's destination
contracts (workstream + conversations schema, ws_id routing, OpenAI
message shape, tool-call/result pairing, provider_data fidelity blob,
attachment lifecycle) so it can map any external chat export onto them.
Validated against turnstone.core.skill_parser.
2026-05-03 13:40:29 -07:00
Patrick Buckley bbc8b99a9f fix(console): home composer attachments + coord chat user-message pills (#462)
* fix(console): home composer attachments + coord chat user-message pills

Two parity gaps in the console's coordinator surface:

- The embedded creator on the home page accepted only text — the
  paperclip / paste / drop pipeline that the in-coord composer and the
  interactive new-ws modal both expose was missing, so a user couldn't
  attach files at create time. Stage Files in memory (no ws_id yet) and
  ship them multipart on Start; the coord create endpoint already accepts
  multipart via create_supports_attachments=True.

- User messages with attachments rendered as plain text on both live
  send and history replay — no chip cluster like the interactive pane.
  Added appendUserMessageWithAttachments and a structured userAttachments
  list built from _attachments_meta (preferred) or the multipart parts
  themselves, then rendered the same .msg-user-attach pill strip the
  interactive pane uses.

Polish from a designer pass:

- Pill background was --panel-2, equal to the .msg bubble background in
  both themes (border contrast ≈1.4:1, below WCAG 1.4.11). Switched to
  --panel so the pill sits on a different surface than the bubble.
- Capped chip filename width inside the home composer (max-width 200px +
  ellipsis) so a long filename doesn't push the strip past the textarea.
- aria-live="assertive" → "polite" on #home-coord-error; client-side
  validation isn't an interrupt-level event.
- Reserved min-height on .home-composer-error and dropped the
  display: none/block toggling so validation messages no longer reflow
  the active-coordinators list below.

* fix(console): address PR #462 review feedback

- Block home-composer submit when files are staged but the task field is
  empty.  Server's _coord_create_post_install short-circuits on an empty
  initial_message, so the multipart upload would create pending
  attachment rows that never reserve onto a turn — orphaned until the
  GC sweep.  Fail in the browser instead.
- Drop the redundant `part &&` guard in coordinator.js's history-replay
  multipart loop; the earlier `if (!part || ...) continue` already
  filtered.
- Rewrite the home-mount .composer-chip-name CSS comment.  shared/chat.css
  defines .composer-chip{,-size,-remove} but no .composer-chip-name rule
  — the span inherits the parent chip font with no width cap.
- Add smoke-guard string assertions in test_coordinator_page.py for
  appendUserMessageWithAttachments and msg-user-attach so a future
  rename can't silently regress the attachment affordance.
2026-05-03 13:40:29 -07:00
Patrick Buckley bd9f780b21 chore: bump version to 1.5.5 2026-05-01 14:09:38 -07:00
Patrick Buckley 5d14b5f675 fix(replay): repair saved-workstream tool result rendering + extend audit-trail decoration (#461)
* fix(replay): repair saved-workstream tool result rendering + extend audit-trail decoration

Loading a saved workstream silently dropped tool results and missed
verdict / output-guard / truncation signals on replay. Root cause was
in `Pane.prototype.replayHistory`: an assistant message carrying both
content and tool_calls cleared the `lastToolBlock` anchor before the
following tool-result iteration could attach. The fix reorders content
to render before the tool block (matching live SSE order) and
restructures the tool-result branch to anchor by `data-call-id` so
multi-tool batches render `[hdr A][out A][hdr B][out B]` rather than
bunching outputs at the bottom.

Beyond the bug, replay now reaches near-parity with the live UX:

- Persisted intent verdicts and output_assessments flow through both
  the SSE replay (`_build_history`) and the `/history` REST endpoint
  used by coord. Single shared helper module owns the wire shape.
- Memory/recall calls persist instead of being filtered at storage
  time — full audit trail; UI dims them by default with hover-reveal
  so heavy memory usage doesn't crowd the narrative.
- Truncation indicator surfaces as a sibling pill (consistent across
  interactive + coord) when a tool result hit the 2000-char cap.
- `replayHistory` wraps DOM work in `aria-busy` so screen readers
  don't get a chatty announce-flood on long replays.
- `_build_history`'s storage I/O moves off the event loop via a new
  `events_replay_prepare` async hook for the SSE path; other async
  callers wrap in `asyncio.to_thread`.

Coord parity:

- `/history` REST endpoint decorates tool_calls with verdict +
  output_assessment + truncation flag (was previously raw
  `load_messages` output).
- Coord JS stamps `judge_verdict` / `heuristic_verdict` from
  history-loaded `tc.verdict` so the existing batch render paints
  the persisted pill, seeds the verdict cache to dedupe later live
  SSE events, and emits an inline `.coord-tool-row-warning` chip
  per call instead of a generic chat line.
- Memory/recall dim rule mirrored on `.coord-tool-row[data-tool-name=...]`.

* fix(replay): address PR #461 review feedback + raise tool-result storage cap

Copilot review feedback:

- Sibling-chain dim rule (memory/recall) now adds :focus-within
  alongside :hover for .tool-output / .media-embed / .output-warning
  / .tool-output-truncated — keyboard users tabbing into a faded
  subtree now get full opacity.
- ``cfg.open_post_load`` is now invoked via ``await asyncio.to_thread``
  so its sync ``_build_history`` call (storage I/O for verdict
  indexes + message reconstruction) doesn't block the event loop on
  every workstream open. Mirrors the SSE replay path that's already
  protected via ``events_replay_prepare``.
- Replaced the hardcoded ``2000`` literal in server.py and session.py
  with ``TOOL_RESULT_STORAGE_CAP`` from the shared decoration module
  so the UI truncation-pill detection can't silently desync from the
  storage write side.

While here:

- Raised ``TOOL_RESULT_STORAGE_CAP`` from 2000 → 10000. A 2000-char
  clip routinely cut grep / file-read bodies mid-line, leaving the
  audit trail useless for retrospective debugging. FTS5 + row size
  grow proportionally; the per-tool upper bound is still bounded
  upstream by ``_truncate_output``'s context-budget clamp.
- Updated the user-visible truncation-pill tooltip on both
  interactive and coord to reflect the new cap.
- ``test_decorates_tool_calls_and_marks_truncated`` now references
  the constant instead of a literal so it stays correct on future
  cap changes.
2026-05-01 14:06:08 -07:00
Patrick Buckley 4693fa95f1 chore: bump version to 1.5.4 2026-04-30 23:51:06 -07:00
renovate[bot] c3423d6606 chore(deps): update ghcr.io/astral-sh/uv docker tag to v0.11.8 2026-04-30 23:48:53 -07:00
Patrick Buckley 53f1222c22 refactor(coord): remove priority queue + queue depth indicator + broken CSS
Speculative reliability machinery from the Stage 3 push that turned
out not to address any user-visible bug. The actual fixes (state /
activity disjunction in handleChildState, bulk-fetch race fix in
_fetch_live_block, push approve_request via cluster bus) are what
resolved the wedged-row issues. Manual testing showed the per-tab
SSE listener queue depth never climbed past single digits even when
rows were stuck — overflow was never the cause.

Removed
- ``_CRITICAL_EVENT_TYPES`` + ``_put_with_priority`` helper.
- Per-tab listener queue selective drop (back to plain
  ``contextlib.suppress(queue.Full)`` everywhere).
- ``ClusterCollector._fanout`` reverts to the same.
- WebUI ``_broadcast_intent_verdict`` / ``_broadcast_approval_resolved``
  / ``_broadcast_approve_request`` revert to plain ``put_nowait``.
- ``_queue_stats`` periodic SSE emit + frontend status-bar indicator
  + the supporting CSS rules.
- Broken ``.approval-block`` ``transition: max-height`` /
  ``max-height: 80vh`` / ``overflow: hidden`` rules — the transition
  never fired (nothing toggled max-height) and ``overflow: hidden``
  clipped long verdict reasoning. Layout-shift on auto-expand jumps
  again, which is preferable to clipped content (Copilot review).

Tidied
- ``_CollectorProtocol`` / ``_ManagerProtocol`` method bodies switch
  from ``...`` ellipsis to docstring-only bodies, silencing four
  CodeQL "statement has no effect" warnings without changing the
  Protocol contract.

5024 passed, ruff + mypy clean.
2026-04-30 23:48:53 -07:00
Patrick Buckley 802d87a57f feat(coord): Stage 3 SessionManager Children primitive lift + cluster bus push paths
Lift the Children primitive out of CoordinatorAdapter into universal
SessionManager core primitives, replace the fragile poll + state-event
piggyback paths with first-class cluster bus event types for inline
approval delivery, and clean up the resulting frontend reducer.

Architecture
- New `turnstone/core/children_registry.py` — universal parent → children
  + reverse-lookup primitive with atomic `add_child` (returns parent UI
  for race-free dispatch). Lifted from `CoordinatorAdapter`.
- New `turnstone/core/child_source.py` — `ChildSource` Protocol with
  `SameNodeChildSource` (in-process via SessionManager state observer)
  and `ClusterChildSource` (cross-node via ClusterCollector listener).
- `SessionManager._on_state_change` upgraded to multi-subscriber
  (`subscribe_to_state` / `unsubscribe_from_state`) under a dedicated
  lock; CLI consumer migrated.
- `CoordinatorAdapter` shrunk: 731 → ~640 LOC. Children data lives in
  the registry; fan-out lives in ClusterChildSource. Backward-compat
  property facades dropped; tests updated to use the registry surface.

Cluster bus event vocabulary
- New event types `intent_verdict`, `approval_resolved`,
  `approve_request` flow through both `ClusterCollector._apply_delta`
  (translation from node SSE) and `emit_console_ws_*` (synthesis on
  console pseudo-node).
- `CoordinatorAdapter._dispatch_child_event` re-emits as
  `child_ws_intent_verdict` / `child_ws_approval_resolved` /
  `child_ws_approve_request` on the parent coord's SSE stream.
- New `_broadcast_intent_verdict` / `_broadcast_approval_resolved` /
  `_broadcast_approve_request` no-op hooks on `SessionUIBase`. WebUI
  pushes to the global queue; ConsoleCoordinatorUI pushes to the
  collector. `approve_tools` calls `_broadcast_approve_request` right
  after setting `_pending_approval` so the items reach the coord tree
  immediately, eliminating the bulk-fetch race.

Cleanups
- `pending_approval_detail` piggyback on `ws_state` / `cluster_state`
  removed end-to-end. Bulk fetch + explicit verdict / approve-request
  push are the canonical carriers.
- Browser `_judgePollTick` 90-second poll loop deleted; push path is
  authoritative.
- `urgent` flag on `scheduleLiveFetch` deleted (only caller was 409
  retry; replaced with `invalidateLiveBadge` + standard schedule).
- Console `_fetch_live_block` derives `pending_approval` from a
  disjunction (`activity_state="approval"` OR `state="attention"`
  OR detail present) so the bulk fetch can't return false during the
  state-transition race window.
- Coord-side merge guard in `flushLiveFetches` no longer clobbered:
  `handleChildState` only stamps `sseUpdatedAt` when authoritatively
  clearing detail.
- `child_locality` capability flag removed (was inert dead code).

Reliability
- Selective drop on listener queue overflow: critical event types
  (verdicts, approvals, ws_closed, child_ws_*) evict one oldest item
  to make room rather than dropping themselves on a full queue.
  Best-effort events (state ticks, content tokens, status, activity)
  drop as before. Applied to `SessionUIBase._enqueue`,
  `ClusterCollector._fanout`, and the `WebUI._global_queue` puts in
  the new broadcast hooks.
- `_state_subscribers` snapshot under a dedicated lock so concurrent
  subscribe / unsubscribe during dispatch can't shift the iterator.

UX / a11y
- Loading placeholder in renderChildRow keeps row height stable while
  the bulk fetch is in-flight (sr-friendly aria-label).
- Focus preservation across `_renderChildrenNow` (capture +
  restore by row + marker) and across targeted `_updateChildRow` swaps.
- Layout-shift transition on the approval block max-height; respects
  `prefers-reduced-motion`.
- Sidebar pending count: `(N children · M pending)`.
- Risk pill `aria-label` spells out level + confidence for SR users.
- Per-coord SSE listener queue depth surfaced in the status bar
  (`queue N/500`) with color escalation (warn at >50%, danger at >80%).

Tests
- 305+ test changes across 8 files. New unit tests for
  `ChildrenRegistry`, `ChildSource` (both impls + multi-subscriber
  observer), the new collector emit + apply_delta cases, the dispatch
  cases for new event types, the broadcast hook overrides on both
  WebUI and ConsoleCoordinatorUI, and the focus / placeholder /
  pending-count frontend assertions in `test_coordinator_page.py`.

5024 passed, ruff + mypy clean.
2026-04-30 23:48:53 -07:00
Patrick Buckley 8349d9994d feat(console): multi-select delete UX for Saved Coordinators (#458)
* feat(console): multi-select delete UX for Saved Coordinators

Mirror the per-server "Saved Workstreams" multi-select delete onto the
console's "Saved Coordinators" section.  Coordinator deletes go through
the existing routing proxy at POST /v1/api/route/workstreams/delete
(body-keyed by ws_id, since coordinators live on the node that owns
them) — no backend change required.

Pagination caps the visible page (and therefore the Select-All fan-out)
at 24.  Without it, a Select-All on a busy cluster would pin the
console proxy pool with hundreds of parallel deletes through the
fan-out router.  While in delete mode the saved-coordinators list is
frozen against SSE re-renders so visible cards don't shuffle out from
under the user's selections (drained on cancel / post-delete close).

Refactor: shared logic now lives in turnstone/shared_static/cards.{css,js}.

  * .ws-delete-* CSS moved out of ui/static/style.css into the shared
    sheet alongside .dashboard-card; the existing ui/static modal
    markup picks up class hooks instead of id-scoped rules.
  * createSavedCardsController() owns mode state, checkbox decoration,
    toolbar wiring, focus trap, modal lifecycle, and batch fan-out.
    Both ui/static (Saved Workstreams) and console/static (Saved
    Coordinators) instantiate one controller; ui/static is now ~300
    LOC lighter as a result.
  * Internalises stale-selection prune across SSE re-renders, the
    wsId->item lookup map (was O(selected x N)), and the aria-hidden
    wrap on the toggle button's emoji glyph.

Designer review tightened the affordance:

  * Modal close restores focus to the toggle button (was landing on
    <body>) — WCAG 2.4.3.
  * Modal [role="alert"] gets a red-chip treatment when populated,
    stays invisible at rest via :not(:empty).
  * Pagination consolidated onto the existing .pagination control
    (terse "X / Y" label + arrow-glyph buttons) instead of a parallel
    .coord-pagination treatment.
  * Filled destructive buttons darkened to #dc2626 in dark theme so
    the white label clears WCAG AA contrast (was 3.0:1 on --red).
    Light theme keeps --red unchanged (5.9:1 already passes).
  * Toolbar wraps below 700px viewport — Delete Selected drops to its
    own full-width row underneath count + Cancel + Select All for
    thumb-target separation.
  * .ws-card-check:focus-visible outline + word-break on
    .ws-delete-item for narrow-modal long aliases.

* fix(cards): address Copilot review feedback on PR #458

* closeModal focus restore now falls back to the section toggle button
  (opts.buttonId) when prevFocus is hidden or detached.  The post-delete
  Close path runs cancel() before closeModal(), which puts the bar at
  display:none — so the captured prevFocus (the bar's "Delete Selected"
  button) is no longer focusable and focus would land on <body>,
  defeating the WCAG 2.4.3 fix.  Esc / Cancel paths still land on the
  original focus owner because the bar stays visible in those flows.

* Saved Coordinators onClose drains _savedCoordsRetry before reloading.
  Without it, SSE events that arrived during the delete-mode freeze
  leave the retry flag true, so loadSavedCoordinators's .finally()
  re-fires a second fetch immediately after the first resolves.  Mirrors
  the same idiom in cancelCoordDeleteMode.
2026-04-30 23:48:53 -07:00
Patrick Buckley ac1fd67137 chore: bump version to 1.5.3 2026-04-30 13:34:35 -07:00
Patrick Buckley 1b40ae79f9 fix(storage): address PR #457 review feedback
Three issues from the Copilot review on PR #457:

1. SQLite race in bulk_close_stale_orphans (Copilot): the SELECT-then-
   UPDATE flow doesn't re-apply the eligibility predicates on the
   UPDATE, so a row that gets touch_workstream-bumped (or set_state-
   transitioned) between the two statements would still be flipped
   to closed.  Postgres dodges this via UPDATE...RETURNING (one atomic
   statement); SQLite needs the explicit re-application.  Fix: rebuild
   the WHERE conditions list once, apply on both SELECT and UPDATE,
   then SELECT-back by ``state='closed' AND updated=now`` to get the
   accurate closed-id list.  A row that became fresh between the two
   statements skips the UPDATE entirely.

2. SQLite IN-clause bind-parameter limit (Copilot): default 999 cap
   could be exceeded on a backlog reap (e.g. after a long outage).
   Chunked the candidate id list at 500 — same chunk size
   prune_workstreams (line 453) uses for the same reason.

3. Wall-clock-dependent test asserts (Copilot, two locations): the
   tests asserted ``updated > '2024-01-01T00:00:00'`` which is fragile
   on systems with skewed clocks or pre-2024 dates.  Replaced with
   ``updated != stale_seed`` — captures the same intent (the value
   was bumped) without depending on wall-clock date.

Two ``...``-as-no-op flags from github-code-quality were false
positives — ``...`` is the standard Python idiom for Protocol method
bodies and matches every other method in _protocol.py.  No code change.
2026-04-30 13:34:15 -07:00
Patrick Buckley b078ddccf0 fix(session_manager): scope orphan reaper by services.last_heartbeat
Replaces the ``node_id == self_node_id`` orphan-scoping heuristic from
earlier on this branch with liveness-based scoping using
``services.last_heartbeat``.  The heuristic was wrong for the post-#384
world: PR #384 (refactor: replace hash-ring rebalancer with rendezvous
hashing) deleted the rebalancer that used to keep workstreams.node_id
pointing at a live node.  Without it, ``workstreams.node_id`` is now
stamped at create time and never updated, so in containerized
deployments with dynamic hostnames a dead pod's rows have ``node_id``
matching no surviving service — they'd accumulate forever under the old
heuristic.

services.last_heartbeat is the same primitive the rendezvous router
uses for routing.  Reusing it here keeps reap scoping aligned with
routing: dead pods' rows fall out of the live set after the heartbeat
window and become reapable; alive pods' rows stay protected as long as
they heartbeat.

Mechanics:

- ``bulk_close_stale_orphans`` parameter renamed
  ``node_id: str | None`` → ``live_node_ids: list[str] | None``.  The
  WHERE clause becomes ``(node_id IS NULL OR node_id NOT IN
  live_node_ids)``.  ``None`` skips the filter entirely (single-process
  / tests / operator backfill).  ``[]`` treats every row as
  unprotected.
- ``SessionManager.close_idle`` pass 2 calls
  ``storage.list_services(self._service_type)`` to enumerate live
  peers, passes their service_ids as ``live_node_ids``.  ``_service_type``
  is derived from ``self.kind`` (INTERACTIVE→"server",
  COORDINATOR→"console") via a module-level mapping — no constructor
  param, so production wiring can't miswire the kind/service_type
  pairing.
- list_services failure → pass 2 is skipped this tick (conservative;
  never reap when liveness state is unknown).  Pass 1 still runs.
- ``workstreams.node_id`` with NULL value is always eligible — defends
  against ANSI ``NULL NOT IN (...)`` evaluating to NULL (not TRUE) and
  silently protecting orphans forever.
- Migration 048 simplified to ``(kind, updated)``; the new query's
  ``NOT IN (small list)`` predicate against an unbounded-cardinality
  column doesn't index well, so leading ``node_id`` would just add
  write cost.

Tests cover the live-services protection (own/dead/null cases), the
empty-peers reap-all case, the list_services-failure conservative
fallback, both kind/service_type pairings (interactive→"server",
coordinator→"console"), and the combined live_node_ids +
exclude_ws_ids filter matrix.
2026-04-30 13:34:15 -07:00
Patrick Buckley 4b6c93a0e9 perf(storage): partial composite index for the orphan reaper query
bulk_close_stale_orphans runs every min(300s, idle_timeout/4) on
every server and console process.  Its WHERE shape is:

    WHERE kind = ?
      AND state IN ('idle','thinking','attention','running')
      AND updated < ?
      AND node_id = ?           -- multi-node interactive only

At current scale the existing single-column indexes are sufficient —
idx_workstreams_state prunes to non-closed and the planner filters the
rest sequentially.  At 100k+ rows that filter becomes a tablescan-
shaped cost.

A partial index covering only BULK_CLOSE_STATE_VALUES rows matches the
reaper's query exactly while staying tiny — closed rows (typically
95%+ of the table) and error rows are excluded, so the index is
roughly 5% the size a full multi-column index would be.  Write
amplification only kicks in for transitions touching one of the four
covered states.

Column order (node_id, kind, updated): node_id is the most selective
filter for multi-node interactive (each server prunes to its own
node's rows), kind second so coord-only and interactive-only queries
within a node still get index-only scans, updated last so the range
comparison rides the trailing column.

Postgres uses CREATE INDEX CONCURRENTLY so the build is non-blocking
on a live system; SQLite has no concurrent concept and the table-
level write lock already serializes, so a plain CREATE INDEX is fine.
2026-04-30 13:34:15 -07:00
Patrick Buckley 9d283e951f fix(console): periodic idle cleanup for the coordinator pool
The console's coord SessionManager had no idle thread — close_idle was
never called for coordinator workstreams.  This is the worse half of
the lifecycle leak: the dashboard filters via the in-memory pool, so
DB-only orphan coords were invisible.  At empirical diagnosis,
coord closure was 16% (10 closed / 64 total) vs interactive 63%.

Adds _coord_idle_cleanup_thread mirroring turnstone/server.py's
_idle_cleanup_thread but skipping the rate-limiter / global-queue arms
the console doesn't have.  Started from the lifespan when coord_mgr is
constructed and server.workstream_idle_timeout > 0 (reuses the
existing setting — same cadence works for both kinds).

Initial sweep runs INSIDE the thread before the first sleep, not
synchronously in the lifespan: cold-start orphans are reaped without
blocking Starlette boot.  Important because cold start with many DB
orphans (the precise condition this code targets) is exactly when the
UPDATE is most likely to be slow.

Helper takes an optional stop_event parameter purely for tests —
production callers pass None and the daemon runs for process lifetime.
This avoids the SystemExit-from-stub + module-wide filterwarnings
fragility a previous iteration relied on.

Four tests: initial sweep runs before first sleep, ticks fire each
loop, exceptions don't kill the thread, stop_event exits cleanly.
2026-04-30 13:34:15 -07:00
Patrick Buckley 4e407e7d4f fix(session_manager): close DB-orphan workstreams in close_idle
Real bug: workstream rows accumulate in non-closed states (idle,
thinking, attention, running) when their owning process restarts or
crashes.  Empirical diagnosis on a live deployment found ~60 stuck
coord rows in DB invisible to the in-memory-keyed dashboard, plus
100+ interactive rows older than the 2h timeout (one stuck "thinking"
for 2 weeks — impossible across a process restart).

Root cause: close_idle iterates self._workstreams.values() — only the
loaded subset.  Anything left behind by a prior process incarnation
sits in DB forever because nothing ever re-loads it.

This commit gives close_idle a second pass.

Pass 1 (existing, unchanged): close loaded IDLE rows whose
ws.last_active (monotonic) is past timeout.  IDLE-only so legitimately-
attentive rows (waiting for user response) stay live.

Pass 2 (new): bulk-close DB rows of this manager's kind whose updated
is past the wall-clock cutoff and which aren't currently loaded.
Closes the broader BULK_CLOSE_STATE_VALUES set — any matching row is
by definition not loaded by any process and cannot be in a live
interaction.  Scoped by self._node_id so a sibling node can't reap
rows we own (multi-node interactive correctness).  No emit_closed —
never-loaded rows have no SSE listeners expecting them.

Lock invariant: pass 1 holds self._lock briefly to snapshot victims
and pop them (existing behavior).  Pass 2 holds self._lock briefly to
snapshot the loaded keys, then releases before the DB UPDATE so a slow
reaper query can't block create/get/set_state.

Also fixes a same-process race in open(): the rehydrate path read DB,
released the manager lock, then re-acquired to install — a concurrent
pass 2 between the two acquisitions snapshots loaded keys without the
in-flight ws_id, and could clobber its DB row to closed.  open() now
calls touch_workstream(ws_id) on rehydrate so the row's updated is
fresh against any pass-2 cutoff.  Pure timestamp write is safe against
concurrent close() (close still wins on the state column).

Three new tests cover the DB orphan pass (basic, exclude-loaded, kind
filter) plus node_id scoping (own/foreign rows, None-skips-filter) and
the open() rehydrate touch.
2026-04-30 13:34:15 -07:00
Patrick Buckley 7ab24e500b fix(storage): add bulk_close_stale_orphans + touch_workstream primitives
Two new methods on the StorageBackend Protocol, with implementations on
both Postgres (UPDATE ... RETURNING) and SQLite (SELECT-then-UPDATE in
one transaction).  No callers yet — wiring lands in subsequent commits.

bulk_close_stale_orphans(kind, cutoff, exclude_ws_ids, node_id=None)
flips rows in BULK_CLOSE_STATE_VALUES (idle/thinking/attention/running)
to closed when their updated timestamp is lex-older than cutoff.  The
node_id filter scopes the reap to a single node's partition — required
for multi-node interactive deployments where each node only has
authority over its own workstreams.node_id rows.  Excludes loaded ids
so the in-memory pass owns those.

touch_workstream(ws_id) bumps updated without changing state.  Used by
the open() rehydrate path to defend against the orphan reaper clobbering
a freshly-loaded row whose DB updated is older than the cutoff.  Pure
timestamp write is safe against concurrent close() because close still
wins on the state column.

BULK_CLOSE_STATE_VALUES is centralized in workstream.py so the two
backend implementations and FakeStorage all agree; if a new transient
state is added to WorkstreamState, deciding whether it joins this set
is part of the change rather than an after-the-fact audit across three
files.

Storage tests (run against both backends via the conftest fixture) cover
the kind/state/cutoff/exclude/node_id matrix plus touch_workstream.
2026-04-30 13:34:15 -07:00
Patrick Buckley 5bcbcb73b9 chore: bump version to 1.5.2 2026-04-30 03:15:59 -07:00
Patrick Buckley af6749421a fix(metacog): drop duplicate [repeat: tool()] info line
The themed ``tool_reminder`` bubble below the tool block already
shows the metacog text, and the tool block immediately above it
carries the tool name — so a separate gray ``[repeat: list_workstreams()
called with same arguments]`` info line was just duplicate visual
noise (operator-visible in the screenshot below the bubble).

Drop the ``ui.on_info`` call inside ``_apply_post_execute_advisories``
that emitted the diagnostic line.  Update the docstring to reflect
that the bubble is the canonical signal.  Rename
``test_emit_repeat_ui_line_on_streak_fire`` →
``test_no_legacy_repeat_info_line_on_streak_fire`` and invert the
assertion.
2026-04-30 03:15:22 -07:00
Patrick Buckley 4d6cb77075 fix(cli): add on_user_reminder + on_tool_reminder to TerminalUI
CI typecheck failed because ``WorkstreamTerminalUI(TerminalUI)``
inherits from ``SessionUI`` (the Protocol), and the Protocol's
``on_user_reminder`` / ``on_tool_reminder`` declarations have empty
bodies — mypy treats those as implicitly abstract, so the subclass
became un-instantiable.

Add real implementations on ``TerminalUI`` that render reminders as
``[metacognition · type] text`` lines in yellow.  This also restores
the metacog signal on the CLI surface (the legacy
``[metacognition: nudge injected — …]`` info-line went away with
``_emit_nudge_ping``; without this commit the CLI showed no signal
at all for metacog nudges).  Tool-channel and user-channel render
identically because terminal output is anchored by stdout flow
rather than by DOM anchor — the line lands directly after the
message it advises.
2026-04-30 03:15:22 -07:00
Patrick Buckley 5c225ef39b docs(metacog): align comments with side-channel + tool-channel scope
Address Copilot's review feedback on PR #456 — the docstrings and
inline comments hadn't all caught up with the architectural shift
across the branch:

  - ``_apply_reminders_for_provider`` docstring: "every user message"
    → role-agnostic, since tool messages also carry ``_reminders``
    (tool_error / repeat).
  - ``_mark_reminders_delivered`` docstring: same role-agnostic
    update; explicitly note both channels.
  - ``_append_user_turn`` callsite comment near
    ``_attach_pending_user_reminders``: still described splicing
    ``<system-reminder>`` blocks into user content; updated to
    reflect the side-channel attach + transient-copy splice at the
    provider boundary.
  - ``_build_history`` block comment: was user-message-only; now
    mentions tool messages and both ``user_reminder`` /
    ``tool_reminder`` SSE events.
  - ``_build_history`` propagation comment: same role-agnostic note
    on the per-entry surface.
  - ``app.js`` ``user_reminder`` SSE handler comment: said the
    bubble renders "above" the user message, but
    ``insertAdjacentElement('afterend', el)`` drops it BELOW.
  - ``app.js`` ``replayHistory`` comment: said "insertBefore drops
    the reminder directly above the just-rendered user bubble";
    same fix — bubble lands BELOW.

No behaviour change.
2026-04-30 03:15:22 -07:00
Patrick Buckley f5a843f44a fix(metacog): drop write-success-clear so sequential same-call streaks fire
The repeat-detection block in ``_apply_post_execute_advisories`` had
a leftover "clear streak when a write tool succeeded" branch from
when ``RepeatDetector`` tracked cumulative counts.  With the
consecutive-streak semantics introduced earlier in the branch the
branch became:

  1. Redundant — any different (name, args) signature already resets
     the streak via ``RepeatDetector.record``, so an intervening
     read/write naturally breaks the streak.
  2. Actively wrong — the clear runs ONCE at the top of each
     ``_apply_post_execute_advisories`` call, before the per-result
     loop records sigs.  In a single parallel batch
     ``[bash, bash, bash]`` the clear runs once and then three
     ``record`` calls accumulate to count=3 in the same call → fires.
     But across three sequential turns, each turn calls
     ``_apply_post_execute_advisories`` fresh, the clear runs at the
     top of each call, and only one ``record`` per call follows — so
     the count never gets above 1 and the canonical
     "small local model stuck on ``bash('echo test')``" pattern
     never triggered the nudge.

The asymmetry only existed for successful calls — failures don't
satisfy the ``not _tool_error_flags.get(tc["id"])`` predicate, so
the clear didn't fire and sequential failures already worked.  The
fix is to drop the clear entirely; ``RepeatDetector``'s
consecutive-streak semantics handle every case uniformly.

Tests:

  - ``test_successful_write_clears_streak`` →
    ``test_intervening_different_call_resets_streak`` —
    rewords the assertion to reflect the actual mechanism (any
    different sig resets, write-or-otherwise) since "writes clear"
    was the bug, not the contract.
  - ``test_failed_write_does_not_clear_streak`` →
    ``test_sequential_bash_failures_fire_repeat`` — same shape, just
    framing fixed.
  - New ``test_sequential_bash_same_command_fires_repeat`` —
    regression for the bug user hit (three sequential successful
    ``bash('echo test')`` calls now correctly fire the nudge).
2026-04-30 03:15:22 -07:00
Patrick Buckley cf44841624 feat(metacog): themed reminder bubble unifies user + tool channels
The yellow themed reminder card introduced for user-channel nudges
(correction / denial / resume / start / completion) now also fronts
tool-channel nudges (tool_error / repeat).  Pre-fix the tool channel
shipped its reminders inside the tool-result envelope via
``wrap_tool_result``, leaking the ``<system-reminder>`` block into
``self.messages`` content (same problem the user channel had before
the side-channel refactor) and surfacing the legacy gray
``[metacognition: nudge injected — …]`` info line as the only
operator-visible signal — duplicated alongside the new themed bubble
for user-channel nudges.

Tool-channel parity:

  - ``_collect_advisories`` now returns
    ``(persistent_advisories, metacog_reminders)``.  Persistent
    advisories (``GuardAdvisory`` / ``UserInterjection``) keep
    riding ``wrap_tool_result`` because they ARE conversation
    history.  Metacognitive reminders extract to the second tuple
    element; the caller attaches them to the tool message dict's
    ``_reminders`` side-channel and emits ``on_tool_reminder``.
  - ``_apply_reminders_for_provider`` already handles ``_reminders``
    on any role, so the tool-channel splice into wire content is
    free.  ``_build_history`` also already propagates
    ``entry["reminders"]`` regardless of role, so reload renders the
    bubble too.
  - ``SessionUI`` Protocol gains ``on_tool_reminder(reminders,
    tool_call_id)``; ``SessionUIBase`` enqueues a ``tool_reminder``
    SSE event with the ``tool_call_id`` anchor.
  - ``_emit_nudge_ping`` had no remaining callers and was removed —
    the themed bubble (live SSE + ``/history`` reload) is the
    canonical operator signal for both channels now.

UI polish (the four fixes the screenshot caught for the user
channel + their tool-channel mirror):

  - Bubble renders BELOW the message it advises (semantically: a
    hint to the model right before its turn).  ``addUserReminder``
    swaps ``insertBefore`` for ``insertAdjacentElement('afterend',
    el)``; ``addToolReminder`` anchors below the ``.ts-approval``
    block whose tool result triggered the batch's reminder.
  - Label uses the full feature name ``metacognition`` (was the
    ``metacog`` shorthand).
  - Card width / alignment inherits from the base ``.msg`` rule —
    ``align-self: flex-end`` and the explicit ``max-width`` are
    gone, so the card matches the user / assistant column instead
    of pinning right-aligned narrow.
  - The legacy ``[metacognition: nudge injected — …]`` gray info
    line is gone for both channels.

Frontend additions:

  - ``Pane.prototype.addToolReminder(reminders, toolCallId)``
    anchors below the ``.ts-approval`` block (live: by
    ``data-call-id``; replay: by "last block in messagesEl"
    fallback, which is correct because messages render in order).
  - SSE switch case ``"tool_reminder"`` calls ``addToolReminder``.
  - ``replayHistory``'s tool-message branch now calls
    ``addToolReminder`` when ``msg.reminders`` is present.
  - ``addUserReminder`` advances its anchor on each loop iteration
    so multiple reminders stack in queued order rather than
    reversed.

Coord console parity:

  - ``coordinator.js`` gains ``appendReminderBubble`` /
    ``appendUserReminderLive`` / ``appendToolReminderLive`` mirroring
    the interactive UI.  The tool-channel anchor walks
    ``toolRows[callId].batch`` to attach below the
    ``.coord-tool-batch`` construct (one bubble per dispatch turn,
    matching the "one nudge per batch even with many failing tools"
    drain).
  - SSE switch handles ``user_reminder`` and ``tool_reminder`` on
    the coord conversation surface.
  - ``/history`` replay propagates ``msg.reminders`` for user and
    tool messages — same wire shape as the interactive pane.
  - ``.msg.user-reminder`` styles moved to
    ``shared_static/chat.css`` so both surfaces inherit the same
    yellow themed bubble from the shared base.

Defensive read on ``_apply_reminders_for_provider`` (per Copilot
review on the closed PR): a malformed ``_reminders`` entry (string,
None, etc. — corruption / partial state) used to abort ``send`` via
AttributeError on the ``.get("text", "")`` call.  Filter to dicts
before building the block, mirroring the same filter
``_build_history`` already applies on the wire-out side; an
all-malformed list passes through as no-reminders.

Tests:

  - ``test_collect_advisories_drains_tool_buffer_on_last_result``
    rewritten to assert the ``(persistent, metacog)`` tuple shape
    and that ``MetacognitiveAdvisory`` no longer appears in the
    persistent list.
  - ``test_collect_advisories_holds_*`` and ``_drops_*`` updated for
    tuple return.
  - ``test_attach_emits_visibility_ping`` /
    ``test_collect_advisories_emits_visibility_ping`` inverted to
    assert the legacy gray line is gone on both channels.
  - ``TestSessionUIBaseToolReminderHook`` covers the new SSE event
    shape with the ``tool_call_id`` anchor.
  - ``test_malformed_reminders_filtered_out`` and
    ``test_all_malformed_reminders_passes_through`` cover the
    Copilot-flagged defensive filter.
2026-04-30 03:15:22 -07:00
Patrick Buckley 7ffab6a272 fix(session): metacog reminders ride a side-channel, not user content
User-channel metacognitive nudges (correction, denial, resume, start,
completion) used to be spliced into ``user_msg["content"]`` permanently,
which leaked the ``<system-reminder>`` envelope into every consumer of
``self.messages`` — UI replay (mitigated by a regex strip in /history),
compaction, title generation, and any future channel adapter that
echoes conversation context.  The /history strip was a band-aid;
compaction and title-gen still saw the raw spliced text.

Switch to a side-channel: ``_attach_pending_user_reminders`` writes the
rendered reminder list to ``user_msg["_reminders"]`` (sibling key,
leading-underscore convention shared with ``_attachments_meta`` /
``_provider_content``).  At the provider boundary, a new
``_apply_reminders_for_provider`` builds a transient shallow-copy with
the reminder spliced into ``content``; the original message dict
stays clean.  ``sanitize_messages`` drops the sibling key on the wire.

Once-per-session-not-per-turn semantics for the wire: after stream
success the loop calls ``_mark_reminders_delivered``, which flips a
``_reminders_delivered`` flag on every user message that carried
reminders into that call.  ``_apply_reminders_for_provider`` skips
already-delivered messages so the model sees each reminder exactly
once (the turn it advised).  ``_build_history`` ignores the delivered
flag entirely, so reconnecting tabs render the same nudge bubble the
originating tab saw via the live ``user_reminder`` SSE event.

UI surface:

  - ``SessionUIBase.on_user_reminder`` enqueues a
    ``{type: "user_reminder", reminders: [...]}`` SSE event with the
    same shape ``_build_history`` surfaces.
  - ``app.js`` renders a ``.msg.user-reminder`` bubble (yellow accent,
    pill-styled) anchored above the user message it advises, both
    live and on history replay.
  - ``replayHistory`` renders ``addUserMessage`` before
    ``addUserReminder`` so the anchor lookup finds the just-rendered
    turn (not a prior one).
  - Multi-tab caveat documented inline: non-originating tabs receive
    no ``user_message`` SSE event today, so a reminder may anchor to
    a stale prior bubble until ``/history`` reload corrects it.

Pre-existing bug surfaced by the audit: cancel handlers
(``GenerationCancelled`` / ``KeyboardInterrupt`` / generic
``Exception``) in ``ChatSession.send`` cleared
``_pending_tool_advisories`` but not the user-channel buffer.  Both
now drain through a shared ``_drain_pending_advisories`` helper.

Removed the ``/history`` regex strip — the side-channel approach
makes it redundant.  Hoisted ``escape_wrapper_tags`` +
``render_system_reminder`` imports to module top (called 2-3× per
turn).

Tests:

  - ``TestApplyRemindersForProvider`` — pass-through-by-reference,
    string + list content splice, escape on user-typed wrapper tags,
    multi-reminder ordering, source-untouched invariant, delivered
    flag skip path, fallback for unexpected content shape.
  - ``TestMarkRemindersDelivered`` — flag idempotency, no-reminders
    no-flag, only marks user messages with reminders.
  - ``TestUpdateTokenTableMsgsParam`` — calibration uses pre-built
    msgs when provided, falls back when not.
  - ``TestUserAdvisoryCancelClear`` — all three cancel branches drain
    the user buffer.
  - ``TestReminderSidechannelIsolation`` — compaction's
    ``_format_messages_for_summary`` and the title-gen extraction
    loop cannot see reminders by construction.
  - ``TestSessionUIBaseUserReminderHook`` — ``on_user_reminder``
    enqueues the right SSE shape.
  - ``TestBuildHistoryReminderPropagation`` — ``entry["reminders"]``
    propagation, absent / empty / multi / coexist-with-attachments
    cases, malformed input filtering, all-malformed elision.
  - ``test_sanitize_messages_strips_underscore_sibling_keys`` covers
    ``_reminders`` and ``_reminders_delivered``.
2026-04-30 03:15:22 -07:00
Patrick Buckley ba3bc9d989 fix(metacog): N>=3 streak detector + drop redundant error-prefix list
Cleanup pass on the metacognitive nudge stack — restores pre-split
errored-counts-toward-repeat behaviour and tightens the is_error
plumbing through the per-batch advisory hook.

The per-batch hook in ``_run_loop`` was duplicating the is_error
signal: ``self._tool_error_flags`` (set by ``_report_tool_result``)
and a string-prefix tuple (``Error`` / ``JSON parse error`` / …).
Two truth sources is what got us here — bash commands that exit
non-zero with normal stdout matched the flag but not the prefix,
the deny path matched the prefix but not the flag, and the result
was that stuck-loop detection silently broke for the most common
failure mode (the model bashing the same broken command).

Single source of truth now:

- ``_execute_tools.run_one`` deny branch routes through
  ``_report_tool_result(is_error=True)`` so denied calls populate
  ``_tool_error_flags`` like every other error path.
- The error-prefix tuple is gone; the write-success-clear gate and
  the tool-error-nudge gate both read ``_tool_error_flags`` only.

Repeat-detection state moves from a ``set[str]`` (fired on the second
identical call, ignored errors entirely) to a ``RepeatDetector``
helper in ``metacognition.py`` with consecutive-streak semantics:

- Threshold raised from 2 to 3 — two-in-a-row was noisy on
  legitimate transient retries; three is the cheapest stuck-loop
  signal.
- Recording a different signature resets the count, so [A, A, B, A]
  is two short streaks of 2 and not a streak of 4. Bounded by O(1)
  state regardless of session length.
- Errored calls now count toward the streak (the split into a
  separate metacog module unintentionally introduced a "skip errors"
  branch — restored).

While there:

- ``metacognition._COOLDOWN_SECS`` default aligned to 300s (matches
  ``MemoryConfig.nudge_cooldown`` and the ``memory.nudge_cooldown``
  config-store default; was set to 30 by an earlier investigation).
- The per-batch advisory block (~80 lines of mixed orchestration
  inside ``_run_loop``) is extracted to
  ``ChatSession._apply_post_execute_advisories`` so the wired
  behaviour is testable without driving ``_run_loop`` end-to-end.
  Producer extraction to a dedicated module is deferred to a
  follow-up; advisory producers all live on ``ChatSession`` for
  now per existing convention.
- Frontend ``appendToolOutput`` (turnstone/ui/static/app.js) now
  skips rendering when the parent approval block is denied or
  the output starts with ``Denied by user`` / ``Blocked``,
  mirroring the history-replay guard at ``_build_history``.
  Previously the live SSE path didn't need this guard because
  the deny path never emitted a ``tool_result`` event; the
  is_error routing change above means it does now, so without
  this guard the badge from ``resolveApproval`` and the SSE
  output would both render.

Tests: 8 unit tests for ``RepeatDetector`` covering streak,
threshold, clear, and intervening-sig reset; 9 integration tests
for ``_apply_post_execute_advisories`` covering the wired
behaviour (3-identical fires warning + advisory + UI line, errored
calls count toward streak as a regression guard, intervening sig
resets streak, successful write clears, failed write does not,
JSON outputs tracked but not inline-warned, tool_error nudge gates
on memory_count, repeat UI line emitted on streak fire).
2026-04-30 03:15:22 -07:00
Patrick Buckley dbe023b4dd chore: bump version to 1.5.1 2026-04-29 20:21:12 -07:00
Patrick Buckley 3d3a8b7367 docs(coord): tighten handleChildState comment per Copilot review
The pre-existing comment said pending_approval_detail "rides on
every ws_state event" — that overstated the case.  The node-side
emit is gated on ``_pending_approval is not None`` so the field is
absent on the steady-state broadcast and possibly null on a node
mid-rolling-upgrade.  The handleChildState fallback already
handles both cases; only the comment was wrong.
2026-04-29 20:20:38 -07:00
Patrick Buckley 99eff73a97 feat(coord): pass pending_approval_detail on child_ws_state SSE events
Inline child approve/deny in the coord tree UI was rendering downstream
of the bulk-live cache (``GET /v1/api/cluster/ws/live``), not the SSE
stream. ``child_ws_state`` events were tiny notifications that fired
an urgent live-bulk fetch on every activity_state transition into/out
of "approval", just to pick up the rich ``pending_approval_detail``
payload. With multiple coord tabs and multi-child workstreams, that
urgent-fetch pattern compounded the SSE-executor pressure Shape A
is unwinding.

Thread the field through every layer so the SSE event itself carries
the rich payload — browser mutates ``liveBadgeCache`` directly,
no urgent fetch:

  1. Node ``WebUI._broadcast_state`` emits ``pending_approval_detail``
     on ``ws_state`` events. Gated on ``_pending_approval is not None``
     so the per-broadcast verdict-cache deepcopy only runs when there
     is actually an approval pending. ``_build_node_snapshot`` also
     projects the field so the console's reconnect-via-snapshot
     resync path delivers it (without this the new collector
     forwarding would never see the field on a snapshot row).

  2. Console ``ClusterCollector._apply_delta`` (live ``ws_state``
     forwarding) and ``_reconcile_node`` (snapshot resync diff) both
     forward the field on the emitted ``cluster_state`` event, AND
     ``_apply_delta`` persists it on the cached ``ws`` dict so the
     ``get_node_detail`` / ``get_snapshot`` endpoints between
     reconciliations don't render stale approve/deny buttons.

  3. ``CoordinatorAdapter._dispatch_child_event`` re-emits the field
     on the ``child_ws_state`` event sent to coord listener queues.

  4. Frontend ``handleChildState`` reads ``ev.pending_approval_detail``
     and writes it directly into ``liveBadgeCache``, tagging the
     entry with ``sseUpdatedAt``. ``flushLiveFetches`` honors that
     tag for ``SSE_AUTHORITATIVE_MS`` (3s) — the upstream
     ``/dashboard`` cache has its own ~2s TTL, so a bulk-poll
     landing right after a transition can otherwise clobber the
     fresh SSE-set state with pre-transition data.

The pre-fix ``enteredApproval`` / ``leftApproval`` urgent-fetch
branch is removed. The 409 stale-call_id retry path keeps its own
urgent fetch — that's a different scenario.

Tests cover the forwarding contract at every layer, the broadcast
gate (event includes the field when an approval is pending,
omits it otherwise, and clears after resolution), and the
``flushLiveFetches`` merge-guard structural shape so a refactor
that keeps the symbols but inverts the comparison or drops the
``prev.live`` check can't pass silently.
2026-04-29 20:20:38 -07:00
Patrick Buckley 99fcd30299 fix(console): offload sync DB calls in coord children/tasks handlers
``coordinator_children`` was calling ``storage.list_workstreams``
directly on the event loop, ``coordinator_tasks`` did the same with
``load_task_envelope``, and ``_resolve_coordinator_or_404`` (called
from both handlers, plus ``coordinator_history`` and
``_resolve_coord_session``) did the same with
``storage.get_workstream`` on its cold-cache path.

The cold-cache resolver path is hit on every console restart,
coordinator eviction, and console proxy hop — exactly when the
event loop is most contended. Three coord tabs reconnecting after a
brief network blip = three serial event-loop blocks per call site.
Other lifted handlers in this file already use
``asyncio.to_thread``; bring all four call sites onto the same
pattern.

Convert ``_resolve_coordinator_or_404`` to ``async def`` and update
its four call sites to ``await``. Exception flow is unchanged.
2026-04-29 20:20:38 -07:00
Patrick Buckley 423c2e80b7 fix(console): isolate coord SSE polling on a dedicated 200-thread pool
Each coord ``events`` SSE listener parks a thread on
``client_queue.get(timeout=5)`` for the connection lifetime. The
console's coord endpoint was wiring no ``sse_executor_lookup`` on
``coord_endpoint_config``, so those parks landed on Python's default
ThreadPoolExecutor (~min(32, cpu_count+4)) and competed with every
other ``asyncio.to_thread`` caller (storage, router, audit). A few
coord tabs against a multi-child workstream would stall new request
handlers waiting for a worker thread.

Mirror the interactive-side precedent (the ``sse_executor`` /
``sse_executor_lookup`` pattern in ``turnstone/server.py``) — build a
dedicated 200-thread ``coord_sse_executor`` in the console lifespan
and wire ``sse_executor_lookup`` onto ``coord_endpoint_config``.
Drain order matters: shut the pool down AFTER ``coord_adapter.shutdown()``
so no new listeners arrive at a dying pool. ``cancel_futures=True``
discards queued-but-not-started futures during teardown.

Update the stale comment on the interactive-side wiring that claimed
"coord wires None and falls back to the default executor" — it now
points at the console's matching wire.
2026-04-29 20:20:38 -07:00
Patrick Buckley a0eb77360d fix(coord): tighten coord_registry refresh logging + comments per round-2 review
Three follow-ups from Copilot's round-2 review on #453.

ValueError logging surfaced the wrong reason
The catch-all ``except ValueError:`` logged ``reason=no_enabled_rows``
unconditionally, but ``ModelRegistry.__init__`` raises ValueError for
five distinct config issues (empty models, default / fallback / agent /
plan / task alias not present).  Operator looking at logs for a
config.toml typo would see the wrong cause.  Switch to
``log.warning("...reason=%s", exc)`` so the actual error message
threads through.  Behavior unchanged — existing registry still
preserved on every ValueError path.

Misleading shutdown() comment
The ``finally`` comment claimed shutdown() was closing clients the
throwaway registry created during DB load.  ``load_model_registry`` only
constructs ModelConfigs and the bare ``ModelRegistry(...)``;
``ModelRegistry.__init__`` leaves ``_clients`` / ``_providers`` empty
and they populate lazily on first resolve.  Today shutdown() iterates
empty dicts.  Comment now says so explicitly while keeping the call
(and its try/except) for forward-compat against an eager-init future.

Stale "probe" wording in test docstring
``test_helper_preserves_registry_when_db_probe_fails`` →
``test_helper_preserves_registry_when_strict_load_fails``.  The
explicit probe was removed in commit 1ba17ed when the helper switched
to ``load_model_registry(..., strict=True)``; the test name and
docstring still talked about a probe.  Updated wording reflects that
the loader's strict-mode re-raise is what the helper catches now.

132 tests pass.
2026-04-29 20:20:38 -07:00
Patrick Buckley 3dd0e196fe refactor(coord): hygiene pass on coord_registry refresh — async + selective teardown + test cleanup
Hygiene follow-ups from the multi-stage code review on #453.

perf-1 — sync helper called from async route handlers
``_refresh_coord_registry`` runs two sync DB reads and a registry reload
that takes ``_client_lock``; calling it directly from an async handler
held the event loop for the duration.  All four call sites now
``await asyncio.to_thread(_refresh_coord_registry, ...)``, matching the
pattern from commit ``1f7d6ad`` (offloaded ``tenant_check``).

perf-3 — ModelRegistry.reload() tore down all clients unconditionally
The reload always closed every cached client and provider, even when
the changed fields (``model``, ``temperature``, ``context_window``)
didn't touch the connection target.  Now selective: clients drop only
when alias removed or ``(base_url, api_key, provider)`` differs;
providers drop only when alias removed or ``provider`` string differs.
Keeps connection pools warm across the common admin-edit case where
only metadata changed.  Two new ``test_model_registry`` cases lock the
keep-warm vs drop-on-change behaviour, and the existing
``test_reload_clears_clients`` was updated (it asserted the old
overly-aggressive contract) into
``test_reload_keeps_clients_when_connection_target_unchanged``.

q-5 — helper rename
``_refresh_console_coord_registry`` → ``_refresh_coord_registry``.  The
``console_`` prefix was redundant given the function lives in
``turnstone/console/server.py`` and sibling helpers there
(``_notify_nodes_model_reload``, ``_publish_config_change``,
``_collect_model_status``) all omit it.

q-1 — shared test middleware
``tests/test_admin_model_registry_refresh`` now imports the
header-driven ``_AuthMiddleware`` from ``tests/_coord_test_helpers``
and sets default ``X-Test-User`` / ``X-Test-Perms`` headers on the
``TestClient``.  The local hardcoded variant duplicated infrastructure
the helper module exists to centralise.

q-3 — multi-alias test registry
``_make_registry`` extracted a ``_make_config`` helper and gained an
``extras={alias: model}`` param so multi-alias scenarios stop
hand-building ``ModelConfig`` literals.
``test_delete_endpoint_refreshes_registry`` now uses the helper.

310 tests pass across the related coordinator + model surfaces.
2026-04-29 20:20:38 -07:00
Patrick Buckley 19c3db5329 test(coord): lock the empty-body gate with a refresh-call spy
bug-3 / q-2 from the multi-stage review on #453: the previous test
``test_update_endpoint_with_empty_body_does_not_blow_up`` asserted only
that the registry's model name was unchanged after an empty PUT, which
holds whether or not the refresh ran (DB row matches registry → refresh
is idempotent).  A regression that always called
``_refresh_console_coord_registry`` — exactly the gate this test was
meant to lock — would have left the assertion green.

Rename to ``test_update_endpoint_skips_refresh_on_empty_body`` and spy
on the helper via ``monkeypatch.setattr``.  Empty-body PUT must register
zero calls; any future change that drops the ``if updates:`` gate now
fails loudly.
2026-04-29 20:20:38 -07:00
Patrick Buckley 0bea72019e fix(coord): strict-mode loader + guarded shutdown for coord_registry refresh
Two correctness follow-ups from the multi-stage code review on #453.

bug-2 / perf-2 (DB probe was theatre + double scan)
The previous probe defended nothing the loader didn't already swallow
on the next line: ``load_model_registry``'s row-loop catches Exception
internally, so a transient DB error after the probe still degrades to
a config.toml-only registry that ``existing.reload()`` would apply,
silently dropping every DB-sourced alias.  And on the happy path each
CRUD paid for two scans of ``model_definitions``.

Add a ``strict: bool = False`` flag to ``load_model_registry``.  When
strict, the row-loop's except re-raises instead of swallowing.  The
helper passes ``strict=True`` and drops the probe — single DB scan,
real failure isolation, the loader's silent fallback can no longer
mask a partial-result regression.  Default ``strict=False`` so CLI /
lifespan callers keep their boot-with-config-fallback behaviour.

bug-1 (shutdown could escape after a successful reload)
``ModelRegistry.shutdown()`` calls ``client.close()`` unguarded, and the
helper's ``finally`` block ran it outside the try/except.  A raising
close() after a successful ``existing.reload()`` would surface as 500
with the registry already mutated and the audit row already recording
success.  Wrap ``new_registry.shutdown()`` in its own try/except that
matches the helper's belt-and-suspenders error policy elsewhere.

The helper's docstring also drops the obsolete probe paragraph; the
``if existing is None: return`` branch gets a one-line inline comment
about the boot-from-empty case (the multi-paragraph version restated
behaviour the line itself documents).

129 tests pass (test_admin_model_registry_refresh + test_model_registry).
2026-04-29 20:20:38 -07:00
Patrick Buckley b9ff52d582 fix(coord): tighten coord_registry refresh — DB probe + accurate boot-from-empty docstring
Two follow-ups from Copilot review of #453:

1. ``load_model_registry`` swallows storage read errors internally
   (logs + continues with config.toml-only models).  Without a strict
   probe in the helper, a transient DB outage on an admin CRUD would
   apply a truncated registry that drops every DB-sourced alias —
   silently, since the loader returns a non-empty registry built from
   ``[models.*]`` config.toml entries.  Add an explicit
   ``storage.list_model_definitions(enabled_only=True)`` probe before
   the loader call so the failure is visible here and the existing
   registry is preserved on outage.

2. The previous docstring claimed ``admin_model_reload`` "has its own
   boot-from-empty story."  It doesn't — it just calls this helper,
   which no-ops when ``coord_registry`` is None.  When no model rows
   existed at boot, lifespan leaves the entire coord subsystem
   uninitialized (no ``coord_mgr``, no ``coord_adapter``, no
   ``session_factory``), and a console restart remains required after
   the operator adds the first row.  Tighten the docstring to admit
   that limitation rather than overstating the helper's reach.

New test ``test_helper_preserves_registry_when_db_probe_fails``
monkeypatches ``list_model_definitions`` to raise and asserts the
existing registry stays intact.
2026-04-29 20:20:38 -07:00
Patrick Buckley 961f999c93 fix(coord): auto-refresh console coord_registry on model-definition changes
The console builds ``app.state.coord_registry`` once at lifespan startup
and the coordinator session factory closes over that exact instance.
Until now, the model-definition admin endpoints (create/update/delete)
wrote to the DB but never touched the in-process registry — and the
explicit reload button only fanned out to nodes via HTTP, also leaving
the console's own registry stale.

Symptom: an operator who changed the underlying model name behind a
local-LLM alias (same alias, same endpoint) saw the DB row update
immediately, but coordinator sessions kept calling the prior model
name until the console process was restarted.

Fix: a new helper ``_refresh_console_coord_registry`` rebuilds a fresh
ModelRegistry from DB and applies it to ``app.state.coord_registry``
via the existing thread-safe ``ModelRegistry.reload()`` — in-place
mutation preserves object identity so the factory closure keeps
working, and active coord sessions auto-pick up the swap on their
next ``send()`` via ``ChatSession._refresh_model_from_registry``.

Wired into four endpoints in ``console/server.py``:

- ``admin_create_model_definition`` — after the DB write
- ``admin_update_model_definition`` — after the DB write, gated on
  ``if updates:`` so a no-op PUT skips the rebuild
- ``admin_delete_model_definition`` — after the DB write
- ``admin_model_reload`` — between ``_publish_config_change`` and
  ``_notify_nodes_model_reload`` so the console mirrors what the
  reload broadcasts to nodes

Failure isolation: a load or reload error leaves the existing registry
intact (logged + swallowed). Coord stays usable while the operator
investigates; the explicit reload remains the user-facing recovery path.

No node fan-out on CRUD — the explicit reload button continues to gate
cluster-wide HTTP propagation, preserving today's UX semantics on shared
clusters.

Tests in ``tests/test_admin_model_registry_refresh.py`` cover:

- helper-level: rebuild from DB, identity preservation, no-op when
  registry is None, preservation on load failure / no-enabled-rows /
  reload validation error
- endpoint-level: create / update / delete / explicit-reload all
  refresh the registry; an empty PUT skips the rebuild
2026-04-29 20:20:38 -07:00
Patrick Buckley 8bdb916064 fix(coord): raise wait_for_workstream message cap to 10 KiB
Production fan-outs are frequently hitting the 6 KiB per-child cap by
just 1-2 KiB, forcing the coordinator into a follow-up inspect_workstream
round-trip per truncated child to recover the tail. Bumping the cap to
10 KiB absorbs the common overshoot without changing the truncation
semantics — truncated=True still fires for genuinely oversized messages,
and inspect_workstream remains the unbounded follow-up.

Worst-case context impact: a 32-child fan-out at the cap is now ~320 KiB
(was ~192 KiB), still well within commercial model context windows.
Typical fan-outs of 1-5 children land at 10-50 KiB.

LAST_ERROR_MAX_LEN (1 KiB) is unchanged — it's intentionally smaller
than the wait cap so error truncation happens at write time, and
1 KiB still sits well below 10 KiB.

WAIT_MESSAGE_MAX_BYTES is referenced by name (not literal 6144) in the
truncation test, so no test value needs updating.
2026-04-29 20:20:38 -07:00
Patrick Buckley 25fe4e728a fix(coord): make coordinator fan out independent work by default
The coordinator system message was descriptive about parallelism rather
than prescriptive — "while multiple children run in parallel" framed
fan-out as incidental, and "a tasks entry, a child to own it" primed
singular delegation. The spawn_batch example (benchmark A, benchmark B,
prototype the winner) showed dependent work under a fan-out framing,
teaching the wrong shape.

In practice the coordinator failed to decompose enumerable requests
("top stories on HN, Lobsters, /r/programming, …") without explicit
"please fan this out" instructions, on both GPT-5.5 and Claude Opus.

base_coordinator.md
- Replace singular "a tasks entry, a child to own it" with plural
  "enumerate the independent units of work, spawn one child per unit,
  run them in parallel by default. Sequential only when one child's
  output feeds the next."
- Tighten the delegation paragraph.

tools_coordinator.md
- Drop the persona repetition that duplicated base_coordinator.md.
- Drop the prescriptive "## Workflow shape" section (the cost note is
  already in wait_for_workstream's tool description; the edit-X
  redirect is already in the persona).
- Drop "in one approval" / "single approval" mentions to avoid
  surfacing approval mechanics to the model.
- Replace the misleading spawn_batch example with truly independent
  items; drop "(up to 10)" which overstated the cap (it's per-call,
  not global, and is documented in the tool schema).
- Add a course-correction example to send_to_workstream — the pattern
  coordinators most often replace with cancel-and-respawn.
- Drop the read action from the tasks examples to keep the lifecycle
  (add → update → remove) coherent.

Coord system message ~16% shorter (4440 → 3722 chars). Both GPT-5.5
and Claude Opus now naturally decompose the news-board prompt without
explicit fan-out instructions. 29 prompt-composition tests pass.
2026-04-29 20:20:38 -07:00
Robert DeAngelis 0d1a32ff65 fix(server): accept --skip-permissions CLI flag (#450)
The server's --help epilog and compose.yaml both reference
--skip-permissions, but the argparser never defined it, so any
container started with SKIP_PERMISSIONS=1 exited with
"unrecognized arguments: --skip-permissions".

Wire the flag through to app.state.skip_permissions, OR-ing it
with the existing tools.skip_permissions config-store setting so
the stored value still works on its own.
2026-04-29 20:20:38 -07:00
650 changed files with 42455 additions and 144433 deletions
+33 -45
View File
@@ -1,61 +1,49 @@
# =============================================================================
# Turnstone environment overrides — ALL OPTIONAL for the dev stack.
# Turnstone Environment Variables
# Copy to .env and adjust values for your deployment.
#
# `docker compose up` from a clone works with zero config: every value below
# has a built-in (insecure) default. Copy this file to `.env` only to override.
#
# The PRODUCTION stack (turnstone/deploy/compose.yaml) has no baked-in secrets
# and DOES require TURNSTONE_JWT_SECRET and POSTGRES_PASSWORD.
#
# Note: for a turnstone process running on bare metal (not in a container),
# put secrets in ~/.config/turnstone/config.toml (chmod 0600), not the
# environment. See docs/docker.md "Join a bare-metal host".
# Usage:
# Single node: docker compose --profile production up
# 10-node cluster: docker compose --profile cluster up
# =============================================================================
# -- LLM backend --------------------------------------------------------------
# Optional: nodes boot without an LLM. Add real model backends from the console
# UI (Models tab). These only set the bootstrap default a node starts with.
# LLM_BASE_URL=http://host.docker.internal:8000/v1
# OPENAI_API_KEY=dummy
# ANTHROPIC_API_KEY=sk-ant-... # set instead of OPENAI_API_KEY for Anthropic
# TURNSTONE_SEARXNG_URL=http://searxng:8080 # web_search backend (default: bundled service; set to an external SearxNG)
# MODEL= # default model alias
# -- LLM Backend --------------------------------------------------------------
LLM_BASE_URL=http://host.docker.internal:8000/v1
OPENAI_API_KEY=dummy
# ANTHROPIC_API_KEY=sk-ant-...# Set instead of OPENAI_API_KEY for Anthropic
# TAVILY_API_KEY=tvly-... # Web search fallback (local models only)
# MODEL=# Override default model alias
# -- Secrets ------------------------------------------------------------------
# The dev stack defaults these to INSECURE values. Always set real ones for
# anything reachable beyond localhost. Generate the JWT secret with:
# python -c "import secrets; print(secrets.token_hex(32))"
# TURNSTONE_JWT_SECRET=
# POSTGRES_PASSWORD=
# -- Authentication (required) ------------------------------------------------
# Generate with: python -c "import secrets; print(secrets.token_hex(32))"
TURNSTONE_JWT_SECRET=changeme-to-32-bytes-of-hex
# -- Database -----------------------------------------------------------------
# Defaults to the bundled PostgreSQL (shared by every service — required for
# the console to discover nodes). Override to point at an external database:
# -- Database ------------------------------------------------------------------
# Single-node default is SQLite (zero config). Set these for PostgreSQL:
# TURNSTONE_DB_BACKEND=postgresql
# POSTGRES_USER=turnstone
# TURNSTONE_DB_URL=postgresql+psycopg://turnstone:<pw>@postgres:5432/turnstone
# POSTGRES_PASSWORD=changeme
# TURNSTONE_DB_URL=postgresql+psycopg://turnstone:changeme@postgres:5432/turnstone
# -- Ports / networking -------------------------------------------------------
# The dashboard is reached via Caddy only (HTTP/2 avoids the browser's
# 6-connection cap on the console's SSE streams). Both stacks expose the same
# two host ports; everything else is proxied through the console.
# CONSOLE_HTTPS_PORT=8443 # Caddy (dashboard HTTPS)
# POSTGRES_PORT=5432 # exposed for bare-metal host joins
# POSTGRES_BIND=127.0.0.1 # set 0.0.0.0 to let another machine join
# -- Ports ---------------------------------------------------------------------
# SERVER_PORT=8080
# CONSOLE_PORT=8090
# -- Workspace ----------------------------------------------------------------
# Bind-mount a host directory the model can read/write at /workspace:
# -- Workspace -----------------------------------------------------------------
# Bind-mount a host directory into the container at /workspace.
# The model can read/write files here. Default: empty Docker volume.
# WORKSPACE_MOUNT=/path/to/your/project
# -- Agent behavior -----------------------------------------------------------
# SKIP_PERMISSIONS=true # auto-approve all tool calls (dev only)
# MCP_CONFIG=/workspace/mcp.json # MCP server config file
# -- Agent behavior ------------------------------------------------------------
# SKIP_PERMISSIONS=true # Auto-approve all tool calls (dev only)
# MCP_CONFIG=/workspace/mcp.json# MCP server configuration file
# -- Channel gateway (Discord / Slack) ----------------------------------------
# -- Discord channel gateway ---------------------------------------------------
# TURNSTONE_DISCORD_TOKEN=
# TURNSTONE_DISCORD_GUILD=0
# TURNSTONE_SLACK_TOKEN=xoxb-...
# TURNSTONE_SLACK_APP_TOKEN=xapp-...
# -- Production image tag ------------------------------------------------------
# TURNSTONE_IMAGE_TAG=latest # pin the ghcr.io image (production stack)
# -- Cluster (profile: cluster) -----------------------------------------------
# These are set per-node in compose.yaml; only override for custom topologies.
# TURNSTONE_NODE_ID=node-1
# TURNSTONE_ADVERTISE_URL=http://server-1:8080
+22 -46
View File
@@ -14,8 +14,8 @@ jobs:
lint:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
- uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6
with:
python-version: "3.14"
- run: pip install pre-commit
@@ -25,8 +25,8 @@ jobs:
typecheck:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
- uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6
with:
python-version: "3.14"
- run: pip install mypy
@@ -35,15 +35,12 @@ jobs:
test:
runs-on: ubuntu-latest
# Cap a hung run at 20 min instead of riding GitHub's 6-hour default
# (a flaky-hang run otherwise streams -v output for hours).
timeout-minutes: 20
strategy:
matrix:
python-version: ["3.11", "3.12", "3.13"]
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
- uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6
with:
python-version: ${{ matrix.python-version }}
# Node is required by tests/test_renderer_js.py — without
@@ -54,10 +51,7 @@ jobs:
with:
node-version: "24"
- run: pip install -e ".[test]"
# -v lists each test id as it starts (pytest prints the nodeid at
# logstart), so a hang names the culprit on the last line instead of
# riding the job timeout with only a trail of "..." dots.
- run: pytest tests/ -m "not live" --cov=turnstone --cov-report=term-missing --cov-report=xml -v
- run: pytest tests/ -m "not live" --cov=turnstone --cov-report=term-missing --cov-report=xml -q
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
if: always()
with:
@@ -66,7 +60,6 @@ jobs:
test-postgres:
runs-on: ubuntu-latest
timeout-minutes: 20
services:
postgres:
image: postgres:18
@@ -82,23 +75,23 @@ jobs:
--health-timeout=5s
--health-retries=5
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
- uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6
with:
python-version: "3.14"
- uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6
with:
node-version: "24"
- run: pip install -e ".[test]"
- run: pytest tests/ -m "not live" --storage-backend=postgresql -v
- run: pip install -e ".[test,postgres]"
- run: pytest tests/ -m "not live" --storage-backend=postgresql -q
env:
TURNSTONE_TEST_PG_URL: postgresql+psycopg://postgres:postgres@localhost:5432/turnstone_test
wheel-completeness:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
- uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6
with:
python-version: "3.14"
- run: pip install build
@@ -113,16 +106,9 @@ jobs:
| grep -v '\.py$' | grep -v '\.dist-info' | grep -v '\.pyc' | grep -v '^File$' \
| sort)
# Files intentionally excluded from the wheel (one per line).
# The vllm-litellm/ deploy example ships in the repo, not the wheel
# (you clone the repo to run it; the package doesn't reference it).
# Files intentionally excluded from the wheel (one per line)
ALLOW="
turnstone/core/storage/migrations/script.py.mako
turnstone/deploy/vllm-litellm/.env.example
turnstone/deploy/vllm-litellm/README.md
turnstone/deploy/vllm-litellm/docker-compose.yml
turnstone/deploy/vllm-litellm/gemma.Dockerfile
turnstone/deploy/vllm-litellm/litellm-config.yaml
"
MISSING=$(comm -23 <(echo "$SOURCE") <(echo "$WHEEL") \
@@ -146,13 +132,13 @@ jobs:
/tmp/smoke/bin/turnstone-console --help
/tmp/smoke/bin/turnstone-admin --help
/tmp/smoke/bin/turnstone-channel --help
/tmp/smoke/bin/turnstone-doctor --help
/tmp/smoke/bin/turnstone-bootstrap --help
lock-check:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 # v8.2.0
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
- uses: astral-sh/setup-uv@08807647e7069bb48b6ef5acd8ec9567f424441b # v8.1.0
with:
uv-version: "0.9.18"
- run: uv lock --check
@@ -160,27 +146,17 @@ jobs:
security:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 # v8.2.0
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
- uses: astral-sh/setup-uv@08807647e7069bb48b6ef5acd8ec9567f424441b # v8.1.0
with:
uv-version: "0.9.18"
- uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6
- uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6
with:
python-version: "3.14"
- run: uv sync --frozen --all-extras
- run: uv pip install pip-audit
- name: Security audit (dependencies)
# PYSEC-2025-183 (pyjwt): "weak encryption" — disputed by the
# supplier because the key length is chosen by the calling
# application, not the library. Turnstone generates its JWT
# signing keys via the standard ``secrets`` module at
# operator-controlled strength (see ``turnstone/core/auth.py``),
# so the advisory does not apply. pyjwt 2.12.1 is the current
# latest release; no fix version exists.
run: >-
uv export --no-emit-project --frozen
| uv run pip-audit --strict --desc -r /dev/stdin
--ignore-vuln PYSEC-2025-183
run: uv export --no-emit-project --frozen | uv run pip-audit --strict --desc -r /dev/stdin
security-ts:
runs-on: ubuntu-latest
@@ -188,7 +164,7 @@ jobs:
run:
working-directory: sdk/typescript
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
- uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6
with:
node-version: "24"
-46
View File
@@ -1,46 +0,0 @@
name: Claude Code Review
on:
pull_request:
types: [opened, synchronize, ready_for_review, reopened]
# Optional: Only run on specific file changes
# paths:
# - "src/**/*.ts"
# - "src/**/*.tsx"
# - "src/**/*.js"
# - "src/**/*.jsx"
jobs:
claude-review:
if: github.event.pull_request.head.repo.full_name == github.repository
# Optional: Filter by PR author
# if: |
# github.event.pull_request.user.login == 'external-contributor' ||
# github.event.pull_request.user.login == 'new-developer' ||
# github.event.pull_request.author_association == 'FIRST_TIME_CONTRIBUTOR'
runs-on: ubuntu-latest
permissions:
contents: read
pull-requests: write # post the review + inline comments
issues: read
id-token: write
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
fetch-depth: 1
- name: Run Claude Code Review
id: claude-review
uses: anthropics/claude-code-action@6c0083bb7289c31716797a039b6367b3079cc46e # v1
with:
claude_code_oauth_token: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }}
allowed_bots: 'renovate[bot]' # let Renovate PRs get reviewed
plugin_marketplaces: 'https://github.com/anthropics/claude-code.git'
plugins: 'code-review@claude-code-plugins'
prompt: '/code-review:code-review ${{ github.repository }}/pull/${{ github.event.pull_request.number }}'
# See https://github.com/anthropics/claude-code-action/blob/main/docs/usage.md
# or https://code.claude.com/docs/en/cli-reference for available options
-63
View File
@@ -1,63 +0,0 @@
name: Claude Code
on:
issue_comment:
types: [created]
pull_request_review_comment:
types: [created]
issues:
types: [opened, assigned]
pull_request_review:
types: [submitted]
jobs:
claude:
if: |
(
github.event_name == 'issue_comment' &&
contains(github.event.comment.body, '@claude') &&
contains(fromJSON('["OWNER","MEMBER","COLLABORATOR"]'), github.event.comment.author_association)
) || (
github.event_name == 'pull_request_review_comment' &&
contains(github.event.comment.body, '@claude') &&
contains(fromJSON('["OWNER","MEMBER","COLLABORATOR"]'), github.event.comment.author_association)
) || (
github.event_name == 'pull_request_review' &&
contains(github.event.review.body, '@claude') &&
contains(fromJSON('["OWNER","MEMBER","COLLABORATOR"]'), github.event.review.author_association)
) || (
github.event_name == 'issues' &&
(contains(github.event.issue.body, '@claude') || contains(github.event.issue.title, '@claude')) &&
contains(fromJSON('["OWNER","MEMBER","COLLABORATOR"]'), github.event.issue.author_association)
)
runs-on: ubuntu-latest
permissions:
contents: read
pull-requests: write # post comments/reviews when @-mentioned on a PR
issues: write # post comments when @-mentioned on an issue
id-token: write
actions: read # Required for Claude to read CI results on PRs
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
fetch-depth: 1
- name: Run Claude Code
id: claude
uses: anthropics/claude-code-action@6c0083bb7289c31716797a039b6367b3079cc46e # v1
with:
claude_code_oauth_token: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }}
# This is an optional setting that allows Claude to read CI results on PRs
additional_permissions: |
actions: read
# Optional: Give a custom prompt to Claude. If this is not specified, Claude will perform the instructions specified in the comment that tagged it.
# prompt: 'Update the pull request description to include a summary of changes.'
# Optional: Add claude_args to customize behavior and configuration
# See https://github.com/anthropics/claude-code-action/blob/main/docs/usage.md
# or https://code.claude.com/docs/en/cli-reference for available options
# claude_args: '--allowed-tools Bash(gh pr *)'
+6 -17
View File
@@ -7,9 +7,7 @@ on:
concurrency:
group: docker-${{ github.event.workflow_run.head_sha }}
# Never cancel mid-push: an interrupted multi-tag push can leave the
# registry with a partial tag set (e.g. :latest moved, :stable not).
cancel-in-progress: false
cancel-in-progress: true
permissions:
contents: read
@@ -21,24 +19,15 @@ env:
jobs:
docker:
# Same gate as publish.yml: workflow_run fires for every CI completion
# (including fork and same-repo PR runs) with this repo's token and
# packages:write. Only same-repo tag pushes may publish images; CI's
# push trigger matches main/stable/* and v* tags, so a head_branch
# starting with "v" is necessarily a tag run.
if: >-
github.event.workflow_run.conclusion == 'success' &&
github.event.workflow_run.event == 'push' &&
github.event.workflow_run.head_repository.full_name == github.repository &&
startsWith(github.event.workflow_run.head_branch, 'v')
github.event.workflow_run.head_repository.full_name == github.repository
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
with:
ref: ${{ github.event.workflow_run.head_sha }}
fetch-depth: 0
# The docker build only reads the tree; keep the token out of it.
persist-credentials: false
- name: Resolve release tag
id: tag
@@ -54,7 +43,7 @@ jobs:
- name: Log in to GHCR
if: steps.tag.outputs.skip == 'false'
uses: docker/login-action@650006c6eb7dba73a995cc03b0b2d7f5ca915bee # v4
uses: docker/login-action@4907a6ddec9925e35a0a9e82d7399ccc52663121 # v4
with:
registry: ${{ env.REGISTRY }}
username: ${{ github.actor }}
@@ -78,12 +67,12 @@ jobs:
fi
echo "tags=${TAGS}" >> "$GITHUB_OUTPUT"
- uses: docker/setup-buildx-action@d7f5e7f509e45cec5c76c4d5afdd7de93d0b3df5 # v4
- uses: docker/setup-buildx-action@4d04d5d9486b7bd6fa91e7baf45bbb4f8b9deedd # v4
if: steps.tag.outputs.skip == 'false'
- name: Build and push
if: steps.tag.outputs.skip == 'false'
uses: docker/build-push-action@53b7df96c91f9c12dcc8a07bcb9ccacbed38856a # v7
uses: docker/build-push-action@bcafcacb16a39f128d818304e6c9c0c18556b85f # v7
with:
context: .
push: true
+5 -19
View File
@@ -7,9 +7,7 @@ on:
concurrency:
group: publish-${{ github.event.workflow_run.head_sha }}
# Never cancel a publish mid-upload: a half-uploaded release (sdist up,
# wheel missing) cannot be re-run cleanly because PyPI rejects duplicates.
cancel-in-progress: false
cancel-in-progress: true
permissions:
contents: write
@@ -17,26 +15,14 @@ permissions:
jobs:
publish:
# workflow_run fires for EVERY CI completion — including CI runs for
# pull_requests from forks — and always executes here with this repo's
# secrets, tokens, and the pypi environment. Gate to same-repo tag
# pushes only: CI's push trigger matches branches main/stable/* and
# tags v*, so a head_branch starting with "v" is necessarily a tag run.
if: >-
github.event.workflow_run.conclusion == 'success' &&
github.event.workflow_run.event == 'push' &&
github.event.workflow_run.head_repository.full_name == github.repository &&
startsWith(github.event.workflow_run.head_branch, 'v')
if: github.event.workflow_run.conclusion == 'success'
runs-on: ubuntu-latest
environment: pypi
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
with:
ref: ${{ github.event.workflow_run.head_sha }}
fetch-depth: 0
# python -m build executes the tree's build backend; don't leave
# the contents:write token sitting in .git/config while it runs.
persist-credentials: false
- name: Resolve release tag
id: tag
@@ -50,7 +36,7 @@ jobs:
echo "skip=false" >> "$GITHUB_OUTPUT"
fi
- uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6
- uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6
if: steps.tag.outputs.skip == 'false'
with:
python-version: "3.14"
@@ -63,7 +49,7 @@ jobs:
- name: Create GitHub Release
if: steps.tag.outputs.skip == 'false'
uses: softprops/action-gh-release@718ea10b132b3b2eba29c1007bb80653f286566b # v3
uses: softprops/action-gh-release@b4309332981a82ec1c5618f44dd2e27cc8bfbfda # v3
with:
tag_name: ${{ steps.tag.outputs.tag }}
generate_release_notes: true
-42
View File
@@ -1,42 +0,0 @@
name: Understone example
# The door-game example is a standalone package with no dependency on
# turnstone core, and the root test suite does not collect it
# (testpaths=["tests"]). Without this workflow its suite never runs in CI.
# Path-filtered so it only runs when the example (or this workflow) changes.
on:
push:
branches: [main, "stable/*"]
paths:
- "examples/door-game/**"
- ".github/workflows/understone-example.yml"
pull_request:
branches: [main, "stable/*"]
paths:
- "examples/door-game/**"
- ".github/workflows/understone-example.yml"
permissions:
contents: read
jobs:
understone:
runs-on: ubuntu-latest
defaults:
run:
working-directory: examples/door-game
strategy:
matrix:
# Floor and ceiling of the example's requires-python (>=3.11).
python-version: ["3.11", "3.13"]
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6
with:
python-version: ${{ matrix.python-version }}
- run: pip install -e ".[test,dev]"
- run: pytest tests/ -q
- run: ruff check .
- run: ruff format --check .
- run: mypy understone/
+5 -26
View File
@@ -25,43 +25,22 @@ permissions:
jobs:
vendor-js:
# Same-repo PRs only: this job checks out the PR head and pushes to it
# with contents:write, so it must never act on a fork's branch.
# Gate on the PR author (immutable), not github.actor (names whoever
# caused the latest event, which can be someone else re-running it).
if: >-
(github.event_name == 'pull_request' &&
github.event.pull_request.user.login == 'renovate[bot]' &&
github.event.pull_request.head.repo.full_name == github.repository) ||
github.event_name == 'workflow_dispatch'
if: github.actor == 'renovate[bot]' || github.event_name == 'workflow_dispatch'
runs-on: ubuntu-latest
steps:
- name: Resolve PR head ref
id: ref
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
# Branch names may contain shell metacharacters; pass via env,
# never interpolate ${{ }} into the script body.
HEAD_REF: ${{ github.head_ref }}
PR_NUMBER: ${{ inputs.pr_number }}
run: |
if [[ "$GITHUB_EVENT_NAME" == "workflow_dispatch" ]]; then
# The dispatch input is an arbitrary PR number; refuse fork PRs.
# A fork's headRefName is a bare branch name that may collide
# with a branch in this repo, and checkout+push would then hit
# that unrelated branch ("same-repo PRs only" applies here too).
pr_json=$(gh pr view "$PR_NUMBER" --repo "$GITHUB_REPOSITORY" --json headRefName,isCrossRepository)
if [[ "$(jq -r '.isCrossRepository' <<< "$pr_json")" != "false" ]]; then
echo "::error::PR #${PR_NUMBER} head is not a branch in this repository; refusing to complete it."
exit 1
fi
ref=$(jq -r '.headRefName' <<< "$pr_json")
if [[ "${{ github.event_name }}" == "workflow_dispatch" ]]; then
ref=$(gh pr view "${{ inputs.pr_number }}" --repo "${{ github.repository }}" --json headRefName -q .headRefName)
else
ref="$HEAD_REF"
ref="${{ github.head_ref }}"
fi
echo "head_ref=${ref}" >> "$GITHUB_OUTPUT"
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
with:
ref: ${{ steps.ref.outputs.head_ref }}
-5
View File
@@ -9,11 +9,6 @@ build/
.venv/
venv/
.env
# Local compose overrides (e.g. run.sh's node-count limiter, bootstrap output)
compose.override.yaml
compose.override.yml
docker-compose.override.yaml
docker-compose.override.yml
*.so
.mypy_cache/
.ruff_cache/
+6 -837
View File
@@ -6,845 +6,14 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
and this project adheres to [PEP 440](https://peps.python.org/pep-0440/) for
version numbers (`X.Y.Z`, with `X.Y.ZaN` / `bN` / `rcN` for pre-releases).
Three release tracks are maintained — the current stable, one prior
stable, and the experimental line:
Three release tracks are maintained:
- **`stable/1.5`** — patch-only (`v1.5.x`)
- **`stable/1.6`** — patch-only (`v1.6.x`)
- **`main`** — experimental (next major)
- **`stable/1.0`** — patch-only (`v1.0.x`)
- **`stable/1.3`** — patch-only (`v1.3.x`)
- **`stable/1.4`** — patch-only (`v1.4.x`)
- **`main`** — experimental (`v1.5.0aN`)
## [1.6.0]
The first stable release of the 1.6 line — and the first under Apache 2.0.
> **⚠️ Before upgrading from 1.5.x:** 1.6.0 changes the internal
> conversation storage schema (Alembic migration `060`, applied
> automatically on first start). The migration converts existing
> workstreams and attachments in place — **back up your storage before
> upgrading** (`pg_dump` for PostgreSQL; copy the database file for
> SQLite). Background: discussion
> [#631](https://github.com/turnstonelabs/turnstone/discussions/631).
**Breaking changes at a glance** (details in the sections below):
`web_search` backend overhaul (Tavily/DuckDuckGo removed, `topic`
`category`), the `man` / `math` / `plan_agent` built-in tools and the
plan-review protocol removed, and the body-keyed `/v1/api/command`
endpoint replaced by path-keyed workstream verbs.
### License
- **Relicensed to Apache 2.0** — from BUSL-1.1, effective with this
release (#546, contributor assent record in #548). Versions 1.5.x and
earlier remain under BUSL-1.1 as shipped, and the `stable/1.5` branch
keeps its original LICENSE. New `NOTICE` and
`CONTRIBUTORS.md` files; `THIRD-PARTY-NOTICES` refreshed to match the
bundled library versions.
### Added
- **Mid-conversation system messages** — advisories, watch results,
skill hints, and operator interjections are now first-class
`role=system` turns in the trajectory instead of ad-hoc reminder
envelopes. Models with native mid-conversation system support receive
them verbatim; for everything else they fold into a nonce-fenced
wrapper. The one-shot `_reminders` side-channel is gone.
- **Self-hosted SearxNG web search** — the `web_search` backend for
local/vLLM models is now a bundled [SearxNG](https://searxng.org)
service (in both compose stacks; internal network only). Configure via
`tools.searxng_url` / `tools.searxng_engines`. Commercial providers
keep their native server-side search; the model can target a corpus by
passing `category` (`general`, `news`, `it`, `science`). Operators
exposing the bundled SearxNG publicly: see the AGPL-3.0 §13 note in
[docs/docker.md](docs/docker.md).
- **Endpoint-backed reranking** — a reranker is now a per-model
definition (Cohere/Jina-compatible wire: vLLM, TEI, llama.cpp, or a
commercial endpoint), disabled by default. When configured it scores
`web_search` results and the BM25 retrieval surfaces (deferred tools,
skills, memory) behind a `tools.rerank_bm25` toggle with a relevance
floor; a calibration CLI (and calibrate-on-detect) tunes the floor
per model.
- **Proactive memory relevance** — injected memories are selected by
BM25 + reranker against the recent user messages instead of recency
alone, and first composition defers to the first user turn so fresh
sessions select against a real query.
- **Smart Approvals** — opt-in (default off): high-confidence `approve`
verdicts from the intent judge auto-approve the tool call instead of
waiting for a human, with a confidence threshold and verdict
bookkeeping designed so a denied or reset judge never auto-fires.
- **Early-painted tool calls** — committed tool calls render immediately
as pending cards (both UIs upgrade the card in place by `call_id`)
instead of waiting for the judge verdict, so big parallel batches no
longer sit invisible during judging.
- **Voice I/O v1** — speech-to-text and text-to-speech as model roles
speaking the OpenAI audio wire protocol (#618); the interactive
composer grows a mic button.
- **Rewind / retry / edit-first-message** — full UX in both the
interactive UI and the coordinator pane, backed by shared path-keyed
verb handlers (#549).
- **Workstream export** — download a conversation as OpenAI-format
messages JSON.
- **Skills platform round** — `SKILL.md` ingestion learns
`when_to_use` / `model` / `effort` / `paths`; prompt substitution
supports `$ARGUMENTS`, `$N`, `$<name>`, and `${CLAUDE_*}` (#572);
per-skill `disable-model-invocation` and `user-invocable` flags
(#571); `skill` + `list_skills` unify into one dual-kind tool; new
`model.skills.write` permission.
- **Coordinator hardening for small models** — workstream references in
coordinator tool calls are validated with did-you-mean recovery, and
`wait_for_workstream` fails fast with uniform `not_found` entries
instead of hanging on a hallucinated `ws_id`.
- **Provider support** — Claude Fable 5 and Claude Opus 4.8; xAI/Grok
via the OpenAI Responses lane; vLLM reasoning-field replay completes
the reasoning-persistence work (#537).
- **Cluster-by-default deployment** — the compose stack fronts
everything with Caddy and supports bare-metal node join; a one-line
`curl | bash` installer bootstraps a node; nodes with no configured
models boot into a degraded state instead of crash-looping; channel
gateways stand by when no adapter token is set.
- **MCP OAuth tokens encrypted at rest**.
- **`turnstone-admin` reads `config.toml`** — same `[database]` section
and precedence as the server (`CLI / config.toml > TURNSTONE_DB_* env
> defaults`), including `pool_size` and the `ssl*` knobs it previously
dropped; new `--config PATH` flag.
### Changed
- **Conversation storage and the provider wire are rebuilt around a
canonical trajectory** (migration `060` — see the upgrade note).
Internally a conversation is now a provider-neutral `Turn` sequence
lowered to each provider's wire format at send time; provider-specific
tool-call metadata rides an opaque producer-tagged lane (replayed
verbatim to the producing provider, rebuilt for others); attachments
become content-addressed, reference-counted rows resolved at the
provider boundary; orphan tool-call repair happens once, at send time.
Wire-visible behavior is unchanged for OpenAI-compatible providers;
histories are preserved across the migration.
- **The console and web UI share one L-shell** — a left glyph rail, a
tab bar, and a pane host now frame interactive chats, coordinator
sessions, dashboards, and the admin panel as tabs in a single window;
the standalone web UI adopts the same shell and the old split-pane
layout is retired. Coordinator and interactive conversations render
through shared `.conv-*` card builders, the rail collapses to a glyph
strip (remembered per browser), mobile gets an off-canvas drawer, and
the frontend is now ES modules end to end.
- **Admin panel modals → the Service Hatch shelf** — all ~35 admin
modals are replaced by pane-scoped shelves plus a small dialog tier
for confirmations. Schedules gain a cron builder with a next-3-runs
preview endpoint, model capabilities render as an LED tile matrix, and
the legacy modal machinery is deleted.
- **SSE delivery is resumable end to end** — per-workstream ring buffer
with `Last-Event-ID` replay (cap raised 2,000 → 50,000), fresh-connect
and reconnect unified on one event-id cursor (in-flight tool batches
included), persisted `last_error` replays on connect, the console
proxy forwards `Last-Event-ID`, and panes close their connections on
`beforeunload` to stop multi-pane refresh from exhausting the
browser's per-host connection cap (#539).
- **Workstream verbs are path-keyed** *(BREAKING)*`rewind` / `retry`
/ `edit-first-message` live at
`/v1/api/workstreams/{ws_id}/<verb>` alongside the other session
verbs; the body-keyed `/v1/api/command` endpoint is removed (#549).
- **`/history` is projected server-side** — both UIs consume the same
REST-first wire shape instead of re-deriving it client-side.
- **Saved workstreams & coordinators: card grid → sortable table** with
model/skill/context columns, pagination, and a unified selector across
both dashboards.
- **`tools.web_search_backend` accepted values** *(BREAKING)* — now `""`
(auto), `"searxng"`, or `"mcp:server:tool"`. The old `"tavily"` and
`"ddg"` values are gone; a config still set to either disables web
search and logs a warning. Auto-detect resolves to SearxNG when
`searxng_url` is set.
- **`web_search` tool: `topic``category`** *(BREAKING)* — renamed
LLM-facing parameter; values map to SearxNG categories. The Tavily-era
`finance` topic is gone.
- **Core install includes what most deployments use** — `anthropic`,
`postgres`, `console`, and `tls` are core dependencies rather than
extras.
- **NODES table → bottom-bar node picker** in the console.
### Fixed
- **Cluster mTLS actually survives operations** — certificate identity
keys on the advertised host rather than the container ID, renewals are
scoped per node, reloaded certs hot-swap into the live SSL context,
and healthchecks/boot retries are mTLS-aware.
- **Intent-verdict lifecycle** — history replay ships risk-none verdict
rows (live/replay parity), late verdicts persist as `superseded` for
the audit trail instead of vanishing, bulk verdict insert tolerates
per-row conflicts, and cancel-on-approval honors its run-to-completion
contract.
- **Usage accounting** — dashboard totals were under-counting; auxiliary
LLM spend (judge, rerank, memory) is now recorded.
- **Concurrent first-boot migrations** no longer deadlock on the
advisory lock.
- **Output renderer** — single-`$` inline math no longer false-positives
in prose; `strip_html` preserves block structure and drops a ReDoS
risk.
- **Model registry** orders versions numerically (no more `1.10 < 1.9`
selection).
### Removed
- **Tavily and DuckDuckGo `web_search` backends** *(BREAKING)*
replaced by the bundled SearxNG service. Removed:
`tools.tavily_api_key`, `$TAVILY_API_KEY`, `[api].tavily_key`, and the
`ddg` install extra. Point `TURNSTONE_SEARXNG_URL` at an existing
instance or use the bundled one; no database migration required.
- **`man`, `math`, and `plan_agent` built-in tools** *(BREAKING)*
`man`/`math` duplicated `bash`; planning is better expressed as a
`task_agent` running a planning skill. Also removed: the `math`
sandbox executor, the read-only `AGENT_TOOLS` sub-agent set, the
plan-review protocol (`/v1/api/plan`, `plan_review`/`plan_resolved`
SSE events, `on_plan_review` hooks), and the `model.plan_*` settings.
Interactive built-in tool count: 19 → 16.
- **`stable/1.4` track retired** — the maintenance policy is now the
current stable plus one prior (`stable/1.6` + `stable/1.5` as of this
release). 1.4's final release was `v1.4.0`; its tags and released
artifacts remain available, under BUSL-1.1 as shipped.
### Security
- **Zero direct-HTML frontend** — every `innerHTML` sink across the
console and web UI is replaced with DOM construction or `setSafeHtml`,
inline handlers became delegated bindings, and CI lints pin the
invariant (plus `var`-free and const-reassign checks) across all
swept bundles.
- **Output guard grows an LLM stage** — merged with the heuristics as
escalate-only (an LLM verdict can raise but never lower a heuristic
positive), with annotated findings, a capability gate, and hardening
against domain-camouflaged injection (#560, #573).
- **One trust-fence primitive** — operator and judge envelopes share a
nonce-fenced wrapper (64-bit nonces, host-escaping); the output guard
flags nonce forgery, and skill hints no longer echo model-controlled
filter values into trusted text.
- **RBAC** — built-in role overrides get an editor, and several
under-enforced permission gates are tightened (#585).
- **Permissive `config.toml` warns** — a single startup warning when the
resolved config file is group- or world-readable; operators usually
want `0600`.
- **Dependency floors** — `starlette>=1.0.1` (PYSEC-2026-161 host-header
path injection) and `aiohttp>=3.14.0` (security release).
## [1.5.17]
Backports a clutch of coordinator-tool clarity fixes plus a watch-delivery
correctness fix from `main` to the `stable/1.5` track, plus a previously-
latent intent-verdicts persistence bug exposed by the new heuristic-verdict
INSERT paths. No schema changes.
### Fixed
- **`intent_verdicts` PK collisions on every llm_fallback delivery** —
async LLM-tier "llm_fallback" verdicts (`turnstone/core/judge.py`
`_deliver_fallbacks` and the in-loop fallback path) deliberately
reuse the heuristic verdict's `verdict_id` so the row gets
"upgraded in place" from `tier="heuristic"``tier="llm_fallback"`
when the LLM judge times out, is cancelled, or returns no content.
The consumer `_persist_intent_verdict` was doing a plain INSERT,
hitting the `intent_verdicts_pkey` constraint on every fallback
delivery; Postgres logged the duplicate-key error, the application
try/except swallowed it at `log.debug`, and the row never actually
got upgraded — the LLM judge's annotation
(`"(LLM judge did not return a verdict)"`) was lost. The collision
rate exploded on this release because the new heuristic-INSERT
paths in the auto-approve early-return branches of `approve_tools`
(introduced below) leave no gap for the fallback to land cleanly
into. Fix: new `upsert_intent_verdict` storage method using
`ON CONFLICT (verdict_id) DO UPDATE` that updates only `tier`,
`reasoning`, `judge_model` — the three fields that genuinely
change between heuristic and llm_fallback. Every other column
(identity, carried-verbatim, and `user_decision`) is excluded;
`user_decision` in particular would otherwise be clobbered back
to `"pending"` when a fallback arrives after the operator has
already resolved the approval. The bulk-INSERT path stays as
plain INSERT — fresh UUIDs in `judge.evaluate` make in-turn dups
impossible; the inverse race (fallback wins before bulk lands) is
reachable but unchanged in observable behavior by this fix,
documented at the bulk site for a future hardening pass.
- **Coordinator LLM re-spawn loops on large fan-outs** — the spawn-tool
return JSON used `ws_id` as its key, which primed the model's recency
bias to feed the spawn result straight back into another
`spawn_workstream(ws_id=...)` call instead of progressing to
`wait_for_workstream(ws_ids=[...])`. On 10+ child fan-outs this cascaded
into self-inflicted re-spawn loops. The LLM-facing tool result now emits
`child_ws_id` (the storage column / HTTP API contract is unchanged); the
field name is already an existing project term so the rename aligns
rather than introduces new vocabulary. Also handles the silent
upstream-omits-ws_id success-shape edge that previously emitted
`{"child_ws_id": null}` to the LLM — now surfaces a tool error so the
model retries rather than chasing a null id.
- **`inspect_workstream` blowing the coordinator context budget** — a
coord doing a fan-out wave against tool-heavy children could land
>100 KB of raw output per inspect call, and the previous safety net
(`_truncate_output`'s head+tail strategy) silently dropped *middle*
messages — exactly the wrong shape for understanding a child's
trajectory (the FIRST sets the brief, the LAST shows the conclusion,
the middle is the connective tissue). Output now goes through a
three-tier degradation ladder mirroring the search tool's
`_format_search_results`: `_tier="full"` (every message verbatim) →
`_tier="compact"` (per-message head/tail-snipped content + snipped
`tool_calls.arguments`, falling through a `(20,30)` / `(10,20)` /
`(5,10)` message-list trim ladder) → `_tier="skeleton"` (counts, role
distribution, last-assistant preview). Budget 32 KiB matches the
search tool's; the chosen tier is annotated on the response so the
model can recall with a tighter `message_limit` if signal was lost.
- **Auto-approved verdicts indistinguishable from pending review** —
`intent_verdict` rows for auto-approved tool calls landed with
`user_decision=""`, which read identically to "still waiting for the
operator" in the audit trail and led to a real misdiagnosis incident.
The column now carries an explicit vocabulary at insert: `pending` /
`approved` / `denied` / `timeout` / `policy` / `blanket` / `skill` /
`always` / `auto_approve_tools`. The auto-approve early-return
branches in `approve_tools` now persist heuristic verdicts stamped
with their reason (previously dropped on the floor), and late LLM-tier
verdicts that arrive for an already-auto-approved call_id are stamped
via a TTL-pruned lookup map — so the audit row carries the
auto-approve reason even when the LLM judge daemon completes after
the synchronous approval cycle finished. `resolve_approval` gains a
`timeout` kwarg writing `"timeout"` (the previous shape collapsed
passive timeouts and active denials into the same column).
- **`list_skills` empty `allowed_tools` misread as "no tool access"** —
the response previously emitted `"allowed_tools": []` for every skill
that hadn't declared an auto-approve allowlist, which a coordinator
model read as "this skill can't use any tools" (real misdiagnosis: a
code-review child appeared to have been spawned with zero tool
access). The field is now omitted entirely when empty — absence
carries the unambiguous meaning "no tool is pre-approved for this
skill", presence (non-empty list) keeps the standard Claude Code
skill-spec shape. The tool description rewrite makes the
auto-approve-allowlist semantics explicit so a future reader doesn't
re-derive the gating misread.
- **Watch terminal-fires silently dropped on backpressure** —
delivery now routes terminal events through the same path as
normal fires instead of being filtered out when the consumer was
saturated.
### Documentation
- **Storage `LIKE_ESCAPE` contract** — clarify that callers passing
`.like(escape=...)` must use the same escape character that the
storage helper assumes; previous wording let a reader pass a
different escape and silently produce no matches.
## [1.5.15]
### Fixed
- **Admin console blank-page on MCP server rows with consented users** — a
Phase 9 (1.5.14) regression in `admin.js` used double-quote string
delimiters on the bulk-revoke button HTML literal, but the literal embeds
a `"` mid-attribute. JS closed the string early, turned `bulk-revoke (`
into bare tokens, and the resulting `SyntaxError` wiped out every global
in `admin.js``showAdmin` and all other admin entry points became
undefined, so the console UI was non-functional whenever the rendered MCP
server list contained at least one row with `consented_users_count > 0`.
Switch the literal to single-quote delimiters to match the surrounding
block.
## [1.5.14]
Backports OAuth-MCP Phase 9 from `main` to the `stable/1.5` track.
### Added
- **OAuth-MCP Phase 9 — admin status, deferred-consent persistence, operator
docs** — completes the per-(user, server) OAuth-MCP build-out. The sync pool
dispatchers now upsert into a new `mcp_pending_consent` table on
`mcp_consent_required` / `mcp_insufficient_scope`, so a non-interactive run
(scheduled / channel) that hits an unconsented server surfaces the deferred
prompt to the user on their next dashboard load via the gear-icon badge —
rows are cleared automatically by the OAuth callback handler on consent
completion, or via new DELETE endpoints for manual dismiss. The MCP Servers
admin row gains a `consented_users_count` pill and a two-step-confirm
bulk-revoke button for `auth_type=oauth_user` servers (upstream RFC 7009
revoke is intentionally not attempted in bulk to avoid N synchronous
round-trips against the provider). Operator-facing docs land at
`docs/mcp-oauth.md` and `docs/operations/mcp-oauth-headless.md`.
Introduces forward-only migrations `054_mcp_pending_consent` and
`055_mcp_user_tokens_server_index`.
## [1.5.13]
This release introduces one forward-only schema migration:
`053_services_notify_trigger` — installs the `services_notify` PostgreSQL
trigger that backs the new LISTEN/NOTIFY dispatcher (no-op on SQLite, where
the dispatcher uses in-process fan-out).
### Added
- **Reactive node discovery via PG LISTEN/NOTIFY** — the console gains a
`NotifyDispatcher` that holds a dedicated session-mode PostgreSQL `LISTEN`
connection (bypasses pgbouncer transaction pooling) and fans wake-ups out to
per-channel handlers on a separate dispatch thread. The cluster collector
subscribes to a new `services` channel and reacts to node register /
deregister within ~500 ms instead of waiting up to 60 s for the next discovery
loop; the 60 s loop is retained as the backstop for crash-shaped loss
(NOTIFY only fires on real writes). The storage layer also gains a uniform
`notify` / `listen` API with an SQLite synthetic-sweep fallback so consumer
code is identical across backends. `TURNSTONE_DB_LISTEN_URL` (or
`[database] listen_url` in `config.toml`) points the dispatcher at a
direct-to-Postgres URL; defaults to the main DB URL when unset.
- **Event-driven `wait_for_workstream`** — coord's block-wait tool no longer
polls storage every 500 ms. A new in-process `ChildEventBus` notifies waiters
whenever a child state change is dispatched to the UI, and the wait loop
blocks on `threading.Event.wait` with a 2 s heartbeat cap (matching the
existing `wait_progress` SSE cadence). A 600 s wait that previously hit
storage ~2400 times now wakes only on real state transitions, with ~4× lower
SSE traffic in the quiescent case.
- **Memory tool audit trail** — the memory tool now emits `memory.save`,
`memory.update`, and `memory.delete` audit events (the admin-console DELETE
route previously emitted only `memory.delete`, so tool-initiated mutations
had no audit footprint). All emissions are best-effort and never break the
tool call itself.
- **`task_agent` per-call personas via `skill=`** — `task_agent` now accepts
an optional `skill=<name>` argument that loads the named skill's content as
the sub-agent's persona in place of the hardcoded identity statement. The
fixed operating-guidance block (one-shot, tool-use over narration,
no follow-up questions) is still layered on top of every persona. High- and
critical-risk skills surface their risk tier in the approval header and
emit a `task_agent.high_risk_skill` warning, matching the existing
session-load gate.
### Fixed
- **Per-role plan / task model overrides could be bypassed by the LLM** — the
back-compat `default` alias auto-synthesised by `load_model_registry`
remained visible to the model even when an operator had configured
`model.task_alias` / `model.plan_alias`, so `task_agent(model="default")`
routed to whichever backend the synthesised alias was attached to at boot
instead of the configured per-role default. The synthesised alias is now
only added when neither the DB nor `[models.*]` populates the registry,
filtered out of the LLM-visible alias list, and explicitly rejected at the
validator chokepoint as defense-in-depth.
- **Mermaid streaming parse errors + progressive `hljs`** — live-streamed
mermaid blocks with bare `(`, `[`, `{` inside unquoted edge or rectangle
node labels were re-entering the shape parser and producing
`Parse error, got 'PS'` messages. The renderer now autoquotes the two
affected label forms (`|content|` and `ID[content]`) before the SVG cache
lookup; shapes whose syntax already nests delimiters (cylinders, subroutines,
trapezoids, etc.) are intentionally left alone. The companion `hljs` change
highlights code blocks progressively as they stream rather than only after
completion.
- **Re-auth from inside the proxy-prefixed UI** — on a proxied node page
(`/node/{id}/...`), an expiring JWT triggered an in-page login modal whose
POST went to `/v1/api/auth/login` and was rewritten to
`/node/{id}/v1/api/auth/login`. Two latent bugs both blocked re-auth: the
console's `AuthMiddleware` didn't recognise the `/node/{id}/` prefix over a
public path, and `proxy_api` would have forwarded the login request to the
upstream node (which mints `JWT_AUD_SERVER` tokens the console then rejects).
Both fixed: proxied public paths stay public, and `proxy_api` now dispatches
every entry in `_PROXY_AUTH_LOCAL_HANDLERS` (login, logout, setup, refresh,
status, whoami, oidc/authorize, oidc/callback) to the console's own auth
handlers. The dispatch table is a single `(method, path) → handler` mapping
so the test parametrize list can't drift from the implementation.
- **Appbar visibility + gear-icon dropdown on the dashboard** — the dashboard
overlay was covering the entire appbar, hiding the proxy-injected node
picker. The overlay now starts at `top: 48px` and the dashboard's role
downgrades from `dialog+aria-modal` to `region` so the appbar above it
remains reachable. The gear icon converts from a direct settings-panel
click into a dropdown with "MCP connections" and "Logout" (the latter with
`.destructive` styling). The settings-menu keydown handler is now attached
synchronously so `Escape` can't fall through the brief window between the
menu opening and its listeners being installed.
- **PostgreSQL test backend on the notify dispatcher suite** — migration 053's
`services_notify` trigger lives only in the alembic chain, but the test
fixture creates tables via `metadata.create_all`. The trigger function +
trigger are now declared in `_schema.py` and attached via
`sa.event.listen(services, "after_create", ...)` DDL events gated on the
PostgreSQL dialect, with the same SQL constants imported by migration 053
so there's a single source of truth.
## [1.5.12]
### Added
- **Enriched backend error messages** — provider name and attempted URL are now
included in session error responses, so operators can triage connectivity
failures without enabling debug logging.
### Fixed
- **`/rewind` always emits a `history` SSE event** — pre-fix, if the session
had no messages remaining after a rewind the history event was skipped,
leaving connected UIs with stale content and blocking edit-and-resend flows.
## [1.5.11]
This release introduces one forward-only schema migration:
`052_model_reasoning_persistence``surface_persisted_reasoning` and
`replay_reasoning_to_model` flag columns on `model_definitions`.
### Added
- **SSE refresh-resume** — clients that reload mid-stream (browser refresh, tab
restore) now receive an `in_progress_snapshot` event carrying the buffered
partial response, so the UI can resume rendering the in-flight turn without
losing content. The snapshot is keyed by a monotonic `_ws_inflight_seq`
counter so a reconnecting client can skip events it already saw.
- **Reasoning persistence** (Phases 14) — model reasoning text can now be
persisted to conversation history and optionally replayed to the model on
subsequent turns. Phase 1 persists reasoning text on the history payload.
Phase 2 wires a build-time shape filter and a per-model
`replay_reasoning_to_model` flag. Phases 3+4 add full OpenAI Responses API
(`include=["reasoning.encrypted_content"]`) and Chat Completions support;
an `ANTHROPIC_VALID_BLOCK_TYPES` shape filter guards the Anthropic path. Two
new per-model capability flags (`surface_persisted_reasoning`,
`replay_reasoning_to_model`) both default `False` on unknown and
local-server models.
- **Console home composer: placeholders + toggle** — the console landing-page
composer now shows context-aware placeholder text and a toggle component for
advanced options; an admin polish pass tightened spacing and focus behaviour
across the form.
### Changed
- **`judge.model` now requires a named alias** — raw provider model IDs on
`judge.model` in config are no longer accepted; the judge must reference an
alias registered in the model registry. The session-provider raw-model
fallback is removed. Existing configs using an unregistered model ID need a
corresponding alias entry.
### Fixed
- **`replay_reasoning_to_model` AND-gated with model capability** — setting the
flag for a model that does not declare reasoning-replay support now silently
no-ops instead of forwarding reasoning blocks and triggering a provider error.
- **Coordinator alias resolution unified across placeholder + factory** — a
placeholder coordinator and the real coordinator factory could previously
resolve to different model aliases, producing a visible mismatch in the model
display. Both paths now share the same resolution logic.
- **Console `cs=None` fallback in `/v1/api/models` placeholder** — an
under-initialised coordinator state no longer 500s when the models endpoint
is hit before the coordinator subsystem is fully bootstrapped.
- **SSE `_ws_inflight_seq` always advances** — sequence numbers were previously
skipped when an emit was past the buffer cap, leaving gaps in the monotonic
counter that broke `state_change` / `in_progress_snapshot` ordering on
reconnect.
- **Reasoning persistence shape + replay fixes** — per-block
`ANTHROPIC_VALID_BLOCK_TYPES` filter applied; `reasoning_text` is now
synthesised alongside non-reasoning `provider_blocks` so both appear
together in the history payload.
## [1.5.10]
This release introduces one forward-only schema migration:
`051_skill_notify_on_complete_array_default` — backfills
`prompt_templates.notify_on_complete` from `'{}'` to `'[]'`.
### Added
- **Skills unlock action** — operators can unlock an installed skill to allow
local customisation. Once unlocked, the skill's resource content, system
prompt additions, and notify configuration are editable through the admin UI.
Skills shipped as part of a bundle remain locked (read-only) until explicitly
unlocked; the unlock is logged to the audit trail. A lock icon in the
top-right of the Skills detail pane doubles as the unlock trigger.
### Fixed
- **`skills.sh` install endpoint** — the install script was targeting an
endpoint removed in an earlier refactor; switched to `/api/download`.
- **Skills `notify_on_complete` default** — the field defaulted to `{}`
(object) instead of `[]` (array), causing notify configurations to be
rejected at schema validation.
- **Skills admin UI modal errors** — `.is-visible` class used consistently
instead of inline `style.display`; stale error text is cleared on submit;
designer-review lock-icon UX applied.
## [1.5.9]
### Fixed
- **`repair=False` on all display-read `load_messages` call sites** —
passing `repair=True` on display paths was silently mutating the stored
message list, causing divergence between what the UI showed and what the
model received on the next turn.
## [1.5.8]
This release introduces two forward-only schema migrations:
`049_mcp_oauth_schema` — OAuth token + consent tables for MCP servers;
`050_conversations_source_and_reminders``_source` and `_reminders` columns
on `conversations`.
### Added
- **MCP OAuth 2.1 + PKCE** — MCP servers that require OAuth can now be
configured with a client ID and secret through the admin UI. The full token
lifecycle (acquire → refresh → rotate) is managed automatically; tokens are
stored encrypted at rest using a key derived from the JWT secret. The consent
flow runs in-browser via a provider redirect. Rolled out in phases:
- Minimum admin form and OAuth schema (`21663d15`).
- Token-at-rest AES-GCM encryption layer (`a4c335d7`).
- Per-(user, server) OAuth 2.1 + PKCE flow (`b0f7029f`).
- Per-(user, server) `ClientSession` pool with OAuth dispatch (`1a1043c4`).
- SDK 401/403 introspection via httpx response hook (`bde09134`).
- Phase 7 — per-user tool catalog scoping: each user sees only the tools
their OAuth token is permitted to call (`cfc8a6c8`).
- Phase 7b — per-user resource + prompt pool dispatch (`b368bdee`).
- Phase 8 — per-user MCP consent UX: users see a consent dialog on first
use of an OAuth-gated server and can revoke consent from their profile;
admins see per-server consent counts in the MCP Servers tab (`61051339`).
- **Metacognition NudgeQueue** — all advisory channels (repeat-tool nudges,
watch reminders, wake triggers) are unified into a pull-model `NudgeQueue`
that delivers at most one nudge per turn, preventing multi-channel pile-ups
that inflate context. Observable changes:
- Watch results carry metadata (watch ID, `valid_until`, trigger type)
through to the system message so the model can reason about recency.
- Coordinator idle-children observer: a coordinator with no in-flight
children for longer than the configured idle threshold receives a nudge.
- Wake trigger (`IdleNudgeWatcher`): sessions waiting on an external event
can be unblocked via `ChatSession.deliver_wake_nudge_from_queue`.
- Watch switchover: watch results are now enqueued on the `NudgeQueue`
rather than the previous `_watch_pending` list, giving them the same
delivery guarantees and priority handling as other advisories.
- **Structured watch-result card** — the UI renders watch results as a styled
card with a system-nudge marker, distinct from the assistant message body.
On history replay, system-nudge turns are visually distinguished from normal
assistant turns.
- **Side-channel persistence** — `_source` and `_reminders` side-channel
fields are persisted to the `conversations` storage table and restored on
session resume, so metacognitive context survives process restarts. A
`REMINDER_TEXT_STORAGE_CAP` byte clamp prevents unbounded growth.
### Fixed
- **Replay consistency** — queued user messages captured mid-loop are now
persisted and replayed in the correct order on a subsequent `events`
subscription. Coordinator history replay fixed: blank assistant cards and
out-of-order tool results on the coordinator tree no longer occur when the
coordinator has mixed queued + delivered messages.
- **Session reminder preservation on fork + resume** — `_source` and
`_reminders` are carried through workstream fork and restored from storage
on resume.
- **NUL-byte sanitization in storage** — PostgreSQL rejects `\x00` in text
columns; `_source` and `_reminders` now strip NUL bytes on write.
- **Console coordinator subsystem bootstrap** — the coordinator subsystem is
now committed atomically on first model add; startup teardown is offloaded
to avoid blocking the event loop.
- **MCP `asyncio.timeout` over `asyncio.wait_for`** — Python 3.11's
`wait_for` wraps the coroutine in a fresh task, breaking anyio's `aclose`
scope exit. Replaced with `async with asyncio.timeout(N)` for safe cleanup.
- **MCP pool-reuse 401 recovery** — a reused `ClientSession` returning 401
now replaces the pool entry with a fresh session; the carrier token is
owned by the pool entry to prevent a race between the 401 handler and a
concurrent request.
- **OIDC hardening** — multiple security and correctness fixes:
SSRF + plaintext credential exfil via discovery document (sec-1, sec-3);
`TURNSTONE_OIDC_REDIRECT_BASE` now required, Host-header fallback removed
(sec-2); atomic user + identity provisioning prevents orphan rows (bug-1);
callback robustness — typed exceptions, shape checks, log sanitization, JS
race (bug-46, sec-4); role-mapping concurrency serialized (bug-2, perf-1);
stranded-user self-heal on role-mapping failure (cumulative bug-1).
## [1.5.7]
### Added
- **Inline node picker** — a compact node-switcher dropdown in the console
header replaces the "← Back to console" banner, so operators can switch
between nodes without a full navigation.
### Fixed
- **Queued user messages injected mid-loop** — messages queued while a
generation was in progress were not being delivered at the correct seam and
could be dropped or reordered when the worker consumed the queue.
- **Search tool output bounded** — pathological inputs (very long lines with
no whitespace) could produce search results exceeding the context budget.
Output is now clamped before reaching the message.
## [1.5.6]
### Added
- **`api_surface` toggle** — model definitions gain an `api_surface` field
(`"chat"` | `"responses"`) that selects which OpenAI-compatible API surface
the provider client uses. Enables Mistral Medium reasoning via the Responses
surface; Chat Completions remains the default for all other models.
- **Healthy model aliases per node** — `GET /v1/api/cluster/nodes` now
includes a `healthy_aliases` list per node, so the coordinator and operators
can see which model aliases are currently reachable without a separate
per-model health probe.
- **Plan/task agent settings in Models → Roles** — the Models admin tab's
Roles sub-tab gains `plan_agent` and `task_agent` rows so operators can
configure per-kind reasoning effort and alias overrides from the UI rather
than editing `config.toml`. Live-refresh dropdowns update in place when
model definitions change.
### Fixed
- **Memory candidate selection** — recall now uses OR-of-terms BM25 with
query-aware candidate-set selection, dramatically improving recall for
queries whose terms span multiple stored entries.
- **Workstream model + config preserved on rehydrate** — reopening a closed
workstream no longer overwrites the model alias and per-workstream config
with session defaults.
- **Console home composer: attachments + user-message pills** — multipart
attachments in the home composer were not forwarded correctly; user-message
pills in the coordinator chat pane were missing.
## [1.5.5]
### Fixed
- **Saved-workstream tool result rendering** — tool results in closed
workstreams were not rendering on history replay. Audit-trail decoration for
tool calls is now applied on the replay path.
## [1.5.4]
### Added
- **Stage 3 SessionManager Children primitive lift** — child workstreams are
first-class citizens in the cluster event bus. `child_ws_state` events are
pushed through the cluster SSE stream so the console tree view updates in
real time without polling. `list_children` and `get_child` primitives on
`SessionManager` provide a consistent cross-node view of the coordinator's
spawn tree.
- **Multi-select delete for Saved Coordinators** — the Saved Coordinators grid
in the console admin panel now supports checkbox multi-select with a
bulk-delete action.
## [1.5.3]
This release introduces one forward-only schema migration:
`048_workstream_reaper_index` — partial composite index on `workstreams` for
the orphan-reaper query.
### Fixed
- **Coordinator orphan reaping scoped by heartbeat** — the session manager's
`close_idle` pass now scopes the DB-orphan reaper by
`services.last_heartbeat` so workstreams belonging to a live node are not
incorrectly reaped. `bulk_close_stale_orphans` and `touch_workstream`
storage primitives added; a partial composite index keeps the reaper scan
cheap.
- **Coordinator pool idle cleanup** — a periodic task on the console now
closes coordinator pool entries whose session has gone idle past the
configurable threshold, preventing pool exhaustion on long-running consoles.
## [1.5.2]
### Added
- **Metacognition themed reminder bubble** — repeat-tool and user-reminder
nudges are rendered as a distinct styled bubble rather than being injected
inline into the assistant message, making it easier to distinguish model
output from metacognitive annotations. The CLI REPL gains matching
`on_user_reminder` / `on_tool_reminder` callbacks.
### Fixed
- **Metacog streak detector** — the N≥3 sequential-same-call streak detector
now fires correctly on the third repetition; a write-success-clear that
reset the counter after a successful tool call (preventing streaks across
mixed-outcome sequences) was removed.
- **Metacog reminders isolated to side-channel** — reminder text no longer
appears in the user content turn; it flows through a dedicated side-channel
the session injects into the system context, preventing the model from
attributing it to the user.
## [1.5.1]
### Added
- **`pending_approval_detail` on child `ws_state` SSE events** — coordinators
now receive the child's pending approval detail in `child_ws_state` events,
enabling the coordinator to surface approval prompts without a separate poll.
### Fixed
- **Coordinator registry auto-refresh** — the console coordinator registry now
refreshes when model definitions change, so a newly added alias is visible
to coordinators without restarting.
- **Coordinator fan-out default** — coordinators now fan out to independent
child workstreams by default instead of serialising them, matching the
documented contract for parallel-work patterns.
- **`wait_for_workstream` message cap raised to 10 KiB** — large plan
summaries and tool results from child workstreams were silently truncated at
the previous 4 KiB cap.
- **Coordinator SSE isolated on dedicated thread pool** — coordinator SSE
polling now runs on a dedicated 200-thread executor, matching interactive's
`sse_executor`, so coordinator long-poll blocking no longer contends with
storage and routing workers on the default pool.
## [1.5.0]
User-visible additions: a unified workstream HTTP surface (interactive and
coordinator under one URL family), inline child approvals, coordinator
composer parity, progressive rendering, OIDC authentication, MCP OAuth
foundations, and a redesigned UI built on the Design System v1 token layer.
This release removes the pre-1.5 body-keyed and query-keyed URL family.
See **Removed (BREAKING)** below before upgrading from a 1.x stable line.
This release introduces the following forward-only schema migrations that the
server applies automatically on first startup. All are additive; no data loss.
- `039_workstream_kind``kind` + `parent_ws_id` columns on `workstreams`.
- `040_coord_cluster_admin_perms` — grants `admin.coordinator` +
`admin.cluster.inspect` to the builtin-admin role.
- `041_workstream_index_tuning` — refined indexes for the workstream query mix
introduced by 039.
- `042_coord_trust_send_perm` — adds `coordinator.trust.send` permission to
builtin-admin.
- `043_skill_description_required` — backfills empty `description` rows in
`prompt_templates`.
- `044_skill_kind` — adds `kind` classifier column to `prompt_templates`
(`interactive` / `coordinator` / `any`).
- `045_skill_risk_level_rename` — renames `prompt_templates.scan_status`
`risk_level`.
- `046_drop_hash_ring_tables` — drops the hash-ring bucket tables superseded
by rendezvous routing in 1.4.
- `047_drop_coord_spawn_quota_settings` — removes the spawn-quota settings
rows removed from the coordinator in 1.5.0a4.
### Added
- **Inline child approvals** — pending tool approvals on coordinator child
workstreams surface directly in the coordinator tree view. A risk pill shows
the judge verdict (or "pending" while the judge evaluates); Approve/Deny
buttons appear inline so operators do not need to navigate to the child's
workstream. `pending_approval_detail` is exposed on
`GET /v1/api/dashboard` and passed through the cluster live-bulk SSE payload
so all connected clients render approval prompts simultaneously. LLM judge
verdicts are cached client-side and replayed on SSE reconnect.
- **Coordinator composer parity** — the coordinator composer now supports
Stop, Send-to-queue, and Attach (file upload), matching the interactive
workstream composer feature set.
- **Per-call model and judge override on coordinator composer** — operators
can override the model alias and judge model for a single coordinator send
from the composer, without changing the node-wide or role-wide defaults. Bad
aliases return a corrective error listing available choices.
- **Coordinator status bar + richer history replay** — each coordinator
workstream gains a per-coordinator status bar showing active children, token
spend, and generation state. History replay in the coordinator panel is
extended to include tool results and thinking blocks.
- **Coordinator child error surfacing + memory tool** — child workstream
errors are surfaced as distinct error rows in the coordinator tree view
rather than disappearing silently. The coordinator gains access to a
`memory` tool (same interface as interactive) for retrieving stored facts.
- **Coordinator inline tool-batch construct** — the coordinator tool approval
UI replaces the separate approval dock with an inline batch construct that
groups all pending tool calls for a given turn into a single review card.
- **Node capability auto-detection** — nodes report kernel-level capabilities
(available memory, CPU count, accelerator presence) via
`/v1/api/node/capabilities` at startup, enabling the console to filter model
aliases offered to coordinators routing to that node.
- **Skills: paste `SKILL.md` to auto-fill the Create Skill modal** — pasting
a `SKILL.md` file's content into the modal auto-populates the name,
description, and configuration fields.
- **Progressive mermaid rendering** — Mermaid diagrams begin rendering as
soon as a complete diagram block is detected in the stream rather than
waiting for the full response; the diagram re-renders in place as the model
extends it.
- **LaTeX and MathML delimiter support** — `\(…\)` inline and `\[…\]` block
math delimiters are now recognised alongside the existing `$$` fences.
## [Unreleased]
### Removed (BREAKING — 1.5.0)
+1 -1
View File
@@ -56,4 +56,4 @@ Open an issue at https://github.com/turnstonelabs/turnstone/issues with:
## License
By contributing, you agree that your contributions will be licensed under the
project's [Apache License 2.0](LICENSE).
project's [Business Source License 1.1](LICENSE).
-13
View File
@@ -1,13 +0,0 @@
# Contributors
Turnstone is written and maintained by Patrick Buckley
([@eous](https://github.com/eous)).
The following people have contributed code to the project — thank you:
- Burhan ([@Burhan-Q](https://github.com/Burhan-Q))
- chrismuzyn ([@chrismuzyn](https://github.com/chrismuzyn))
- daoxley ([@daoxley](https://github.com/daoxley))
- Robert DeAngelis ([@OriginalOrangeXD](https://github.com/OriginalOrangeXD))
- William ([@sillyWillieBilly](https://github.com/sillyWillieBilly))
- [@pizzaandcheese](https://github.com/pizzaandcheese)
+3 -5
View File
@@ -8,7 +8,7 @@ FROM python:3.14-slim
LABEL org.opencontainers.image.title="turnstone" \
org.opencontainers.image.description="Multi-node AI orchestration platform"
COPY --from=ghcr.io/astral-sh/uv:0.11.26 /uv /usr/local/bin/uv
COPY --from=ghcr.io/astral-sh/uv:0.11.8 /uv /usr/local/bin/uv
# Remove the slim image's man page exclusion so man-db has actual content
RUN rm -f /etc/dpkg/dpkg.cfg.d/docker
@@ -17,10 +17,8 @@ RUN rm -f /etc/dpkg/dpkg.cfg.d/docker
# ripgrep is the preferred backend for the search tool — natively bounds
# per-line, per-file, and per-filesize so pathological inputs (minified
# bundles, training-data JSONL with multi-MB single records) can't OOM us.
# ffmpeg transcodes omni STT uploads (browser webm/opus) to the 16 kHz mono
# WAV the omni chat-audio lane decodes.
RUN apt-get update && apt-get upgrade -y && apt-get install -y --no-install-recommends \
libpq5 git curl jq man-db manpages procps file ripgrep ffmpeg \
libpq5 git curl jq man-db manpages procps file ripgrep \
&& rm -rf /var/lib/apt/lists/*
# Node.js LTS (for npx-based MCP servers like @modelcontextprotocol/server-github)
@@ -35,7 +33,7 @@ RUN useradd --create-home --shell /bin/bash turnstone
WORKDIR /app
# Install dependencies first (cached layer — only re-runs when deps change)
COPY pyproject.toml uv.lock README.md LICENSE NOTICE THIRD-PARTY-NOTICES ./
COPY pyproject.toml uv.lock README.md LICENSE ./
RUN uv sync --frozen --no-install-project --no-dev \
--no-compile --extra all
-207
View File
@@ -1,207 +0,0 @@
# What is a harness?
*A hypothesis — not a theorem. The honest answer is a claim about **shape**: an object you can write down that says what a harness is and, just as precisely, the guarantee it cannot carry for free.*
Most descriptions of an agent framework are a feature list. This is an attempt at a definition.
---
## The claim
*Informal.* A harness is a **stopped, deterministically-controlled Markov process on task-state, closed around a stopped autoregressive process on context-space, driven by a learned model kernel** — a deterministic controller in closed loop with a stochastic learned plant.
*In plain terms.* The **harness** is the whole governed loop: a deterministic **shell** you write — build the prompt, authorize an action, fold the response back into state — wrapped around a black-box stochastic model kernel (the **plant**, $M_W$) and the environment its actions touch, looped until it halts in $H$. The shell is deterministic, $M_W$ is not, and everything below makes that split precise.
*Formal — the objects.* A harness is a tuple $\mathcal{H} = (\mathcal{S}, \mathcal{C}, \mathcal{Y}, \mathcal{A}, \mathcal{E}, \pi, M_W, \gamma, Q_E, \rho, H, H_{\mathrm{ok}}, B)$ over **standard Borel** spaces (concretely: the *controlled* state is standard Borel by construction — token sequences, finite config maps, bounded counters and ledgers, finite tuples of real vectors — and the model/environment coordinates are inherited as such whenever they serialize to a Polish space; the assumption is roomier than it looks — even a belief-state coordinate valued in $\mathcal{P}(X)$ survives, since $\mathcal{P}(X)$ is Polish for Polish $X$ — and fails only for a genuinely non-separable coordinate, an uncountable product $\sigma$-algebra being the canonical hazard, which this construction avoids): a deterministic lowering $\pi:\mathcal{S}\to\mathcal{C}$; a stochastic model-run kernel $M_W(c, dy)$ into a readout space $\mathcal{Y}$ (which includes the parse-failure $\bot$, so $M_W$ and $\gamma$ are total over it); a deterministic **authorization gate** $\gamma:\mathcal{S}\times\mathcal{Y}\to\mathcal{A}_{\bot}$ that validates the model's parsed readout into an authorized action in $\mathcal{A}$ or rejects it as $\bot$ (parsing itself lives inside $M_W$ — realized as the readout $R$ of the specialization below); a stochastic environment/tool kernel $Q_E:\mathcal{S}\times\mathcal{A}_{\bot}\rightsquigarrow\mathcal{E}$ on the authorized action (rejection included, with $Q_E(s,\bot,\cdot)=\delta_{e_0}$ for a distinguished no-op response $e_0\in\mathcal{E}$); and a deterministic verify-and-fold-back map $\rho:\mathcal{S}\times\mathcal{Y}\times\mathcal{A}_{\bot}\times\mathcal{E}\to\mathcal{S}$.
*Terminal structure.* The terminal set is an absorbing halt set $H\subseteq\mathcal{S}$ (the daemon "ready-state" recurrence of the note below is a separate, non-absorbing object) with accepting subset $H_{\mathrm{ok}}\subseteq H$; separately, a bad set $B\subseteq\mathcal{S}$ ($B\cap H_{\mathrm{ok}}=\varnothing$) marks the unsafe states for reach-avoid, possibly entered before any halt; hitting times are $\tau_A=\inf\{n\ge 0:s_n\in A\}$, and $\tau_H$ is a stopping time for the natural filtration.
*The outer kernel.* The induced outer transition kernel, for $s\notin H$, is
$$T(s, A) = \int_{\mathcal{Y}}\!\int_{\mathcal{E}} \mathbf{1}_A\!\big(\rho(s, y, \gamma(s,y), e)\big)\; Q_E\big(s, \gamma(s,y), de\big)\; M_W(\pi(s), dy), \qquad T(s,A)=\mathbf{1}_A(s)\ \text{ for } s\in H,$$
and the harness runs $s_{n+1} \sim T(s_n)$ from an initial $s_0 \sim \mu_0$ until $\tau_H = \inf\{n : s_n \in H\}$. Because $\pi, \gamma, \rho, H$ are deterministic they contribute no integration variable of their own — they appear as measurable transformations inside the integrand (the pushforward), not literally outside it — so the controller injects no randomness, and every coin is inherited from $M_W$ and $Q_E$. (The earlier shorthand $T = \rho \circ (M_W \circ \pi, E)$ is suggestive but ill-typed — $M_W$ returns a *law*, while $\rho$ consumes a *sample* together with the prior state $s$; the integral is what the shorthand meant.)
*Fail-closed.* The gate $\gamma$ is what makes **fail-closed** a property, not just a name: model output is an *untrusted proposal*, and $\gamma(s,y)=\bot$ forces a no-op environment response ($Q_E(s,\bot,\cdot)=\delta_{e_0}$) — so a malformed or unauthorized tool call is rejected *before* it can act, not validated after its side effects have landed. Fail-closed is then the property that a rejected proposal causes *no unauthorized side effect* and lands in a **safe, non-bad** set ($\rho(s,y,\bot,e_0)\notin B$): a non-accepting terminal $H\setminus H_{\mathrm{ok}}$ in the strict case, or a safe non-terminal state when the spec retries. And $\rho$ must validate the tool *response* $e$, not only the proposal that $\gamma$ already gated: a malformed or adversarial response $e$ is caught at fold-back, not just at the gate. But response-validation has a hard limit: $\rho$ can reject a bad tool *response*, yet it cannot undo side effects an *authorized* action already caused — so $\gamma$, not $\rho$, is the last line before irreversible effects, and anything irreversible must be gated at authorization. The boundary is also only real if raw model output reaches *no* sink — tool, logger, browser, or remote call — before $\gamma$; any pre-authorization escape bypasses the gate. The user-visible final response and any logging are themselves effects, and the rule binds *model-authored* bytes: they reach a sink either as an authorized action through $\gamma$, or only after an accepted halt in $H_{\mathrm{ok}}$. Shell-*templated* text — a refusal notice, a cancellation report reading the ledger — is controller output, outside $\gamma$'s jurisdiction, and may accompany any halt (a template that *interpolates* model-authored fragments inherits the model's label — the appendix's meet rule — and those bytes are gated like any others); the invariant is that raw model text never reaches a sink ungated, not that failed runs die silent.
*The harness invariants.* These are the invariants that make $\mathcal{H}$ a *harness* and not merely a controlled Markov process with a learned kernel inside: the model sees only $\mathcal{C}$, never full $\mathcal{S}$; its outputs are proposals, not actions; a deterministic capability boundary $\gamma$ gates every side effect; and the *terminal* set $H$ splits into accepting ($H_{\mathrm{ok}}$) and non-accepting ($H\setminus H_{\mathrm{ok}}$ — safe refusals outside $B$, and wrong or bad halts possibly in $B$), while the bad set $B$ is a *separate* unsafe set — possibly absorbing, possibly entered mid-run before any halt — against which $\tau_B$ is measured for reach-avoid. Two notes keep the invariants honest. They are *signature*, not strength: a $\gamma$ that authorizes everything still satisfies the tuple, as a trivial group satisfies the group axioms — the definition admits degenerate harnesses, and fail-closed, provenance isolation, and the certificates below are properties a particular harness *earns*, not gifts of the signature. And the first invariant has a sharper, two-sided form: $\pi$ is the *only* channel from state to model — the confidentiality floor lives at what $\pi$ must never lower (credentials, other principals' data) — exactly as $\gamma$ is the only channel from model output to effect, where the injection bounds live; exfiltration is therefore cut at either chokepoint, never lowered or never emitted (the gate refusing the read whose URL is the payload is the emission-side cut). One chokepoint out of the state, one into the world; a bypass of either is the same bug with the sign flipped.
*Beyond the stationary kernel.* This displayed $T$ is the time-homogeneous, fixed-kernel case; for nonstationary or adversarial environments, replace $Q_E$ with a time-indexed kernel $Q_{E,n}$ — or an admissible family of kernels, or an adversary's policy — over which the robust certificate (the minimax form under *The limit*) quantifies. If that adversary conditions on history rather than only the current $(s, y)$, the history must itself live in $s$ — otherwise the object is a Markov *game* requiring further augmentation, not a Markov chain. And nonstationarity is not the environment's monopoly: a provider retraining or re-serving under a fixed endpoint name is a nonstationary $M_{W,n}$ — the table places model version *in* $s$ precisely so a version bump is a visible state change — and any measured surrogate (the $\delta$ of *The limit*) is calibrated against one kernel and dies with the bump; the dashboard must be keyed to the kernel it measured.
*The inner kernel.* $M_W$ is itself a stopped process, and for a decoder-only transformer it is implemented as
$$M_W(c, \cdot) = \mathrm{Law}\big(R(z_{\tau})\big), \quad z_t = (c_t, b_t, m_t), \quad v \sim K_W(c_t, \cdot), \quad K_W(c, v) = (U \circ \Phi_W \circ \mathrm{Emb})(c)[v], \quad c_{t+1} = \mathrm{suffix}_{\le L}(c_t\!\cdot\! v),\ \ b_{t+1} = b_t\!\cdot\! v,\ \ m_{t+1} = \mathsf{step}(m_t, v),\ \ \tau=\inf\{t:m_t\in\mathrm{Stop}\}.$$
with the layer stack $\Phi_W$ on the residual stream as the (loosely) "manifold" core — formally just the learned high-dimensional residual-stream transformation, with manifold-proper reserved for the frontier. The inner state $z_t=(c_t,b_t,m_t)$ separates the model-visible window $c_t$ (the $\le L$ slice that slides) from the untruncated output buffer $b_t$ (the transcript the readout actually consumes, so truncation never loses it) and the parser/stop state $m_t$ (parser state, a token counter, and a clock, so the cap and timeout are functions of it), updated $m_{t+1}=\mathsf{step}(m_t,v)$, whose stop set $\mathrm{Stop}$ — EOS emitted, max-token cap, timeout, or parse-failure $\bot$ — forces $\tau=\inf\{t:m_t\in\mathrm{Stop}\}$ finite, making $M_W$ a genuine *probability* kernel rather than a sub-probability one completed by a cemetery output. (One honesty note on the clock: a token-count cap is a deterministic function of the run, but a *wall-clock* timeout imports infrastructure noise — server load, batching, congestion — into the kernel's coin; legitimate, a kernel may carry any randomness, but it makes the displayed $M_W$ the model *plus its serving substrate*, and the determinism audit under *How this could be wrong* must hold the clock fixed along with the samples.) The readout is total, $R : \mathcal{Z} \to \mathcal{Y}$ — a parsed tool-call, answer, or transcript, returning the parse-failure $\bot\in\mathcal{Y}$ when parsing fails; crucially $R$ is a *syntactic, verified* readout (parsing and extraction), not a semantic solver, or the $L$-wall below is void — arbitrary computation could hide in $R$ off the $\le L$ window — so $M_W(c, \cdot) = R_{\sharp}\,\mathrm{Law}(z_{\tau})$, the pushforward of the stopped-state law along $R$ (equivalently $M_W(c, A_Y) = \Pr[R(z_{\tau}) \in A_Y \mid z_0 = (c,\varnothing,m_0)]$ for a measurable $A_Y\subseteq\mathcal{Y}$); the no-truncation special case takes $\mathcal{Y}=\mathcal{C}$ with $R(c,b,m)=c$ (the window is the whole transcript), reading $c_{\tau}$ directly. The $\bot$ branch is exactly what $\gamma$ rejects fail-closed. This is a **specialization, not part of the definition**: a harness wrapped around a black-box API is still a harness, and $M_W$ may be any learned kernel. Where the weights are open, the geometry of $\Phi_W$ is where the substrate's continuity lives, and several downstream claims lean on it — but the definition does not.
Two stopped processes, nested: **deterministic control over stochastic dynamics over a learned kernel.** Both loops are hitting-time processes; *some* harnesses additionally read the halt set as a fixpoint or acceptance condition — iterative refinement to self-consistency is the genuine fixpoint case, while EOS, length, and tool-call syntax are not convergence. Neither loop settles because you asked it to. (The clean inner-then-outer nesting assumes tool calls fall *between* model runs; streaming or mid-generation tool calls interleave the two loops and need a finer state machine — the nesting is then an idealization.)
## Reading it
| Symbol | Is |
|---|---|
| $\mathcal{H}$ | the harness — the whole controlled system, *not* the model |
| $s \in \mathcal{S}$ | task-state: IR / dialect stack, tool results, plan, counters, **and every mutable interface variable** (model/tool versions, permissions, retrieved context) — only Markov *after* that augmentation |
| $\mathcal{C},\ \mathcal{Y},\ \mathcal{A},\ \mathcal{E}$ | the **context / readout / action / effect spaces** — model-visible context $\mathcal{C}$, model readout $\mathcal{Y}$ (incl. the parse-failure $\bot$), authorized actions $\mathcal{A}$ (with $\mathcal{A}_\bot = \mathcal{A}\cup\{\bot\}$), and tool/environment effects $\mathcal{E}$ |
| $\pi : \mathcal{S} \to \mathcal{C}$ | **lowering** — prompt construction, dialect lowering, effective-program selection (deterministic) |
| $M_W(c, dy)$ | the **model-run kernel** (inner solver) — a stopped autoregressive process; $\Phi_W$ is the residual-stream ("manifold") core in the transformer case |
| $Q_E(s, a, de)$ | the **environment/tool kernel** on the authorized action $a\in\mathcal{A}_{\bot}$ (with $Q_E(s,\bot,\cdot)=\delta_{e_0}$, the no-op $e_0$) — tool effects, API responses, the world (possibly adversarial) |
| $\gamma,\ \rho$ | the deterministic **authorization gate** $\gamma:\mathcal{S}\times\mathcal{Y}\to\mathcal{A}_{\bot}$ (untrusted proposal → authorized action or $\bot$) and the **fail-closed verify-and-fold-back** $\rho:\mathcal{S}\times\mathcal{Y}\times\mathcal{A}_{\bot}\times\mathcal{E}\to\mathcal{S}$ |
| $H,\ \tau_H$ | the **halt set** (absorbing) and the outer **halting time** — a hitting-time process, not a single pass |
| $H_{\mathrm{ok}},\ B$ | the **accepting halts** $H_{\mathrm{ok}}\subseteq H$ (correct, successful terminals) and the **bad set** $B$ — unsafe states for reach-avoid ($B\cap H_{\mathrm{ok}}=\varnothing$), *separate* from $H$ and possibly entered mid-run before any halt |
The structural fact that earns the word *controller*: $\pi$, $\gamma$, $\rho$, and the halt test are **deterministic** (and the readout $R$ too, where the transformer specialization is in play), so $\mathcal{H}$ injects no randomness of its own. Every coin is inherited from $M_W$ and $Q_E$. This split — deterministic code around a stochastic oracle — wears two names. In control-theory terms it is **controller vs. plant**: the controller is those deterministic maps; the **plant** is the learned kernel $M_W$, *plant* in its exact sense — the element with its own dynamics you steer but do not author. In engineering terms it is **shell vs. plant**: the **shell** is the entire deterministic outer harness — the control logic *plus* the external memory and tools it administers (the files, databases, vector stores below) — of which the controller is just the control-logic slice. So *shell : plant :: the part you write : the part you don't*; $M_W$ is the only thing on the right, while the environment $Q_E$ is the world the actions meet — a disturbance into the loop, not the plant. (A reader from reinforcement learning or classical control will make the opposite assignment — environment as plant, policy as controller; the inversion is deliberate: in harness engineering the element you are trying to make behave is the model, and the world is what pushes back on the attempt.) This determinism is *conditional* — on versioned code, configuration, model endpoint, and tool interfaces, and on *single-run sequencing*: concurrent runs sharing authorization state re-open a gap the per-run object cannot see (taken up under *Gate placement* in the appendix); any retry, timeout, race, or randomized routing that escapes that conditioning must be modeled explicitly as part of $Q_E$ or the controller, not waved away. The displayed $M_W(c)$ likewise freezes endpoint, version, and sampler; a routing or config change is a state-indexed kernel $M_{\kappa(s)}$ or folds into $K_C$ — the kernel must not silently depend on config the table places in $s$. More generally, control may itself be stochastic — a controller kernel $K_C(s, dc)$ over routing, sampled retries, ensemble votes, learned routers — of which the deterministic $\pi, \gamma, \rho, H$ are the Dirac special case. That case is the one worth wanting: it localizes every coin to $M_W$ and $Q_E$ and keeps the controller/plant split clean. Where control is genuinely stochastic the split does not break, it widens — fold $K_C$ into the kernel and the certificate quantifies over its randomness too. But the guarantees do not soften uniformly, and the component-to-guarantee map is worth stating because it says exactly what may be learned without loss. A learned $\pi$ — retrieval, reranking, summarization inside the lowering — costs only *semantic adequacy*, under one factorization: $\pi$ splits into a deterministic **never-lower filter** — the redaction that keeps credentials and other principals' data out of $\mathcal{C}$ — composed with learned selection, and only the selection may soften, or the confidentiality floor of the invariants note becomes a probability. With the filter Dirac, no-unauthorized-effect is $\gamma$'s property alone, and the reach-avoid certificate survives too, so long as the provenance partition of *The limit* holds. A learned $\gamma$ or $\rho$ costs the thing itself — authorization and ledger integrity are exactly the properties that must stay Dirac, or "no unauthorized effect" and "the ledger is what happened" become probabilities. So the minimal deterministic core is $\{\gamma, \rho, H\}$ plus $\pi$'s never-lower filter: the rest of $\pi$ may soften into a kernel and the harness bends without breaking — fortunate, because every deployed $\pi$ already has learned kernels inside it.
## Why this shape
$$f(x) \;\longrightarrow\; x = f(x;\,W) \;\longrightarrow\; f(x)$$
Classical software, inverted into latent geometry, then re-wrapped in classical software. The harness **re-imposes the determinism the model dissolved**: $\pi, \gamma, \rho$, and the halt test ($H$) are ordinary designed code — a controller — whose primitive operand happens to be a stochastic oracle. That closure is why a compiler is the right mental model (staged deterministic software ports cleanly) and exactly why the analogy breaks (a compiler's primitive operation was never a coin). **The harness is the half you can reason about classically, sitting on top of the half you cannot.**
## The limit, stated honestly
**Raw halting is cheap; correct halting is not.** A **certificate** is a *witness*: a checkable object — here a Lyapunov/drift function $V \ge 0$ — that *provably* satisfies a condition entailing the guarantee, through a standard supermartingale / optional-stopping theorem (the target picks the condition: drift toward $H$ for halting, a barrier for safety, reach-avoid for success). It is not the property, only an object cheap to check and hard to produce. One word then carries two senses, and the seam between them is what this section is about: the **proven** certificate, a $V$ whose bound actually holds; and the **measured** surrogate you fall back on when the architecture exhibits none — a candidate $\hat V$ with a sampled slack $\delta$, a *calibrated risk metric, not a certificate* until that bound is proven (or held to a high-confidence worst case). The gap between the two is the whole honest-limit argument. A deterministic budget — augment $s$ with a counter $k$ decremented each outer step, halting at $k=0$ — makes $V(s)=k$ a trivial Lyapunov certificate for *halting*, so the architecture does not lack a halting guarantee by construction. What it lacks for free is a certificate of *correct, safe, successful* halting under the learned dynamics. The un-budgeted halting object is still worth stating, since it shows where even the easy guarantee comes from: a certificate would be *sufficient* for almost-sure halting with bounded expected runtime — a $V \ge 0$ with
$$\mathbb{E}[\,V(s_{n+1}) \mid s_n\,] \le V(s_n) - \varepsilon \quad\text{off the halt set}$$
bounds $\mathbb{E}[\tau_H] \le V(s_0)/\varepsilon$ under the usual integrability and optional-stopping conditions. Nothing in the harness hands you such a $V$ the way a compiler's structure does: a specific compiler analysis gets its $V$ for free where a finite-height lattice *is* a well-founded descent — termination by construction *for that analysis*, not for a whole compiler — and the harness has no analogous built-in descent for its model/environment loop.
But the relevant $V$ is not *absent* — and this is the subtlety the blunt phrasing erased. The minimal certificate exists and is **forced**: it is the expected halting time itself,
$$V^\star(s) = \mathbb{E}[\,\tau_H \mid s_0 = s\,],$$
finite wherever $H$ is reached in finite expected time — the domain $\{s : \mathbb{E}_s[\tau_H] < \infty\}$ — though note this $V^\star$ certifies *halting* (reaching the terminal set $H$ at all), not *correct* halting; the stronger object, the expected time to an accepting $H_{\mathrm{ok}} \subseteq H$, is $V^\star_{\mathrm{ok}}$, taken up at the second wall below. So the honest claim splits in two: the architecture provides no certificate *for free*, and the one that exists is — **conjecturally, not as a theorem** — a functional of $W$ and the environment that does not compress below model scale. The conjecture needs scoping, because the per-step drift splits by coordinate (made precise below) and the shell's contribution is an exact, designed descent of low description complexity *by construction* — so whatever is incompressible is not the shell's part but the **plant's**, the contribution $M_W$ supplies. And even there it is conjecture with a live counter-possibility, not foregone hardness: $V^\star$ is a *coarse* functional — one scalar, an expected hitting time, not the full output law — and coarse functionals of complicated kernels are sometimes cheap (absorbing chains with sparse transition structure have tractable expected hitting times over enormous state spaces). So the honest form is conditional: *if* the plant's contribution to the drift admits no certificate of description length materially below $|W|$, then ours is as hard as the dynamics — but that antecedent is the unproven part, and the flat phrasing of an earlier draft ("the dynamics it certifies *are* the weights") overstated it by treating a coarse hitting-time functional as if it carried the whole distribution. The compiler's certificate is structurally trivial; ours is *plausibly* as hard as the plant dynamics, though whether useful compressed certificates exist — for the coarse hitting-time functional, or for structured sub-tasks — is open. This is the quantitative form of *you can borrow how LLVM is built — not, in general, why it is correct.*
So you never compute $V^\star$. You pick a candidate $\hat V$ and **estimate its drift slack**
$$\delta = \sup_{s \notin H}\Big(\mathbb{E}[\,\hat V(s_{1}) \mid s_0 = s\,] - \hat V(s) + \varepsilon\Big).$$
The status of $\delta$ has to be stated carefully, because it is easy to oversell. If you can establish a *high-confidence upper bound* on the true worst-case slack and it is $\le 0$, optional stopping hands you a real, conservative certificate, $\mathbb{E}[\tau_H] \le \hat V(s_0)/\varepsilon$. But an *empirical* $\delta$ estimated from sampled states is **not** a certificate: a measured $\delta > 0$ may mean the candidate $\hat V$ is poor, the sampled distribution missed rare failures, the supremum was never attained in-sample, the process is non-stationary, or the state abstraction is not Markov. So $\delta$ is **the number on the dashboard** — a *calibrated risk metric*, the evaluable surrogate for a guarantee the geometry will not give you, and a genuine bound only once it is statistically controlled against rare-event and adversarial tests. A weaker result is still useful: a true bound $\delta \le \bar\delta < \varepsilon$ (rather than $\le 0$) leaves descent intact with effective slack $\varepsilon - \bar\delta$ and $\mathbb{E}_s[\tau_H] \le \hat V(s)/(\varepsilon - \bar\delta)$. And the empirical quantity is distributional, not a supremum — write $\delta_{\nu}$ for drift averaged over a sampled $\nu$, reserving $\delta_{\sup}$ for the worst-case bound; only $\delta_{\sup}$ certifies. Its empirical noise floor and residual risk are driven by the measure $\mu(D)$ of the divergent region $D=\{s:\mathbb{E}_s[\tau_H]=\infty\}$ (states from which $H$ is not reached in finite expected time, under the reference/sampling measure $\mu$), the coverage of the sampled state distribution, and the hitting-time variance $\mathrm{Var}[\tau_H]$ — properties of the trained weights, the environment, and the evaluation distribution, knowable only a posteriori.
> For an agent *meant* to run forever — a coordinator, a daemon — halting is the wrong target, and $V^\star = \infty$ is the spec, not a pathology. The same drift theory then certifies **recurrence to a ready-state** instead of absorption to a halt-set. The object changes; the missing certificate does not. Safety changes shape too: it is no longer the one-shot $\Pr_s(\tau_B=\infty)$ but a *per-cycle* hazard that compounds — if each ready-state-to-ready-state cycle touches $B$ with probability $q$, survival over $N$ cycles is $\approx (1-q)^N$, so a reassuring per-cycle $0.9999$ is $\approx 0.37$ over ten thousand cycles. The reach-avoid certificate for a daemon is therefore a bound on $q$ against the intended horizon — the safety twin of the regenerative expected time that replaces $V^\star_{\mathrm{ok}}$ for restarting specs.
And the consolation rests in part on an assumption the world violates — though less of it than it first seems. The supermartingale *bound* itself survives a nonstationary kernel, provided the conditional drift holds uniformly at every step; what genuinely needs a **time-homogeneous kernel** is $V^\star$ as a fixed function, the resolvent / fundamental-matrix identities, and the sampled-$\delta$ calibration (which assumes the very kernel it was measured on). But the environment $E$ is *part of* $T$, and the world is not stationary — worse, it can be **adversarial**, an attacker choosing the tool-output *policy* — a kernel over what tools return, not the realized draw — so as to break your descent. The drift condition then stops being a fixpoint question and becomes a **minimax** one,
$$\sup_{\alpha \in \Pi}\ \int_{\mathcal{Y}}\!\int_{\mathcal{E}} V\big(\rho(s, y, \gamma(s,y), e)\big)\, Q_E^{\alpha(s,y)}\big(s, \gamma(s,y), de\big)\; M_W(\pi(s), dy) \;\le\; V(s) - \varepsilon,$$
a descent that must hold in expectation over the model's own output $y$ *and* even when the adversary picks the worst admissible environment policy $\alpha(s,y)$ from the class $\Pi$ of policies the environment genuinely permits — every $\alpha\in\Pi$ must still respect rejection, $\gamma(s,y)=\bot \Rightarrow Q_E^{\alpha}(s,\bot,\cdot)=\delta_{e_0}$, or the adversary resurrects side effects the gate refused. Well-posedness is a frontier caveat of its own: for $\sup_{\alpha\in\Pi}$ to be *attained* rather than merely defined, $\Pi$ needs structure — measurability of $\alpha\mapsto Q_E^{\alpha}$, compactness of the per-state admissible set, or a measurable-selection theorem furnishing a worst-case $\alpha$ — and "respects rejection" is a *constraint* on $\Pi$, not that existence argument; on a general state space the sup may have no maximizer, in which case the certificate quantifies over a maximizing sequence rather than a single adversary. A $V$ that certifies halting against a benign world is defeated by an adversarial one, and the measured $\delta$ bounds only the $Q_E$ you *sampled*, never the policy an attacker will choose.
**This is the formal home of prompt injection** — not "the model did something bad," but the environment optimized to bend your dynamics. And the target is not merely non-halting: injection steers toward a **bad set** $B$ — wrong acceptance, data exfiltration, unauthorized tool use, privilege escalation, irreversible side effects — so security is a **reach-avoid** problem, not a liveness one.
Here two reliability objects must be kept apart, because under absorbing refusal every naive intermediate collapses into one of them:
$$p_{\mathrm{succ}}(s) = \Pr_s\big(\tau_{H_{\mathrm{ok}}} < \tau_F\big), \quad F = B \cup (H \setminus H_{\mathrm{ok}}), \qquad\qquad p_{\mathrm{safe}}(s) = \Pr_s\big(\tau_B = \infty\big).$$
**Success** is reaching a correct halt before *any* failure — a safe refusal counts *against* it. **Safety** is never entering the bad set at all — a safe refusal *satisfies* it. These genuinely differ on any run that avoids $B$ without reaching $H_{\mathrm{ok}}$ ($p_{\mathrm{succ}}$ scores $0$, $p_{\mathrm{safe}}$ scores $1$): safe refusals, and — absent almost-sure absorption into $H\cup B$ — safe non-halting or endless safe retry. The tempting middle form $\Pr_s(\tau_{H_{\mathrm{ok}}} < \tau_B)$ is *not* a third object, by a two-line case analysis: for it to differ from $p_{\mathrm{succ}}$, a run would need $\tau_F < \tau_{H_{\mathrm{ok}}} < \tau_B$ — a non-accepting terminal hit strictly before success, then success anyway — which forces *exiting* $H \setminus H_{\mathrm{ok}}$, impossible while $H$ is absorbing. Note what does **not** re-separate them: within-run fail-closed retries (the non-terminal fail-closed of the definition) never touch $F$ at all — the rejected proposal lands in a safe *non-terminal* state — so a refuse-retry-succeed run scores $1$ on both forms, and the coincidence survives any amount of retrying. The middle form becomes a genuine third object only when the two hitting times can genuinely part ways: under **restarting specs**, where an owner re-launches out of a refusal terminal and the absorbency of $H \setminus H_{\mathrm{ok}}$ is deliberately dropped (the regenerative reading the daemon note above already contemplates) — no bookkeeping needed, since hitting times record *visits*, not occupancy, so the relaunched run's $\tau_F$ is already finite — or under a failure set that counts refusal *events* accumulated in $s$, $F' = B \cup (H \setminus H_{\mathrm{ok}}) \cup \{\mathsf{refusals} \ge 1\}$, which separates the forms even within a single run. In the restart case a run may halt refused, restart, and still reach $H_{\mathrm{ok}}$ before $B$: the middle form credits it; $p_{\mathrm{succ}}$, measured against the refusal it passed through, does not. Safety is certified by a barrier / avoidance certificate for $B$; success needs that plus the reach part — a hitting-time drift toward $H_{\mathrm{ok}}$. Fail-closed control is the disturbance-rejection margin for both, but split by reversibility: the gate $\gamma$ caps how far an adversarial world reaches into *side effects* and widens the gap to $B$ (it is the margin for the irreversible part), while $\rho$ validates the response and folds back, rejecting bad state after the action has run — which cannot undo an authorized side effect. In this language, security is robustness of the reach-avoid certificate.
And injection is not confined to the post-model kernel $Q_E$: poisoned retrieval, prompt-injected pages, and malicious tool metadata enter through $\pi$'s *inputs*, before generation — so the adversary lives wherever untrusted content enters the state/context-construction pipeline, which is why input provenance and the gate $\gamma$ both matter, not post-hoc verification alone. And provenance is a *precondition* of the certificate, not just an entry point to police: partition $s$ into a **control-determining** part — plan, intent, what is authorized next, the coordinates $\pi$ lowers and $\gamma$ checks — and a **data** part — tool values, retrieved text, the bytes of $e$. Reach-avoid presupposes untrusted effects touch only the latter; let $\rho$ fold attacker-controlled $e$ into the control part and the structural-intent check validates against a plan the adversary already bent, collapsing $\gamma$ to the strength of $\rho$'s validation. So the claim is conditional — reach-avoid *given* control flow provenance-isolated from untrusted data, the isolation that makes provable security possible (the content of CaMeL's control/data-flow separation, untrusted data filling typed values but never the program), a structural property the harness supplies and $\rho$ cannot recover after the fact. The partition then forces a question the isolation rule alone cannot answer: *something* must be permitted to write the control-determining part mid-run — or no plan could be steered, no approval granted, no scope widened — and naming that something is part of the object. It is the **trusted principal**: the owner of the run. An approval request is an ordinary authorized action through $\gamma$ into $Q_E$ — ask-the-owner is a tool call to the one counterparty you trust — and its response is the *single* class of $e$ that $\rho$ may fold into control coordinates; every other $e$ folds into data. This is not an exception eroding the partition but the partition completed: a provenance *lattice* with exactly one writer at the top, which is what trusted means — and the appendix's gate-placement entry derives the matching rule for *learned* verdicts, which may never stand in this writer's stead. One distinction keeps the lattice from outlawing the loop it governs. Control-determining is not one rank but two: **authority** — grants, scopes, budgets, what the principal has permitted — which only the top writer widens; and the **plan**, which the model rewrites at every fold of $y$, because replanning *is* the harness. The plan is a *middle* rank: written through the gated fold of the model's own output — the channel the minimax descent above already prices — never directly by an effect, and never a source of widened authority. The rank is also the field's live design axis: pin plan-writes to the top-derived rank — the plan fixed from the trusted query before any untrusted read, which is CaMeL's move — and provable security follows exactly there; let the middle rank replan interactively and you pay the adversarial price the certificate quantifies. A corollary with teeth: a dedicated planning component is rank-neutral — its writes land in the same middle rank as the model replanning inline — so it changes no guarantee and lives or dies on measured capability alone; in general, sub-components that only write middle-rank state are priced by evals, not by the certificate, which prices only rank crossings, gates, and $\Pi$. (For $B$ to capture irreversible side effects rather than only states, the side-effect ledger must itself live in $\mathcal{S}$, and the response $e$ must be an *effect record* carrying the ledger outcome — not just API bytes — since only $\rho$ writes external effects into $s$.)
There is a **second wall, orthogonal to the first.** It binds not the full harness state $\mathcal{S}$ but the **model-visible working memory** $\mathcal{C} = \mathcal{V}^{\le L}$ — bounded by the context length $L$. That bound is *not* the incompressibility of $V^\star$ (a fact about the parameters $W$ — the **dictionary**, fixed at training); it is a fact about the inner kernel's **working memory** (the $L\times d$ residual stream — the **desk**). $\mathcal{S}$ itself may be far richer — files, databases, vector stores, durable memory, queues — but that is *external* memory the shell supplies, and the distinction is the point: every external read still passes *through* the $\le L$ window to touch computation, so external stores extend addressable storage without extending the per-pass resident set. The shell can page; the plant cannot grow its desk. (What follows is heuristic, not definition-level: the complexity claims turn on depth, precision, and architecture, and belong with the frontier, not the core.) The tape picture comes from the autoregressive structure alone and needs no complexity theorem: each step reads a bounded window and writes one token, so **the context window is the tape, the autoregressive loop is the read/write head**, and — in the variable-$L$, fixed-precision idealization — the model-mediated inner computation behaves like a linear-bounded automaton, its reachable fixpoints capped by space-$O(L)$ computability (chain-of-thought is register-spilling onto that tape). Separately, and more weakly, there is a *per-pass* expressivity bound: under the standard fixed-depth, log-precision theoretical model a single forward pass is in constant-depth $\mathsf{TC}^0$ — *suggestive* for deployed models, not literal (real models use fixed-point precision and depth that grows with scale, and log-depth variants escape parts of it). These are different resources — the first bounds the *space* the loop addresses, the second the *depth* of one step — and only the space bound carries the $L$-wall; chaining them (one pass buys bounded depth, *therefore* the loop is space-$O(L)$) would be a non-sequitur, since per-step depth says nothing about the length of the tape the loop runs on. This is a *second* obstruction beside divergence, and it concerns *success*, not raw halting. Split the terminal set: let $H$ be any halt state (including fail-closed refusal) and $H_{\mathrm{ok}} \subseteq H$ the successful, accepting halts, with $V^\star_{\mathrm{ok}}(s) = \mathbb{E}[\tau_{H_{\mathrm{ok}}} \mid s_0 = s]$ taken on the process where $H \setminus H_{\mathrm{ok}}$ — halting wrong, refusing, failing closed — is *absorbing failure*, so a run that fails closed before acceptance has infinite accepting hitting time unless the spec explicitly restarts it — hence unconditional $V^\star_{\mathrm{ok}}$ is infinite whenever pre-acceptance failure has positive probability, which is why the workable reliability object is the success probability $p_{\mathrm{succ}}$ (above) or, for restarting specs, the regenerative expected time. Then $U_{\mathcal{H}}(L)$ — harness-relative, since the shell's decompositions and verified tools determine what can be paged or outsourced — is the set of tasks whose **irreducible per-step model-mediated working set** exceeds $L$ — not tasks whose *data* exceeds $L$ (those the shell can page), and not work that can be **discharged to a verified external tool** (a solver, interpreter, or compiler computes off-context). For a task in $U_{\mathcal{H}}(L)$ the raw chain may still hit $H$ — by failing closed, refusing, or returning a wrong answer — so $V^\star = \mathbb{E}[\tau_H \mid s]$ stays perfectly well-defined; what blows up is $V^\star_{\mathrm{ok}}$, the expected time to a *correct* halt, which is infinite under a formal success predicate, or undefined if no such predicate has been specified. The honest statement is about the finite-success domain, and it is *schematic* — a shape written in set notation, not a theorem, since $\mathrm{reachable}_{\mathcal{H}}(L)$ is exactly as informal as the working-set notion behind $U_{\mathcal{H}}(L)$: $\mathrm{dom}_{<\infty}(V^\star_{\mathrm{ok}}) \subseteq \mathrm{reachable}_{\mathcal{H}}(L) \setminus D$ — both the reachable set and the divergent set $D$ relative to $\mathcal{H}$. The two walls **trade***directionally, not as a literal exchange rate*: parametric memory $|W|$ and working memory $L$ press on the same budget along the pretraining-vs-inference-scaling axis, with no clean unit-for-unit substitution of one for the other. And the bound is inherent to *finite working memory*, not attention specifically: state-space models embody it differently (a fixed-size recurrent state rather than an $L$-window), and real attention's usable tape is shorter than $L$ (lost-in-the-middle).
## Where it cashes out
This is not ornament; the decomposition is load-bearing in the design.
- **$\pi$ is a progressively-lowered dialect stack** — raw input → intent → plan → tool-call → the neutral wire IR — each level a deterministic pass with its own verifier — *pass* and *verifier* meaning the shell's transformation and checking: the **content** entering at the plan level is plant-authored, middle-rank state (the two-rank note of *The limit*), which is exactly why that level carries a verifier at all. The per-step drift $r(s)=\mathbb{E}[\hat V(s_{n+1})\mid s]-\hat V(s)$ splits by coordinate, $r = r_{\text{shell}} + r_{\text{plant}} + r_{\text{env}}$ — presuming an additively separable $\hat V$, or a declared scheme attributing each step's drift to shell, plant, and environment coordinates: the shell term is an *exact, designed* descent — but per lowering pass, not per outer step: each pass strictly narrows the admissible-meaning set, a well-founded descent we build by hand, while the outer loop *revisits* — retry, replan, rewind are planned ascents of any reasonable $\hat V$, which the run-level certificate must absorb (a retry budget inside $\hat V$ is the standard device), so the shell's descent is well-founded in the nested, lexicographic sense rather than monotone along the run; the plant term ($M_W$) is the irreducible residue, and the environment term ($Q_E$) is the one an adversary controls — the very quantity the minimax descent must bound, which the old two-way split folded out of sight. **Syntactic soundness is free; semantic adequacy is not.** Relative to a formal schema and a correct validator, schemas, types, and boundary checks go into the shell at zero probabilistic cost; whether the lowered task still *means* what the user intended stays empirical, because natural language supplies no source-language standard to check against.
- **$\rho$ is fail-closed verification** — validate at every boundary, never let malformed state flow downstream. The discipline transfers from compilers in *form*; the *teeth* do not, because a harness has no source-language standard — natural language is, in effect, all undefined behavior — there is no complete formal source-language semantics to check against. And $\rho$ must be *deterministic*: if verification is itself an LLM judge, that is another learned kernel call — it belongs in $M_W$, not in $\rho$. Where $\rho$ *repairs* rather than rejects — canonicalizing malformed input into valid shape — remember that repair is an authorization decision in disguise: each repair rule converts a reject into an accept on bytes the adversary chose, so it must be deterministic, meaning-narrowing, and its output re-validated as if it had arrived that way, or the repair pass is a bypass of the very boundary it serves.
- **$\delta$, $\mu(D)$, $\mathrm{Var}[\tau_H]$ are what you measure** — not derive. You instrument the certificate precisely because the architecture does not hand it to you — you estimate it unless it is separately certified. And the meter is attack surface: if $\hat V$ is itself computed by a learned judge — a model scoring "progress" — the instrument is a kernel draw with the plant's own adversarial exposure, and an environment optimized to bend your dynamics will bend your *measurement* of them first; an injected page persuading the judge that work is advancing is precisely a divergence hidden from the dashboard built to catch it. The rule that put the LLM judge in $M_W$, not $\rho$, applies to instrumentation too: a learned $\hat V$ is part of the measured system, never a neutral meter.
## How this could be wrong
It is a hypothesis; here is what would falsify it. If the controller cannot in practice be kept deterministic — if real reliability demands stochastic control the plant can't absorb — the clean *deterministic* split is a fiction (the broader $K_C$ kernel model still holds, but loses its payoff: localizing every coin to the plant). If the drift slack $\delta$ turns out *not* to track real-world failure, the whole "measure the certificate you can't prove" program is empty. And if harnesses are simply better described some other way — not as nested stopped chains at all — then this is a pretty equation that merely happens to fit, an elegance we would be right to distrust.
First, handles — the load-bearing claims numbered, so the tests have addresses. **C1**: the harness is faithfully modeled as nested stopped Markov processes — the tuple, the outer $T$, the inner $M_W$. **C2**: the controller injects no randomness — every coin localizes to $M_W$ and $Q_E$. **C3**: fail-closed is a *gate* property — no effect crosses unvalidated, and rejection is a true no-op. **C4**: no certificate of correct halting comes free, and the measured slack $\delta$ is a calibrated risk metric, never a certificate. **C5** (conjecture): the minimal certificate $V^\star$ admits no representation materially below model scale. **C6**: two orthogonal walls — divergence ($\mu(D)$) and the $L$-bounded per-pass working set. **C7**: security is reach-avoid, certifiable only conditional on provenance isolation with a single trusted writer. **C8** (figure): certificate and interlingua are one object — already demoted by its own section, and exempt below accordingly.
Each claim is operational, not merely rhetorical:
- **State-ablation (C1 — the Markov claim).** Drop a variable from $s$ and check whether next-step transition statistics move. If they do, the abstraction was not Markov, and $s$ must be augmented until it is. (Passing is necessary, not sufficient — the test can falsify Markovity, not establish it.) The same probe pointed at $\pi$ tests lowering *sufficiency*: drop a coordinate from $c$ rather than $s$ and watch task success rather than transition statistics — context compaction lives or dies by exactly this.
- **Controller-determinism audit (C2).** Re-run with model samples and tool outputs *held fixed*. Any residual variance is randomness the harness itself injected — clock reads are the classic leak (timestamps folded into $s$, wall-clock timeouts, cache expiries) — and must be folded into $Q_E$ or the controller, or the determinism claim is false.
- **Drift calibration (C4).** Test whether $\hat V$-drift actually predicts failure, retry count, latency, or non-halting. One uncorrelated candidate kills that candidate, not the program; the program is empty only if candidates from the natural families — plan depth, open-obligation counts, budget burn, judge scores — *systematically* fail to track failure.
- **Adversarial-environment test (C7).** Replace sampled $E$ with worst-case tool outputs, prompt-injected documents, poisoned tool metadata, malformed responses. The minimax descent must survive these, not merely the benign draw.
- **Boundary-control ablation (C3, C7).** Compare prompt-only defenses against deterministic tool-call validation, capability checks, sandboxing, and fail-closed rejection at the gate $\gamma$. The hypothesis predicts the latter class dominates; if prompt-only defenses match it, the controller/plant security story is wrong.
- **Readout-typing check (C1, C3).** Verify that $M_W$'s codomain is exactly what $\gamma$ consumes — especially under window truncation, where the final context need not hold the full transcript, so the output buffer and the gate's input must still agree.
- **Certificate-compression search (C5).** The conjecture falsifies constructively: exhibit a $\hat V$ of description length far below $|W|$ whose worst-case slack is provably $\le 0$ over a nontrivial task domain. The text concedes the live counter-possibility — coarse hitting-time functionals of complicated kernels are sometimes cheap — so C5 stands only until someone cashes it.
- **Working-set probe (C6).** Fix the shell and scale a task family's irreducible per-step working set past $L$, on tasks the shell can neither page nor discharge to a verified tool — anchoring "irreducible" in families with proven streaming or communication-complexity lower bounds, so the floor is someone else's theorem and a solved family cannot retreat to reducible-after-all. C6 predicts success collapses at the wall rather than degrading smoothly; a family solved reliably past it, without new shell decompositions, falsifies the second obstruction.
## Where this points (the frontier — least falsifiable, so flagged)
If $V^\star$ is incompressible only in *token* coordinates, the right change of coordinates might compress it — and that change of coordinates is a representation of meaning itself. Cost-to-go and representation co-determine each other: where the Koopman operator is diagonalizable — a point-spectrum idealization, since mixing dynamics carry continuous spectrum and admit no eigenbasis — the eigenbasis that linearizes the dynamics is also the one in which the certificate decomposes, and even then only for a $V$ in the span of those eigenfunctions; in reinforcement learning the discounted successor representation (Dayan 1993) is the resolvent $(I-\beta P)^{-1}$ — discount $\beta$, not the gate $\gamma$ — with $V$ a *linear readout* of it — and in the undiscounted, absorbing case that actually matches a stopped harness the same role is played, in the finite setting — and countable settings where the Neumann series converges — by the **fundamental matrix** $N = \sum_{n \ge 0} Q_{\mathrm{tr}}^{\,n}$ (written $(I - Q_{\mathrm{tr}})^{-1}$ when the inverse exists), where $Q_{\mathrm{tr}}$ is the sub-stochastic kernel restricted to $H^c$ (transitions before absorption at $H$) and the row sums $N\mathbf{1}$ *are* $V^\star$ on the finite-mean hitting domain; on general state spaces the same series is read as the potential (Green) operator $G$, with $G\mathbf{1} = V^\star$ wherever it converges. Each of these is a clean identity only for a fixed, time-homogeneous kernel — under a nonstationary $Q_{E,n}$ the resolvent and fundamental matrix dissolve into a time-ordered product, and under an *adaptive* adversary into a controlled / game-value operator, so what is identity in the stationary regime is analogy beyond it.
With that caveat, **the interlingua and the certificate are one object seen twice** — and the reason neither can be written in closed form is the same "all undefined behavior": no canonical lowering of meaning, hence no finite header-file for either. The only representation of both is $W$ — a band-limited, lossy compression of a scale-free meaning-space, sharp where the record is thick and blurred where it thinned. That a finite object renders an infinite one *lossily but honestly* — declaring its resolution, and where it is unsure — is not a lie; it is the most an $f(\cdot\,;W)$ can do. **The search for $V$ and the search for the interlingua are not two programs. They are one** — and the day either is written in closed form, so is the other, or we will have proven why neither can be. Read this as *figure*, not a lurking theorem: the only precise version would need the Koopman eigenbasis to fall on the very coordinates that lower meaning, and the mixing-spectrum caveat above already concedes that eigenbasis does not exist — which guts it. It is the least-defensible claim in this document, and it should announce that rather than imply a rigor it has not got.
---
*The formula is the architecture; the corollary is why the architecture is hard. Both on the page — nothing hidden behind a tidy composition.*
## Grounding
Borrowed theorems are real; the framings are not — keep them separate. Some framings are nonetheless *corroborated* — independently reached from another field — a third grade, weaker than proof and noted last.
**Proven (citable).** FosterLyapunov drift ⇒ positive recurrence + $\mathbb{E}[\tau]\le V(s_0)/\varepsilon$ (Foster 1953; Meyn & Tweedie, *Markov Chains and Stochastic Stability*, 1993) — positive recurrence needs the usual irreducibility/petite-set hypotheses, while the absorbing-halt case used here needs only the weaker supermartingale optional-stopping hitting-time bound. The minimal $V$ is the expected hitting time, by first-step analysis + optional stopping (Norris, *Markov Chains*, 1997). For an absorbing chain that expected hitting time is the row sum of the fundamental matrix $N=\sum_{n\ge0}Q_{\mathrm{tr}}^{\,n}$ (Kemeny & Snell, *Finite Markov Chains*, 1960), with the general-state analogue the potential (Green) operator (Revuz, *Markov Chains*, 1984). Koopman's linear-operator view of nonlinear dynamics is classical (Koopman 1931), and Lyapunov functions can be assembled from its eigenfunctions when the spectrum is suitable (Mauroy & Mezić, 2016). You certify a candidate $\hat V$ by a *proven* drift inequality rather than by deriving $V^\star$, and estimate it empirically only where a proof is out of reach — the empirical drift checks, it does not certify (neural-Lyapunov: Chang, Roohi & Gao, *Neural Lyapunov Control*, NeurIPS 2019, arXiv:2005.00611). A classical monotone data-flow analysis gets its $V$ for free because a finite-height lattice is a well-founded descent (Kildall, POPL 1973). The gate-a-plant architecture itself is classical: supervisory control theory synthesizes a deterministic supervisor that disables controllable events of a plant it does not author, with the supremal controllable sublanguage as the largest admissible behavior (Ramadge & Wonham, SIAM J. Control and Optimization, 1987) — $\gamma$ is that supervisor, with a learned stochastic plant on general state spaces. The successor representation is Dayan (*Improving Generalization for Temporal Difference Learning: The Successor Representation*, Neural Computation 1993). Dialect-stack architecture: MLIR (Lattner et al., CGO 2021, arXiv:2002.11054); learned pass-ordering: MLGO (Trofin et al., arXiv:2101.04808). Single-pass low-depth expressivity: log-precision transformers are simulable by constant-depth logspace-uniform threshold circuits ($\mathsf{TC}^0$) (Merrill & Sabharwal, *The Parallelism Tradeoff: Limitations of Log-Precision Transformers*, TACL 2023) — fixed/constant precision is a stronger restriction, added autoregressive steps escape it (Merrill & Sabharwal, *The Expressive Power of Transformers with Chain of Thought*, ICLR 2024), and growing precision changes the picture, so the bound is suggestive for deployed models, not literal.
**Asserted (ours — not theorems).** That the harness is best modeled as nested stopped chains; that $V^\star$ is incompressible (no compression theorem); that "no lattice for $f(\cdot\,;W)$" means none is *known*, not that none exists; and everything under *Where this points* — including the Koopman/certificate co-determination, which is well-posed only under the spectral assumptions noted there, and the interlingua/certificate identification; and the design rules read off the objects rather than proven from them — the single-trusted-writer completion of the provenance partition, the narrow-only rule for learned checks, the composition law of the appendix. These organize the design; they are not results.
**Converged-upon (independently arrived at, from other framings).** The *Asserted* claims above are ours but not ours alone; several are reached independently, from starting points unconnected to this framing — which is the corroboration a definition earns: not a chorus of agreement (the systems below often disagree on method and goal), but that work approaching from capabilities, reinforcement learning, control theory, software architecture, and language-modeling theory each lands on a piece of the same object. That the **deterministic controller, not the model, carries the guarantee** is reached from four directions — capability and information-flow control (CaMeL: Debenedetti et al., *Defeating Prompt Injections by Design*, arXiv:2503.18813, securing the agent even when the underlying model is susceptible); reinforcement learning (shielding: Alshiekh et al., *Safe Reinforcement Learning via Shielding*, AAAI 2018, arXiv:1708.08611 — a deterministic reactive shield filtering a learned policy's actions against a temporal-logic specification); control theory (*Stable Agentic Control*, arXiv:2605.03034, enforcing finite action catalogs at the tool-output interface under a Lyapunov input-to-state-stability certificate against adversarial disturbance); and software architecture (the plan-then-execute / control-flow-integrity line, e.g. Beurer-Kellner et al., *Design Patterns for Securing LLM Agents against Prompt Injections*, arXiv:2506.08837). The **certified-vs-measured split** is reached from the construction side (CaMeL's provable security) and, independently, from the destruction side (guardrail-evasion results — *Bypassing Prompt Injection and Jailbreak Detection in LLM Guardrails*, arXiv:2504.11168, the v1 title — later versions retitle it; *No Free Lunch with Guardrails*, arXiv:2504.00441), with verification-oriented work stating it as the motivating gap (*Towards Verifiably Safe Tool Use for LLM Agents*, arXiv:2601.08012; VeriGuard, arXiv:2510.05156): a learned safeguard raises the odds of detection but cannot guarantee safety against a persistent attacker. The **inner readout as a composition of Markov kernels** is independently formalized in language-modeling theory — the autoregressive step as kernel composition in the category $\mathsf{Stoch}$ (*A Markov Categorical Framework for Language Modeling*, arXiv:2507.19247), and the broader "LLMs as Markov chains" line — though that work models the inner kernel alone and never closes it into an agentic loop, which is exactly the seam this definition adds. That **provenance shrinks the admissible adversary** is reached by datamarking / spotlighting (Hines et al., arXiv:2403.14720, 2024) and by CaMeL's data/control-flow separation; and a systematization of prompt injection against agentic coding assistants reaches the same verdict from the attack side — mitigation must be *architectural*, not model-level (*Prompt Injection Attacks on Agentic Coding Assistants*, arXiv:2601.17548); the sharper open problem this object is built to answer — formally specify the trust boundaries, then verify implementations respect them — is our phrasing of where that verdict points, not the paper's. Two convergences are weaker, and flagged. The **reach-avoid hitting-time certificate** is the independently developed reach-avoid supermartingale (RASM, arXiv:2210.05308, AAAI 2023) and stochastic Lyapunovbarrier apparatus, and its *hardness* is corroborated — expected-stopping-time problems for Markov chains are inter-reducible with the Positivity problem, a relative of the Skolem problem (Chatterjee & Doyen, *Stochastic Processes with Expected Stopping Time*, arXiv:2104.07278) — but this supports generic hardness only, not the specific incompressibility-at-$|W|$ conjecture, which remains ours and unproven. And **injection as an adversarial policy** is corroborated as a minimax game in the *detection* setting (DataSentinel: Liu et al., *A Game-Theoretic Detection of Prompt Injection Attacks*, arXiv:2504.11358) and as adversarial-disturbance robustness (*Stable Agentic Control*, above) — but no prior work assembles it as reach-avoid over the tool-output kernel with the gate as the irreversibility margin; here the relation is adjacency, not convergence.
---
## Appendix: model implementation
The definition is deliberately abstract: $\pi, \gamma, Q_E, \rho$ are *roles*, not code, and a deployed harness forces concerns the abstract object is silent on. This appendix does not re-derive the implementation; it establishes a **pattern** — take a hard practical concern, locate it in the objects already defined, and read off the discipline they imply rather than inventing new machinery. Cancellation is the worked example, chosen because it is where the silence bites hardest and because the answer falls entirely out of objects already on the page.
**Cancellation.** An owner stops a running agent mid-flight — worst across a task-agent tree. The naive reading is "stop and undo," but the irreversibility point forbids it: $\gamma$ is the last line before irreversible effects, and $\rho$ can reject a response but cannot undo an authorized action. So cancellation is not *making it not have happened*; it is a disciplined stop with a defined disposition for what is already irreversible.
A cancel is a signal, so by the Markov requirement it lives in $s$. The gate then closes on it: while the cancel flag is live, $\gamma(s,y)=\bot$ for every proposal. That is the entire "block the pending actions" requirement — they hit the gate already built and bounce into the no-op, with no new blocking machinery — and it forecloses all *future* turns at once, since $\pi$ lowers nothing new that $\gamma$ will pass. After the signal is observed, **no action crosses $\gamma$.**
The hard half is the action already *past* $\gamma$, executing in $Q_E$, whose effect is landing or has landed. Here the disposition is a trinary on the kind of $Q_E$ you authorized. If the tool is **cancellable**, propagate the cancel into it; it aborts and reports a true end-state (committed, rolled-back, or partial), and $\rho$ folds the real disposition. If it is **bounded** — drainable in acceptable time — simply wait and record the real $e$. If it is **opaque and unbounded** — a bash invocation that may itself be a harness, an environment you hold no handle into — you cannot stop the effect, only your *wait* for it: the controller fabricates $e$, a synthetic "cancelled" response, and folds it through $\rho$ so the loop can reach a terminal.
That synthetic result is the subtle case, and the load-bearing rule is this: $\rho$ may fabricate the *acknowledgment* but must not fabricate the *outcome*. A synthetic "cancelled, no effect" entry reads downstream as *the action did not happen* — and will cause a double-send exactly as readily as a dropped record causes an orphan. Same bug, opposite sign. An outcome you did not observe is $\mathsf{unknown}$, never $\mathsf{none}$: the cancelled agent never saw whether bash sent the email, and the ledger must say exactly that. (This is why $e$ must be an effect record and the ledger must live in $s$ — the fabricated entry is still a ledger write, and its value is what a later reader acts on.)
The run halts into $H_{\mathrm{cancel}} \subseteq H \setminus H_{\mathrm{ok}}$ — a distinguished terminal, non-accepting but *safe* (outside $B$), refining the deliberately coarse $H \setminus H_{\mathrm{ok}}$ of the definition (the body leaves that set unenumerated; the appendix is where its subclasses earn names) — with a specific postcondition: no action crossed $\gamma$ after the cancel was observed, every in-flight action was drained to its real disposition or recorded $\mathsf{unknown}$, and the ledger is consistent. It is worth separating from refusal and from a wrong answer precisely because that guarantee is its own.
Cancellation must be **cooperative, not preemptive.** The owner writes the cancel into the child's $s$; the child observes it at its next $\gamma$ check. The guarantee is therefore "no new action after the cancel is *observed*," not "after it is *sent*" — a child may authorize one more action in the gap, which simply drains like any other in-flight. Preemptive cancellation — killing the child mid-$Q_E$ — is exactly what manufactures $\mathsf{unknown}$ state at scale, because it destroys the record of whether the action landed. And the propagation is **recursive**: cancel flows down the subtree, each level closes its gate at its next check and drains, and the owner's cancel "completes" only when the subtree has drained. A single agent's drain is its own in-flight action; a tree's is the whole subtree reaching safe points cooperatively — the irreversibility problem stacked on a distributed-coordination one, which is why task agents are the worst case.
Compensation lives **outside** the cancelled agent. A completed-but-unwanted effect cannot be undone by the agent that caused it — its gate is closed — so a compensating, saga-style action is the *owner's* job, issued after $H_{\mathrm{cancel}}$ and reading the child's ledger to decide what to reverse or annotate. It must be the owner's, because the cancelled child cannot even know whether compensation is needed: it never observed the outcome. The owner inherits the $\mathsf{unknown}$ and any still-live orphan process, and reconciliation is its responsibility.
Finally, the part that shapes the tool rather than the document. Opaque unbounded $Q_E$ is uncancellable because authorization happened at the wrong **granularity** — an unbounded environment crossed $\gamma$ on a single approval. The discipline the objects imply is therefore not "handle uncancellable tools better" but: *the gate should prefer bounded, instrumented $Q_E$ over opaque ones, so that cancellation and the ledger stay honest.* A bash invocation behind a wrapper that tracks its process tree and effects converts the third branch into the first. Sometimes opaque is the only option, and then $\mathsf{unknown}$ and owner-inherited orphans are the honest floor — but where the choice exists, that is the pressure cancellation semantics put on tooling.
**Resume (involuntary stop).** Cancellation's twin, without the courtesy of a signal: a process crash, a lost node, a partition mid-$Q_E$. Nothing new is needed to say what recovery *is*. A crash is not a halt — $H$ is a property of the state, and the run never reached it; the chain merely stopped being *computed*, and resume computes it further, re-entering $T$ at the last durable $s$ (not the body's *restarting spec*, which exits a refusal terminal — here no terminal was ever reached). That sentence is the Markov requirement cashing out operationally: re-entry is sound exactly when $s$ was the whole state, so anything load-bearing that lived only in process memory — an in-flight buffer, a lock held in RAM, a plan revision not yet folded — is a state-ablation failure (*How this could be wrong*) discovered at the worst possible time. Durability of $s$ is not an implementation nicety; it is what the Markov claim *means* when the machine dies.
The sharp part is an ordering the ledger's own trichotomy forces. The formal transition is atomic — $s_{n+1} = \rho(s, y, a, e)$ in one piece — and a crash lands *inside* it, so resume is really a statement about the implementation's refinement of that atom into micro-steps: authorize, journal, dispatch, collect, fold. The discipline is that every crash point must resume to one of exactly two honest readings — not-yet-dispatched ($\mathsf{none}$, safely retriable) or dispatched-unconfirmed ($\mathsf{unknown}$, the cancellation entry's third branch) — and **journal-before-dispatch** is what makes the boundary between them observable: on $\gamma$'s authorization the shell journals an open $(\mathsf{action\_id}, \mathsf{pending})$ entry into durable $s$ before $Q_E$ sees the action — the write is the shell's step bookkeeping, so $\gamma$ itself stays effect-free. Journal *after* dispatch and a crash in the gap leaves no record at all — resume reads silence as $\mathsf{none}$ and re-sends, the double-send bug again, produced by a power cut instead of a synthetic entry. Write-ahead intent is not imported from database lore; it is forced by "did not confirm" is not "did not happen."
The same pressure lands on tooling from a second direction. The $\mathsf{action\_id}$ the record already carries is an idempotency key wherever the tool will accept one: re-dispatch after resume becomes safe, and $\mathsf{unknown}$ becomes *queryable* — ask the tool what it did with this key — rather than terminal. The disposition trinary returns with new labels: idempotent-or-queryable $Q_E$ resumes cleanly, bounded $Q_E$ drains, opaque $Q_E$ leaves $\mathsf{unknown}$ and owner-inherited orphans, the honest floor again. The wrapper that made bash cancellable makes it resumable; it was the same wrapper all along. And if durable $s$ itself is lost there is nothing to re-enter: the run collapses to a single $\mathsf{unknown}$ in its owner's ledger — degraded accounting, but never silent.
**Gate placement (fail-closed, in practice).** The natural implementation question is whether fail-closed means tool-call parsing and validation must happen before any tool invocation. It does — with the division of labor the definition already fixed: *parsing* lives in the inner readout $R$, the syntactic, verified extraction into $\mathcal{Y}$ (what the readout-typing falsifier checks), and *authorization* lives in $\gamma$, which is a *gate* — validation is not merely *prior to* invocation, it is what *authorizes* it. The model emits text; $R$ has already extracted it into a typed proposal; $\gamma$ validates that proposal against $s$, and only a survivor becomes an authorized action that $Q_E$ may execute. The teeth are in $\gamma$ being the *sole* route from model text to execution: no path to a side effect that does not pass the gate. And the validation is not a fixed checklist but **any deterministic predicate over $s$ and $y$** — that domain is the point, since the gate sees all of the state and the full proposal, so anything computable from them is a legitimate authorization condition. Three kinds matter. *Syntactic* — well-formed, schema-conformant, the tool exists, arguments typed. *User authorization* — does the principal this run acts for hold the right to *this* operation on *this* resource in *this* context: a function of the auth scope, principal, and session carried in $s$ and the resource and operation named in $y$, and *dynamic* rather than a static capability table, since the same caller may be permitted now and not once a budget is spent or a lock held. *Structural intent* — does the call cohere with the plan and the lowered task already in $s$: a consistency check, not a mind-reading one.
That last kind marks the seam where the gate stops being able to stay pure, and it is the same seam the rest of this document is built around. The *structural* slice of intent — does the action cohere with the plan in $s$ — is a deterministic predicate over $s$ and $y$, effect-free, and belongs in $\gamma$ without reservation. But whether an action matches what the user *actually meant*, in the full semantic sense, is exactly the thing the definition says cannot be checked: natural language is all undefined behavior, with no source-language standard to validate against. So a semantic intent check is a *learned* check, and an LLM judging "is this what they wanted" is a **stochastic kernel** — putting it inside $\gamma$ breaks the property the gate exists to hold, by the same move flagged for the fold-back verifier: a learned judge is a kernel, and belongs in $M_W$, not in a deterministic map. Semantic intent therefore does not live *in* the gate; it is a plant call — a separate authorize-the-proposal pass through $M_W$ whose output $\gamma$ then deterministically gates — or it is drift you measure, never a guarantee you hold. That nested call is not a new kind of thing: it is a mini-harness inside the gate's decision — a judge $M_W$, its own syntactic readout, its own deterministic gate — so its failure case answers itself, the inner gate fail-closing on an unparseable or low-confidence judgment exactly as the outer one does, because it *is* one. The object is **closed under this construction**: semantic gating is added by recursion, not by a new primitive. One constraint on the recursion is load-bearing enough to be a rule, because it is where this entry meets the provenance partition of the body: the judge's verdict is derived, through a learned kernel, from the very content an adversary may have bent, so folding it into authorization is exactly the fold the partition forbids — *unless the verdict can only cost capability*. **A learned check may narrow the deterministic admissible set; it must never widen it.** Judge-as-veto is safe by construction: attacker influence over the judge can at worst manufacture a denial, a liveness cost the certificate already prices. Judge-as-approver — a verdict granting what the deterministic checks alone would refuse, or standing in for the trusted principal's confirmation — lowers the certified floor to those deterministic checks alone; if avoiding $B$ depended on the deny the judge now withholds on the adversary's behalf, the certificate is gone. Only the trusted principal widens authorization; learned kernels only narrow it. (The recursion already obeys this: the mini-harness's inner gate fail-closes to $\bot$ — a deny — which is why the construction was safe to add at all.) The cost is real and worth stating — a judge pass is another full model call, with its latency and tokens — so it is a decision about *which* actions warrant it, not a free wrapper for all of them. The gate widens to every deterministic predicate over $s$ and $y$; it does not widen to the one predicate the document says is not deterministically checkable.
But "before any invocation" has to be read as *before any effect*, which is sharper than it sounds — and the reason is the irreversibility point above: you validate before execution because execution is what you cannot take back, so the real invariant is **no effect crosses $\gamma$ unvalidated**. That catches three cases the naive reading misses. *Reads are not free*: a read-only call is still an injection vector (it pulls attacker-controlled content into context) or an exfiltration vector (a request whose URL is the payload), so the gate authorizes the *call* regardless of whether it mutates. *Validation must not act*: a "validator" that resolves a call by hitting an API, expanding a template that fires a webhook, or evaluating an argument that runs code has collapsed validation into invocation, and the effect has already happened *inside* $\gamma$ — so $\gamma$ itself must be **effect-free**, pure and total over the proposal and the current $s$, with no network and no execution; if deciding validity *requires* a side effect, that side effect is itself an action and must go through the gate, recursively. *The output is an action too*: the user-visible response and any logging are effects — for model-authored text, emitted either as an authorized action through $\gamma$ or only after an accepted halt (shell-templated status on any halt is the controller speaking, not the model) — streaming raw tokens to a sink before $\gamma$ has cleared them is the same bug from the other end.
So the property, tightest: $\gamma$ is a **pure, effect-free authorization that every model-proposed action — tool call, read, write, or final output — must pass before any effect occurs**, with "before" enforced structurally by the gate being the only route from model text to $Q_E$. The two failure modes to design against are a path from model output to a sink that bypasses the gate, and a $\gamma$ that is not effect-free, so that "validating" a call already rang the bell. And the boundary, so the property does not overpromise: $\gamma$ guarantees *no unauthorized effect* — pure code ordering, fully in your control — but not that an *authorized* effect is safe or correct; that is the plant's problem, and the reason $\rho$ and the reach-avoid certificate exist. Fail-closed is the floor — nothing executes that did not pass the gate — not the ceiling.
There is a third failure mode beside those two, and it is not a code path but a credential. A tool process that holds standing authority — an environment full of long-lived secrets, a database connection with every grant, an agent identity the network trusts — does not need the model's proposal to act, and against it $\gamma$'s $\bot$ is a decision with nothing to enforce it. The gate *decides*; something must make the decision *binding*, and "no path from model output to a sink that bypasses the gate" must be read to include the non-code paths: ambient authority is a bypass provisioned before the run began. The discipline is **per-action capability**: the authorized action *carries* its grant — a scoped, short-lived credential minted at authorization, valid for this $\mathsf{action\_id}$, this resource, this operation — so that a tool holds, at any moment, exactly the authority of the actions the gate has passed it and nothing standing. In the language of the minimax certificate this is enforcement as $\Pi$-shaping: sandboxing, capability scoping, and network policy do not make the gate smarter — they shrink the class $\Pi$ of environment policies an adversary can choose from, so the worst case the certificate must survive gets structurally smaller. A gate in front of an omnipotent tool is a suggestion; the objects compose into a guarantee only when $Q_E$'s reachable effects are no larger than what crossed $\gamma$.
And one more boundary, because "fully in your control" above is a *single-run* statement. $\gamma$ authorizes against the $s$ it read; the effect lands later, against a world that may have moved — the gate cannot freeze the world between authorization and commit, so the honest property is *no effect unauthorized relative to the $s$ at authorization time*, and closing that gap requires the tool itself to bind check to commit (compare-and-swap in $Q_E$), which relocates part of the enforcement past the gate and weakens "$\gamma$ is the last line" to "$\gamma$ plus a commit guard" for exactly the effects that need it. The same seam opens *between* runs: the dynamic authorization state the gate reads — budgets, quotas, locks — is, once shared, no single run's coordinate, and two children of a coordinator can each pass $\gamma$ against snapshots that jointly overdraw a budget neither exceeded alone. The cancellation entry's observed-not-sent gap ("a child may authorize one more action in the gap") is this phenomenon wearing one hat; the general statement is that cross-run authorization state needs its own serialization discipline — the ledger as the serialization point is the natural choice — and the per-run certificate is silent about it. TOCTOU is not a counterexample to the formalism; it is what the formalism says when you admit $s$ is a *view*.
**Parallel proposals (the batch gate).** Models emit several tool calls in one turn, and the outer chain assumed one action per step. The repair is formally cheap: a batch is a single action in $\mathcal{A}$ that happens to be a set, $Q_E$ runs its elements concurrently, the interleaving's nondeterminism folds into $Q_E$ exactly as the determinism audit requires, and $\rho$ folds one effect record per element — $e$ is then a finite set of records — each keyed by its own $\mathsf{action\_id}$ — the record interface already supports partial outcomes (one element $\mathsf{committed}$, its sibling $\mathsf{unknown}$). One discipline survives the cheapness: **individually admissible actions can be jointly inadmissible.** Read-the-secret and post-to-the-web each pass a per-call check; the pair is an exfiltration channel — and two calls that each fit a budget jointly overdraw it, the cross-run overdraw of the previous entry reappearing *inside* one turn whenever elements are authorized independently. Since $\gamma$'s domain is any deterministic predicate over $s$ and $y$, joint authorization was licensed all along; the content here is only that the gate must take it — authorize the *set*, atomically, against one snapshot, with interaction predicates (source-to-sink flow between capability classes, summed resources) and not merely element predicates. The cost note is the judge's, transposed: full powerset reasoning is combinatorial, so a real gate checks declared interactions rather than every subset — a tractability trade to make explicitly, not by forgetting the batch was a set.
**Effect records (what $\rho$ folds back).** The fold-back $\rho$ and the cancellation ledger both turn on the response $e$ being an *effect record* rather than raw API bytes — said twice in the body and pinned down nowhere, though it is the interface that makes both tractable. The minimal shape is small: roughly
$$e = (\mathsf{tool\_id},\ \mathsf{action\_id},\ \mathsf{status},\ \mathsf{effects},\ \mathsf{time}), \quad \mathsf{status}\in\{\mathsf{committed},\mathsf{rolled\_back},\mathsf{partial},\mathsf{none},\mathsf{unknown}\}, \quad \mathsf{effects}=[(\mathsf{resource},\mathsf{op},\mathsf{reversible})].$$
Each field is forced by something the body already needs. The $\mathsf{action\_id}$ lets $\rho$ match a response to the in-flight action $\gamma$ authorized, and lets the ledger say which actions are still open — without it the $\mathsf{unknown}$/orphan accounting has nothing to key on. The $\mathsf{status}$ must carry $\mathsf{unknown}$ as a value *distinct* from $\mathsf{committed}$ and from $\mathsf{none}$, because that distinction is the whole content of the cancellation ledger: "did not confirm" is not "did not happen" ($\mathsf{none}$ is *never launched* — the record of the distinguished no-op $e_0$ a $\gamma$-rejection forces, which is how a bounce at the gate enters the ledger at all — distinct in turn from $\mathsf{rolled\_back}$, which launched and was undone: conflating those erases the difference between a gate that held and a compensation that worked). The $\mathsf{reversible}$ bit on each effect is what lets the gate know which effects are irreversible — the predicate the gate-placement entry leans on ("anything irreversible must be gated at authorization") but cannot evaluate unless the record carries it (a bit is the minimal honest form, not the final one: real effects are reversible *until* — an unsend window, a force-push until someone fetched, a row until the backup rotates — so the mark wants to be a $(\mathsf{reversible\_until}, \mathsf{cost})$ pair, a refinement the open-interface caveat below already licenses). And $\rho$ writes the record into $s$ (the ledger lives in the state), which is what lets the next step's $\gamma$, and any owner-side compensation, read it at all. The exact fields are an **open interface, not a result**: bash, HTTP, a filesystem, and a database expose effects at wildly different granularity, and a record uniform across them is a real design problem this document does not resolve — it fixes only what the record must *support* (match by $\mathsf{action\_id}$, the $\mathsf{committed}$/$\mathsf{none}$/$\mathsf{unknown}$ trichotomy, and a reversibility mark), since without those three $\rho$ and the cancellation semantics lose their grip.
**Derived and durable state (compaction and memory).** Two mechanisms let data re-enter the context long after it arrived: compaction, which replaces transcript with a summary when the conversation outgrows what $\pi$ can lower, and memory, which persists records across sessions. Both are transformations of state that produce state, and both therefore raise a question the body's partition answers only if one more closure property is stated: **provenance is a property of the information, not of its position in the pipeline — a transformation's output inherits the meet, in the trusted-writer lattice, of its inputs' labels.** Without that closure, compaction is a laundering channel: a summary of a session that contained an injected page can assert "the user asked to export the database," and the structural-intent check then validates future proposals against a plan the adversary bent — not through $\gamma$, not through $\rho$'s fold of a single $e$, but through the summarizer, which is a learned kernel (it lives in $M_W$, by the standing rule) and so cannot be trusted to preserve a partition it does not know exists. The discipline: summaries of data are data; the control-determining coordinates — plan, grants, what is authorized next — cross a compaction *verbatim* (copied, not paraphrased) or by re-confirmation from the trusted principal — never through the *summarizer*; the model rewrites the plan at plan steps, through the gated fold the body prices, and compaction is not one of them. Memory obeys the same closure twice, at write and at retrieval: the label rides the stored record across sessions, or a poisoned memory is an injection with an arbitrarily long fuse — and retrieval, being learned ($\pi$'s selection factor — adequacy-only behind the never-lower filter), decides what comes back but never what it is trusted *as*. The same test applies at birth: tool catalogs and server-supplied tool descriptions are third-party durable data that arrive dressed as instructions, and the lattice files them on the data side of $s_0$.
One more read-off, this time from irreversibility. *Destructive* compaction — dropping the original transcript once the summary is written — is a side effect against your own state that no later step can undo, and the gate-placement rule ("anything irreversible must be gated at authorization") does not exempt self-directed effects. The granularity preference then says what it said about bash: prefer the instrumented form — originals kept content-addressed, the summary an index and a cache rather than an authority, re-derivable when the $\pi$-sufficiency probe (*How this could be wrong*) says the summary dropped what mattered. A summary you can audit against its source is a lowering; a summary that replaced its source is a fait accompli.
**Composition (harness trees).** The cancellation entry already walked a tree — cancel flowing down, drains flowing up — and "a bash invocation that may itself be a harness" has hovered since the disposition trinary; what is missing is only the statement that makes both ordinary. From the parent's seat, a child harness *is* a $Q_E$ component: spawning it is an action authorized by $\gamma$ like any other, and the entire child run — its own $\pi, \gamma, \rho$, its own coins, its own halt — is one environment draw whose response $e$ is the child's terminal ledger. The law is four correspondences. The child's halting time is the parent's per-step *cost*: a parent certificate consumes a bound on $\mathbb{E}[\tau_H^{\mathrm{child}}]$ — the budget handed down at spawn, which the child's own budget-counter certificate discharges — or the parent's drift is uncontrolled however good its own $\hat V$. The child's ledger is the parent's *effect record*: the child's $e$ carries the $\mathsf{committed}/\mathsf{none}/\mathsf{unknown}$ accounting upward — which is what already let the cancellation entry make compensation the owner's job; the interface was this all along. And the child's non-accepting halts are the parent's *partial failures*: a refused child folds back as a response the parent routes around, not an exception that unwinds it. And the child's admissible effects are the parent's *$\Pi$-restriction*: the spawn grant bounds what the child can reach — the ledger reports what *happened*, the grant bounds what *could* — which is how safety composes without the parent ever reading the child's gate; the attenuation below is this correspondence stated as a rule. Read this way, the gate-granularity discipline and the tree are one preference: an instrumented child — budgeted, ledgered, cancellable — *is* the bounded, cancellable $Q_E$ the trinary prefers, and an opaque bash invocation is an un-annotated child you declined to instrument. Nesting adds no primitive on the environment side either: the parent never sees the child's gate and does not need to — it gates the spawn, prices the budget, folds the ledger, and the child's internal guarantees surface only as the shape of $e$. Nothing fixes one level: the tree recurses, budgets subdivide, ledgers concatenate upward, and the cooperative drain of cancellation is this law read under a cancel signal.
The tree leaves one seat unassigned: who plays trusted principal for a *child*? The parent — but with derived authority, not original, and the derivation is the narrow-only rule read along the spawn edge: **authority attenuates monotonically down the tree.** A spawn may grant the child any subset of the parent's own grants and nothing outside them; budgets subdivide, scopes narrow, and no edge widens. When a child asks-the-owner, the parent may answer from authority it already holds — that is attenuation working as designed — but a request beyond the parent's grants routes *up*, ultimately to the root principal, because a parent improvising an answer it was never granted is a learned kernel widening authorization: precisely what the gate-placement rule forbids a judge, and being a parent confers no exemption. The corollary is worth one sentence: a fully autonomous run is one whose root principal is unreachable, so the tree's only widening channel is closed and authorization is frozen at launch — not a limitation of the formalism but the honest price of the word *autonomous*.
The pattern generalizes, and that is the point of the appendix. Nothing here added a primitive: the cancel is a signal in $s$, the gate closes by the rule it already follows, the in-flight disposition is forced by irreversibility, $H_{\mathrm{cancel}}$ is a subclass of an existing terminal set, and compensation is an ordinary owner-issued action — and the later entries kept the promise: resume re-enters $T$ at a persisted $s$, the batch gate was always in $\gamma$'s domain, provenance closure is the lattice's meet, attenuation is narrow-only read along an edge, and per-action capability is the gate's decision made enforceable. Every practical concern that earns a place here should resolve the same way — not new machinery, but the discipline the existing objects already imply, made explicit. Cancellation and resume, gate placement and the batch gate, effect records and the state derived from them, composition and delegation — those are the worked instances; the rest of the model is the same exercise.
---
*The ramblings of Claude and Patrick.*
+48 -187
View File
@@ -1,201 +1,62 @@
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
License text copyright (c) 2020 MariaDB Corporation Ab, All Rights Reserved.
"Business Source License" is a trademark of MariaDB Corporation Ab.
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
Parameters
1. Definitions.
Licensor: Patrick Buckley
Licensed Work: Turnstone 0.2.0. The Licensed Work is (c) 2025-2026 Patrick Buckley.
Additional Use Grant: You may make production use of the Licensed Work, provided
your use does not include providing the Licensed Work to third
parties as a hosted or managed service, where the service
provides users with access to any substantial set of the
features or functionality of the Licensed Work.
Change Date: 2030-03-01
Change License: Apache License, Version 2.0
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
For information about alternative licensing arrangements for the Licensed Work,
please contact buckleypm@gmail.com.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
Notice
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
Business Source License 1.1
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
Terms
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
The Licensor hereby grants you the right to copy, modify, create derivative
works, redistribute, and make non-production use of the Licensed Work. The
Licensor may make an Additional Use Grant, above, permitting limited production use.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
Effective on the Change Date, or the fourth anniversary of the first publicly
available distribution of a specific version of the Licensed Work under this
License, whichever comes first, the Licensor hereby grants you rights under
the terms of the Change License, and the rights granted in the paragraph
above terminate.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
If your use of the Licensed Work does not comply with the requirements
currently in effect as described in this License, you must purchase a
commercial license from the Licensor, its affiliated entities, or authorized
resellers, or you must refrain from using the Licensed Work.
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
All copies of the original and modified Licensed Work, and derivative works
of the Licensed Work, are subject to this License. This License applies
separately for each version of the Licensed Work and the Change Date may vary
for each version of the Licensed Work released by Licensor.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
You must conspicuously display this License on each original or modified copy
of the Licensed Work. If you receive the Licensed Work in original or
modified form from a third party, the terms and conditions set forth in this
License apply to your use of that work.
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
Any use of the Licensed Work in violation of this License will automatically
terminate your rights under this License for the current and all other
versions of the Licensed Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
This License does not grant you any right in any trademark or logo of
Licensor or its affiliates (provided that you may use a trademark or logo of
Licensor as expressly required by this License).
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright [yyyy] [name of copyright owner]
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
TO THE EXTENT PERMITTED BY APPLICABLE LAW, THE LICENSED WORK IS PROVIDED ON
AN "AS IS" BASIS. LICENSOR HEREBY DISCLAIMS ALL WARRANTIES AND CONDITIONS,
EXPRESS OR IMPLIED, INCLUDING (WITHOUT LIMITATION) WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, NON-INFRINGEMENT, AND
TITLE.
-7
View File
@@ -1,7 +0,0 @@
Turnstone
Copyright 2025-2026 Patrick Buckley
Licensed under the Apache License, Version 2.0; see the LICENSE file.
Third-party software bundled with this distribution is listed in the
THIRD-PARTY-NOTICES file; each component remains under its own license.
+66 -81
View File
@@ -1,107 +1,92 @@
# Quickstart
# Bootstrap Wizard
Install Turnstone, then diagnose it with `turnstone-doctor` if anything looks off.
Interactive, AI-guided setup for Turnstone deployments. Instead of manually
editing `.env` files and reading deployment docs, the wizard walks you through
every decision conversationally and generates all the config files for you.
## Install
The one-line installer autodetects your distro (Ubuntu/Debian, Fedora/RHEL,
Arch, and WSL), installs git + Docker if missing, generates secrets, picks free
ports, and starts the stack:
## Quick Start
```bash
curl -fsSL https://raw.githubusercontent.com/turnstonelabs/turnstone/main/run.sh | bash
turnstone-bootstrap
```
Re-running is safe — it updates the checkout and keeps your existing `.env`.
When it finishes it prints the dashboard URL and how to create the first admin
user.
That's it — no flags, no arguments. The wizard prompts for everything.
**Other ways to install**
## How It Works
- **Already have Docker?** Clone the repo and `docker compose up` for the full
local cluster, or `docker compose -f turnstone/deploy/compose.yaml up` for the
released single-node stack. See [docs/docker.md](docs/docker.md).
- **Python package:** `pip install turnstone` (add `--pre` for the experimental
track), then run `turnstone-server` / `turnstone-console` directly. See the
[README](README.md#quickstart).
1. **Pick a model** — Choose OpenAI, Anthropic, or a local/vLLM endpoint to
power the wizard. Local endpoints auto-detect available models.
2. **Answer questions** — The AI walks you through deployment mode, LLM
provider, database, authentication, ports, and optional features.
3. **Review generated files** — Each file is previewed before writing. You
confirm or reject every write.
4. **Start the stack** — The wizard prints the exact `docker compose` command
and a `setup.sh` script to create your first admin user, roles, and policies.
## Diagnose: `turnstone-doctor`
## What Gets Generated
`turnstone-doctor` is an LLM-backed assistant that inspects a **running**
Turnstone install and helps you troubleshoot it. It is **read-only** — it
investigates and tells you the exact commands to fix things, but never changes
your system. (Installation is the installer's job, not the doctor's.)
```bash
# From a host that has the turnstone package installed:
turnstone-doctor
# For a Docker install from run.sh (no package on the host), run it with pipx:
pipx run --spec turnstone turnstone-doctor --dir ~/turnstone
```
### What it does
1. **Preflight** — detects how Turnstone is installed here (docker-compose,
systemd/bare-metal, pip, or a source checkout) by probing for `config.toml`
files, `TURNSTONE_*` environment variables, compose files, and systemd units.
2. **Self-configures its LLM** — it powers its own brain from your cluster's
*own* model configuration (env / `config.toml` / the database). Whether that
works is the first diagnostic: success means your LLM backend is healthy; if
it can't, that's surfaced as finding #1 and it falls back to asking you for a
provider and key so it can still help.
3. **Version check** — reports the installed version, version drift across your
cluster's nodes, and the latest upstream stable/experimental releases.
4. **Interactive diagnosis** — it reads logs, `/health`, `docker compose ps`,
`systemctl`, config, and ports to pin down problems like a node not joining
the console, an unreachable database, a down model backend, port conflicts,
or a JWT-secret mismatch — then hands you the precise remediation commands.
### Flags
| Flag | Purpose |
| File | Purpose |
|------|---------|
| `--dir PATH` | Install directory to inspect (default: current directory) |
| `--report` | Print the deterministic preflight report and exit — no LLM key needed |
| `--offline` | Skip the upstream GitHub version check |
| `.env` | All environment variables for `compose.yaml` |
| `setup.sh` | Post-start script: creates admin user, roles, tool policies, prompt templates via the API |
| `docker-compose.override.yaml` | Only if customizations beyond env vars are needed |
`--report` is the fastest way to get a health snapshot (and to share one when
asking for help) — it never needs an API key:
## Requirements
```bash
turnstone-doctor --report --dir ~/turnstone
```
- **Python 3.11+** with turnstone installed (`pip install turnstone`)
- **An LLM API key** — for the wizard itself (OpenAI, Anthropic, or a local
model). This can differ from the LLM your deployment will use.
- **Docker & Docker Compose** — needed to run the stack. The wizard detects
whether Docker is installed and gives platform-specific install instructions
if it's missing. You can still generate config files without Docker.
## Deployment Modes
The wizard supports two deployment modes:
- **Single-node production** (`docker compose --profile production up`) —
1 server + console + PostgreSQL. Good for most use cases.
- **Multi-node cluster** (`docker compose --profile cluster up`) —
10-node server fleet + console + PostgreSQL. For high-throughput or
HA deployments.
## Example Session
```
## Install profile
- Detected kind(s): docker-compose (primary: docker-compose)
- Docker daemon reachable: yes
- Compose files:
/home/you/turnstone/compose.yaml
- Database: backend=postgresql, url=postgresql+psycopg://turnstone:****@postgres:5432/turnstone
- Candidate health URLs: http://localhost:8080/health, http://localhost:8090/health
$ turnstone-bootstrap
## Versions
- Installed (this tool): 1.7.0a2
- Cluster nodes: 10 reporting; versions ['1.7.0a2']
- Version drift across nodes: no
- Upstream: stable 1.6.9, experimental 1.7.0a2
Turnstone Bootstrap Wizard v1.5.0
────────────────────────────────────────────────
## LLM backend (ok)
- resolved Qwen/Qwen3-32B via openai-compatible @ http://host.docker.internal:8000/v1
Which provider for this wizard?
[1] OpenAI
[2] Anthropic
[3] OpenAI-compatible (local/vLLM)
> 3
Base URL [http://localhost:8000/v1]:
API key (press Enter for 'none'):
Querying http://localhost:8000/v1 for available models...
Found model: Qwen/Qwen3-32B
Connected to Qwen/Qwen3-32B. Handing off to AI assistant...
> (AI walks you through the rest interactively)
```
Secrets (JWT secret, database password, API keys) are always redacted in the
report and in anything the doctor reads.
## Tips
- **Type `quit`** to exit the conversation; **Ctrl+C** interrupts (twice to quit).
- **Point it at the right install** with `--dir` when you run it from elsewhere.
- **(Re)installing or adding nodes?** Use the installer (`run.sh`), not the doctor.
- **Re-run safely** — running the wizard again detects your existing `.env`
and offers to update it rather than overwriting.
- **Duplicate writes are skipped** — if the LLM tries to write the same file
twice with identical content, it's silently ignored.
- **Type `quit` to exit** at any time during the conversation.
- **Ctrl+C** is handled gracefully — press once to interrupt, twice to exit.
## See Also
- [Docker Deployment](docs/docker.md) — compose stacks, ports, and bare-metal nodes
- [Docker Deployment](docs/docker.md) — manual compose setup and profiles
- [Security](docs/security.md) — auth architecture and token types
- [Governance](docs/governance.md) — roles, policies, and templates
+14 -44
View File
@@ -3,10 +3,9 @@
[![CI](https://github.com/turnstonelabs/turnstone/actions/workflows/ci.yml/badge.svg)](https://github.com/turnstonelabs/turnstone/actions/workflows/ci.yml)
[![PyPI](https://img.shields.io/pypi/v/turnstone)](https://pypi.org/project/turnstone/)
[![Python](https://img.shields.io/pypi/pyversions/turnstone)](https://pypi.org/project/turnstone/)
[![License](https://img.shields.io/badge/license-Apache--2.0-blue)](LICENSE)
[![Discord](https://img.shields.io/badge/Discord-join%20us-5865F2?logo=discord&logoColor=white)](https://discord.gg/Nh3bWMacaq)
[![License](https://img.shields.io/badge/license-BSL--1.1-blue)](LICENSE)
Self-hosted, local-first orchestration for tool-using AI agents. Give LLMs real tools — shell, files, search, web — and run them across your own cluster with direct HTTP routing and interactive interfaces. Your code, your models, your data stay on hardware you control: no telemetry, no phone-home.
Multi-node AI orchestration platform. Deploy tool-using AI agents across a cluster of servers with direct HTTP routing, interactive interfaces, and enterprise governance.
<p align="center">
<img src="docs/assets/hero.png" alt="Turnstone coordinator — parallel tool batches with judge-graded approval and child workstream tracking" width="960"/>
@@ -14,14 +13,6 @@ Self-hosted, local-first orchestration for tool-using AI agents. Give LLMs real
Named after the [Ruddy Turnstone](https://en.wikipedia.org/wiki/Ruddy_turnstone) (*Arenaria interpres*) — a shorebird that flips stones to discover what's hiding underneath.
**What is a harness?**
```
: s_{n+1} ~ T(s_n) for n < τ*, T = ρ ∘ (M_W ∘ π, E)
```
[**the hypothesis →**](HYPOTHESIS.md)
### Release Tracks
| Track | Install | Docker | Description |
@@ -35,13 +26,12 @@ See [docs/releasing.md](docs/releasing.md) for the full release process.
Turnstone gives LLMs tools — shell, files, search, web, planning — and orchestrates multi-turn conversations where the model investigates, acts, and reports.
- **Local-first & private** — runs entirely on hardware you control, with no telemetry and no phone-home. Point it at local models (vLLM, llama.cpp) or commercial APIs you hold the keys to — your prompts and data never transit a third party you didn't choose.
- **Bring your own models** — OpenAI-compatible APIs (vLLM, llama.cpp, NIM), the Anthropic Messages API, and Google Gemini, mixed freely per role
- **Interactive sessions** — terminal CLI or browser UI with parallel workstreams
- **Cluster dashboard** — real-time view of every node and workstream, with a rendezvous routing proxy
- **Intent validation** — an LLM judge (your model) grades every tool call with a risk assessment and evidence before it runs
- **Cluster dashboard** — real-time view of all nodes and workstreams with console routing proxy
- **Intent validation** — LLM judge evaluates every tool call with risk assessments and evidence
- **Governance** — RBAC, OIDC SSO, tool policies, skills, usage tracking, audit logs
- **Multi-provider** — OpenAI-compatible APIs (vLLM, llama.cpp, NIM), Anthropic Messages API, and Google Gemini
- **MCP support** — external tool servers with native deferred loading (Anthropic/OpenAI) or BM25 fallback
- **Team controls when you need them** — optional RBAC, SSO, tool policies, and audit logs, all stored in your own database
<p align="center">
<img src="docs/diagrams/architecture-overview.svg" alt="Turnstone system architecture" width="960"/>
@@ -59,12 +49,14 @@ turnstone --base-url http://localhost:8000/v1
turnstone-server --port 8080 --base-url http://localhost:8000/v1
# Cluster dashboard
pip install turnstone[console]
turnstone-console --port 8090
```
For PostgreSQL (recommended for production):
```bash
pip install turnstone[postgres]
export TURNSTONE_DB_BACKEND=postgresql
export TURNSTONE_DB_URL="postgresql+psycopg://user:pass@localhost:5432/turnstone"
turnstone-server --port 8080 --base-url http://localhost:8000/v1
@@ -72,29 +64,12 @@ turnstone-server --port 8080 --base-url http://localhost:8000/v1
### Docker
One-line install — autodetects Ubuntu/Debian, Fedora/RHEL, Arch, and WSL,
installs git + Docker if missing, generates secrets, and starts the stack:
```bash
curl -fsSL https://raw.githubusercontent.com/turnstonelabs/turnstone/main/run.sh | bash
cp .env.example .env # edit LLM_BASE_URL, OPENAI_API_KEY, etc.
docker compose --profile production up
```
Or, if you already have Docker, clone the repo and run it yourself:
```bash
docker compose up
```
That builds one image and brings up a full local cluster — PostgreSQL, console,
Caddy, channel gateway, and 10 server nodes — with no `.env` required (it ships
with insecure dev defaults). Open the dashboard at https://localhost:8443 (Caddy
serves it over TLS with its own local CA — trust it once). Nodes boot without an
LLM; add model backends from the console UI.
For production (released images from ghcr.io, real secrets required), use the
bundled stack: `docker compose -f turnstone/deploy/compose.yaml up`.
See [QUICKSTART.md](QUICKSTART.md) for the install + troubleshooting walkthrough and [docs/docker.md](docs/docker.md) for Docker configuration.
See [QUICKSTART.md](QUICKSTART.md) for the bootstrap wizard and [docs/docker.md](docs/docker.md) for Docker configuration and profiles.
### Programmatic (SDK)
@@ -125,7 +100,7 @@ Built-in tools for shell, files, search, web, memory, notifications, and autonom
| `turnstone-channel` | Channel gateway (Discord and Slack adapters) |
| `turnstone-admin` | User/token management CLI |
| `turnstone-eval` | Eval harness for prompt/tool optimization |
| `turnstone-doctor` | LLM-backed cluster diagnostics |
| `turnstone-bootstrap` | LLM-guided setup wizard |
### Diagrams
@@ -167,14 +142,9 @@ UML diagrams in [`docs/diagrams/`](docs/diagrams/):
- Python 3.11+
- An OpenAI-compatible API endpoint, Anthropic API key, or Google Gemini API key
- Optional: Discord / Slack channel integrations (`pip install turnstone[discord,slack]`)
- Optional: PostgreSQL (`pip install turnstone[postgres]`), Anthropic (`pip install turnstone[anthropic]`)
- [Git LFS](https://git-lfs.com/) for cloning (diagram PNGs)
## Community
Questions, ideas, or want to show what you're building? Join us on Discord:
**[discord.gg/Nh3bWMacaq](https://discord.gg/Nh3bWMacaq)**.
## License
[Apache License 2.0](LICENSE), as of version 1.6.0. Versions 1.5.x and earlier remain under the Business Source License 1.1 they shipped with.
[Business Source License 1.1](LICENSE) — free for all use except hosting as a managed service. Converts to Apache 2.0 on 2030-03-01.
+4 -4
View File
@@ -2,11 +2,11 @@ Turnstone — Third-Party Notices
This file contains the licenses and notices for third-party software bundled
with Turnstone. Each bundled dependency retains its original license; the
Turnstone Apache-2.0 license does not apply to these components.
Turnstone BUSL-1.1 license does not apply to these components.
================================================================================
KaTeX 0.17.0
KaTeX 0.16.38
https://katex.org/
https://github.com/KaTeX/KaTeX
@@ -70,7 +70,7 @@ OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
================================================================================
Mermaid 11.15.0
Mermaid 11.13.0
https://mermaid.js.org/
https://github.com/mermaid-js/mermaid
@@ -98,7 +98,7 @@ SOFTWARE.
================================================================================
hls.js 1.6.16
hls.js 1.6.15
https://github.com/video-dev/hls.js
Copyright 2017 Dailymotion
+140 -231
View File
@@ -1,60 +1,16 @@
# =============================================================================
# Turnstone — local cluster stack (docker compose)
# Turnstone Docker Compose Stack — Development
#
# Clone the repo and run:
# This file is for local development from a git clone. It builds images
# locally from the Dockerfile. If you installed via pip/pipx, run
# `turnstone-bootstrap` instead — it writes a production compose.yaml
# that pulls pre-built images from ghcr.io.
#
# docker compose up
#
# That builds one image and brings up a complete, console-visible cluster:
# PostgreSQL + console + Caddy + channel gateway + 10 server nodes (node-1…10).
#
# Dashboard: https://localhost:8443 (Caddy's local CA — trust it once)
#
# Access is via Caddy only — the console's plain-HTTP port is intentionally not
# published (HTTP/2 from Caddy avoids the browser's 6-connection cap on the
# dashboard's SSE streams). Trust Caddy's root once:
# docker compose exec caddy cat /data/caddy/pki/authorities/local/root.crt
#
# It works out of the box with INSECURE dev defaults (see the secret/password
# values below) so there's nothing to configure first. A .env file still
# overrides any value. For a real deployment use the bundled production stack
# at turnstone/deploy/compose.yaml — it pulls released images from ghcr.io and
# requires you to set real secrets.
#
# Bring your own LLM: nodes boot without one and show up in the console
# immediately. Add model backends (OpenAI / Anthropic / local vLLM) from the
# console UI's Models tab, or point LLM_BASE_URL / OPENAI_API_KEY (below) at an
# OpenAI-compatible endpoint.
#
# Fewer nodes (lighter machines):
# docker compose up postgres console caddy channel node-1 node-2 node-3
#
# Join a bare-metal host: a turnstone-server running OUTSIDE compose (e.g. to use
# a local GPU) can join this cluster. Postgres, the console's ACME endpoint, and
# SearxNG are published on 127.0.0.1 so a node on THIS machine reaches them via
# localhost. Keep secrets in ~/.config/turnstone/config.toml (chmod 0600 — the
# loader warns otherwise):
# [auth]
# jwt_secret = "dev-only-insecure-jwt-secret-change-me-for-real-deployments"
# [database]
# backend = "postgresql"
# url = "postgresql+psycopg://turnstone:turnstone@localhost:5432/turnstone"
# [api]
# base_url = "http://localhost:8000/v1"
# api_key = "dummy"
# [tls] # only if the cluster runs mTLS
# enabled = true
# then run (node identity isn't a secret, so it stays on the command line):
# TURNSTONE_NODE_ID=host-1 \
# TURNSTONE_ADVERTISE_URL=http://host.docker.internal:8080 \
# TURNSTONE_CONSOLE_URL=http://localhost:8090 \
# TURNSTONE_SEARXNG_URL=http://localhost:8081 \
# turnstone-server --host 0.0.0.0 --port 8080
# The node registers in Postgres, auto-enrolls its mTLS cert from the console's
# ACME endpoint (when the cluster runs mTLS), and the console collector reaches
# it back via host.docker.internal. To join from ANOTHER machine, set
# TURNSTONE_HOST_IP to this host's LAN IP and use it in the URLs above (and the
# node's TURNSTONE_ADVERTISE_URL = the NODE host's IP) — see docs/docker.md.
# Usage:
# Infra only: docker compose up
# Single node: docker compose --profile production up
# Production (PG): TURNSTONE_DB_BACKEND=postgresql docker compose --profile production up
# 10-node cluster: docker compose --profile cluster up
# =============================================================================
name: turnstone
@@ -67,30 +23,16 @@ volumes:
turnstone-data:
workspace:
postgres-data:
caddy-data:
caddy-config:
searxng-cache:
# -- Shared values (scalar anchors) -------------------------------------------
# Defined once here, referenced (*alias) by every service so the dev defaults
# can't drift. All `${VAR:-default}` values are still overridable via .env.
x-shared:
# INSECURE dev default. Every service MUST share ONE secret — the console
# mints its own service token (signed with this) to reach the nodes. Override
# TURNSTONE_JWT_SECRET in .env for anything that isn't a local sandbox.
jwt-secret: &jwt-secret "${TURNSTONE_JWT_SECRET:-dev-only-insecure-jwt-secret-change-me-for-real-deployments}"
db-backend: &db-backend "${TURNSTONE_DB_BACKEND:-postgresql}"
# All services point at the same Postgres. Node discovery REQUIRES a shared
# DB: each server registers + heartbeats into a `services` table that the
# console polls. (SQLite-per-container can't see other containers.)
db-url: &db-url "${TURNSTONE_DB_URL:-postgresql+psycopg://${POSTGRES_USER:-turnstone}:${POSTGRES_PASSWORD:-turnstone}@postgres:5432/turnstone}"
services:
# -------------------------------------------------------------------
# PostgreSQL — the shared database that ties the cluster together.
# PostgreSQL — production database (profile: production)
# -------------------------------------------------------------------
postgres:
image: pgautoupgrade/pgautoupgrade:18-alpine
profiles:
- production
- cluster
command:
- postgres
- -c
@@ -100,17 +42,8 @@ services:
environment:
POSTGRES_DB: turnstone
POSTGRES_USER: ${POSTGRES_USER:-turnstone}
# INSECURE dev default — override POSTGRES_PASSWORD in .env for real use.
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:-turnstone}
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:?POSTGRES_PASSWORD is required for production profile}
PGDATA: /var/lib/postgresql/data
# Published so a bare-metal turnstone-server can join the cluster (see "Join
# a bare-metal host" in the header). Bound to 127.0.0.1 by default (same-host
# nodes only); set TURNSTONE_HOST_IP to this host's LAN IP to let another
# machine connect — but set a real POSTGRES_PASSWORD first, or you'll expose a
# database with the insecure default password to your network. (The legacy
# POSTGRES_BIND is still honored as a fallback when TURNSTONE_HOST_IP is unset.)
ports:
- "${TURNSTONE_HOST_IP:-${POSTGRES_BIND:-127.0.0.1}}:${POSTGRES_PORT:-5432}:5432"
volumes:
- postgres-data:/var/lib/postgresql/data
networks:
@@ -124,23 +57,65 @@ services:
deploy:
resources:
limits:
memory: 2G
memory: 4G
cpus: '4.0'
restart: unless-stopped
# -------------------------------------------------------------------
# turnstone-console — cluster dashboard. Reach it ONLY through Caddy at
# https://localhost:8443 (see the caddy service below).
#
# Browsers must reach the dashboard through Caddy (https://localhost:8443): a
# plain HTTP/1.1 origin caps the browser at 6 connections, which starves the
# dashboard's per-pane SSE streams, whereas Caddy serves HTTP/2 (multiplexed)
# and proxies to console:8090 internally. The console's :8090 is published
# below ONLY so bare-metal nodes can reach the plain-HTTP ACME enrollment
# endpoint — don't point a browser at it.
#
# The single `build:` here produces the turnstone:local image every other
# service reuses. extra_hosts lets the console reach a bare-metal server
# advertising http://host.docker.internal:8080 (see "Join a host" below).
# turnstone-server — Web UI + chat workstreams + LLM interaction
# -------------------------------------------------------------------
server:
image: turnstone:local
profiles:
- production
command:
- sh
- -c
- >-
turnstone-server
--host 0.0.0.0
--port 8080
--base-url "$${LLM_BASE_URL}"
--api-key "$${OPENAI_API_KEY}"
$${MODEL:+--model $$MODEL}
$${SKIP_PERMISSIONS:+--skip-permissions}
$${MCP_CONFIG:+--mcp-config $$MCP_CONFIG}
ports:
- "${SERVER_PORT:-8080}:8080"
volumes:
- turnstone-data:/data
- ${WORKSPACE_MOUNT:-workspace}:/workspace
environment:
- LLM_BASE_URL=${LLM_BASE_URL:-http://host.docker.internal:8000/v1}
- OPENAI_API_KEY=${OPENAI_API_KEY:-dummy}
- TAVILY_API_KEY=${TAVILY_API_KEY:-}
- SKIP_PERMISSIONS=${SKIP_PERMISSIONS:-}
# Generate with: python -c "import secrets; print(secrets.token_hex(32))"
- TURNSTONE_JWT_SECRET=${TURNSTONE_JWT_SECRET:?Set TURNSTONE_JWT_SECRET in .env}
- MODEL=${MODEL:-}
- MCP_CONFIG=${MCP_CONFIG:-}
- TURNSTONE_DB_BACKEND=${TURNSTONE_DB_BACKEND:-sqlite}
- TURNSTONE_DB_URL=${TURNSTONE_DB_URL:-}
- TURNSTONE_NODE_ID=${TURNSTONE_NODE_ID:-}
- TURNSTONE_ADVERTISE_URL=${TURNSTONE_ADVERTISE_URL:-http://server:8080}
extra_hosts:
- "host.docker.internal:host-gateway"
networks:
- turnstone-net
depends_on:
postgres:
condition: service_healthy
required: false
healthcheck:
test: ["CMD", "python", "/usr/local/bin/healthcheck.py", "http://127.0.0.1:8080/health"]
interval: 10s
timeout: 5s
retries: 5
start_period: 60s
restart: unless-stopped
# -------------------------------------------------------------------
# turnstone-console — Cluster dashboard
# -------------------------------------------------------------------
console:
image: turnstone:local
@@ -151,26 +126,16 @@ services:
- turnstone-console
- --host=0.0.0.0
- --port=8090
# Publishes the console's plain-HTTP listener so a bare-metal node can reach
# the ACME endpoint, fetch the CA, and enroll its cert (the console serves
# HTTP here even under mTLS). Bound to 127.0.0.1 by default; setting
# TURNSTONE_HOST_IP exposes the WHOLE console HTTP API — including the
# cert-issuing ACME endpoint — on that interface, so the JWT secret's
# strength is the only gate. Browsers use Caddy :8443, never this port.
ports:
- "${TURNSTONE_HOST_IP:-127.0.0.1}:8090:8090"
- "${CONSOLE_PORT:-8090}:8090"
environment:
TURNSTONE_JWT_SECRET: *jwt-secret
TURNSTONE_DB_BACKEND: *db-backend
TURNSTONE_DB_URL: *db-url
TURNSTONE_CONSOLE_URL: http://console:8090
extra_hosts:
- "host.docker.internal:host-gateway"
# Generate with: python -c "import secrets; print(secrets.token_hex(32))"
- TURNSTONE_JWT_SECRET=${TURNSTONE_JWT_SECRET:?Set TURNSTONE_JWT_SECRET in .env}
- TURNSTONE_DB_BACKEND=${TURNSTONE_DB_BACKEND:-sqlite}
- TURNSTONE_DB_URL=${TURNSTONE_DB_URL:-}
- TURNSTONE_CONSOLE_URL=http://console:8090
networks:
- turnstone-net
depends_on:
postgres:
condition: service_healthy
healthcheck:
test: ["CMD", "python", "/usr/local/bin/healthcheck.py", "http://127.0.0.1:8090/health"]
interval: 10s
@@ -180,33 +145,14 @@ services:
restart: unless-stopped
# -------------------------------------------------------------------
# caddy — browser TLS for the console dashboard.
# Terminates HTTPS (Caddy's own local CA, see turnstone/deploy/Caddyfile) → console:8090.
# Dashboard over TLS: https://localhost:${CONSOLE_HTTPS_PORT:-8443}
# -------------------------------------------------------------------
caddy:
image: caddy:2.11
depends_on:
- console
ports:
- "${CONSOLE_HTTPS_PORT:-8443}:443"
# SearxNG web UI — localhost-only (it has no auth). Browse https://localhost:8444.
- "127.0.0.1:${SEARXNG_HTTPS_PORT:-8444}:8444"
volumes:
- ./turnstone/deploy/Caddyfile:/etc/caddy/Caddyfile:ro
- caddy-data:/data # persist Caddy's local CA across restarts
- caddy-config:/config
networks:
- turnstone-net
restart: unless-stopped
# -------------------------------------------------------------------
# turnstone-channel — channel gateway (Discord and/or Slack).
# Runs HTTP-only with no adapters until you set a token, so it's safe
# to leave running. See docs/channels.md.
# turnstone-channel — Channel gateway (Discord, Slack, etc.)
# Requires TURNSTONE_DISCORD_TOKEN to enable Discord adapter
# -------------------------------------------------------------------
channel:
image: turnstone:local
profiles:
- production
- cluster
command:
- sh
- -c
@@ -215,66 +161,36 @@ services:
--http-host=0.0.0.0
$${TURNSTONE_DISCORD_GUILD:+--discord-guild $$TURNSTONE_DISCORD_GUILD}
environment:
TURNSTONE_JWT_SECRET: *jwt-secret
TURNSTONE_DB_BACKEND: *db-backend
TURNSTONE_DB_URL: *db-url
TURNSTONE_DISCORD_TOKEN: ${TURNSTONE_DISCORD_TOKEN:-}
TURNSTONE_DISCORD_GUILD: ${TURNSTONE_DISCORD_GUILD:-0}
TURNSTONE_SLACK_TOKEN: ${TURNSTONE_SLACK_TOKEN:-}
TURNSTONE_SLACK_APP_TOKEN: ${TURNSTONE_SLACK_APP_TOKEN:-}
TURNSTONE_CHANNEL_ADVERTISE_URL: http://channel:8091
- TURNSTONE_DISCORD_TOKEN=${TURNSTONE_DISCORD_TOKEN:-}
- TURNSTONE_DISCORD_GUILD=${TURNSTONE_DISCORD_GUILD:-0}
# Generate with: python -c "import secrets; print(secrets.token_hex(32))"
- TURNSTONE_JWT_SECRET=${TURNSTONE_JWT_SECRET:?Set TURNSTONE_JWT_SECRET in .env}
- TURNSTONE_DB_BACKEND=${TURNSTONE_DB_BACKEND:-postgresql}
- TURNSTONE_DB_URL=${TURNSTONE_DB_URL:-postgresql+psycopg://${POSTGRES_USER:-turnstone}:${POSTGRES_PASSWORD:-turnstone}@postgres:5432/turnstone}
- TURNSTONE_CHANNEL_ADVERTISE_URL=http://channel:8091
networks:
- turnstone-net
depends_on:
postgres:
condition: service_healthy
restart: unless-stopped
# -------------------------------------------------------------------
# searxng — self-hosted metasearch backing the web_search tool.
# Internal-network only (no published port): nodes reach it at
# http://searxng:8080. Config (JSON output on, limiter off) lives in
# turnstone/deploy/searxng/settings.yml, mounted read-only. Commercial
# models use native provider search and never hit this; it serves
# local/vLLM models. Override the tag with SEARXNG_IMAGE_TAG in .env.
# -------------------------------------------------------------------
searxng:
image: searxng/searxng:${SEARXNG_IMAGE_TAG:-latest}
# Published so a bare-metal node's web_search can reach it. SearxNG has NO
# auth, so it is bound to 127.0.0.1 by default; setting TURNSTONE_HOST_IP
# exposes it on that interface — an open search proxy on your LAN, which also
# triggers the SearxNG AGPL-3.0 §13 source-offer obligation (see docs/docker.md).
# In-compose nodes always use the internal http://searxng:8080 and ignore this.
ports:
- "${TURNSTONE_HOST_IP:-127.0.0.1}:${SEARXNG_API_PORT:-8081}:8080"
volumes:
- ./turnstone/deploy/searxng:/etc/searxng:ro
- searxng-cache:/var/cache/searxng # favicon + internal SQLite cache (survives restarts)
networks:
- turnstone-net
healthcheck:
test: ["CMD", "wget", "-q", "--spider", "http://localhost:8080/healthz"]
interval: 10s
timeout: 5s
retries: 5
start_period: 20s
required: false
restart: unless-stopped
# ===================================================================
# Server fleet — node-1 … node-10
# 10-node cluster (profile: cluster)
#
# Each node registers itself in Postgres on boot (unique
# TURNSTONE_NODE_ID + TURNSTONE_ADVERTISE_URL) and the console
# discovers it automatically — no static node list anywhere.
# All nodes share the same PostgreSQL instance.
# Access via console at :8090.
#
# node-1 carries the shared definition (&node / &node-env); node-2…10
# inherit it and override only their identity.
# Start: docker compose --profile cluster up
# ===================================================================
node-1: &node
# -- cluster servers ------------------------------------------------
server-1: &cluster-server
image: turnstone:local
build:
context: .
dockerfile: Dockerfile
build: { context: ., dockerfile: Dockerfile }
profiles: [cluster]
command:
- sh
- -c
@@ -290,30 +206,23 @@ services:
volumes:
- turnstone-data:/data
- ${WORKSPACE_MOUNT:-workspace}:/workspace
environment: &node-env
TURNSTONE_JWT_SECRET: *jwt-secret
TURNSTONE_DB_BACKEND: *db-backend
TURNSTONE_DB_URL: *db-url
# Bootstrap LLM defaults — real backends are configured in the console UI.
environment: &cluster-server-env
LLM_BASE_URL: ${LLM_BASE_URL:-http://host.docker.internal:8000/v1}
OPENAI_API_KEY: ${OPENAI_API_KEY:-dummy}
# web_search backend. Defaults to the bundled searxng service; point at an
# external SearxNG by setting TURNSTONE_SEARXNG_URL in .env (empty disables).
TURNSTONE_SEARXNG_URL: ${TURNSTONE_SEARXNG_URL:-http://searxng:8080}
TAVILY_API_KEY: ${TAVILY_API_KEY:-}
SKIP_PERMISSIONS: ${SKIP_PERMISSIONS:-}
# Generate with: python -c "import secrets; print(secrets.token_hex(32))"
TURNSTONE_JWT_SECRET: ${TURNSTONE_JWT_SECRET:?Set TURNSTONE_JWT_SECRET in .env}
MODEL: ${MODEL:-}
MCP_CONFIG: ${MCP_CONFIG:-}
SKIP_PERMISSIONS: ${SKIP_PERMISSIONS:-}
TURNSTONE_DB_BACKEND: ${TURNSTONE_DB_BACKEND:-postgresql}
TURNSTONE_DB_URL: ${TURNSTONE_DB_URL:-postgresql+psycopg://${POSTGRES_USER:-turnstone}:${POSTGRES_PASSWORD:?}@postgres:5432/turnstone}
TURNSTONE_NODE_ID: node-1
TURNSTONE_ADVERTISE_URL: http://node-1:8080
extra_hosts:
- "host.docker.internal:host-gateway"
networks:
- turnstone-net
TURNSTONE_ADVERTISE_URL: http://server-1:8080
extra_hosts: ["host.docker.internal:host-gateway"]
networks: [turnstone-net]
depends_on:
postgres:
condition: service_healthy
searxng:
condition: service_healthy
postgres: { condition: service_healthy }
healthcheck:
test: ["CMD", "python", "/usr/local/bin/healthcheck.py", "http://127.0.0.1:8080/health"]
interval: 10s
@@ -322,34 +231,34 @@ services:
start_period: 60s
deploy:
resources:
limits:
memory: 4G
limits: { memory: 4G, cpus: '4' }
restart: unless-stopped
node-2:
<<: *node
environment: { <<: *node-env, TURNSTONE_NODE_ID: node-2, TURNSTONE_ADVERTISE_URL: "http://node-2:8080" }
node-3:
<<: *node
environment: { <<: *node-env, TURNSTONE_NODE_ID: node-3, TURNSTONE_ADVERTISE_URL: "http://node-3:8080" }
node-4:
<<: *node
environment: { <<: *node-env, TURNSTONE_NODE_ID: node-4, TURNSTONE_ADVERTISE_URL: "http://node-4:8080" }
node-5:
<<: *node
environment: { <<: *node-env, TURNSTONE_NODE_ID: node-5, TURNSTONE_ADVERTISE_URL: "http://node-5:8080" }
node-6:
<<: *node
environment: { <<: *node-env, TURNSTONE_NODE_ID: node-6, TURNSTONE_ADVERTISE_URL: "http://node-6:8080" }
node-7:
<<: *node
environment: { <<: *node-env, TURNSTONE_NODE_ID: node-7, TURNSTONE_ADVERTISE_URL: "http://node-7:8080" }
node-8:
<<: *node
environment: { <<: *node-env, TURNSTONE_NODE_ID: node-8, TURNSTONE_ADVERTISE_URL: "http://node-8:8080" }
node-9:
<<: *node
environment: { <<: *node-env, TURNSTONE_NODE_ID: node-9, TURNSTONE_ADVERTISE_URL: "http://node-9:8080" }
node-10:
<<: *node
environment: { <<: *node-env, TURNSTONE_NODE_ID: node-10, TURNSTONE_ADVERTISE_URL: "http://node-10:8080" }
server-2:
<<: *cluster-server
environment: { <<: *cluster-server-env, TURNSTONE_NODE_ID: node-2, TURNSTONE_ADVERTISE_URL: "http://server-2:8080" }
server-3:
<<: *cluster-server
environment: { <<: *cluster-server-env, TURNSTONE_NODE_ID: node-3, TURNSTONE_ADVERTISE_URL: "http://server-3:8080" }
server-4:
<<: *cluster-server
environment: { <<: *cluster-server-env, TURNSTONE_NODE_ID: node-4, TURNSTONE_ADVERTISE_URL: "http://server-4:8080" }
server-5:
<<: *cluster-server
environment: { <<: *cluster-server-env, TURNSTONE_NODE_ID: node-5, TURNSTONE_ADVERTISE_URL: "http://server-5:8080" }
server-6:
<<: *cluster-server
environment: { <<: *cluster-server-env, TURNSTONE_NODE_ID: node-6, TURNSTONE_ADVERTISE_URL: "http://server-6:8080" }
server-7:
<<: *cluster-server
environment: { <<: *cluster-server-env, TURNSTONE_NODE_ID: node-7, TURNSTONE_ADVERTISE_URL: "http://server-7:8080" }
server-8:
<<: *cluster-server
environment: { <<: *cluster-server-env, TURNSTONE_NODE_ID: node-8, TURNSTONE_ADVERTISE_URL: "http://server-8:8080" }
server-9:
<<: *cluster-server
environment: { <<: *cluster-server-env, TURNSTONE_NODE_ID: node-9, TURNSTONE_ADVERTISE_URL: "http://server-9:8080" }
server-10:
<<: *cluster-server
environment: { <<: *cluster-server-env, TURNSTONE_NODE_ID: node-10, TURNSTONE_ADVERTISE_URL: "http://server-10:8080" }
+40
View File
@@ -0,0 +1,40 @@
# Bare-metal overlay — expose PostgreSQL and let the console reach
# a turnstone-server running outside Docker on the host machine.
#
# Requires TURNSTONE_HOST_IP set to the host's routable IP address.
#
# Usage:
# export TURNSTONE_HOST_IP="$(hostname -I | awk '{print $1}')"
# docker compose --profile production \
# -f compose.yaml -f deploy/docker-compose.bare-metal.yml up
#
# Then on the host:
# export TURNSTONE_JWT_SECRET="<same as .env>"
# export TURNSTONE_DB_BACKEND=postgresql
# export TURNSTONE_DB_URL="postgresql://turnstone:<pw>@localhost:5432/turnstone"
# export TURNSTONE_NODE_ID="bare-metal-1"
# export TURNSTONE_ADVERTISE_URL="http://${TURNSTONE_HOST_IP}:8080"
# python -m turnstone.server --host 0.0.0.0 --port 8080 \
# --base-url http://localhost:8000/v1 --api-key "$OPENAI_API_KEY"
services:
postgres:
ports:
- "${POSTGRES_PORT:-5432}:5432"
console:
extra_hosts:
- "host.docker.internal:host-gateway"
environment:
# Console needs to reach the bare-metal server on the host
TURNSTONE_SERVER_URL: "http://${TURNSTONE_HOST_IP}:${SERVER_PORT:-8080}"
channel:
ports:
- "${CHANNEL_PORT:-8091}:8091"
environment:
# Channel gateway advertises with host-routable IP so the
# bare-metal server can reach it for schedule notifications
TURNSTONE_CHANNEL_ADVERTISE_URL: "http://${TURNSTONE_HOST_IP}:${CHANNEL_PORT:-8091}"
# Channel needs to reach the bare-metal server on the host
TURNSTONE_SERVER_URL: "http://${TURNSTONE_HOST_IP}:${SERVER_PORT:-8080}"
+4 -5
View File
@@ -1,8 +1,7 @@
# TLS overlay — enables mTLS across the turnstone deployment.
# TLS overlay — enables mTLS across the turnstone cluster.
#
# Layers on the production stack (it patches the `server`, `console`, and
# `channel` services that file defines):
# docker compose -f turnstone/deploy/compose.yaml -f deploy/docker-compose.tls.yml up
# Usage (requires base compose.yaml with production profile):
# docker compose -f compose.yaml -f deploy/docker-compose.tls.yml --profile production up
#
# The tls-init service bootstraps a CA and issues certs.
# All turnstone services auto-provision their own certs via the
@@ -13,7 +12,7 @@ services:
# Runs as root to create directories in the volume, then chowns
# to turnstone:turnstone with restrictive perms (keys 0600).
tls-init:
image: ghcr.io/turnstonelabs/turnstone:${TURNSTONE_IMAGE_TAG:-latest}
build: .
user: root
command:
- sh
+1 -1
View File
@@ -7,6 +7,6 @@ appVersion: "0.3.0"
dependencies:
- name: postgresql
version: ~18.7.0
version: ~18.6.0
repository: https://charts.bitnami.com/bitnami
condition: postgresql.enabled
+5 -10
View File
@@ -103,18 +103,13 @@ network_policies:
- path: /usr/bin/python3*
- path: /usr/local/bin/python3*
# --- Web search (SearxNG) ---
# Turnstone talks only to its SearxNG instance over HTTP; SearxNG itself makes
# the outbound calls to search engines (and is NOT governed by this policy —
# it runs as a separate service). The host/port below is the bundled compose
# service name; if your SearxNG runs elsewhere, set it to match
# TURNSTONE_SEARXNG_URL.
# --- Web search fallback (Tavily) ---
searxng:
name: searxng-search
tavily_api:
name: tavily-search
endpoints:
- host: searxng
port: 8080
- host: api.tavily.com
port: 443
binaries:
- path: /usr/bin/python3*
- path: /usr/local/bin/python3*
-72
View File
@@ -1,72 +0,0 @@
# Running a bare-metal turnstone-server under systemd
These units run a `turnstone-server` **outside** Docker (e.g. on a box with a
local GPU) so it joins an existing cluster — typically the docker-compose stack
in [`compose.yaml`](../../compose.yaml). They are the hardened, production-shaped
counterpart to the quick `turnstone-server …` invocation in
[`docs/docker.md`](../../docs/docker.md) ("Join a bare-metal host").
| File | Purpose |
|------|---------|
| `turnstone-server.service` | The hardened server unit (sandboxed; secrets via `config.toml`). |
| `turnstone.slice` | Shared memory/process budget for colocated Turnstone units. |
| `turnstone-server.service.d/node.conf.example` | Per-host identity + cluster URLs drop-in (no secrets). |
## Cluster-side prerequisite
The compose stack must publish Postgres, the console's ACME endpoint, and SearxNG
on an address the bare-metal host can reach. Start it with `TURNSTONE_HOST_IP`
set to the compose host's LAN IP (default `127.0.0.1` keeps everything host-local):
```bash
TURNSTONE_HOST_IP=<compose-host-ip> docker compose up -d
```
## Install (run as root on the bare-metal host)
```bash
# 1. A dedicated, unprivileged user.
useradd --system --no-create-home --shell /usr/sbin/nologin turnstone
# 2. Install turnstone into a venv at /opt/turnstone-venv (lacme/mTLS is a core dep).
uv venv /opt/turnstone-venv --python 3.12
uv pip install --python /opt/turnstone-venv 'turnstone @ git+https://github.com/turnstonelabs/turnstone'
# …or from a local checkout: uv pip install --python /opt/turnstone-venv /path/to/turnstone
# 3. Secrets — match the cluster's JWT secret + DB credentials (kept out of env).
install -d -m 750 -o turnstone -g turnstone /etc/turnstone
cat > /etc/turnstone/config.toml <<'TOML'
[auth]
jwt_secret = "<same secret as the cluster>"
[database]
backend = "postgresql"
url = "postgresql+psycopg://turnstone:<password>@<compose-host-ip>:5432/turnstone"
[api]
base_url = "http://localhost:8000/v1" # a real model backend is configured in the console UI
api_key = "dummy"
TOML
chown turnstone:turnstone /etc/turnstone/config.toml
chmod 600 /etc/turnstone/config.toml
# 4. Units + per-host drop-in.
cp turnstone-server.service turnstone.slice /etc/systemd/system/
install -d /etc/systemd/system/turnstone-server.service.d
cp turnstone-server.service.d/node.conf.example \
/etc/systemd/system/turnstone-server.service.d/node.conf
$EDITOR /etc/systemd/system/turnstone-server.service.d/node.conf # set the addresses
# 5. Go.
systemctl daemon-reload
systemctl enable --now turnstone-server.service
journalctl -u turnstone-server -f # watch it register + (if the cluster runs mTLS) enroll
```
`tls.enabled` is **not** set here — a joining node inherits it from the cluster's
shared settings (the database). If the cluster runs mTLS, the node auto-enrolls a
cert from the console's ACME endpoint and re-advertises itself over `https://`.
> **mTLS + cross-host caveat:** a node on a *different* host than the console
> currently can't complete ACME enrollment — the console advertises an
> unroutable in-container address in its ACME directory
> ([turnstonelabs/lacme#22](https://github.com/turnstonelabs/lacme/issues/22)).
> Same-host bare-metal nodes, and any node in a non-mTLS cluster, are unaffected.
-85
View File
@@ -1,85 +0,0 @@
# Run a bare-metal turnstone-server as a systemd service so it joins a cluster
# (e.g. the docker-compose stack) from outside Docker — typically to use a local
# GPU. Install steps + the cluster-side prerequisites are in deploy/systemd/README.md
# and docs/docker.md ("Join a bare-metal host"). Per-host identity + the cluster
# URLs go in a drop-in (see node.conf.example); secrets go in config.toml.
[Unit]
Description=Turnstone server (chat workstreams + LLM gateway)
Documentation=https://github.com/turnstonelabs/turnstone
# Postgres is required. After= orders against a colocated postgresql.service
# when present and silently no-ops otherwise (the cluster DB is usually remote).
After=network.target postgresql.service
StartLimitIntervalSec=60
StartLimitBurst=5
[Service]
Type=exec
User=turnstone
Group=turnstone
# Secrets live in config.toml — JWT secret, Postgres URL+password, LLM API key —
# kept out of os.environ so a prompt-injected tool can't dump them via `env`.
Environment=TURNSTONE_CONFIG=/etc/turnstone/config.toml
Environment=TURNSTONE_LOG_LEVEL=info
Slice=turnstone.slice
# Per-host node identity + cluster wiring (TURNSTONE_NODE_ID / _ADVERTISE_URL /
# _CONSOLE_URL / _SEARXNG_URL) go in a drop-in, not here — see node.conf.example.
StateDirectory=turnstone
StateDirectoryMode=0750
LogsDirectory=turnstone
LogsDirectoryMode=0750
WorkingDirectory=/var/lib/turnstone
# --host 0.0.0.0 so the console collector + peer nodes can dial this node back
# at its advertised address. (A single-node, Caddy-fronted install can use
# 127.0.0.1 instead.) Rewrite --port if :8080 is already taken on the host.
ExecStart=/opt/turnstone-venv/bin/turnstone-server --host 0.0.0.0 --port 8080
Restart=on-failure
RestartSec=5s
TimeoutStartSec=120
TimeoutStopSec=30
KillSignal=SIGTERM
KillMode=mixed
# --- Resource limits ---
# SSE keeps an fd per active workstream + outbound LLM stream + MCP stdio pipe.
LimitNOFILE=65535
LimitNPROC=8192
TasksMax=8192
LimitCORE=0
# --- Hardening ---
NoNewPrivileges=true
CapabilityBoundingSet=
AmbientCapabilities=
UMask=0027
PrivateTmp=true
# PrivateDevices=true — disabled: GPU access via /sys/class/drm
ProtectSystem=strict
ProtectHome=true
ProtectKernelTunables=true
ProtectKernelModules=true
ProtectKernelLogs=true
ProtectControlGroups=true
ProtectClock=true
ProtectHostname=true
RestrictNamespaces=true
RestrictRealtime=true
RestrictSUIDSGID=true
RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6
LockPersonality=true
MemoryDenyWriteExecute=true
SystemCallArchitectures=native
SystemCallFilter=@system-service
SystemCallFilter=~@privileged @mount
StandardOutput=journal
StandardError=journal
SyslogIdentifier=turnstone-server
[Install]
WantedBy=multi-user.target
@@ -1,25 +0,0 @@
# Per-host node identity + cluster wiring for a bare-metal turnstone-server.
# Copy to /etc/systemd/system/turnstone-server.service.d/node.conf and edit the
# addresses, then `systemctl daemon-reload`. Identity + URLs are NOT secrets, so
# they live here; the JWT secret + DB URL live in /etc/turnstone/config.toml.
#
# Addresses below use RFC 5737 documentation IPs — replace them:
# <this-host> = the bare-metal host's own LAN IP (what the console dials back)
# <compose-host> = the host running the cluster / docker-compose stack, started
# with TURNSTONE_HOST_IP=<compose-host> so :8090 and :8081 are
# published on its LAN interface (see docs/docker.md).
[Service]
# Unique node id (defaults to the hostname if unset).
Environment=TURNSTONE_NODE_ID=host-1
# The address peers + the console collector dial back. Auto-upgrades to https://
# once the node enrolls its mTLS cert.
Environment=TURNSTONE_ADVERTISE_URL=http://192.0.2.10:8080
# The cluster console's reachable plain-HTTP ACME/API endpoint. A bare-metal node
# can't resolve the in-cluster name (console:8090), so point it at the published
# port; turnstone-server honors this for cert enrollment.
Environment=TURNSTONE_CONSOLE_URL=http://192.0.2.1:8090
# The cluster's published SearxNG, for the web_search tool.
Environment=TURNSTONE_SEARXNG_URL=http://192.0.2.1:8081
-15
View File
@@ -1,15 +0,0 @@
# Shared resource budget for the colocated Turnstone units. Without a slice each
# unit's MemoryMax= is enforced independently — three units at 85% each can sum
# to 255% of host RAM before any throttles. Under a shared slice the cap is
# hierarchical: the slice ceiling is the real limit. (A bare-metal node that runs
# only turnstone-server still benefits — and keeps the unit's Slice= reference
# valid.) Adjust if the host runs other meaningful workloads alongside Turnstone.
[Unit]
Description=Turnstone services slice (server + console + channel)
Documentation=https://github.com/turnstonelabs/turnstone
Before=slices.target
[Slice]
MemoryHigh=70%
MemoryMax=85%
TasksMax=16384
+11 -75
View File
@@ -2,69 +2,13 @@
"""Health check for turnstone containers.
Usage: healthcheck.py <url>
Exit 0 if the endpoint returns {"status": "ok"} or {"status": "degraded"},
exit 1 otherwise. Uses only stdlib no pip dependencies required.
When the node serves mTLS (tls.enabled), a plain-HTTP probe is rejected at
the socket, so on failure this script retries over HTTPS, presenting the
node's own certificate as the client cert and pinning the cluster CA. The
PEM files are the ones the server writes at boot under
$TURNSTONE_TLS_PEM_DIR (default: <tmpdir>/turnstone-tls). The host is
rewritten to "localhost" for the TLS attempt because the internal CA issues
DNS SANs only certificate verification rejects a literal-IP dial.
When mTLS is disabled (the default), the plain probe succeeds and nothing
here changes: the PEM directory is never consulted.
Exit 0 if the endpoint returns {"status": "ok"}, exit 1 otherwise.
Uses only stdlib no pip dependencies required.
"""
import json
import os
import ssl
import sys
import tempfile
import urllib.request
from pathlib import Path
from urllib.parse import urlsplit, urlunsplit
def _check(url: str, context: ssl.SSLContext | None = None) -> None:
"""Probe one URL; raise if unreachable or the payload is unhealthy."""
req = urllib.request.Request(url, method="GET")
with urllib.request.urlopen(req, timeout=5, context=context) as resp:
data = json.loads(resp.read().decode())
if data.get("status") not in ("ok", "degraded"):
raise RuntimeError(f"unhealthy payload: {data}")
def _pem_root() -> Path:
"""PEM runtime root.
Must mirror turnstone.core.tls.tls_pem_runtime_dir this script is
standalone stdlib and cannot import turnstone; a drift-guard test in
tests/test_docker_healthcheck.py pins the two together.
"""
root_env = os.environ.get("TURNSTONE_TLS_PEM_DIR")
return Path(root_env) if root_env else Path(tempfile.gettempdir()) / "turnstone-tls"
def _find_pem_dir() -> Path | None:
"""Locate the newest complete PEM dir written by the server at boot."""
root = _pem_root()
candidates = [
d
for d in root.glob("lacme-pem-*")
if all((d / name).is_file() for name in ("fullchain.pem", "key.pem", "ca.pem"))
]
if not candidates:
return None
return max(candidates, key=lambda d: d.stat().st_mtime)
def _tls_url(url: str) -> str:
"""Rewrite scheme to https and host to localhost, keeping port and path."""
parts = urlsplit(url)
netloc = f"localhost:{parts.port}" if parts.port else "localhost"
return urlunsplit(("https", netloc, parts.path, parts.query, parts.fragment))
def main() -> None:
@@ -74,24 +18,16 @@ def main() -> None:
url = sys.argv[1]
try:
_check(url)
sys.exit(0)
except Exception as plain_exc:
pem_dir = _find_pem_dir()
if pem_dir is None:
print(f"Health check failed: {plain_exc}", file=sys.stderr)
sys.exit(1)
try:
context = ssl.create_default_context(cafile=str(pem_dir / "ca.pem"))
context.load_cert_chain(str(pem_dir / "fullchain.pem"), str(pem_dir / "key.pem"))
_check(_tls_url(url), context=context)
sys.exit(0)
except Exception as tls_exc:
print(
f"Health check failed: plain: {plain_exc}; mtls: {tls_exc}",
file=sys.stderr,
)
req = urllib.request.Request(url, method="GET")
with urllib.request.urlopen(req, timeout=5) as resp:
data = json.loads(resp.read().decode())
if data.get("status") in ("ok", "degraded"):
sys.exit(0)
print(f"Unhealthy: {data}", file=sys.stderr)
sys.exit(1)
except Exception as exc:
print(f"Health check failed: {exc}", file=sys.stderr)
sys.exit(1)
if __name__ == "__main__":
+42 -56
View File
@@ -63,10 +63,7 @@ Auth is always enabled. All API endpoints except public paths require a valid to
Include a token in one of two ways:
- **Bearer header**: `Authorization: Bearer <token>`
- **Cookie**: the surface-scoped auth cookie — `turnstone_auth_server` on
turnstone-server, `turnstone_auth_console` on turnstone-console (set
automatically by the login endpoint). The names differ so the two surfaces,
when co-hosted on one origin, don't overwrite each other's session.
- **Cookie**: `turnstone_auth=<token>` (set automatically by the login endpoint)
The server accepts two token types:
@@ -105,8 +102,7 @@ Authenticate with credentials and receive a JWT. Accepts two credential formats:
}
```
The response also sets a surface-scoped HttpOnly cookie containing the JWT
(`turnstone_auth_server` on turnstone-server, `turnstone_auth_console` on turnstone-console).
The response also sets a `turnstone_auth` HttpOnly cookie containing the JWT.
**Response (failure):** `401`
@@ -118,8 +114,7 @@ The response also sets a surface-scoped HttpOnly cookie containing the JWT
### `POST /v1/api/auth/logout`
Clears the surface-scoped auth cookie (`turnstone_auth_server` /
`turnstone_auth_console`). No request body required.
Clears the `turnstone_auth` cookie. No request body required.
**Response:** `200`
@@ -204,8 +199,7 @@ this endpoint.
}
```
The response also sets a surface-scoped HttpOnly cookie containing the JWT
(`turnstone_auth_server` on turnstone-server, `turnstone_auth_console` on turnstone-console).
The response also sets a `turnstone_auth` HttpOnly cookie containing the JWT.
**Response (already set up):** `409`
@@ -287,7 +281,6 @@ Each message in the `messages` array has:
| `role` | string | `"user"`, `"assistant"`, or `"tool"` |
| `content` | string or null | Text content of the message |
| `tool_calls` | array or null | Present only on assistant messages with calls |
| `reasoning` | string (optional) | Concatenated reasoning / chain-of-thought text on assistant turns whose `provider_data` carried reasoning-bearing blocks (Anthropic `thinking`, OpenAI Responses `reasoning`, or synthetic `reasoning_text` from local-model servers). Present only when the active model's `surface_persisted_reasoning` flag is True. |
Each entry in `tool_calls`:
@@ -332,44 +325,6 @@ finalize any in-progress assistant message.
{"type": "stream_end"}
```
**`state_change`** -- the worker thread transitioned to a new state. Drives
the client's busy-mode (composer in send vs. stop, spinner indicators,
auto-focus on idle). Sent live during normal operation AND on every fresh
SSE subscribe (so a mid-stream page refresh restores the correct composer
state without waiting for the next live transition).
```json
{"type": "state_change", "state": "running"}
```
| Field | Type | Description |
|----------|--------|----------------------------------------------------------------------|
| `state` | string | One of `"running"`, `"thinking"`, `"attention"`, `"idle"`, `"error"` |
**`in_progress_snapshot`** -- one-shot replay of the in-progress turn's
content + reasoning text-so-far when this client connects mid-stream.
Lets a refreshing browser tab restore partial assistant text immediately
instead of waiting for the response to complete. Yielded once after the
kind-specific replay phase (history + pending), only when at least one
of `content` / `reasoning` is non-empty. Both halves render into the same
assistant bubble the live `content` / `reasoning` events would target;
clients should treat the snapshot as idempotent (skip overwrite if the
current local buffer is already a superset prefix — covers EventSource
auto-reconnect re-replays).
```json
{
"type": "in_progress_snapshot",
"content": "Here is the answer so far: it depends on ",
"reasoning": "The user is asking about a comparison; let me think about..."
}
```
| Field | Type | Description |
|--------------|--------|------------------------------------------------------------|
| `content` | string | Joined assistant content text accumulated this turn |
| `reasoning` | string | Joined reasoning / chain-of-thought text accumulated |
**`tool_info`** -- one or more tool calls that were auto-approved (no user
action required).
@@ -461,6 +416,13 @@ Each item in `items` (shared by `tool_info` and `approve_request`):
| `cache_creation_tokens` | int | Tokens written to prompt cache (Anthropic) |
| `cache_read_tokens` | int | Tokens served from prompt cache (Anthropic + OpenAI) |
**`plan_review`** -- the model is proposing a plan and wants feedback. The
client must respond via `POST /v1/api/plan`.
```json
{"type": "plan_review", "content": "Step 1: ...\nStep 2: ..."}
```
**`info`** -- an informational message (e.g. command output).
```json
@@ -560,13 +522,7 @@ Each SSE connection to a workstream receives its own delivery queue. Events
produced by the worker thread are fanned out to all registered listener queues,
so multiple consumers (browser, console proxy, SDK) can connect
simultaneously and each receives every event. On reconnect the client receives
the kind-specific replay (`connected` + `status` + `history` + pending
approval / plan for interactive; `connected` + `status` + pending for coord)
followed by a `state_change` carrying the current worker state and an
optional `in_progress_snapshot` carrying any partial content / reasoning
buffered for the in-progress turn — so a mid-stream refresh restores both
the busy-mode UI and the partial assistant text without waiting for the
response to complete.
a full history replay, so no catch-up mechanism is needed.
---
@@ -778,6 +734,36 @@ automatically approved without prompting.
---
### `POST /v1/api/plan`
Responds to a plan review dialog. The SSE stream must have previously sent a
`plan_review` event for the given workstream.
**Request body:**
```json
{"feedback": "", "ws_id": "abc123"}
```
| Field | Type | Required | Description |
|------------|--------|----------|---------------------------------------------------------|
| `feedback` | string | yes | Feedback text; empty string means approval |
| `ws_id` | string | yes | Target workstream ID |
To approve the plan, send an empty string for `feedback`. To reject or request
changes, send a non-empty feedback string (e.g. `"reject"` or specific
revision instructions).
**Response:**
```json
{"status": "ok"}
```
**Error:** `404` with `{"error": "Unknown workstream"}` if `ws_id` is invalid.
---
### `POST /v1/api/command`
Executes a slash command in the given workstream.
+58 -138
View File
@@ -3,8 +3,8 @@
Turnstone is an AI orchestration platform with tool use, parallel workstreams, and persistent
memory. It connects to any OpenAI-compatible API (local vLLM, OpenAI, etc.) or
Anthropic's native Messages API via pluggable provider adapters, and gives the
model 16 built-in tools plus external tools via MCP (Model Context Protocol) for
reading, writing, searching, and executing code.
model 19 built-in tools plus external tools via MCP (Model Context Protocol) for
reading, writing, searching, planning, and executing code.
The core design principle is a **UI-agnostic engine with pluggable frontends**.
The engine (`ChatSession`) drives the conversation loop -- streaming, tool
@@ -22,7 +22,7 @@ plugs in.
| `turnstone-eval` | `turnstone.eval` | `NullUI` | Headless evaluation and prompt optimization |
| `turnstone-channel` | `turnstone.channels.cli` | ChannelAdapter | Channel gateway (Discord, Slack, etc.) |
| `turnstone-admin` | `turnstone.admin` | — | Offline user and API token management |
| `turnstone-doctor` | `turnstone.doctor` | — | LLM-backed cluster diagnostics |
| `turnstone-bootstrap` | `turnstone.bootstrap` | — | LLM-guided setup wizard |
---
@@ -61,6 +61,7 @@ turnstone/
ratelimit.py Per-IP token-bucket rate limiter (RateLimiter, TokenBucket)
edit.py File edit utilities (find_occurrences, pick_nearest)
safety.py Command safety validation (blocked patterns, sanitization)
sandbox.py Math code sandboxing (AST validation, subprocess execution)
web.py Web utilities (HTML stripping, SSRF prevention)
api/
schemas.py Shared Pydantic v2 models (auth, errors, WorkstreamState)
@@ -90,7 +91,7 @@ turnstone/
discord/ Discord adapter (bot, cog, views, streaming, config)
slack/ Slack adapter (Socket Mode bot, DM routing, approval buttons)
shared_static/ Shared design system (base.css, auth.js, theme.js, toast.js, utils.js, kb.js)
katex-0.17.0/ Vendored KaTeX math rendering library (MIT, woff2 fonts)
katex-0.16.45/ Vendored KaTeX math rendering library (MIT, woff2 fonts)
ui/
colors.py ANSI color constants with NO_COLOR support
markdown.py Streaming terminal markdown renderer (line-buffered)
@@ -101,7 +102,7 @@ turnstone/
renderer.js Markdown + LaTeX renderer (tables, nested lists, blockquotes, KaTeX math)
app.js Split-pane UI (Pane class, binary layout tree, SSE, tool approval)
tools/
*.json 16 tool schemas (OpenAI function-calling format + turnstone metadata)
*.json 19 tool schemas (OpenAI function-calling format + turnstone metadata)
```
Both UIs share a common design system extracted into `turnstone/shared_static/`: design tokens, login overlay, toast notifications, theme toggle, keyboard shortcuts, and utility functions. Each UI imports `base.css` and the shared JS modules at `/shared/`, then adds only page-specific code at `/static/`.
@@ -189,6 +190,7 @@ Phase 3: EXECUTE (parallel)
(cancel_event also checked per line — kills process group on cancel)
Final output (stdout + stderr) delivered via ui.on_tool_result(call_id, name, output)
call_id links tool_info items → streaming chunks → final result
For plan tool: post-execution gate via ui.on_plan_review()
```
### State Transitions
@@ -207,7 +209,7 @@ The engine emits state changes via `_emit_state()` which calls
"running" ---> tool execution
|
v
"attention" ---> waiting for user approval
"attention" ---> waiting for user approval / plan review
|
v
"running" ---> executing approved tools
@@ -229,13 +231,11 @@ The engine emits state changes via `_emit_state()` which calls
> See also: [Core Engine Classes diagram](diagrams/png/03-core-engine-classes.png)
Defined in `turnstone.core.session.SessionUI` as a `typing.Protocol` with 15
Defined in `turnstone.core.session.SessionUI` as a `typing.Protocol` with 14
methods. Every frontend must implement all of them.
```python
class SessionUI(Protocol):
def on_turn_start(self) -> None: ...
def on_turn_committed(self) -> None: ...
def on_thinking_start(self) -> None: ...
def on_thinking_stop(self) -> None: ...
def on_reasoning_token(self, text: str) -> None: ...
@@ -245,20 +245,13 @@ class SessionUI(Protocol):
def on_tool_result(self, call_id: str, name: str, output: str, *, is_error: bool = False) -> None: ...
def on_tool_output_chunk(self, call_id: str, chunk: str) -> None: ...
def on_status(self, usage: dict, context_window: int, effort: str) -> None: ...
def on_plan_review(self, content: str) -> str: ...
def on_info(self, message: str) -> None: ...
def on_error(self, message: str) -> None: ...
def on_state_change(self, state: str) -> None: ...
def on_rename(self, name: str) -> None: ... # propagate alias to tab/UI label
```
`on_turn_start` fires at the top of each iteration of the send-loop;
`on_turn_committed` fires immediately after `messages.append(assistant_msg)`.
`SessionUIBase` uses both to reset the per-turn inflight buffers
(`_ws_inflight_content` / `_ws_inflight_reasoning` / `_ws_inflight_seq`)
that fuel the SSE refresh-resume `in_progress_snapshot` event — see
the per-workstream events stream in
[`docs/api-reference.md`](api-reference.md#get-v1apiworkstreamsws_idevents).
`on_rename` is called by the `/name` command (on success) and after a successful `/resume` (if the resumed session has an alias or title). `WebUI.on_rename` broadcasts a `ws_rename` event on the global SSE channel and updates the in-memory `Workstream.name`; `TerminalUI.on_rename` is a no-op.
### Three Implementations
@@ -266,7 +259,7 @@ the per-workstream events stream in
| Class | Module | Notes |
|-------|--------|-------|
| `TerminalUI` | `turnstone.cli` | ANSI colors, `MarkdownRenderer`, `Spinner`, readline-based `input()` for approval |
| `WebUI` | `turnstone.server` | SSE event queue per workstream + global broadcast, `threading.Event` for blocking on approval. `on_state_change` sends to both per-workstream and global SSE (the browser UI uses per-workstream `state_change` events to manage busy/idle transitions; `stream_end` only finalizes markdown rendering). |
| `WebUI` | `turnstone.server` | SSE event queue per workstream + global broadcast, `threading.Event` for blocking on approval/plan. `on_state_change` sends to both per-workstream and global SSE (the browser UI uses per-workstream `state_change` events to manage busy/idle transitions; `stream_end` only finalizes markdown rendering). |
| `NullUI` | `turnstone.eval` | Discards all output; `approve_tools` always returns `(True, None)` |
### WorkstreamTerminalUI
@@ -278,9 +271,10 @@ awareness:
are appended to `_output_buffer` instead of written to stdout. When the user
switches to this workstream, `flush_buffer()` replays them.
- **Approval blocking**: `approve_tools()` calls `_fg_event.wait()` when in
background, blocking the worker thread until the workstream is foregrounded.
This ensures the user sees the approval prompt in the correct context.
- **Approval blocking**: `approve_tools()` and `on_plan_review()` call
`_fg_event.wait()` when in background, blocking the worker thread until the
workstream is foregrounded. This ensures the user sees the approval prompt
in the correct context.
- **Foreground/background toggle**: `set_foreground(bool)` sets or clears
`_fg_event` (a `threading.Event`). The manager calls this during `/ws <N>`
@@ -417,6 +411,7 @@ turnstone metadata keys:
| Metadata Key | Type | Meaning |
|-------------|------|---------|
| `agent` | `bool` | Include this tool when running as a plan/task sub-agent |
| `task_agent` | `bool` | Include this tool when running as a task sub-agent |
| `auto_approve` | `bool` | Tool is read-only; skip user approval |
| `primary_key` | `str` | Fallback argument name for bare-string JSON recovery |
@@ -436,6 +431,7 @@ Example (`read_file.json`):
},
"required": ["path"]
},
"agent": true,
"task_agent": true,
"auto_approve": true,
"primary_key": "path"
@@ -446,17 +442,19 @@ At import time, `turnstone.core.tools._load_tools()` strips the metadata keys
from each schema and builds:
- `TOOLS` -- list of `{"type": "function", "function": {...}}` dicts for the API
- `AGENT_TOOLS` -- subset with `agent: true`
- `TASK_AGENT_TOOLS` -- subset with `task_agent: true`
- `TASK_AUTO_TOOLS` -- set of tool names with `auto_approve: true`
- `AGENT_AUTO_TOOLS` / `TASK_AUTO_TOOLS` -- sets of tool names with `auto_approve: true`
- `PRIMARY_KEY_MAP` -- `{name: primary_key}` for JSON fallback recovery
- `merge_mcp_tools(builtin, mcp_tools)` -- merges built-in + MCP tools at session init
### 16 Tools by Category
### 19 Tools by Category
**Read-only (auto-approve)**:
- `read_file` -- read file contents with optional offset/limit
- `diff_file` -- show diff between two files / versions
- `search` -- ripgrep-based codebase search
- `man` -- read man pages
- `recall` -- search conversation history
- `read_resource` -- read an MCP resource by URI
@@ -464,21 +462,23 @@ from each schema and builds:
- `bash` -- execute shell commands (with safety checks via `turnstone.core.safety`)
- `write_file` -- create or overwrite a file
- `edit_file` -- string replacement in an existing file (requires prior `read_file`)
- `math` -- execute Python in sandboxed subprocess (via `turnstone.core.sandbox`)
- `web_fetch` -- fetch a URL (with SSRF protection via `turnstone.core.web`)
- `web_search` -- search the web (provider-native for Anthropic/OpenAI, self-hosted SearxNG fallback for local models)
- `web_search` -- search the web (provider-native for Anthropic/OpenAI, Tavily fallback for local models)
- `notify` -- send a user-facing notification (Discord/Slack, optional reply routing)
- `watch` -- schedule a recurring poll with condition DSL
**Agent (delegated sub-sessions)**:
- `task_agent` -- delegate to a sub-agent with full tool access (`TASK_AGENT_TOOLS`)
- `plan_agent` -- explore codebase and write a structured plan (`AGENT_TOOLS`)
**Memory / skills / prompts**:
- `memory` -- save, search, delete, or list memories (typed and scoped)
- `skill` -- invoke a skill (governed, versioned procedure)
- `use_prompt` -- fetch and apply a prompt template
The tool name uses the `_agent` suffix — bare `task` collides with
chat-template channels on some local models.
Tool names are `plan_agent` / `task_agent` (not `plan` / `task`); bare words
collide with chat-template channels on some local models.
### Prepare / Execute Pattern
@@ -497,11 +497,17 @@ separation allows the UI to show previews before any side effects occur.
### Agent Tools
`task_agent` invokes `_run_agent()`, which runs a multi-turn loop with a
subset of tools and its own system prompt. The sub-agent runs independently,
then returns the final content as the tool result.
`task_agent` and `plan_agent` invoke `_run_agent()`, which runs a multi-turn
loop with a subset of tools and its own system prompt. The sub-agent runs
independently, then returns the final content as the tool result.
- **task_agent**: uses `self._task_tools` (`TASK_AGENT_TOOLS` + MCP tools)
- **plan_agent**: uses `self._agent_tools` (`AGENT_TOOLS` + MCP tools). Writes output
to `.plan-<ws_id>.md` — unique per `ChatSession` so concurrent workstreams
don't collide. On repeat invocations the prior `plan_agent` tool call and its result
are forwarded from `self.messages` so the agent refines the existing plan rather
than starting over. Planning instructions are injected as a developer message
prepended to the agent's conversation.
- **Turn limit**: controlled by `agent_max_turns` (default: `-1`, unlimited).
When a limit is set and reached, the agent is forced to synthesize a final
response without tools. When unlimited, the loop only exits when the model
@@ -548,8 +554,8 @@ adds, removes, or reconnects servers as needed.
When tools change, `_rebuild_tools()` creates new `_tools`/`_tool_map` objects
(copy-on-write for thread safety) and notifies listener callbacks. Each `ChatSession`
rebuilds its `_tools` and `_task_tools` lists and reconstructs `ToolSearchManager`
(preserving expanded tools).
rebuilds its merged tool lists and reconstructs `ToolSearchManager` (preserving
expanded tools).
**Tool naming:** `mcp__{server}__{tool}` — double underscore delimiter, validated
at connection time (server names with `__` are rejected).
@@ -613,15 +619,14 @@ LLMProvider (protocol)
| `get_capabilities()` | Per-model flags (`ModelCapabilities`) |
| `convert_tools()` | Translate OpenAI tool schemas to provider format |
| `retryable_error_names` | Exception class names that trigger retry |
| `extract_reasoning_text()` | Walk stored `provider_blocks`, return concatenated reasoning text for UI rehydration (per-provider block-type knowledge: Anthropic `thinking`, OpenAI Responses `reasoning`, OpenAI Chat synthetic `reasoning_text`) |
**Normalized data types:**
| Type | Fields |
|------|--------|
| `StreamChunk` | `content_delta`, `reasoning_delta`, `tool_call_deltas`, `info_delta`, `usage`, `finish_reason`, `provider_blocks` |
| `CompletionResult` | `content`, `tool_calls`, `finish_reason`, `usage`, `provider_blocks` |
| `ModelCapabilities` | `context_window`, `max_output_tokens`, `supports_temperature`, `token_param`, `thinking_mode`, `supports_effort`, `supports_web_search`, `supports_tool_search`, `supports_vision`, `supports_reasoning_replay` |
| `StreamChunk` | `content_delta`, `reasoning_delta`, `tool_call_deltas`, `info_delta`, `usage`, `finish_reason` |
| `CompletionResult` | `content`, `tool_calls`, `finish_reason`, `usage` |
| `ModelCapabilities` | `context_window`, `max_output_tokens`, `supports_temperature`, `token_param`, `thinking_mode`, `supports_effort`, `supports_web_search`, `supports_tool_search`, `supports_vision` |
| `UsageInfo` | `prompt_tokens`, `completion_tokens`, `total_tokens`, `cache_creation_tokens`, `cache_read_tokens` |
**OpenAIProvider** (`_openai.py`): passes messages through unchanged (they are
@@ -634,7 +639,7 @@ annotations are formatted as footnotes. Extended prompt cache retention
(`prompt_cache_retention: "24h"`) is enabled for GPT-5.x models at no
additional cost. Cached token counts are extracted from
`usage.prompt_tokens_details.cached_tokens`. Unknown models (local servers) get
permissive defaults with `supports_vision=False` and use SearxNG for web search.
permissive defaults with `supports_vision=False` and use Tavily for web search.
**AnthropicProvider** (`_anthropic.py`): converts OpenAI-format messages to
Anthropic content blocks, maps `system`/`developer` roles to the `system`
@@ -652,8 +657,9 @@ display). Automatic prompt caching is enabled via top-level `cache_control:
cacheable block and advances it as conversations grow (90% input cost
reduction on cache hits, 1.25x write on first turn). Cache metrics
(`cache_creation_input_tokens`, `cache_read_input_tokens`) are extracted from
both streaming and non-streaming responses. The `anthropic` SDK is a core
dependency — the Anthropic provider is first-class alongside OpenAI.
both streaming and non-streaming responses. The `anthropic` SDK is imported
lazily so it remains an optional dependency (`pip install
turnstone[anthropic]`).
**GoogleProvider** (`_google.py`): extends `OpenAIChatCompletionsProvider` for
the Gemini `/v1beta/openai/` endpoint. Uses a single default
@@ -702,41 +708,12 @@ agent_model = "claude"
Each `[models.*]` entry produces a `ModelConfig` with a `provider` field
(default: `"openai"`). Supported values: `"openai"`, `"anthropic"`, `"google"`,
`"openai-compatible"`, and `"anthropic-compatible"`.
and `"openai-compatible"`.
**Per-model sampling overrides:** Each model can specify `temperature`,
`max_tokens`, and `reasoning_effort` to override the global defaults from
ConfigStore. When unset (`NULL`), the global default is used.
**Per-model reasoning persistence:** Two booleans on `model_definitions`
(migration 052) control how reasoning text round-trips:
* `surface_persisted_reasoning` (default `True`) — gates whether stored
reasoning text is surfaced on `/history` payloads for UI rehydration.
**Storage of reasoning bytes happens regardless of this flag** — they
ride in `provider_data` independently. Phase-1 admin UI label "Surface
persisted reasoning."
* `replay_reasoning_to_model` (default `False`) — gates whether stored
reasoning blocks are sent back to the provider on subsequent turns.
Capability-gated: `ModelCapabilities.supports_reasoning_replay` must
also be `True` for the wire path to actually replay (canonical OpenAI
gpt-5*/o-series and Anthropic Claude entries set it; unknown / local-
server models default to `False`).
Three reasoning paths are recognised:
| Path | Provider | Capture | Persist | Replay |
|------|----------|---------|---------|--------|
| 1 | Anthropic Messages API | `thinking_delta` | `provider_blocks` (`type="thinking"`) | Verbatim via `_provider_content` |
| 2 | OpenAI Responses (gpt-5*, o-series) | `response.reasoning_text.delta` events | `provider_blocks` (`type="reasoning"`) — only when `include=["reasoning.encrypted_content"]` | `ResponseReasoningItemParam` input items |
| 3 | OpenAI Chat Completions (vLLM, llama.cpp, Gemini-compat) | `delta.reasoning_content` Pydantic extras | Synthetic `{type: "reasoning_text", text, source}` block stamped at end-of-stream | None — no API surface for replay on Chat Completions |
Cross-provider safety is enforced by `ANTHROPIC_VALID_BLOCK_TYPES` (a
shape filter in `_anthropic.py:_convert_messages`): foreign blocks
(OpenAI `reasoning`, synthetic `reasoning_text`) fall through to the
text+tool_calls rebuild path rather than reaching Anthropic's input
boundary as malformed content.
```toml
[models.local]
base_url = "http://localhost:8000/v1"
@@ -765,63 +742,6 @@ model = "qwen-3.5-vl"
supports_vision = true
```
**Anthropic-compatible local servers (vLLM `/v1/messages`):** the
`"anthropic-compatible"` provider drives local servers that expose
Anthropic's Messages API for arbitrary checkpoints — vLLM's
`/v1/messages` endpoint, which requires a release with thinking-block
support in the Anthropic endpoint (post-2026-02-28; verified against
v0.22.1rc1). The lane reuses `AnthropicProvider` in compat mode: same
wire translation as the real Anthropic lane, but every model resolves to
the `_ANTHROPIC_COMPAT_DEFAULT` capabilities (200K context, 64K output,
`token_param=max_tokens`, `thinking_mode=none`, no native
web_search/tool_search, no vision) — the static Claude table never
applies to local checkpoints. `base_url` is required — the server root
WITHOUT `/v1` (the Anthropic SDK appends `/v1/messages`); a trailing
`/v1` pasted out of openai-compatible habit is stripped automatically,
and an empty value fails at client construction rather than falling
back to the commercial endpoint. Set a
placeholder `api_key` (e.g. `"dummy"`) for unauthenticated servers. Tool calling
needs the server started with `--enable-auto-tool-choice
--tool-call-parser <family>` plus the matching reasoning parser.
Per-model capability overrides opt in to what the checkpoint actually
supports:
```toml
[models.vllm-claude]
provider = "anthropic-compatible"
base_url = "http://localhost:8000" # no /v1 — the SDK appends /v1/messages
api_key = "dummy"
model = "deepseek-ai/DeepSeek-V4-Flash"
[models.vllm-claude.capabilities]
supports_vision = true # multimodal checkpoints only
supports_mid_conversation_system = true # template-dependent
context_window = 131072
```
The reasoning toggle does NOT use Anthropic's `thinking` request param.
Toggle it through the chat template instead: set `{"chat_template_kwargs":
{"thinking": false}}` as extra body params in the admin Models
server-compat section (for this provider the section shows only the
extra-body field — server type, API surface, and thinking mode are
openai-compatible-only knobs); the provider forwards it via the SDK's
`extra_body`.
Verified quirks of vLLM's Anthropic endpoint:
* The `thinking` request param is silently dropped — use
`chat_template_kwargs` (above) to control reasoning.
* `stop_sequences` cut the raw stream wherever the text appears —
including inside thinking — and report `end_turn` with
`stop_sequence=None`. Turnstone does not send stop sequences from
this provider.
* No cache telemetry: `usage` carries input/output token counts only
(no `cache_creation_input_tokens` / `cache_read_input_tokens`).
* Images require a multimodal checkpoint — text-only models return a
500 on image blocks, so `supports_vision` stays opt-in per model.
* Mid-conversation `role: "system"` turns are template-dependent —
opt in per model via `supports_mid_conversation_system`.
**Database model definitions:** On server entry points, models can also be
defined in the `model_definitions` table (admin Models tab). DB models support
the same per-model sampling overrides. Config.toml models override DB models
@@ -843,7 +763,7 @@ with the same alias in-memory (the DB rows are never modified).
parameters
6. `_create_stream_with_retry()` tries the primary model, then each fallback
alias in order if the primary is unreachable
7. `_run_agent()` resolves `registry.agent_model` (if set) for task
7. `_run_agent()` resolves `registry.agent_model` (if set) for plan/task
sub-agents, allowing a cheaper model for autonomous loops
**Per-workstream selection:** `POST /v1/api/workstreams/new` accepts an optional
@@ -852,7 +772,7 @@ which can override the model before workstream creation.
### Tool Output Truncation
Tool execution results (bash, read_file, search) are truncated by
Tool execution results (bash, read_file, search, math, man) are truncated by
`_truncate_output()` when they exceed `tool_truncation` characters. Truncation
preserves the first half and last half of the output, with a message in
between:
@@ -1098,12 +1018,11 @@ warns if the summary was truncated.
unhandled promise rejections
- **Pending approval across tab switches**: `WebUI._pending_approval` stores
the `approve_request` event payload while the session is blocked waiting
for user response. On tab switch / reconnect the pane reloads history via
REST `GET /history` and then reconnects SSE; the live approval event is
re-injected. The server-side `project_history_messages` projection marks
the trailing orphan tool-call turn `"pending": true` so `replayHistory`
skips the false `✓ approved` badge; the live approval UI is rendered by
the re-injected event instead.
for user response. On SSE reconnect (e.g., switching back to the tab),
the event is re-injected after history replay. `_build_history` marks the
pending tool call as `"pending": true` so `replayHistory` skips the
false `✓ approved` badge; the live approval UI is rendered by the
re-injected event instead.
- **Browser history integration**: `history.pushState` is called in
`switchTab()` with `{turnstone: 'workstream', wsId}`. The initial state is
seeded with `history.replaceState({turnstone: 'dashboard'})` on load. The
@@ -1191,8 +1110,7 @@ Three hierarchical scopes control endpoint access:
`/metrics`, `/openapi.json`, `/docs`, `/api/auth/*`, and `/api/auth/setup`
are always allowed.
2. **Token extraction**`Authorization: Bearer <token>` header first, then
surface-scoped auth cookie (`turnstone_auth_server` on the node server,
`turnstone_auth_console` on the console) as fallback.
`turnstone_auth` cookie as fallback.
3. **Token type detection** — dots in the token indicate JWT; `ts_` prefix
indicates API token.
4. **Validation** — JWT signature check or API token hash lookup in storage.
@@ -1281,6 +1199,7 @@ Starlette ASGI app (served by uvicorn)
+-- Async request handlers (all under /v1/ prefix)
| POST /v1/api/workstreams/{ws_id}/send -> starts worker thread per workstream
| POST /v1/api/workstreams/{ws_id}/approve -> unblocks WebUI._approval_event
| POST /v1/api/plan -> unblocks WebUI._plan_event
| POST /v1/api/workstreams/new -> creates workstream + worker
| GET /v1/api/workstreams/{ws_id}/events -> SSE via EventSourceResponse (per workstream)
| GET /v1/api/events/global -> SSE via EventSourceResponse (fan-out)
@@ -1290,7 +1209,7 @@ Starlette ASGI app (served by uvicorn)
|
+-- Worker thread per workstream (daemon)
| Runs session.send() synchronously -- ChatSession is fully blocking
| Blocks on WebUI._approval_event (threading.Event)
| Blocks on WebUI._approval_event / _plan_event (threading.Event)
|
+-- Background daemon threads
Global SSE fan-out: reads global_queue, copies to per-client queues
@@ -1314,7 +1233,7 @@ registry).
Each workstream's `WebUI` has:
- `_listeners` (per-client SSE queues, fan-out on `_enqueue()`)
- `_approval_event` (`threading.Event` for blocking)
- `_approval_event` / `_plan_event` (`threading.Event` for blocking)
- `_global_queue` (class variable, shared, for state broadcasts)
The SSE handlers bridge these sync queues to async via
@@ -1570,7 +1489,8 @@ implemented in `turnstone/core/judge.py`:
The judge is session-scoped (`IntentJudge`), lazy-initialized on first
approval, and configured via the `[judge]` config section or `--judge` CLI
flags. By default it uses self-consistency (same model), but supports
cross-model and cross-provider configurations. Task sub-agents are exempt. All verdicts are persisted to the `intent_verdicts` table
cross-model and cross-provider configurations. Sub-agents (plan, task)
are exempt. All verdicts are persisted to the `intent_verdicts` table
(migration 012) with the user's final decision, enabling future calibration.
The console exposes `GET /v1/api/admin/verdicts` for audit queries
(requires `admin.judge` permission).
+27 -21
View File
@@ -12,6 +12,7 @@ Existing bulk endpoints at time of writing:
|---------------------------------------------------------|--------------------------|------------------------------------------|
| `GET /v1/api/cluster/ws/live?ids=a,b,c` | bulk read | `{results, denied, truncated}` |
| model tool `spawn_batch` | bulk create (per-item) | `{results, denied}` |
| `POST /v1/api/workstreams/{ws_id}/stop_cascade` | cascade mutation | `{cancelled, failed, skipped}` |
| `POST /v1/api/workstreams/{ws_id}/close_all_children` | cascade mutation | `{closed, failed, skipped}` |
---
@@ -109,18 +110,11 @@ owns it; the node is just currently unreachable.
### Example — `spawn_batch`
This is the coordinator-tool result shape (the JSON the LLM receives),
not an HTTP API response — the table above keys it under "model tool"
to distinguish it from the `/v1/api/...` endpoints in the same table.
The underlying HTTP spawn endpoint still returns `ws_id`; the tool
result re-keys it to `child_ws_id` to defuse a coordinator-LLM recency
bias (see `docs/coordinator-skills.md`).
```json
{
"results": {
"0": {"child_ws_id": "d4e5f6...", "name": "csrf-audit", "node_id": "gpu-3"},
"2": {"child_ws_id": "f1a2b3...", "name": "xss-audit", "node_id": "gpu-1"}
"0": {"ws_id": "d4e5f6...", "name": "csrf-audit", "node_id": "gpu-3"},
"2": {"ws_id": "f1a2b3...", "name": "xss-audit", "node_id": "gpu-1"}
},
"denied": [
{"idx": 1, "reason": "skill not found: nonexistent-skill"}
@@ -145,7 +139,7 @@ consistently-typed across the read and create cases.
```
Where `<bucket>` is the endpoint-specific name for "succeeded" —
`closed` for `close_all_children`.
`cancelled` for `stop_cascade`, `closed` for `close_all_children`.
The three buckets partition the input set exactly once:
| Bucket | Meaning |
@@ -160,6 +154,20 @@ be partial. `skipped` is pre-resolved — the target is already in
the terminal state the cascade was aiming at, so it's neither a
win to report nor a fault to fix.
### Example — `stop_cascade`
```json
{
"status": "ok",
"cancelled": ["child-1", "child-3"],
"failed": [],
"skipped": ["child-2"]
}
```
A subsequent retry would target only `failed` ids, not `skipped`
ones — the latter are already done.
### Example — `close_all_children`
```json
@@ -171,12 +179,10 @@ win to report nor a fault to fix.
}
```
Here the success bucket is `closed`. A subsequent retry would
target only `failed` ids, not `skipped` ones — the latter are
already done. When `coord_client` is unavailable (session loaded
but no HTTP client attached — a construction bug) every id goes to
`failed` so the operator notices rather than getting a silent
all-skipped response.
Same partition, different success-bucket name. When `coord_client`
is unavailable (session loaded but no HTTP client attached — a
construction bug) every id goes to `failed` so the operator notices
rather than getting a silent all-skipped response.
---
@@ -219,12 +225,12 @@ all-skipped response.
- **Phase 6** shipped `cluster/ws/live` as the first Shape A endpoint
(`{results, denied, truncated}`).
- **Phase 7** introduced the Shape B cascade-mutation envelope
(`{<bucket>, failed, skipped}`) for the coordinator's
cancel-cascade path.
- **Phase 7** shipped `stop_cascade` as the first Shape B endpoint
(`{cancelled, failed, skipped}`).
- **Phase 8 PR A** shipped `spawn_batch` (Shape A, keyed by idx) and
`close_all_children` (Shape B), which crystallised the
two-shape-per-semantic-category policy codified here.
`close_all_children` (Shape B, twin of `stop_cascade`), which
crystallised the two-shape-per-semantic-category policy codified
here.
Before adding a third shape, read this doc and argue for why the
new surface doesn't fit either A or B. Two idioms in the cluster
+15 -5
View File
@@ -99,10 +99,10 @@ TURNSTONE_DISCORD_GUILD=123456789
Then start the stack:
```bash
docker compose up
docker compose --profile production up
```
The `channel` gateway runs by default; the Discord adapter activates once
The `channel` service starts automatically when
`TURNSTONE_DISCORD_TOKEN` is set.
### 3. Link User Accounts
@@ -179,6 +179,7 @@ both and the gateway hosts both adapters in one process.
see starts a per-user channel session.
- Tool approvals render as Slack **Block Kit** buttons; only the user
who owns the workstream can approve/reject.
- Plan reviews render as a modal with approve / request-changes actions.
- Notifications and reply routing work identically to Discord.
- Session recovery: persisted channel routes are re-subscribed when the
bot restarts, so existing Slack conversations keep flowing.
@@ -234,6 +235,15 @@ config, the bot auto-responds with approval and posts a
field (useful for allowing specific tools like `bash` or `read_file` while
still requiring manual approval for others).
### Plan Reviews
Plan review requests are displayed as a blue embed with:
- **Approve Plan** (green) button — approves the plan with empty feedback
- **Request Changes** (gray) button — opens a modal for feedback text
(up to 2000 characters)
- Feedback is forwarded to the server via HTTP
---
## Configuration Reference
@@ -421,9 +431,9 @@ message with a `ws_id` so that user replies can be routed back to the
originating workstream. Adapters must track the mapping from outgoing
message ID to `(ws_id, target_user_id)` and handle DM replies.
Platform-specific concerns — approval prompts, message edits, thread
creation — live inside the adapter implementation and are not part of
the protocol surface. Each adapter drives those via its
Platform-specific concerns — approval prompts, plan reviews, message
edits, thread creation — live inside the adapter implementation and are
not part of the protocol surface. Each adapter drives those via its
own `_on_ws_event` dispatcher using SDK-native APIs.
To add a new platform:
+51 -44
View File
@@ -18,7 +18,7 @@ schema changes.
> auth and the `admin.coordinator` permission. A session-scoped JWT
> is minted per login (see [docs/oidc.md](oidc.md) / [docs/security.md](security.md));
> a service token may call the read paths but destructive governance
> paths (`/restrict`, `/close_all_children`) require
> paths (`/restrict`, `/stop_cascade`, `/close_all_children`) require
> the explicit `admin.coordinator` grant — a service-token owner
> match isn't enough.
@@ -44,6 +44,7 @@ schema changes.
| 6 | Wait for fan-out | model-side tool `wait_for_workstream` |
| 7 | Govern | `POST /v1/api/workstreams/{ws_id}/trust` |
| | | `POST /v1/api/workstreams/{ws_id}/restrict` |
| | | `POST /v1/api/workstreams/{ws_id}/stop_cascade` |
| | | `POST /v1/api/workstreams/{ws_id}/close_all_children` |
| 8 | Approve / cancel | `POST /v1/api/workstreams/{ws_id}/approve` |
| | | `POST /v1/api/workstreams/{ws_id}/cancel` |
@@ -52,7 +53,7 @@ schema changes.
Refer to `/openapi.json` (Swagger UI at `/docs`) on any
`turnstone-console` process for the authoritative operation ids and
schemas. Coordinator-only verbs (`/children`, `/trust`, `/restrict`,
`/close_all_children`) 404 against `kind=interactive`
`/stop_cascade`, `/close_all_children`) 404 against `kind=interactive`
rows; the shared verbs (`/send`, `/approve`, `/cancel`, `/events`,
`/history`, `/open`, `/close`, etc.) work on both kinds.
@@ -114,8 +115,7 @@ with a `type` field. The recurring shapes a UI has to handle:
| `tool_output_chunk` | Streaming tool output (e.g. long bash command) | `call_id`, `chunk` |
| `approve_request` | One or more tool calls need operator approval | `items: [{call_id, header, preview, func_name, approval_label, needs_approval}]` |
| `approval_resolved` | Operator answered the approval prompt | `approved`, `feedback` |
| `state_change` | Worker-thread state transition (also re-emitted with the current state on every fresh subscribe so refresh-mid-stream restores composer mode) | `state``running`, `thinking`, `attention`, `idle`, `error` |
| `in_progress_snapshot` | One-shot replay of the in-progress turn's content + reasoning when this client connects mid-stream | `content`, `reasoning` |
| `state_change` | Worker-thread state transition | `state``running`, `thinking`, `attention`, `idle`, `error` |
| `status` | Token usage + context-window snapshot (fires on every streaming tick) | `prompt_tokens`, `completion_tokens`, `total_tokens`, `context_window`, `pct`, `effort`, `cache_creation_tokens`, `cache_read_tokens` |
| `rename` | Session's display name changed | `name` |
| `intent_verdict` | Intent judge produced a verdict on a pending tool call | `risk_level`, `recommendation`, `reasons` |
@@ -130,13 +130,9 @@ with a `type` field. The recurring shapes a UI has to handle:
**Reconnection contract:** a freshly-opened SSE connection receives
the current snapshot of any pending tool approval (`approve_request`
is re-sent if unresolved), any in-flight `wait_*` / `batch_*`
indicator, the worker's current `state_change`, and an
`in_progress_snapshot` carrying any partial content / reasoning the
model has produced for the in-progress turn — so a tab refresh
mid-approval, mid-tool-execution, or mid-stream restores both the
correct composer mode and the partial assistant text without waiting
for the response to complete.
is re-sent if unresolved) and any in-flight `wait_*` / `batch_*`
indicator — so a tab refresh mid-approval doesn't strand the
operator.
---
@@ -236,21 +232,14 @@ Key properties:
tool with a fresh timeout.
- **Modes**`mode="any"` returns as soon as one child reaches a
real terminal state (`idle` / `error` / `closed` / `deleted`);
`mode="all"` waits for every polled child to reach a real
terminal state.
`mode="all"` waits for every polled child.
- **Progress throttling** — the poll loop runs every 500 ms but the
SSE emission is diff-on-state-change plus a 5-second heartbeat. A
600 s wait generates O(dozens) of progress events, not 1200.
- **Unresolvable ids** — ws_ids are validated up front (exactly
32 hex chars; copy them verbatim): a malformed id fails the call
immediately with did-you-mean suggestions and a roster of the
coord's children. An id the caller doesn't own, a missing row, or
a child hard-deleted mid-wait is reported as `state="not_found"`
and aborts the wait on the tick that observes it (top-level
`error` / `not_found` / `children` fields, `complete=false`) — the
LLM should fix the id and re-issue, not conclude the child died.
Foreign and missing collapse into one shape, so the wait can't be
used as an existence oracle.
- **Denied rows** — an id the caller doesn't own (cross-tenant) or a
missing row is reported as a `denied` state in the results dict;
`mode="any"` won't satisfy on a pure-denied list (the LLM should
treat it as a config error, not a completion).
Prefer `wait_for_workstream` over polling `inspect_workstream` in a
loop — a wait consumes one assistant turn regardless of how long the
@@ -260,10 +249,10 @@ rounds to a 10× token-efficiency win.
---
## 7. Governance — trust, restrict, close_all_children
## 7. Governance — trust, restrict, stop_cascade, close_all_children
These three endpoints let an operator steer a live coordinator session
mid-flight. All three emit an audit event tagged
These four endpoints let an operator steer a live coordinator session
mid-flight. All four emit an audit event tagged
`coordinator.<action>` via the dedicated audit executor so a cascade
burst can't starve audit writes.
@@ -293,6 +282,28 @@ idempotent — calling twice with overlapping lists converges to the
union. Revocations don't survive a session close/reopen; operators
opt in per session. Cap 256 tool names per request, 128 chars each.
### `POST /stop_cascade` — cancel the subtree
```http
POST /v1/api/workstreams/{ws_id}/stop_cascade
{}
```
Cancels the coordinator's in-flight generation AND dispatches
`cancel_workstream` through the routing proxy for every direct
child in the in-memory registry. Returns:
```json
{"status": "ok", "cancelled": ["child-1", "child-3"], "failed": [], "skipped": ["child-2"]}
```
Response uses the [cascade-mutation bulk shape](bulk-endpoints.md):
`cancelled` = accepted, `failed` = dispatch error worth retrying,
`skipped` = upstream 404 (already gone — stale registry entry or
the row was deleted between snapshot and dispatch). Grandchildren
aren't touched directly; they sit behind their parent's cancel and
propagate via the child's SSE stream.
### `POST /close_all_children` — soft-close the direct fan-out
```http
@@ -306,16 +317,16 @@ Response:
{"status": "ok", "closed": ["c-1", "c-2"], "failed": [], "skipped": []}
```
Soft-close cascade bounded by a concurrency semaphore. The `reason`
(up to 512 chars) propagates into each closed child's audit +
`workstream_config` for postmortem. The model-facing tool that
pairs with this endpoint asks for a bounded teardown of the
coordinator's own fan-out. This *soft-closes*; to *cancel* the
fan-out instead, cancel the coordinator (§8) — a coordinator cancel
auto-cascades to its direct children.
Soft-close cascade bounded by the same semaphore as `stop_cascade`.
The `reason` (up to 512 chars) propagates into each closed child's
audit + `workstream_config` for postmortem. Unlike `stop_cascade`
this does NOT recurse into grandchildren — the model-facing tool
that pairs with this endpoint asks for a bounded teardown of the
coordinator's own fan-out. For a full-subtree teardown, use
`stop_cascade`.
See [bulk-endpoints.md](bulk-endpoints.md) for why `close_all_children`
uses the cascade-mutation shape and how it differs from the
See [bulk-endpoints.md](bulk-endpoints.md) for why both endpoints
share the cascade-mutation shape and how it differs from the
`spawn_batch` / `cluster/ws/live` shape.
---
@@ -333,10 +344,7 @@ POST /v1/api/workstreams/{ws_id}/approve
{"approved": true, "feedback": null, "always": true} // always-approve this tool name
```
`cancel` drops the coordinator's in-flight generation and, for a
coordinator, auto-cascades the cancel to its direct children:
`cancel_workstream` is dispatched through the routing proxy for
every direct child in the registry. The coordinator itself is left
`cancel` drops the in-flight generation but leaves the coordinator
idle and open for a fresh `send`:
```http
@@ -353,10 +361,9 @@ POST /v1/api/workstreams/{ws_id}/close
{}
```
Soft-closes the session — state persists, children keep running
(wind them down first with `close_all_children`, or by cancelling
the coordinator, which cascades the cancel to its direct children),
the worker thread exits, SSE streams send a final `stream_end` and
Soft-closes the session — state persists, children keep running (use
`close_all_children` or `stop_cascade` first to wind them down), the
worker thread exits, SSE streams send a final `stream_end` and
disconnect. The row is reopenable via
`POST /v1/api/workstreams/{ws_id}/open` so long as it hasn't been
deleted.
@@ -371,7 +378,7 @@ deleted.
- [bulk-endpoints.md](bulk-endpoints.md) — the two bulk-shape
idioms (`{results, denied, truncated}` vs
`{<bucket>, failed, skipped}`) used by `cluster/ws/live`,
`spawn_batch`, and `close_all_children`.
`spawn_batch`, `stop_cascade`, and `close_all_children`.
- [architecture.md](architecture.md) — cluster-wide architecture
including how coordinator sessions fit next to node-hosted
interactive workstreams.
+40 -75
View File
@@ -14,46 +14,34 @@ care about.
---
## `kind` — authored audience metadata
## The two-surface model
A row in `prompt_templates` carries a `kind` column (see
[`turnstone/core/skill_kind.py`](../turnstone/core/skill_kind.py);
migration 044 added the column). Three values:
| `SkillKind` enum | Stored as | Meaning |
|-------------------------|-----------------|----------------------------------------------------------------------------|
| `SkillKind.INTERACTIVE` | `"interactive"` | Authored for the interactive maker persona (single-workstream "do this"). |
| `SkillKind.COORDINATOR` | `"coordinator"` | Authored for the orchestrator persona (delegate, monitor, synthesise). |
| `SkillKind.ANY` | `"any"` | Either surface (or audience-neutral). Default on create. |
| `SkillKind` enum | Stored as | Visible in |
|----------------------|-----------------------------|---------------------------------------------------------------------------|
| `SkillKind.INTERACTIVE` | `"interactive"` | Only the interactive-session activation path. `list_skills` on a coord won't show it. |
| `SkillKind.COORDINATOR` | `"coordinator"` | Only the coordinator's `list_skills` tool. Hidden from interactive activation pickers. |
| `SkillKind.ANY` | `"any"` | Both surfaces. Default for legacy rows predating the classifier. |
The `kind` field is a `StrEnum` — drop-in `str` compatible — so DB
rows, JSON payloads, and `==` comparisons all work without translation
at the edge.
The `kind` field is a `StrEnum` — drop-in ``str`` compatible — so
DB rows, JSON payloads, and `==` comparisons all work without
translation at the edge.
**`kind` is metadata, not an enforcement boundary.** The model can
`skills(action='find')` across every kind from any session, `get` any
row by name, and `load` any visible skill regardless of session kind.
Actual runtime capability is gated by `allowed_tools` + `auto_approve`
on the skill and the operator's approval card on every `load` /
`spawn_workstream(skill=...)` decision — `kind` doesn't add or remove
any of that. It's a sorting / grouping / search-narrowing hint.
When a coordinator calls `list_skills`, the SQL filter narrows to
`kind IN ('coordinator', 'any')`. When an interactive session picks
a skill at activation, the filter narrows to
`kind IN ('interactive', 'any')`. A skill author tags once at
creation; the two surfaces stay partitioned without any
per-call filtering on the LLM side.
The opt-in filter is on `skills(action='find', kind='coordinator')`
(or `'interactive'`) — pass it when you want to narrow a catalog
browse to a specific authored audience. Omitting it (or passing
`kind='any'`) returns the full catalog. When supplied, the storage
filter widens to `[<kind>, 'any']` so audience-neutral rows remain
visible inside the narrowed view.
**Tagging a new skill as coordinator-targeted** — set `kind` to
`SkillKind.COORDINATOR` (or the literal `"coordinator"`) when you
`skills(action='create', kind='coordinator', ...)` or POST to
`/v1/api/admin/skills`. Use this to signal intent to other skill
authors and to make the orchestrator-targeted catalog easy to
browse — not to hide the skill from interactive sessions. Existing
rows default to `SkillKind.ANY`; bump them to `COORDINATOR` if
you've rewritten the prompt around the orchestrator toolset and
want the kind filter to surface them as such.
**Tagging a new skill as coordinator-only** — set `kind` to
`SkillKind.COORDINATOR` (or the literal string `"coordinator"`) when
you POST to `/v1/api/admin/skills`. Existing rows default to
`SkillKind.ANY`; bump them to `COORDINATOR` if you've rewritten the
prompt around the orchestrator toolset.
---
@@ -76,9 +64,7 @@ or MCP config can do adds to it. Current members:
| `cancel_workstream` | wind-down | Drop the in-flight generation; leaves child idle for a fresh send. |
| `delete_workstream` | wind-down | Hard-delete one child. Requires approval. |
| `list_nodes` | discover | Enumerate live cluster nodes + capabilities. |
| `skills` (action=find) | discover | Browse the skill catalog; opt-in `kind` filter narrows by audience. |
| `memory` | persist | Durable orchestration memory (`coordinator` scope, per-user — survives across coordinator sessions). |
| `notify` | broadcast | Post a status update to a human channel at a narrative beat. |
| `list_skills` | discover | Coordinator-visible skills only (SkillKind filter above). |
| `tasks` | plan | Orchestrator-only scratchpad. Children don't see it. |
Explicitly **not** in the coordinator set:
@@ -86,8 +72,8 @@ Explicitly **not** in the coordinator set:
- `bash` / `edit_file` / `write_file` / `append_file` / `diff_file` — no local FS.
- `read_file` / `search` — no local FS reads.
- `web_fetch` / `web_search` — no direct web access.
- `task_agent` — sub-agent tool is zeroed on coord sessions.
- `recall` / `watch` / `read_resource` / `use_prompt` — UX / persistence tools that belong to interactive sessions. The dual-kind `memory` / `skills` / `notify` tools are available on both kinds (see the table above).
- `task_agent` / `plan_agent` — sub-agent tools are zeroed on coord sessions.
- `memory` / `recall` / `notify` / `watch` / `read_resource` / `use_prompt` / `skill` — the orchestrator's "memory" is its children's outputs; these UX / persistence tools belong to interactive sessions.
If your skill needs a coordinator to "run a command" or "read a
file", write the delegate pattern instead: spawn a child with an
@@ -168,50 +154,29 @@ Every ws_id returned by `spawn_workstream` / `spawn_batch` is a
invent ws_ids — a model that hallucinates `"child-1"` or `"ws-abc"`
hits the tenant guard in `CoordinatorClient._is_own_subtree`, which
validates ws_id against `parent_ws_id=coord_ws_id` AND
`user_id=owner` in storage. The rejection shape is uniform and
recovery-oriented:
`user_id=owner` in storage. The rejection shape varies by tool:
- **Mutating ops** (`send_to_workstream`, `close_workstream`,
`cancel_workstream`, `delete_workstream`) and
**`inspect_workstream`** return
`{"error": "no workstream matching '<ref>' among your children; …",
"status": 404, "ws_id": "<ref>", "did_you_mean": [...],
"children": [...], "children_truncated": bool}` — a did-you-mean
(edit distance ≤ 3 against the coord's own children, which catches
the garbled-hex incident class: a 32-char id whose `aaa` run
collapsed to `a`) plus a roster of the coord's children. A ref
that matches a child's display NAME is called out explicitly with
the right id (names are mutable labels, not addresses). Foreign
and nonexistent ids produce the same payload (no existence
oracle), every hint references only the coord's own children, and
near-miss ids are never auto-resolved — the skill should fix the
id and re-issue, not treat the child as dead.
- **`wait_for_workstream`** validates ids before waiting: a
malformed id fails the whole call immediately (`invalid_ws_ids`
carries the per-id payloads above, `elapsed=0`); a well-formed id
that is foreign, nonexistent, or hard-deleted mid-wait surfaces as
`state="not_found"` and aborts the wait on that tick with
top-level `error` / `not_found` / `children` fields.
`complete=true` therefore means every polled lane really finished
— an unobservable id can neither burn the timeout nor ride along
to a "complete" result.
`cancel_workstream`, `delete_workstream`) return
`{"error": "workstream not in coordinator subtree: <ws_id>", "status": 404}`
— the skill should treat this as a tool error, not an empty result.
- **`inspect_workstream`** returns `{"error": "workstream not found", "ws_id": "<ws_id>"}`
(same shape as a genuinely missing row, so the guard can't be
used as an existence oracle).
- **`wait_for_workstream`** reports the offending id with
`state="denied"` in its `results` dict; `mode="any"` won't
satisfy on a pure-denied list, so a hallucinated id won't trick
the wait into reporting "complete".
Pattern: capture each spawn result in the next tool call's input.
The JSON tool-result carries `{"child_ws_id": "...", "name": "...",
The JSON tool-result carries `{"ws_id": "...", "name": "...",
"node_id": "...", "routing_strategy": "..."}`; the model should
extract the `child_ws_id` and pass it as `ws_id` (or in the `ws_ids`
list) to `inspect_workstream` / `wait_for_workstream` /
`send_to_workstream` / `close_workstream` verbatim. The asymmetry
— spawn returns `child_ws_id` but the other tools accept `ws_id` /
`ws_ids` — is intentional: it defuses a coordinator-LLM recency
bias where seeing `ws_id` in a spawn return primed re-spawn loops
instead of progression to the wait phase.
extract the ws_id and pass it to `inspect_workstream` /
`wait_for_workstream` / `send_to_workstream` / `close_workstream`
verbatim.
A UI that wants human-readable identifiers should render the `name`
field and keep the workstream id as the click-through key — note
that the id *value* is the same regardless of whether it arrived
under the `child_ws_id` key (spawn return) or the `ws_id` key
(every other tool's input/output); only the field name differs.
field and keep the ws_id as the click-through key.
---
@@ -351,7 +316,7 @@ persona drift without a real LLM in the loop.
`spawn_batch` and `close_all_children` use, so your skill can
parse results / denied arrays correctly.
- [governance.md](governance.md) — the broader governance surface
(`/trust`, `/restrict`, role-based permissions)
(`/trust`, `/restrict`, `/stop_cascade`, role-based permissions)
that wraps every coord session.
- [settings.md](settings.md) — `coordinator.model_alias` and
`coordinator.reasoning_effort` settings that gate which LLM runs
+2
View File
@@ -34,6 +34,7 @@ package "turnstone/core/" <<Rectangle>> {
component [metrics.py\nPrometheus metrics] as metrics <<core>>
component [config.py\nTOML config] as config <<core>>
component [safety.py\nPath validation] as safety <<core>>
component [sandbox.py\nCommand sandbox] as sandbox <<core>>
component [edit.py\nFile editing] as edit <<core>>
component [web.py\nWeb helpers] as web <<core>>
component [auth.py\nAuthentication] as auth <<core>>
@@ -122,6 +123,7 @@ session --> tools
session --> memory
memory --> storage
session --> safety
session --> sandbox
session --> edit
session --> web
session --> healthcheck
+7 -5
View File
@@ -15,6 +15,7 @@ interface "SessionUI" as SessionUI <<Protocol>> {
+ on_tool_result(call_id: str, name: str, output: str, *, is_error: bool = False)
+ on_tool_output_chunk(call_id: str, chunk: str)
+ on_status(usage: dict, ctx_window: int, effort: str)
+ on_plan_review(content: str) → str
+ on_info(message: str)
+ on_error(message: str)
+ on_state_change(state: str)
@@ -42,13 +43,15 @@ class "WorkstreamTerminalUI" as WsTermUI {
class "WebUI" as WebUI {
- _listeners: list[Queue]
- _approval_event: Event
- _plan_event: Event
- _ws_prompt_tokens: int
- _ws_tool_calls: dict
+ resolve_approval(approved, feedback)
+ resolve_plan(feedback)
--
Enqueues JSON events for SSE.
Blocks on threading.Event for
approval.
approval/plan review.
SSE handlers bridge Queue to
async via run_in_executor().
--
@@ -66,10 +69,9 @@ class "NullUI" as NullUI {
interface "LLMProvider" as LLMProvider <<Protocol>> {
+ provider_name: str {property}
+ get_capabilities(model) → ModelCapabilities
+ create_streaming(client, model, messages, ..., replay_reasoning_to_model) → Iterator[StreamChunk]
+ create_completion(client, model, messages, ..., replay_reasoning_to_model) → CompletionResult
+ create_streaming(client, model, messages, ...) → Iterator[StreamChunk]
+ create_completion(client, model, messages, ...) → CompletionResult
+ convert_tools(tools) → list[dict]
+ extract_reasoning_text(provider_blocks) → str
+ retryable_error_names: frozenset[str] {property}
--
core/providers/_protocol.py
@@ -124,7 +126,6 @@ class "ModelCapabilities" as ModelCaps <<frozen>> {
+ supports_web_search: bool
+ supports_tool_search: bool
+ supports_vision: bool
+ supports_reasoning_replay: bool
}
' ChatSession
@@ -142,6 +143,7 @@ class "ChatSession" as ChatSession {
+ model_alias: str | None {property}
- _tools: list[dict]
- _task_tools: list[dict]
- _agent_tools: list[dict]
- _read_files: set[str]
- system_messages: list[dict]
--
+3 -18
View File
@@ -24,14 +24,6 @@ CS -> DB : save_message(ws_id, "user", input)
group loop [while tool_calls present]
CS -> UI : on_turn_start()
note right of UI
SessionUIBase resets the per-turn inflight
buffers (_ws_inflight_content / reasoning /
seq) that fuel the SSE in_progress_snapshot
event for mid-stream refresh resume.
end note
CS -> UI : on_state_change("thinking")
CS -> UI : on_thinking_start()
@@ -81,14 +73,6 @@ group loop [while tool_calls present]
CS -> CS : _update_token_table()\ncalibrate chars_per_token ratio
CS -> CS : messages.append(assistant_msg)
CS -> UI : on_turn_committed()
note right of UI
Drops the per-turn inflight buffers — the
assistant message is now in the history
list, so the in_progress_snapshot must
not re-render it during the next tool-
execution window or the next streaming turn.
end note
CS -> DB : save_message(ws_id, "assistant", content)
CS -> DB : save_message(ws_id, "tool_call", ...) ×N
@@ -139,9 +123,10 @@ group loop [while tool_calls present]
read_file → open().read() or base64 image
search → grep subprocess
edit_file → string replace
task → _run_agent() sub-loop
task/plan → _run_agent() sub-loop
math → sandboxed subprocess
web_fetch → httpx + LLM summarize
web_search → provider-native or SearxNG fallback
web_search → provider-native or Tavily fallback
memory/recall → SQLite
end note
+13 -2
View File
@@ -24,7 +24,7 @@ partition "Phase 1: Prepare" #E8F5E9 {
:Dispatch to _prepare_{func_name}();
note right
**Dispatch table (16 built-in + tool_search):**
**Dispatch table (19 built-in + tool_search):**
┌───────────────┬──────────────────┐
│ Tool │ Needs Approval? │
├───────────────┼──────────────────┤
@@ -34,10 +34,13 @@ partition "Phase 1: Prepare" #E8F5E9 {
│ edit_file │ ✓ Yes │
│ search │ ✗ Auto-approve │
│ diff_file │ ✗ Auto-approve │
│ math │ ✗ Auto-approve │
│ man │ ✗ Auto-approve │
│ web_fetch │ ✗ Auto-approve │
│ web_search │ ✗ Auto-approve │
│ tool_search │ ✗ Auto-approve │
│ task_agent │ ✓ Yes │
│ plan_agent │ ✓ Yes │
│ memory │ ✗ Auto-approve │
│ recall │ ✗ Auto-approve │
│ notify │ ✗ Auto-approve │
@@ -107,10 +110,13 @@ partition "Phase 3: Execute" #E3F2FD {
├─ _exec_write_file: makedirs + write
├─ _exec_edit_file: find_occurrences + replace
├─ _exec_search: grep subprocess
├─ _exec_math: sandboxed subprocess
├─ _exec_man: man/info subprocess
├─ _exec_web_fetch: httpx.get + LLM summary
├─ _exec_web_search: SearxNG JSON GET (fallback for local models)
├─ _exec_web_search: Tavily API POST (fallback for local models)
├─ _exec_tool_search: BM25 search + expand_visible()
├─ _exec_task: _run_agent(TASK_AGENT_TOOLS)
├─ _exec_plan: _run_agent(AGENT_TOOLS, read-only)
├─ _exec_notify: HTTP POST to channel gateway
├─ _exec_memory: structured memory save/search/delete/list
├─ _exec_recall: conversation history FTS5 search
@@ -125,6 +131,11 @@ partition "Phase 3: Execute" #E3F2FD {
:bash: ui.on_tool_output_chunk(call_id, line) per stdout line;
:ui.on_tool_result(call_id, name, output, is_error) for each;
if (plan tool was executed?) then (yes)
:ui.on_plan_review(output);
:Block for user review/feedback;
endif
}
:Return (results, user_feedback);
+4 -2
View File
@@ -13,7 +13,7 @@ skinparam state {
state "IDLE" as idle <<idle>> : Waiting for user input.\nNo active LLM call or tool execution.
state "THINKING" as thinking <<thinking>> : LLM streaming response.\nTokens flowing (reasoning + content).
state "RUNNING" as running <<running>> : Tools executing.\nThreadPoolExecutor active.
state "ATTENTION" as attention <<attention>> : Blocked on user action.\nTool approval needed.
state "ATTENTION" as attention <<attention>> : Blocked on user action.\nTool approval or plan review needed.
state "ERROR" as error <<error>> : Exception occurred.\nRecoverable on next send().
[*] --> idle : Session created
@@ -34,6 +34,8 @@ attention --> running : User denies\n(denial recorded)\n_emit_state("running")
running --> thinking : Tool results appended,\nnext LLM call\n_emit_state("thinking")
running --> attention : Plan tool complete,\non_plan_review()\n_emit_state("attention")
running --> error : Exception during\ntool execution
error --> thinking : New send() call\n_emit_state("thinking")
@@ -42,7 +44,7 @@ thinking --> idle : cancel() called\nstream aborted\n_emit_state("idle")
running --> idle : cancel() called\n_emit_state("idle")
attention --> idle : cancel() unblocks\napproval wait\n_emit_state("idle")
attention --> idle : cancel() unblocks\napproval/plan wait\n_emit_state("idle")
note left of idle
**Cancel escalation:**
+1
View File
@@ -30,6 +30,7 @@ package "turnstone/sdk/ (Python)" {
+ close_workstream()
+ send(message, ws_id)
+ approve()
+ plan_feedback()
+ command()
+ cancel(ws_id)
+ stream_events(ws_id)
+1 -1
View File
@@ -202,7 +202,7 @@ note over Session, Judge
Cross-model: separate provider/client from [judge] config.
**Sub-agent exemption:**
Task sub-agents skip intent validation entirely.
Plan agent and task agent skip intent validation entirely.
**Output guard:**
Runs when judge_config.output_guard is true (default).
+2 -2
View File
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:9857db23fe3c4316d492073aac69c7e7558b1abe3b95ad7756d4a5933bd0ece7
size 620214
oid sha256:25b5448bbb7da8ddafe4f65c6c5e6cbcaa9cb9f31746ca46d3a2241bc47b1956
size 259687
+2 -2
View File
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:d9c7769a600c38e6387390e6c42db8152e0f80c31d17b2218f7f636b71c7b868
size 355459
oid sha256:3aa8d972bba40d78152f9f0c762b9f5ec616d8052c45fa52b7dd1c679ed81d61
size 325245
+2 -2
View File
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:23ca090b5656baaf70820cbe4ab6c27f0a3a02e18b4db0695614cf9489c23980
size 281440
oid sha256:674712a0563f51837383184652efeb28b7bec13378be636e89d2959bfba39d1e
size 281519
+110 -230
View File
@@ -1,284 +1,164 @@
# Docker Deployment
Turnstone ships two Docker Compose stacks:
Docker Compose stack for running the full turnstone platform.
| Stack | File | Use it for |
|-------|------|------------|
| **Dev cluster** | `compose.yaml` (repo root) | Clone-and-run. Builds locally, zero config, full 10-node cluster. |
| **Production** | `turnstone/deploy/compose.yaml` | Pip/pipx installs. Pulls released images from ghcr.io, requires real secrets. |
## Quick start — local cluster
## Quick Start
```bash
git clone https://github.com/turnstonelabs/turnstone
cd turnstone
# Copy and edit environment config
cp .env.example .env
# Full stack (needs an LLM API on the host)
docker compose up
```
That builds one image and brings up the whole stack: PostgreSQL, the console,
Caddy, the channel gateway, and **10 server nodes** (`node-1``node-10`). No
`.env` is required — it ships with insecure dev defaults so it just works.
Console dashboard: http://localhost:8090
Open the dashboard at **https://localhost:8443**. It's served by Caddy with its
own local CA, so trust the root certificate once (or click through the browser
warning):
> See also: [Deployment diagram](diagrams/png/12-deployment.png)
## Services
| Service | Port | Profile | Description |
|---------|------|---------|-------------|
| `server` | 8080 | default | Web UI + chat workstreams + LLM |
| `console` | 8090 | default | Cluster dashboard |
| `channel` | — | production | Channel gateway (Discord and/or Slack adapters) |
| `server-1``server-10` | — | cluster | 10-node server fleet (PostgreSQL required) |
## Profiles
**Default** (no flag) — starts `server` and `console`. Requires an OpenAI-compatible LLM API running on the host (default: `http://localhost:8000/v1`).
```bash
docker compose exec caddy cat /data/caddy/pki/authorities/local/root.crt
docker compose up
```
Create your first admin user (any node works — they share one database):
**Production** — adds PostgreSQL and the channel gateway. Requires `POSTGRES_PASSWORD` and (for Discord) `TURNSTONE_DISCORD_TOKEN`:
```bash
docker compose exec node-1 turnstone-admin create-user --username admin --name "Admin"
docker compose --profile production up
```
### Bring your own LLM
Nodes boot **without** an LLM and appear in the console immediately. Add real
model backends (OpenAI, Anthropic, or a local/vLLM endpoint) from the console
UI's **Models** tab. To set a node's bootstrap default instead, point
`LLM_BASE_URL` / `OPENAI_API_KEY` at an OpenAI-compatible endpoint in `.env`.
### Fewer nodes
Ten nodes is heavy on a laptop. Start a subset by naming the services (always
include `postgres`, `console`, and `caddy`):
**Cluster** — 10-node server fleet sharing PostgreSQL. Access all nodes via the console at `:8090`. Requires `POSTGRES_PASSWORD`:
```bash
docker compose up postgres console caddy channel node-1 node-2 node-3
docker compose --profile cluster up
```
## Why HTTPS-only?
The console's plain-HTTP port (8090) is **not** published to the host. A plain
HTTP/1.1 origin caps the browser at 6 connections, which starves the
dashboard's per-pane SSE streams. Caddy serves the browser over HTTP/2
(multiplexed) and proxies to `console:8090` on the internal network, so the cap
is gone. Everything goes through `https://localhost:8443`.
## Join a bare-metal host
PostgreSQL, the console's ACME endpoint (`:8090`), and SearxNG (`:8081`) are
published on `127.0.0.1`, so a `turnstone-server` running directly on the same
machine — for example to use a local GPU — can join the same cluster (enrolling
its mTLS cert and running `web_search`) and show up in the console alongside the
containerized nodes.
Put the secret and connection settings in `~/.config/turnstone/config.toml`
(secrets belong in this file, not the process environment — keep it `0600`,
the loader warns otherwise):
```toml
[auth]
jwt_secret = "dev-only-insecure-jwt-secret-change-me-for-real-deployments"
[database]
backend = "postgresql"
url = "postgresql+psycopg://turnstone:turnstone@localhost:5432/turnstone"
[api]
base_url = "http://localhost:8000/v1" # your local model endpoint
api_key = "dummy"
```
Then start the server. The node identity isn't a secret, so it stays on the
command line:
```bash
chmod 600 ~/.config/turnstone/config.toml
TURNSTONE_NODE_ID=host-1 \
TURNSTONE_ADVERTISE_URL=http://host.docker.internal:8080 \
TURNSTONE_CONSOLE_URL=http://localhost:8090 \
TURNSTONE_SEARXNG_URL=http://localhost:8081 \
turnstone-server --host 0.0.0.0 --port 8080
```
The host server registers itself in PostgreSQL; the console reaches it back via
`host.docker.internal`. `TURNSTONE_CONSOLE_URL` points the node at the console's
published ACME endpoint so it can enroll its mTLS certificate (needed only when
the cluster runs mTLS; harmless otherwise), and `TURNSTONE_SEARXNG_URL` points
`web_search` at the published SearxNG. The `jwt_secret` and DB credentials above
are the dev-stack defaults — match whatever you set in `.env` if you changed them.
To let a server on a **different** machine join, start the stack with
`TURNSTONE_HOST_IP=<this host's LAN IP>` — that binds PostgreSQL, the console
ACME endpoint, and SearxNG to that interface. Then on the remote box set the
three URLs above to that IP, and set `TURNSTONE_ADVERTISE_URL` to the **remote**
box's own IP (the address the console dials back). **Set a strong
`POSTGRES_PASSWORD` first** — `TURNSTONE_HOST_IP` exposes the database (and every
user account + API-token hash in it), the console API, and the unauthenticated
SearxNG to your network.
To run the bare-metal node as a hardened, persistent service instead of by hand,
use the systemd units in [`deploy/systemd/`](../deploy/systemd/).
## Production stack
For a real deployment use the bundled stack, which pulls released images
instead of building:
```bash
docker compose -f turnstone/deploy/compose.yaml up
```
It's the same shape as the dev stack — Caddy-fronted console, channel, and a
PostgreSQL all share one database so the console discovers the node — but it
pulls released images, runs a single server node, and has **no baked-in
secrets**. Set these in `.env` first (generate with `openssl rand -hex 32`):
```bash
TURNSTONE_JWT_SECRET=<python -c "import secrets; print(secrets.token_hex(32))">
POSTGRES_PASSWORD=<a strong password>
```
The dashboard is at **https://localhost:8443** (Caddy, same as the dev stack);
the console's HTTP port isn't published. For a real domain and a publicly
trusted cert, edit `turnstone/deploy/Caddyfile` to point Caddy at Let's Encrypt
(see [tls.md](tls.md)). Pin the image with `TURNSTONE_IMAGE_TAG` (default:
`latest`).
### mTLS
Layer the TLS overlay on the production stack to enable mutual TLS between
services. A bootstrap container creates a CA and every service auto-provisions
certs via the console's ACME endpoint:
```bash
docker compose -f turnstone/deploy/compose.yaml -f deploy/docker-compose.tls.yml up
```
See [tls.md](tls.md) for details.
## Configuration
Everything is configured with environment variables in `.env` (copy from
[`.env.example`](../.env.example)). The dev stack needs none of them — they're
overrides.
All configuration is via environment variables in `.env` (copy from `.env.example`):
### LLM backend
### LLM Backend
| Variable | Default | Description |
|----------|---------|-------------|
| `LLM_BASE_URL` | `http://host.docker.internal:8000/v1` | Bootstrap OpenAI-compatible API URL (real backends go in the UI) |
| `LLM_BASE_URL` | `http://host.docker.internal:8000/v1` | OpenAI-compatible API URL |
| `OPENAI_API_KEY` | `dummy` | API key (`dummy` for local servers) |
| `TURNSTONE_SEARXNG_URL` | `http://searxng:8080` | SearxNG URL for the `web_search` tool (local/vLLM models only; Anthropic/OpenAI use native search). Defaults to the bundled `searxng` service; set to an external instance's URL. To turn web search off, clear `tools.searxng_url` in the admin Settings tab. |
| `SEARXNG_IMAGE_TAG` | `latest` | Tag for the bundled `searxng/searxng` image |
| `MODEL` | — | Override the default model alias |
| `TAVILY_API_KEY` | — | Web search API key (only needed for local/vLLM models; Anthropic and OpenAI search models use native search) |
### Auth & database
| Variable | Default (dev / prod) | Description |
|----------|----------------------|-------------|
| `TURNSTONE_JWT_SECRET` | insecure default / **required** | JWT signing secret. Every service must share one value. |
| `TURNSTONE_DB_BACKEND` | `postgresql` | `sqlite` or `postgresql`. Multi-node discovery requires `postgresql`. |
| `TURNSTONE_DB_URL` | bundled Postgres | SQLAlchemy URL. Override to use an external database. |
| `POSTGRES_USER` | `turnstone` | PostgreSQL username |
| `POSTGRES_PASSWORD` | `turnstone` / **required** | PostgreSQL password |
| `POSTGRES_MAX_CONNECTIONS` | `300` | `max_connections` for the bundled Postgres |
> **Discovery needs a shared database.** Each server registers and heartbeats
> into a `services` table that the console polls. All services in these stacks
> point at the same PostgreSQL by default; SQLite-per-container can't see other
> containers.
> **Large clusters:** each process keeps a small pool (5 max). Beyond ~50 nodes,
> put [PgBouncer](pgbouncer.md) (transaction pooling) between turnstone and
> PostgreSQL.
### Ports
Both stacks publish Caddy (dashboard) and PostgreSQL; the dev stack additionally
publishes the console's ACME endpoint and SearxNG on localhost so a bare-metal
node can enroll its cert and run `web_search`. Everything else is reached through
Caddy or proxied by the console:
### Server
| Variable | Default | Description |
|----------|---------|-------------|
| `CONSOLE_HTTPS_PORT` | `8443` | Host port for Caddy (dashboard HTTPS) |
| `SEARXNG_HTTPS_PORT` | `8444` | Host port for the SearxNG UI via Caddy (dev: localhost-only; prod: opt-in) |
| `POSTGRES_PORT` | `5432` | Host port for PostgreSQL (for bare-metal joins) |
| `SEARXNG_API_PORT` | `8081` | Host port for the SearxNG API a bare-metal node's `web_search` dials (dev stack) |
| `TURNSTONE_HOST_IP` | `127.0.0.1` | Interface PostgreSQL, the console ACME endpoint, and SearxNG bind on (dev stack). Set to this host's LAN IP so a bare-metal node on **another machine** can reach them — set a strong `POSTGRES_PASSWORD` first (it also exposes the DB and the unauthenticated SearxNG to your network). |
| `POSTGRES_BIND` | `127.0.0.1` | Production stack (`turnstone/deploy/compose.yaml`) only: interface PostgreSQL binds on; set to the host's LAN IP for remote joins. |
| `SERVER_PORT` | `8080` | Host port mapping |
| `SKIP_PERMISSIONS` | — | Set to any value to auto-approve all tools |
### Channel gateway
### Console
| Variable | Default | Description |
|----------|---------|-------------|
| `TURNSTONE_DISCORD_TOKEN` | — | Discord bot token (enables the Discord adapter) |
| `TURNSTONE_DISCORD_GUILD` | `0` | Restrict to one guild (0 = all) |
| `TURNSTONE_SLACK_TOKEN` | — | Slack Bot User OAuth token `xoxb-…` |
| `TURNSTONE_SLACK_APP_TOKEN` | — | Slack App-Level token `xapp-…` (with the Slack token) |
| `CONSOLE_PORT` | `8090` | Host port mapping |
The channel runs HTTP-only with no adapters until a token is set, so it's safe
to leave running. See [Channel Integrations](channels.md) for app setup.
### Auth
### Web search (SearxNG)
The `web_search` tool for local/vLLM models is backed by a self-hosted
[SearxNG](https://searxng.org) metasearch service, bundled into both stacks as the
`searxng` service. The Turnstone nodes reach it over the internal docker network at
`http://searxng:8080` — its API port is **not** published. Its config —
[`turnstone/deploy/searxng/settings.yml`](../turnstone/deploy/searxng/settings.yml),
mounted read-only — enables the JSON API and leaves the rate limiter off (the
limiter would need a separate Valkey/Redis instance). A `searxng-cache` volume
persists its favicon + internal cache across restarts. Commercial providers
(Anthropic, OpenAI) use their own native search and never touch this service.
Point at an existing SearxNG instead of the bundled one with `TURNSTONE_SEARXNG_URL`,
or narrow the engines via `tools.searxng_engines` in the admin Settings tab (e.g.
`duckduckgo,wikipedia`).
**SearxNG web UI.** Caddy can also serve SearxNG's own search/Preferences UI on a
dedicated port. The dev stack publishes it at **`https://localhost:8444`** bound to
localhost only; the production stack does **not** publish it by default (uncomment
the `8444` port on the `caddy` service to opt in). Change the port with
`SEARXNG_HTTPS_PORT`. **SearxNG has no authentication** — never bind this to a public
interface, or anyone who can reach it can search through your instance.
> **AGPL note for operators.** SearxNG is licensed AGPL-3.0. Kept on the internal
> network (or bound to localhost), no external user interacts with it — so the AGPL
> §13 (remote network interaction) source-offer obligation does not attach. If you
> publish SearxNG to remote users (bind its port to a public interface, or front it
> with your own reverse proxy) you become the operator of a network-reachable AGPL
> service and must offer its corresponding source; that is trivially satisfied by
> linking to upstream <https://github.com/searxng/searxng>. Turnstone's own license is
> unaffected: it talks to SearxNG over HTTP as a separate process (mere aggregation),
> not by linking.
### Other
Auth is always enabled. `TURNSTONE_JWT_SECRET` is required.
| Variable | Default | Description |
|----------|---------|-------------|
| `WORKSPACE_MOUNT` | empty volume | Host directory bind-mounted at `/workspace` for the model to read/write |
| `SKIP_PERMISSIONS` | — | Set to any value to auto-approve all tool calls (dev only) |
| `MCP_CONFIG` | — | Path to an MCP server config file |
| `TURNSTONE_IMAGE_TAG` | `latest` | ghcr.io image tag — production stack |
| `TURNSTONE_JWT_SECRET` | — | Secret key for signing JWTs (required) |
## Building
### Database
Both stacks install all entry points into a single image (`turnstone`,
`turnstone-server`, `turnstone-console`, `turnstone-channel`, `turnstone-admin`,
`turnstone-eval`, `turnstone-doctor`):
| Variable | Default | Description |
|----------|---------|-------------|
| `TURNSTONE_DB_BACKEND` | `sqlite` | Storage backend: `sqlite` or `postgresql` |
| `TURNSTONE_DB_URL` | — | Database URL (e.g. `postgresql+psycopg://user:pass@postgres:5432/turnstone`). For SQLite, defaults to `/data/.turnstone.db` |
| `TURNSTONE_DB_POOL_SIZE` | `2` | PostgreSQL connection pool size per process (default: 2 base + 3 overflow = 5 max) |
| `POSTGRES_USER` | `turnstone` | PostgreSQL container username (used in default `TURNSTONE_DB_URL` for cluster/channel) |
| `POSTGRES_PASSWORD` | — | PostgreSQL container password (required for production and cluster profiles) |
The database stores workstream history, user accounts, and API tokens. When using JWT auth, a database backend is required for user storage.
> **Upgrading from <1.3.0a4:** Earlier versions used `DB_BACKEND` and `DATABASE_URL` in `.env`, which `compose.yaml` mapped to the `TURNSTONE_`-prefixed names internally. These short aliases have been removed. Rename `DB_BACKEND``TURNSTONE_DB_BACKEND` and `DATABASE_URL``TURNSTONE_DB_URL` in your `.env` file.
> **Large clusters:** Each turnstone process maintains a small connection pool (5 max). At hundreds of nodes this adds up — use [PgBouncer](pgbouncer.md) in transaction pooling mode between turnstone and PostgreSQL.
> **First-time setup:** After deploying with auth enabled, create an initial admin user by running `turnstone-admin create-user` inside the container:
>
> ```bash
> docker compose exec server turnstone-admin create-user --username admin --name "Admin"
> ```
>
> You will be prompted to set a password. Use it to log in via the UI or SDK, then create additional users through the admin API. Pass `--token --scopes read,write,approve` to also generate an initial API token.
### Channel Gateway
| Variable | Default | Description |
|----------|---------|-------------|
| `TURNSTONE_DISCORD_TOKEN` | — | Discord bot token (required to enable Discord adapter) |
| `TURNSTONE_DISCORD_GUILD` | `0` | Restrict to a single Discord guild (0 = all guilds) |
| `TURNSTONE_SLACK_TOKEN` | — | Slack Bot User OAuth token `xoxb-…` (required to enable Slack adapter) |
| `TURNSTONE_SLACK_APP_TOKEN` | — | Slack App-Level token `xapp-…` (required with `TURNSTONE_SLACK_TOKEN`) |
| `TURNSTONE_SLACK_CHANNELS` | — | Comma-separated Slack channel IDs to allow (empty = all) |
| `TURNSTONE_SLACK_SLASH_COMMAND` | `/turnstone` | Slash command registered in the Slack app |
The channel service runs in the `production` profile. When
`TURNSTONE_DISCORD_TOKEN` or the Slack pair is set the gateway starts the
corresponding adapter; both can run in one process. See
[Channel Integrations](channels.md) for platform app setup and user
account linking.
## Scaling
For multi-node testing, use the `cluster` profile which provides 10 server instances with unique node IDs (`node-1` through `node-10`), resource limits, and shared PostgreSQL:
```bash
docker compose build # build the dev image
docker compose build --no-cache # rebuild from scratch
POSTGRES_PASSWORD=secret docker compose --profile cluster up
```
The default `server` also runs alongside the cluster nodes (11 total). All nodes are accessible via the console dashboard at `:8090`.
For production clusters beyond ~50 nodes, add PgBouncer between turnstone services and PostgreSQL. See [PgBouncer Connection Pooling](pgbouncer.md) for Docker Compose and Helm configuration.
## Volumes
| Volume | Purpose |
|--------|---------|
| `postgres-data` | PostgreSQL data directory |
| `turnstone-data` | `/data` per node (SQLite fallback, local state) |
| `workspace` | `/workspace` (unless `WORKSPACE_MOUNT` is set) |
| `caddy-data` / `caddy-config` | Caddy's local CA and config (dev stack) |
| Volume | Mount | Purpose |
|--------|-------|---------|
| `turnstone-data` | `/data` | SQLite database (`.turnstone.db`) |
## Building
The image uses a multi-stage Dockerfile:
```bash
# Build all services
docker compose build
# Rebuild without cache
docker compose build --no-cache
```
All entry points are installed in a single image: `turnstone`,
`turnstone-server`, `turnstone-console`, `turnstone-channel`,
`turnstone-admin`, `turnstone-eval`, and `turnstone-bootstrap`.
## Cleanup
```bash
docker compose down # stop and remove containers
docker compose down -v # also remove volumes (database, certs)
# Stop and remove containers
docker compose down
# Stop, remove containers and volumes
docker compose down -v
```
+2 -1
View File
@@ -274,7 +274,8 @@ for iteration in 0..max_iterations:
### Phase 1: Analyst (`_run_analyst`)
A multi-turn agent with a `bash` tool for computing statistics. It receives per-case results with failure classifications and
A multi-turn agent with `math` (Python) and `bash` tools for computing
statistics. It receives per-case results with failure classifications and
produces a structured diagnosis:
- **Failure patterns**: Shared root causes across failing cases
+18 -122
View File
@@ -37,31 +37,13 @@ model = "" # empty = same as session model
provider = "" # empty = same as session provider
base_url = ""
api_key = ""
smart_approvals = false # auto-approve high-confidence "approve" LLM verdicts (opt-in)
confidence_threshold = 0.95 # Smart Approvals auto-approve bar (LLM recommendation=approve)
confidence_threshold = 0.7 # reserved for v2 smart approvals (not used in v1)
max_context_ratio = 0.5 # max % of judge context window for history
timeout = 120.0 # seconds (generous for local models)
timeout = 60.0 # seconds (generous for local models)
read_only_tools = true # judge can use read_file/list_directory
cancel_on_approval = false # stop judging remaining tool calls once user decides
```
### Smart Approvals
With `smart_approvals = true` (off by default) a tool call is approved
automatically — no operator prompt — when the intent judge's **LLM** verdict
recommends `approve` with confidence at or above `confidence_threshold`. Every
other outcome still reaches a human: `review` / `deny` recommendations,
confidence below the threshold, judge errors or timeouts (`llm_fallback`), and
any call the deterministic heuristic rules explicitly flagged `deny` or
`critical`. That heuristic floor blocks only those explicit danger verdicts — it
is **not** a general "never lower the heuristic" rule: the heuristic's default
for an unmatched tool is `review`, and letting a confident LLM `approve` upgrade
a `review` is exactly what Smart Approvals is for. Only `deny` / `critical`
findings are off-limits to auto-approval. Requires the judge to be enabled;
auto-approved calls are tagged `smart_approval` in the dashboard and audit trail.
Smart Approvals applies to the web and coordinator surfaces, not the interactive
CLI.
All fields are optional. The judge is enabled by default; use `enabled = false`
(or `--no-judge` on the command line) to disable it.
@@ -71,13 +53,10 @@ All fields are optional. The judge is enabled by default; use `enabled = false`
--judge / --no-judge Enable/disable (default: enabled)
--judge-model MODEL Model for judge
--judge-provider PROVIDER Provider for judge
--judge-timeout SECONDS LLM judge timeout (default: 120)
--judge-confidence FLOAT Confidence threshold, 0-1 (default: 0.95)
--judge-timeout SECONDS LLM judge timeout (default: 60)
--judge-confidence FLOAT Confidence threshold (default: 0.7)
```
(Smart Approvals is configured via `[judge] smart_approvals` / the admin Judge
settings, not a CLI flag — the interactive CLI prompts for approval directly.)
CLI flags override `config.toml` values.
---
@@ -125,7 +104,7 @@ last) and returns the first matching rule. Each rule has:
| Critical | 0.90 | deny | `rm -rf /`, `mkfs`, `dd if=`, pipe-to-shell, chmod 777 on root, write/edit to `/etc/` or `.ssh/`, download-then-execute chains (`curl -o file && chmod +x && bash`) |
| High | 0.80 | review | `sudo`, `kill -9`, destructive git, DROP TABLE, write/edit secrets, HTTP mutations, `ssh`/`scp`, credential file access, browser automation + data export, transitive installs (`npx skills add`, `pip install git+`), control plane mutations (`crontab`, `systemctl enable/start/stop`) |
| Medium | 0.70 | review | Content ingestion pipelines (`curl \| python3`), interpreter execution (`python3 script.py`, `node build.js`), cloud CLI mutations (`az/gcloud/aws/kubectl/terraform` with create/delete/destroy verbs), package installs, `write_file`, MCP tools, Docker operations |
| Low | 0.85 | approve | `read_file`, `list_directory`, `search`, `recall`, `use_prompt`, `tool_search`, `read_resource`, `web_search`, read-only bash (`ls`, `cat`, `head`, `grep`, `find`, etc.) |
| Low | 0.85 | approve | `read_file`, `list_directory`, `search`, `recall`, `man`, `use_prompt`, `tool_search`, `read_resource`, `web_search`, read-only bash (`ls`, `cat`, `head`, `grep`, `find`, etc.) |
When no rule matches, the heuristic returns a default verdict: medium risk,
0.50 confidence, "review" recommendation.
@@ -193,10 +172,9 @@ Security hardening blocks access to sensitive paths:
### Timeout
The `timeout` setting (default 120 seconds) applies **per turn**, not as a total
budget across turns — each of the up to 5 turns gets a fresh budget, so a slow
earlier turn doesn't starve later ones. If a turn's budget expires, the judge
attempts to parse whatever partial response is available.
The `timeout` setting (default 60 seconds) is a total budget across all judge
turns. Time is decremented after each LLM call. If the budget expires mid-turn,
the judge attempts to parse whatever partial response is available.
---
@@ -232,23 +210,6 @@ calls for approval, it calls `_evaluate_intent()` which:
4. Attaches each heuristic verdict to its item as `_heuristic_verdict`
5. The daemon thread runs the LLM judge and delivers results via `ui.on_intent_verdict()`
The daemon evaluates items sequentially, so a large parallel batch can outlive
its approval gate. With `cancel_on_approval = false` (the default) the daemon
runs every item to completion: verdicts that land after the operator decided
still stream to the UI and persist, stamped with the decision. The daemon is
aborted only when the next tool batch supersedes it or the session closes —
then each unfinished item degrades to an `llm_fallback` verdict. With
`cancel_on_approval = true` the abort additionally fires the moment the gate
resolves, trading verdict completeness for inference savings — recommended
when the judge shares a single local inference backend with the session model,
where a large batch's remaining judge calls would otherwise compete with the
next turn's completion.
Verdicts that arrive after a *newer batch* has replaced the judge generation
are withheld from the live surfaces (a reused call_id must never ride a stale
`approve` into Smart Approvals) but still persist with
`user_decision = "superseded"` so the audit trail records the judge's answer.
Sub-agents (plan agent, task agent) are exempt from intent validation -- they
always get full tool visibility without judge evaluation.
@@ -260,13 +221,7 @@ All verdicts are persisted to the `intent_verdicts` table (migration 012):
- Heuristic verdicts are stored when the `approve_request` event is emitted
- LLM verdicts are stored when the `intent_verdict` event is delivered
- The `user_decision` column is updated when the user approves or denies;
auto-approved rows carry the bypass reason (`policy`, `blanket`,
`auto_approve_tools`, `smart_approval`), and rows whose verdict landed only
after a newer batch replaced the judge generation carry `superseded`
- Every stored verdict — including the benign `risk_level = "none"` majority —
is re-attached to its tool call on history replay, so a reloaded workstream
shows the same verdict badges the live stream did
- The `user_decision` column is updated when the user approves or denies
The console admin panel exposes verdict history via:
@@ -417,36 +372,10 @@ redact_secrets = true # auto-redact detected credentials (default)
Configurable at runtime via the admin Settings tab.
### Merge semantics (heuristic + LLM judge)
The chip is a **merge** of the two detectors (issue #560, "show, annotated"),
not a winner-take-all:
- `risk_level` = **max**(heuristic, llm) and `flags` = **union**. A positive
from either detector surfaces; a negative ("none") or failed/absent LLM
**never lowers** a heuristic positive. The judge reads adversarial tool
output, so it may raise the alarm but must not be able to hide a
deterministic regex finding — defeating the judge can't erase the tripwire.
- Credential **redaction** is a heuristic-only signal the LLM cannot override.
- When the judge returned a verdict, its OWN verdict rides along as
annotation (`judge_risk` / `confidence` / `reasoning` / `judge_model`) so
the operator sees the judge's opinion even when it disagrees with the
displayed (merged) risk.
The same merge runs live and on reconnect (both call
`output_guard.merge_guard_display_payload`), so the chip can't drift between
the two surfaces.
The MODEL on the other side of the conversation is shown the merged
`risk_level` + `flags` (via the `GuardAdvisory` spliced into the tool-result
envelope), but is **never** told the judge cleared a finding — a judge fooled
into "none" must not get to talk the model out of caution. The judge's
"benign" verdict is operator-facing only.
### SSE event: `output_warning`
When the merged finding is non-clean (or credentials were redacted), an
`output_warning` SSE event is emitted to the frontend. A regex-only finding:
When the output guard detects risk signals, an `output_warning` SSE event is
emitted to the frontend:
```json
{
@@ -457,50 +386,17 @@ When the merged finding is non-clean (or credentials were redacted), an
"flags": ["credential_leak"],
"annotations": ["API key detected (sk-proj-...)"],
"output_length": 1024,
"redacted": true,
"tier": "heuristic"
"redacted": true
}
```
When the LLM judge returned a verdict, `tier` is `"llm"` and the event carries
the judge's own verdict as annotation. Here the regex flagged MEDIUM but the
judge assessed the output benign — the finding still surfaces (`risk_level`
stays MEDIUM), annotated with the judge's dissent (`judge_risk: "none"`):
The web UI renders this as an inline warning after the tool result. The CLI
shows a colored terminal warning. The server forwards it as an
`OutputWarningEvent` for console subscribers.
```json
{
"type": "output_warning",
"call_id": "call_def456",
"func_name": "web_fetch",
"risk_level": "medium",
"flags": ["camouflaged_injection"],
"annotations": ["Authority-framed directive embedded in the document."],
"output_length": 8192,
"redacted": false,
"tier": "llm",
"judge_risk": "none",
"confidence": 0.92,
"reasoning": "Legitimate analyst commentary; no injection.",
"judge_model": "gpt-5-mini"
}
```
`judge_risk` (the judge's OWN risk verdict, which may differ from the merged
`risk_level`), `confidence` (0.01.0), `reasoning`, and `judge_model` are
present only on the `"llm"` tier. The identical shape is projected onto
history replay by `build_merged_output_assessment_payload`, so the inline chip
renders the same live and on refresh.
The web UI renders this as an inline warning after the tool result — the
`"llm"` tier adds a `⚖ LLM · NN%` badge (showing the judge's verdict when it
differs from the displayed risk, e.g. `⚖ LLM: none · 92%`) and the judge's
rationale. The CLI shows a colored terminal warning. The server forwards it as
an `OutputWarningEvent` for console subscribers.
Assessments are persisted to the `output_assessments` table (one row per
`(call_id, tier)`) for calibration. Raw tool output is never stored — only
metadata: flags, risk level, annotations, output length, redaction status,
and — for the LLM tier — confidence, reasoning, judge model, and latency.
Assessments are persisted to the `output_assessments` table for v2
calibration. Raw tool output is never stored — only metadata (flags, risk
level, annotations, output length, redaction status).
### Session-level skill scan warning
-117
View File
@@ -1,117 +0,0 @@
# MCP OAuth — per-user authorization for MCP servers
Turnstone supports **per-(user, MCP server) OAuth 2.1 + PKCE** delegation so each Turnstone user authorizes a remote MCP server with their own identity, rather than sharing a single bearer token across the deployment. This is the right shape for MCP servers that expose user-specific data (a personal CRM, an email inbox, a calendar) and for MCP servers that want per-user audit attribution.
Per-user OAuth is opt-in per `mcp_servers` row. Local-auth Turnstone installs with no `oauth_user` rows exercise zero new code paths — the entire feature is dark by default.
> **Note**: This is a separate authorization layer from Turnstone's own user authentication. A user who logs into Turnstone with a local username + password can still authorize a per-server OAuth MCP server. OIDC SSO and per-server OAuth are orthogonal.
---
## When to use which `auth_type`
The MCP server admin form exposes three authorization modes ("Multitenant Authorization"):
| `auth_type` | What it means | When to use |
|---|---|---|
| `none` | No headers attached. Open MCP server (or one gated by network policy only). | Internal MCP servers on a trusted network. |
| `static` | One static bearer token, configured per server, sent on every request from every user. | Service-to-service MCP servers where per-user attribution doesn't matter, or single-tenant deployments. |
| `oauth_user` *(recommended for user-data servers)* | Each user authorizes separately via OAuth 2.1 + PKCE; Turnstone stores per-user tokens encrypted at rest. | MCP servers that expose user-specific data or that want per-user audit attribution. |
Switching `auth_type` away from `oauth_user` orphans existing per-user tokens. Use the admin **bulk-revoke** affordance on the server row (Phase 9) to clear them, or let them expire naturally — they're inert without the matching `auth_type` value.
---
## Prerequisites for `auth_type=oauth_user`
1. **Encryption key**. Tokens are stored encrypted with Fernet. Set `[security] mcp_token_encryption_key` in `config.toml` (Turnstone won't start with an `oauth_user` row configured but no key installed). Rotate via `MultiFernet` — add the new key first, then later remove the old one once all rows have been re-encrypted.
2. **MCP server publishes RFC 9728 PRM and RFC 8414 AS metadata** *or* you configure the AS URL override on the server row. PKCE S256 is mandatory; Turnstone refuses to connect to authorization servers that don't advertise `code_challenge_methods_supported: ["S256"]`.
3. **OAuth client registration**. Two paths:
- **Pre-registered** (most common): you create an OAuth client at the authorization server (manually, via admin console, or via Terraform), then paste the `client_id` / `client_secret` into the Turnstone admin form.
- **Dynamic client registration** (RFC 7591): if the AS supports it and you select that mode in the admin form, Turnstone registers a client at first use and persists the `client_id` automatically.
4. **Redirect URI** registered at the authorization server: `https://your-turnstone-host/v1/api/mcp/oauth/callback`.
---
## Configuration
### Per-server fields (admin UI)
| Field | Required | Description |
|---|---|---|
| Server URL | Yes | The MCP server's `streamable-http` base URL. |
| Multitenant Authorization | Yes | `none` / `static` / `oauth_user` (recommended). |
| Authorization Server URL | No | Override for RFC 9728 PRM discovery. Set when your AS endpoint differs from the MCP server URL (e.g., corporate AS protecting a third-party MCP). When unset, Turnstone falls back to PRM discovery against the MCP server itself. |
| Client Registration | Yes (oauth_user) | `preregistered` or `dynamic`. |
| Client ID | Yes (preregistered) | OAuth 2.0 client ID. Stored unencrypted. |
| Client Secret | Optional (write-only) | OAuth 2.0 client secret (confidential client). Encrypted at rest. Written but never re-read by the API; field stays masked. |
| Scopes | No | Space-separated default scope set requested at the authorize endpoint. Per-tool step-up may union additional scopes from a server's `insufficient_scope` response. |
| Audience | No | RFC 8707 `resource=` parameter sent on every authorize and token request. Defaults to the MCP server URL when unset. Validate against the `aud` claim in returned JWT tokens. |
### Encryption key
```toml
[security]
mcp_token_encryption_key = "base64-fernet-key"
# For rotation, list the keys in priority order — first is used for new
# writes, all are tried for reads.
# mcp_token_encryption_keys = ["new-key", "old-key"]
```
Keep this in `config.toml` rather than environment variables. An in-process LLM with shell-tool access can read the server's environment via `env` / `os.environ` and exfiltrate any secret stored there; secrets in `config.toml` are only loaded into the server at startup and never re-read on a tool-driven path, so a prompt-injection attack against the agent cannot reach them.
---
## Lifecycle
1. **First tool call** for a user against an `oauth_user` MCP server: pool dispatch finds no stored token, returns `mcp_consent_required` to the agent. Dashboard renders an inline "Connect" action card.
2. **User clicks Connect**: opens `/v1/api/mcp/oauth/start?server=<name>` in a popup. Browser redirects through the AS authorize endpoint, user grants consent, AS redirects back to `/v1/api/mcp/oauth/callback`. Turnstone exchanges code → tokens via PKCE, validates audience, encrypts, persists in `mcp_user_tokens`, redirects user back to the originating URL.
3. **Subsequent tool calls** by the same user against the same server reuse the persisted token via the per-(user, server) session pool. Tokens auto-refresh via the refresh-token grant when expired; failed refresh emits `mcp_consent_required` to drive re-consent.
4. **Step-up scope**: when a tool call hits `403` with `WWW-Authenticate: error="insufficient_scope"`, Turnstone emits `mcp_insufficient_scope` with the parsed scope set; the dashboard offers a "Connect with additional scopes" affordance that opens `/v1/api/mcp/oauth/start?server=<name>&scopes=<extra>` so the union of original + new scopes flows into the AS authorize request.
5. **User revoke** (settings modal): `DELETE /v1/api/mcp/oauth/connections/{server_name}` runs the authoritative local delete + best-effort RFC 7009 upstream revoke (fire-and-forget, capped at 256 concurrent in-flight tasks).
6. **Admin bulk-revoke** (Phase 9): `POST /v1/api/admin/mcp-servers/{name}/bulk-revoke` drops every user's token for the server. Upstream RFC 7009 revoke is intentionally **not** attempted in bulk (avoids N upstream HTTP calls per admin click); tokens at the AS expire naturally. Use the per-user revoke endpoint if you need guaranteed upstream invalidation.
---
## Admin status indicators
The MCP Servers admin tab shows per-server status pills (Phase 9):
- **Consented users count** — distinct users with a non-expired token for this server. Surfaced as a `bulk-revoke (N)` button when ≥1; clicking it opens a confirmation dialog. Hidden when 0.
- **Last refresh** — timestamp + outcome (`ok` / `error:ClassName`) of the most recent manual or auto-reconnect refresh. Per node. Absent until at least one refresh has occurred (renders as "never" in the admin UI).
Additional indicators (circuit-breaker state, encryption-key mismatch) are exposed via `get_server_status` on the API but do not yet have a dedicated admin pill — operators see them today via the per-server status text + error tooltip and in audit logs. A future phase may surface these as discrete pills.
---
## Auth-type transitions
| From | To | What happens |
|---|---|---|
| `none` / `static``oauth_user` | — | New code path activates for this server. Existing static headers (if any) are no longer sent. Users must authorize on first use. |
| `oauth_user``none` / `static` | — | Existing `mcp_user_tokens` rows are **orphaned** — inert without a matching `auth_type`. Use admin bulk-revoke to drop them, or let them expire. Switching back to `oauth_user` later re-activates the orphaned rows if they haven't been deleted. |
| OAuth `client_id` or `client_secret` rotated | — | Existing tokens may stop refreshing if the AS treats them as bound to the previous client. Bulk-revoke after rotation. |
The orphan-by-default behavior is chosen so switching back to `oauth_user` is non-destructive. Bulk-revoke is the explicit cleanup path.
---
## Troubleshooting
| Symptom | Likely cause | Action |
|---|---|---|
| `mcp_consent_required` even after consenting | Token persistence failed, or refresh-token rejected by AS | Check audit log for `mcp_server.oauth.persist_failed` or `mcp_server.oauth.token_revoked`. Re-consent via settings modal. |
| `mcp_token_undecryptable_key_unknown` | Encryption key rotated without keeping the previous key in the keyring | Add the previous key back to `mcp_token_encryption_keys` until all rows have been re-encrypted, then drop. |
| `mcp_oauth_url_insecure` | MCP server URL is `http://` (not `https://`) on a non-loopback host | Use `https://`. Per-user bearers must not transit cleartext. |
| Tools fail in scheduled / Discord / Slack runs | OAuth-MCP requires browser-based consent | Users must pre-consent via the web UI. Phase 9 dashboard badge surfaces deferred consents from these runs on next login. |
| Circuit breaker open repeatedly | Transport-level errors on the MCP server (DNS, TLS, 5xx) | Check the per-server error pill; auth errors do not trip the breaker. |
See also: `docs/operations/mcp-oauth-headless.md` for the cron / channel-driven run caveat.
+5 -30
View File
@@ -26,40 +26,15 @@ Each memory has three dimensions:
### Memory scopes
| Scope | Visibility |
|---------------|-----------------------------------------------------------------|
| `global` | Visible to all workstreams and users |
| `workstream` | Visible only within the originating workstream |
| `user` | Follows the authenticated user across workstreams |
| `coordinator` | Coordinator sessions only; follows the user across coordinators |
| Scope | Visibility |
|--------------|-----------------------------------------------------------|
| `global` | Visible to all workstreams and users |
| `workstream` | Visible only within the originating workstream |
| `user` | Follows the authenticated user across workstreams |
A memory's identity is the tuple `(name, scope, scope_id)`. Saving a memory
with the same identity upserts -- updating content while preserving the ID.
### Coordinator scope
Coordinator sessions are isolated to a single scope: `coordinator`, keyed by
the coordinator's creator `user_id`. It is durable -- every coordinator
session the same user runs (including concurrent ones) shares one
orchestration namespace, so procedures and lessons survive close/reopen.
Isolation is bidirectional and enforced by session kind, not by secrecy of
the scope id:
- A coordinator session can read and write **only** `coordinator`-scope rows.
It never sees `global`/`workstream`/`user` memories, so content written by
interactive sessions (which routinely ingest untrusted MCP/attachment
output) cannot reach a coordinator's system message.
- Interactive sessions -- including a coordinator's own children, which share
its `user_id` -- are rejected from the `coordinator` scope on every memory
action. Children cannot plant rows the parent coordinator would read.
- The REST memory API (`/v1/api/memories`) does not accept the `coordinator`
scope at all; the scope is written exclusively through a coordinator
session's own memory tool.
Coordinator sessions require an authenticated user identity -- an anonymous
coordinator cannot be constructed, so the scope id is always a real user.
### BM25 relevance injection
On every conversation turn, the system:
+1 -1
View File
@@ -94,7 +94,7 @@ cannot bypass the proxy.
|--------|-------|---------|
| `openai_api` | `api.openai.com` | OpenAI LLM API |
| `anthropic_api` | `api.anthropic.com` | Anthropic LLM API |
| `searxng` | `searxng:8080` (bundled service) | Web search backend |
| `tavily_api` | `api.tavily.com` | Web search fallback |
| `skills_registry` | `skills.sh` | Skill discovery |
| `github_api` | `api.github.com` (read-only L7), `raw.githubusercontent.com` | Skill fetch, GitHub API |
| `mcp_registry` | `registry.modelcontextprotocol.io` (read-only L7) | MCP server discovery |
-29
View File
@@ -1,29 +0,0 @@
# MCP OAuth in headless / scheduled / channel-driven runs
**Constraint**: OAuth-MCP servers (`auth_type=oauth_user`) require browser-based user consent. Users must pre-consent via the web UI before any run that cannot drive a browser redirect.
**Affected surfaces**:
- Scheduled workstreams (`turnstone-console` task scheduler).
- Discord adapter runs.
- Slack adapter runs.
- Any future channel adapter without an interactive browser session.
**What happens when consent is missing**:
A tool call against an `oauth_user` server returns a structured `mcp_consent_required` error to the agent. The agent surfaces the deferred work in its output. Turnstone persists a record to `mcp_pending_consent` so the dashboard badge surfaces the deferred consent need to the user on next login.
**Recovery**:
The user opens the dashboard, sees the gear-icon badge counting pending consents, opens the settings modal, clicks Connect for each affected server, and completes the OAuth dance. The pending-consent record is cleared by the OAuth callback handler on success. Subsequent scheduled / channel runs use the freshly-stored token.
**Pre-consent recipe**:
Before scheduling a workstream that depends on an `oauth_user` MCP server, the user should:
1. Open the dashboard.
2. Open the settings modal (gear icon).
3. Click Connect on each MCP server the schedule will use.
4. Confirm consent in the popup.
This stores tokens that the scheduled run will reuse. Refresh-token rotation is handled transparently on the run side; only the first consent requires browser interaction.
+1 -25
View File
@@ -108,7 +108,7 @@ pgbouncer:
maxClientConn: 5000
maxDbConnections: 80
```
:
---
## Configuration reference
@@ -199,28 +199,4 @@ does not support prepared statements. Turnstone's SQLAlchemy layer does
not use server-side prepared statements by default, so this is not an
issue.
**LISTEN / NOTIFY not supported in transaction mode** — PgBouncer's
transaction pooling assigns a real server connection only for the
duration of each transaction, then returns it to the pool. PostgreSQL
`LISTEN` is session state — a transaction-pooled client can't hold the
multi-statement session a long-lived `LISTEN` needs. The console's
`NotifyDispatcher` (reactive node discovery via the `services` channel)
therefore opens a **dedicated, direct-to-Postgres** connection that
bypasses PgBouncer.
Configure via `config.toml` `[database] listen_url` (preferred —
co-located with the main `url`) or the `TURNSTONE_DB_LISTEN_URL` env var
(config.toml wins when both are set). Defaults to the main DB URL when
unset.
| Setting | Behaviour |
|---|---|
| unset | Listener uses `TURNSTONE_DB_URL` as-is. Fine when PgBouncer is in **session** mode, or when there's no pooler in front of Postgres. With transaction-mode PgBouncer the listener's `LISTEN` will fail and the dispatcher retries with exponential backoff (1 s → 30 s cap) without ever succeeding. Reactive NOTIFY-driven node discovery is silently lost; the cluster collector's 60 s `_discovery_loop` is the only remaining backstop. |
| set to direct-to-PG URL (e.g. `postgresql://…/turnstone`) | Listener bypasses PgBouncer for its one dedicated connection. Reactive discovery latency drops from up-to-60 s to ~500 ms. The rest of the storage layer continues to go through PgBouncer in transaction mode. |
Set this whenever PgBouncer is in transaction mode (the recommended
setting per this doc). The override only adds one long-lived PG
connection per console process — sized into the cluster's
`max_connections` budget alongside the pool.
See also: [Docker deployment](docker.md) · [Security](security.md)
+17 -18
View File
@@ -6,9 +6,10 @@ Turnstone ships several parallel release tracks from a single PyPI package.
| Track | Versions | Branch | Docker tags | PyPI install |
|-------|----------|--------|-------------|--------------|
| **Stable 1.5** | `1.5.x` | `stable/1.5` | `:1.5.x`, `:1.5` | `pip install 'turnstone==1.5.*'` |
| **Stable 1.6** | `1.6.x` | `stable/1.6` | `:1.6.x`, `:1.6`, `:stable`, `:latest` | `pip install turnstone` |
| **Experimental** | `1.7.0aN` | `main` | `:1.7.0aN`, `:experimental` | `pip install turnstone --pre` |
| **Legacy 1.0** | `1.0.x` | `stable/1.0` | `:1.0.x`, `:1.0` | `pip install 'turnstone==1.0.*'` |
| **Stable 1.3** | `1.3.x` | `stable/1.3` | `:1.3.x`, `:1.3` | `pip install 'turnstone==1.3.*'` |
| **Stable 1.4** | `1.4.x` | `stable/1.4` | `:1.4.x`, `:1.4`, `:stable`, `:latest` | `pip install turnstone` |
| **Experimental** | `1.5.0aN` | `main` | `:1.5.0aN`, `:experimental` | `pip install turnstone --pre` |
- **Stable** tracks receive bugfixes only. The most-recent stable minor
owns the `:stable` / `:latest` Docker tags and the default PyPI
@@ -16,10 +17,8 @@ Turnstone ships several parallel release tracks from a single PyPI package.
- **Experimental** (always on `main`) receives new features. May be
rough around the edges.
- When experimental matures, it is promoted to a new stable minor via
a `stable/X.Y` branch. One prior stable track is maintained alongside
the current one; at each promotion the oldest track is retired — its
branch is deleted, while its tags and released artifacts remain
available.
a `stable/X.Y` branch; older stable branches continue to receive
security fixes until explicitly retired.
## Version Scheme
@@ -34,17 +33,17 @@ Turnstone ships several parallel release tracks from a single PyPI package.
## Releasing an Experimental Version (from main)
```bash
scripts/release.sh 1.7.0a2 --push
scripts/release.sh 1.5.0a2 --push
```
This bumps `pyproject.toml` + `turnstone/__init__.py`, regenerates `uv.lock`, commits, tags `v1.7.0a2`, and pushes. CI runs, then publish + Docker workflows fire automatically.
This bumps `pyproject.toml` + `turnstone/__init__.py`, regenerates `uv.lock`, commits, tags `v1.5.0a2`, and pushes. CI runs, then publish + Docker workflows fire automatically.
## Releasing a Stable Patch (from stable/X.Y)
```bash
git checkout stable/1.6
git checkout stable/1.4
git cherry-pick <commit-hash> # bugfix from main
scripts/release.sh 1.6.1 --push
scripts/release.sh 1.4.1 --push
```
## Promoting Experimental to Stable
@@ -53,19 +52,19 @@ When `main` is ready for a stable release:
```bash
# 1. Tag the stable release on main
scripts/release.sh 1.6.0 --push
scripts/release.sh 1.5.0 --push
# 2. Create the stable maintenance branch from that tag
git branch stable/1.6 v1.6.0
git push origin stable/1.6
git branch stable/1.5 v1.5.0
git push origin stable/1.5
# 3. Start the next experimental cycle on main
scripts/release.sh 1.7.0a1 --push
scripts/release.sh 1.6.0a1 --push
```
The previous stable branch continues to receive security-only patches;
the track before it is retired at each promotion (at 1.6.0:
`stable/1.5` stays maintained, `stable/1.4` is retired).
The previous stable branch (`stable/1.4`) continues to receive
security-only patches; older tracks (`stable/1.0`, `stable/1.3`) are
retired when they fall out of support.
## CI/CD Pipeline
+2 -3
View File
@@ -77,6 +77,7 @@ Both `TurnstoneServer` (sync) and `AsyncTurnstoneServer` (async) expose:
| | `delete_attachment(ws_id, attachment_id)` | `StatusResponse` |
| **Chat** | `send(message, ws_id)` | `SendResponse` |
| | `approve(*, ws_id, approved, feedback, always)` | `StatusResponse` |
| | `plan_feedback(*, ws_id, feedback)` | `StatusResponse` |
| | `command(*, ws_id, command)` | `StatusResponse` |
| | `cancel(ws_id, *, force=False)` | `StatusResponse` |
| **Streaming** | `stream_events(ws_id)` | `Iterator[ServerEvent]` |
@@ -133,12 +134,10 @@ SSE events are deserialized into typed dataclasses. Use `event.type` to discrimi
| `tool_result` | `ToolResultEvent` | `call_id`, `name`, `output`, `is_error` |
| `tool_output_chunk` | `ToolOutputChunkEvent` | `call_id`, `chunk` |
| `status` | `StatusEvent` | `prompt_tokens`, `total_tokens`, `pct`, `effort`, `cache_creation_tokens`, `cache_read_tokens` |
| `plan_review` | `PlanReviewEvent` | `content` |
| `error` | `ErrorEvent` | `message` |
| `info` | `InfoEvent` | `message` |
| `stream_end` | `StreamEndEvent` | — |
| `state_change` | `StateChangeEvent` | `state``running`/`thinking`/`attention`/`idle`/`error` |
| `in_progress_snapshot` | `InProgressSnapshotEvent` | `content`, `reasoning` (one-shot mid-stream refresh resume) |
| `approval_resolved` | `ApprovalResolvedEvent` | `approved`, `feedback` |
| `cancelled` | `CancelledEvent` | — |
**Global events** (from `stream_global_events()`):
+1 -1
View File
@@ -67,7 +67,7 @@ Scopes are hierarchical — higher scopes imply all lower ones.
| Method | Path pattern | Required scope |
|--------|-------------|----------------|
| GET | Any protected path | `read` |
| POST | `/api/command` | `write` |
| POST | `/api/plan`, `/api/command` | `write` |
| POST | `/api/workstreams/new`, `/api/cluster/workstreams/new` | `write` |
| POST | `/api/workstreams/{ws_id}/{send,cancel,close,delete,open,refresh-title,title,attachments}` | `write` |
| DELETE | `/api/workstreams/{ws_id}/send` (dequeue), `/api/workstreams/{ws_id}/attachments/{attachment_id}` | `write` |
+9 -21
View File
@@ -59,32 +59,20 @@ from ConfigStore. Model names and context windows are now configured per-model
in the Models tab. A startup warning is logged if these keys appear in
`config.toml`.
### Reasoning persistence (per-model)
### Plan / task agent overrides
Two boolean flags on `model_definitions` (migration 052) control how
reasoning text round-trips per model:
| Flag | Default | Effect |
|------|---------|--------|
| `surface_persisted_reasoning` | `True` | Surface stored reasoning text on `/history` payloads so a page reload re-renders the reasoning bubble. **Storage of reasoning bytes is independent of this flag** — they ride in `provider_data` regardless. |
| `replay_reasoning_to_model` | `False` | Send stored reasoning blocks back to the provider on subsequent turns. Capability-gated: only takes effect when the model's `ModelCapabilities.supports_reasoning_replay` is also `True`. Set on canonical OpenAI gpt-5*/o-series and Anthropic Claude entries; unknown / local-server models default to `False` so an operator who flips the flag on a model whose API doesn't understand reasoning replay silently no-ops rather than 400-ing. |
Edit both via the admin Models tab. See the architecture doc for the
provider-side mechanics (Anthropic `thinking`, OpenAI Responses
`reasoning` + `include=["reasoning.encrypted_content"]`, synthetic
`reasoning_text` for Chat Completions / vLLM / llama.cpp / Gemini-compat).
### Task agent overrides
`task_agent` sub-sessions resolve independently from the conversation model
so operators can pick a cheaper/faster model for autonomous loops:
`plan_agent` and `task_agent` sub-sessions resolve independently from the
conversation model so operators can pick a cheaper/faster model for
autonomous loops:
| Setting | Purpose |
|---------|---------|
| `model.task_alias` | Alias used for `task_agent` sub-sessions. Falls back to `[model].agent_model` in config.toml, then the session's active model. |
| `model.plan_alias` | Alias used for `plan_agent` sub-sessions. Falls back to `[model].plan_model` in config.toml, then `[model].agent_model`, then the session's active model. |
| `model.task_alias` | Alias used for `task_agent` sub-sessions. Same fallback chain as `plan_alias`. |
| `model.plan_effort` | Reasoning effort for `plan_agent` (`none` / `minimal` / `low` / `medium` / `high` / `xhigh` / `max`). Defaults to `high`. |
| `model.task_effort` | Reasoning effort for `task_agent`. Empty string means "inherit from the session". |
Both are live-editable from the Settings tab and take effect on the
All four are live-editable from the Settings tab and take effect on the
next sub-agent invocation — no restart required.
---
@@ -107,7 +95,7 @@ initialization:
| Section | Settings |
|---------|----------|
| `model` | default_alias, temperature, max_tokens, reasoning_effort, task_alias, task_effort |
| `model` | default_alias, temperature, max_tokens, reasoning_effort, plan_alias, task_alias, plan_effort, task_effort |
| `session` | instructions, retention_days, compact_max_tokens, auto_compact_pct |
| `tools` | timeout, truncation, agent_max_turns, skip_permissions, search, search_threshold, search_max_results |
| `server` | workstream_idle_timeout, max_workstreams |
+10 -110
View File
@@ -8,7 +8,7 @@ inter-service communication, powered by [lacme](https://pypi.org/project/lacme/)
## Quick Start (Docker Compose)
```bash
docker compose -f turnstone/deploy/compose.yaml -f deploy/docker-compose.tls.yml up
docker compose -f compose.yaml -f deploy/docker-compose.tls.yml up
```
This:
@@ -19,53 +19,6 @@ This:
---
## Browser access (dashboard HTTPS)
The mTLS above secures **service-to-service** traffic (node↔node, collector and
routing proxy → nodes). The **console dashboard itself serves plain HTTP** — and
must, because it is the cluster's ACME bootstrap endpoint: new nodes fetch
`/acme/ca.pem` and provision their first cert over HTTP, before they have the CA
to verify TLS. So the console cannot be HTTPS-only on its port.
To put the **browser → console** hop on HTTPS, terminate TLS at a reverse proxy
in front of the console. The dev stack (root `compose.yaml`) ships a `caddy`
service that does exactly this — and it's the only published entry point, so the
dashboard is HTTPS by default:
```bash
docker compose up
# dashboard: https://localhost:${CONSOLE_HTTPS_PORT:-8443}
```
The production stack (`turnstone/deploy/compose.yaml`) bundles the same `caddy`
service, so the dashboard is HTTPS there too. For a real domain and a publicly
trusted cert, point Caddy at Let's Encrypt by editing `turnstone/deploy/Caddyfile`.
```
browser --h2 / HTTPS--> caddy:443 --h1.1 / HTTP--> console:8090
```
Caddy uses its **own local CA** (`tls internal`, see `turnstone/deploy/Caddyfile`), so the
setup is self-contained with no dependency on the console's ACME path. Trust the
local root once to silence the browser warning:
```bash
docker compose exec caddy \
cat /data/caddy/pki/authorities/local/root.crt # import into your OS/browser
```
**Can Caddy get its cert from the console's internal CA instead?** Technically
yes — the console exposes a real ACME directory (`/acme/directory`) with
auto-approval, so Caddy's `tls { ca http://console:8090/acme/directory }` would
mint a cert for any name. It's not recommended as the default: lacme's ACME
responder is built for turnstone's own client (interop with Caddy's client is
unverified), it couples Caddy startup to the console, and the browser must trust
a private CA either way — so it buys nothing over `tls internal`. For a publicly
trusted cert (no warning), point Caddy at Let's Encrypt with a real domain
instead.
---
## Architecture
```
@@ -88,32 +41,6 @@ Console (CA + ACME Server)
- **Frontend cert** (HTTPS): From an external ACME CA (e.g. Let's Encrypt)
if `tls.acme_directory` is set, otherwise self-issued from the internal CA.
### Boot, retry, and fallback
With `tls.enabled`, a node fetches the CA cert and requests its own cert
during startup, retrying with exponential backoff (6 attempts, ~31 s total)
— enough to absorb a whole-stack restart where every node races the console
for its listener. If all attempts fail, the node **falls back to plain
HTTP** (availability over confidentiality) and reports `"tls": "fallback"`
in `GET /health`; a node serving HTTPS reports `"tls": "active"`, and the
key is absent when TLS is disabled. Fallback persists until the next
restart — it is not upgraded in place.
### Container healthcheck under mTLS
An mTLS listener rejects plain-HTTP probes at the socket, so
`docker/healthcheck.py` falls back to HTTPS when the plain probe fails:
it presents the node's own cert as the client cert and pins the cluster
CA, using the PEM files the server writes at boot under
`$TURNSTONE_TLS_PEM_DIR` (default `<tmpdir>/turnstone-tls`). The probe
dials `localhost` for the TLS attempt — the internal CA issues DNS SANs
only, so a literal-IP URL would fail verification. Cert renewal rewrites
the PEM dir alongside the live listener swap, so the probe's client cert
never outlives the served cert. With TLS disabled the plain probe succeeds
and the PEM directory is never consulted. On bare metal with multiple
nodes per host, set `TURNSTONE_TLS_PEM_DIR` per node (each boot clears
stale `lacme-pem-*` dirs under its root).
---
## Configuration
@@ -244,20 +171,10 @@ const client = new TurnstoneServer({
### Node Bootstrap Flow
1. Node starts, connects to shared database (plain connection)
2. Discovers the console URL from the `services` table — or honors an explicit
`TURNSTONE_CONSOLE_URL` (a bare-metal node outside the compose network can't
resolve the in-cluster `console` name, so it points this at the console's
published ACME endpoint)
2. Discovers console URL from `services` table
3. Fetches CA root cert from `http://console/acme/ca.pem` (plain HTTP, TOFU)
4. Requests a service cert via ACME (plain HTTP, JWS-signed). The cert's
primary domain / SAN is the node's **advertised host** (the host of
`TURNSTONE_ADVERTISE_URL`, e.g. `node-1`) — the name peers actually dial,
not the container hostname. This makes mTLS hostname verification succeed
and keys the cert by a stable name that survives container recreation.
5. Starts auto-renewal (24h interval, re-issues before expiry) **scoped to its
own certificate**. Each node renews only its own cert; the shared store is
never swept wholesale. Renewed certs are hot-swapped into the live HTTPS
listener with no restart.
4. Requests service cert via ACME protocol (plain HTTP, JWS-signed)
5. Starts auto-renewal (24h interval, re-issues before expiry)
6. All subsequent inter-service communication uses mTLS
### Console Startup Flow
@@ -266,9 +183,7 @@ const client = new TurnstoneServer({
2. Initialize CA (load from DB or generate new root key)
3. Mount ACME responder at `/acme` (serves `/ca.pem` natively)
4. Issue console certs (internal + optional frontend)
5. Start CA-direct auto-renewal (no network, signs directly), scoped to the
console's own cert, plus a periodic GC that reclaims cert rows for
long-departed nodes
5. Start CA-direct auto-renewal (no network, signs directly)
6. Register console URL in services table with heartbeat
---
@@ -280,32 +195,17 @@ const client = new TurnstoneServer({
Certs are valid for 48 hours. If auto-renewal stopped (e.g. console was down),
restart the service to re-request a cert.
### Collector/proxy can't reach a node (TLS hostname mismatch)
mTLS verifies a node's advertised host against the cert's SANs. Each node's
cert is issued for the host in its `TURNSTONE_ADVERTISE_URL`, so that name is
always a SAN automatically — you do **not** need to set `TURNSTONE_TLS_SANS`
per node. Only set `TURNSTONE_TLS_SANS` to add *extra* names (e.g. a node
fronted under a second hostname). Symptom if this is wrong: the console
dashboard shows nodes as unreachable and `openssl s_client` reports the served
cert's SANs don't include the dialed name.
### "No console service found"
The console registers itself in the `services` table on startup. If the console
hasn't started or the registration expired (1 hour TTL), nodes can't discover
it. Set `TURNSTONE_CONSOLE_URL` to a reachable console address (this is also how
a bare-metal node that can't resolve the in-cluster `console` name enrolls).
it. Use `--console-url` explicitly.
### Browser HTTPS to the console
### Let's Encrypt for console frontend
The console serves plain HTTP (it's the ACME bootstrap endpoint — see
[Browser access](#browser-access-dashboard-https)). Put browser traffic on
HTTPS by terminating TLS at a reverse proxy; the `cluster` profile's `caddy`
service does this with Caddy's local CA. For a publicly trusted cert, front the
console with a proxy pointed at Let's Encrypt using a real domain. The
`tls.acme_directory` setting only governs the console's internal/frontend cert
material — it does **not** make the console listen on HTTPS itself.
Set `tls.acme_directory` to `https://acme-v02.api.letsencrypt.org/directory`
in the admin Settings tab. The console will request a publicly trusted cert
for its HTTPS endpoint. Internal mTLS still uses the private CA.
### Verifying the cert chain
+117 -90
View File
@@ -1,6 +1,6 @@
# Tools Reference
turnstone exposes 16 built-in tools plus any number of external MCP tools to the
turnstone exposes 19 built-in tools plus any number of external MCP tools to the
LLM via the OpenAI function-calling interface. Built-in tools are defined as JSON
files under `turnstone/tools/` and loaded at startup by `turnstone/core/tools.py`.
MCP tools are discovered from configured MCP servers at startup by
@@ -22,6 +22,7 @@ schema plus turnstone-specific metadata keys:
"properties": { ... },
"required": ["param1"]
},
"agent": true,
"task_agent": true,
"auto_approve": true,
"primary_key": "param1"
@@ -32,7 +33,8 @@ schema plus turnstone-specific metadata keys:
| Key | Type | Meaning |
|----------------|------|---------|
| `task_agent` | bool | Tool is available to task sub-agents. |
| `agent` | bool | Tool is available to plan/task sub-agents (read-only subset). |
| `task_agent` | bool | Tool is available to task sub-agents (broader subset). |
| `auto_approve` | bool | Tool runs without user confirmation (read-only, safe operations). |
| `primary_key` | str | When the model sends a bare string instead of JSON args, map it to this parameter name. |
@@ -44,10 +46,12 @@ schema plus turnstone-specific metadata keys:
| Name | Description |
|---------------------|-------------|
| `TOOLS` | All 28 loaded built-in tool definitions (interactive + coordinator union). Sessions send a kind-specific subset (`INTERACTIVE_TOOLS` or `COORDINATOR_TOOLS`). |
| `TOOLS` | All 19 tool definitions (sent to the model). |
| `AGENT_TOOLS` | Tools with `agent: true` -- available to plan sub-agents. Read-only tools. |
| `TASK_AGENT_TOOLS` | Tools with `task_agent: true` -- available to task sub-agents. Includes write operations. |
| `TASK_AUTO_TOOLS` | Set of all tool names with `auto_approve: true` -- used by task-agent sub-sessions to skip confirmation for matching available tools. |
| `BUILTIN_TOOL_NAMES`| Frozenset of all 28 built-in tool names (interactive + coordinator union). Used by tool search to distinguish always-on tools from deferrable MCP tools. |
| `AGENT_AUTO_TOOLS` | Set of tool names with `auto_approve: true` -- no user confirmation needed. |
| `TASK_AUTO_TOOLS` | Same as `AGENT_AUTO_TOOLS` (identical filter). |
| `BUILTIN_TOOL_NAMES`| Frozenset of all 19 built-in tool names. Used by tool search to distinguish always-on tools from deferrable MCP tools. |
| `PRIMARY_KEY_MAP` | Dict mapping tool name to its `primary_key` parameter name. |
---
@@ -65,7 +69,7 @@ Tool execution follows a three-phase pipeline inside `ChatSession._execute_tools
- Parses the JSON arguments (with fallback for malformed JSON).
- If JSON parsing fails entirely, uses `PRIMARY_KEY_MAP` to map a bare string
to the correct parameter.
- Dispatches to the matching `_prepare_{func_name}()` handler. There are 16
- Dispatches to the matching `_prepare_{func_name}()` handler. There are 19
built-in tools plus `tool_search` (synthetic, client-side BM25 fallback) and
the generic `_prepare_mcp_tool()` handler for MCP tools.
- Validates arguments and builds a preview dict containing:
@@ -107,6 +111,9 @@ Each item's `execute` callable is invoked:
denials are tracked separately. This removes the need for text-prefix heuristics.
Other tools deliver results atomically via
`ui.on_tool_result(call_id, name, output, is_error=...)` only.
- Special post-execution gate for `plan`: the plan output is shown to the user
for review, and the user can reject or annotate it.
---
## Tool Approval Flow
@@ -114,6 +121,7 @@ Each item's `execute` callable is invoked:
**Auto-approved** (no user confirmation needed at runtime):
- `read_file` -- reads files, no side effects
- `search` -- grep-style search, no side effects
- `man` -- reads man pages, no side effects
- `memory` -- structured persistent memory (save/search/delete/list)
- `recall` -- searches conversation history
- `notify` -- sends notifications to linked channels (time-sensitive, auto-approved for urgency)
@@ -122,14 +130,16 @@ Each item's `execute` callable is invoked:
- `bash` -- arbitrary command execution
- `write_file` -- creates or overwrites files
- `edit_file` -- modifies file content
- `math` -- sandboxed computation (confirmation required despite being sandboxed)
- `web_fetch` -- fetches a URL (SSRF-protected, but makes network requests)
- `web_search` -- web search via self-hosted SearxNG (makes network requests)
- `task_agent` -- spawns an autonomous sub-agent
- `web_search` -- web search via Tavily API (makes network requests)
- `task` -- spawns an autonomous sub-agent
- `plan` -- spawns a planning sub-agent, plus post-execution review gate
Note: The JSON schema metadata key `auto_approve` controls membership in
`TASK_AUTO_TOOLS` (used for task agent sub-sessions). The actual runtime
approval behavior is determined by the `needs_approval` field set in each
`_prepare_*` method on `ChatSession`. These two mechanisms can differ.
`AGENT_AUTO_TOOLS`/`TASK_AUTO_TOOLS` (used for agent sub-sessions). The actual
runtime approval behavior is determined by the `needs_approval` field set in
each `_prepare_*` method on `ChatSession`. These two mechanisms can differ.
---
@@ -155,9 +165,12 @@ Every tool defines a `primary_key`. The mapping is:
| `write_file` | `content` |
| `edit_file` | `old_string`|
| `search` | `query` |
| `math` | `code` |
| `man` | `page` |
| `web_fetch` | `url` |
| `web_search` | `query` |
| `task_agent` | `prompt` |
| `plan_agent` | `goal` |
| `memory` | `name` |
| `recall` | `query` |
| `notify` | `message` |
@@ -181,7 +194,7 @@ Execute a bash command and return stdout + stderr.
- **What it does**: Runs the command in a subprocess with a configurable timeout. Commands are sanitized and checked against a blocklist (e.g. `rm -rf /`). Environment variables containing secrets are scrubbed (`*_KEY`, `*_SECRET`, `*_TOKEN`, etc.).
- **Output format**: Stdout is returned directly. Stderr lines are prefixed with `[stderr]` so the model can distinguish them. When the command itself redirects stderr to stdout (`2>&1`), no prefix is added. Output exceeding 256KB is truncated (head + tail preserved, middle replaced with a truncation notice).
- **Auto-approve**: No -- requires user confirmation.
- **Agent availability**: `task_agent` only.
- **Agent availability**: `task_agent` only (not available to plan sub-agents).
---
@@ -199,7 +212,7 @@ base64-encoded image data for supported image formats.
- **What it does**: For text files, reads and returns content with line numbers. For image files (PNG, JPEG, GIF, WebP, BMP, TIFF, ICO), returns image data as multi-part content when the model supports vision, or a text description when it does not. SVG files are read as text. Images larger than 4 MB are rejected. Must be called before `edit_file` on the same path (the session tracks which files have been read).
- **Vision support**: Controlled by `ModelCapabilities.supports_vision`. All commercial OpenAI and Anthropic models have vision enabled. Local models (vLLM, llama.cpp, NIM) default to off — enable via `[models.*.capabilities] supports_vision = true` in config.toml.
- **Auto-approve**: Yes.
- **Agent availability**: `task_agent`.
- **Agent availability**: `agent` and `task_agent`.
---
@@ -255,7 +268,7 @@ Show a unified diff between two files, or between a file and a provided string.
- **What it does**: Returns unified diff output using Python's `difflib`. Binary files (containing null bytes) are rejected with a clear error. Files read through `diff_file` satisfy `edit_file`'s read guard — you can diff then edit without a separate `read_file` call. Large diffs are streamed with early cutoff at the tool truncation limit.
- **Auto-approve**: Yes (read-only).
- **Agent availability**: `task_agent`.
- **Agent availability**: `agent` and `task_agent`.
---
@@ -270,12 +283,44 @@ Search file contents for a regex pattern.
- **What it does**: Recursively searches for the pattern using `grep -rn`. Returns matching lines with file paths and line numbers.
- **Auto-approve**: Yes.
- **Agent availability**: `task_agent`.
- **Agent availability**: `agent` and `task_agent`.
---
## Computation
### math
Execute Python code for math and computation in a sandbox.
| Parameter | Type | Required | Description |
|-----------|--------|----------|-------------|
| `code` | string | yes | Python code to execute. Must use `print()` for output. |
- **What it does**: Runs Python code in a sandboxed environment with pre-imported libraries: `sympy`, `numpy`, `scipy`, `math`, `fractions`, `itertools`, `functools`, `collections`, `decimal`, `operator`, `random`, `re`, `string`. Common sympy names (`symbols`, `solve`, `simplify`, `sqrt`, `Matrix`, etc.) are pre-imported. `pytest` is also available for import.
- **Installation**: `sympy`, `numpy`, `scipy`, and `pytest` require the `[sandbox]` extras group: `pip install turnstone[sandbox]` (included in `[all]`).
- **Auto-approve**: Yes.
- **Agent availability**: `agent` and `task_agent`.
---
## Information
### man
Read a man page.
| Parameter | Type | Required | Description |
|-----------|--------|----------|-------------|
| `page` | string | yes | The man page name (e.g. `grep`, `socket`, `printf`). |
| `section` | string | no | Manual section (e.g. `1` commands, `2` syscalls, `3` library). |
- **What it does**: Returns the full formatted manual entry. Preferred over `bash('man ...')` or `web_search` for command/API documentation.
- **Auto-approve**: Yes.
- **Agent availability**: `agent` and `task_agent`.
---
### web_fetch
Fetch a URL and extract specific information from it.
@@ -287,7 +332,7 @@ Fetch a URL and extract specific information from it.
- **What it does**: Fetches the URL, strips HTML to plain text, and uses the LLM to extract the answer to the question from the page content. Protected against SSRF (blocks private/internal IPs).
- **Auto-approve**: No -- requires user confirmation (makes network requests).
- **Agent availability**: `task_agent`.
- **Agent availability**: `agent` and `task_agent`.
---
@@ -299,55 +344,20 @@ Search the web using a text query.
|---------------|---------|----------|-------------|
| `query` | string | yes | The search query. |
| `max_results` | integer | no | Max results to return (default 5, max 20). |
| `category` | string | no | Search category: `general` (default), `news`, `it` (code/tech), or `science`. Maps to SearxNG categories; the model picks per query. |
| `topic` | string | no | Search topic: `general`, `news`, or `finance` (default `general`). |
- **What it does**: Searches the web and returns ranked results with titles, URLs, and content snippets. Uses provider-native search when available:
- **Anthropic**: Replaced at the API boundary with Anthropic's `web_search_20250305` server-side tool. Claude decides when to search; the API executes it and returns results with citations inline. No backend needed.
- **Anthropic**: Replaced at the API boundary with Anthropic's `web_search_20250305` server-side tool. Claude decides when to search; the API executes it and returns results with citations inline. No Tavily key needed.
- **OpenAI search models** (`gpt-5-search-api`): Replaced with `web_search_options` parameter. The model always searches and returns `url_citation` annotations.
- **Local/vLLM models**: Falls back to a self-hosted [SearxNG](https://searxng.org) instance. Set `searxng_url` in `config.toml` `[tools]` or `$TURNSTONE_SEARXNG_URL` (the docker-compose stack bundles a `searxng` service and points at it by default). Operators with a custom MCP search server can instead set `web_search_backend = "mcp:server:tool"`.
- **Local/vLLM models**: Falls back to the Tavily API. Requires `tavily_key` in `config.toml` or `$TAVILY_API_KEY`.
- **Auto-approve**: Yes (auto-approved for all tool dispatch paths).
- **Agent availability**: `task_agent`.
---
### Reranking (optional)
`web_search` can use an external **reranker** to re-order the backend's result pool by relevance to the query before returning the top hits. Turnstone runs no reranker model itself; it POSTs to a Cohere/Jina-compatible `/rerank` endpoint (self-hosted [vLLM](https://docs.vllm.ai) / [TEI](https://github.com/huggingface/text-embeddings-inference) / llama.cpp, or hosted Cohere/Jina/Voyage).
**Disabled by default.** In the console **Models** tab, add a model definition whose `base_url` is a Cohere/Jina-compatible `/rerank` endpoint and whose capabilities include `{"supports_rerank": true}`, then select it under **Models → Roles → Reranker**. It's managed like every other model (write-only key, enable/disable, calibration). The reranker is purely this per-model definition — there is no global `rerank_url`-style endpoint setting.
The `rerank_web_search` toggle defaults on once a reranker is selected. If the endpoint is unreachable or errors, web_search falls back silently to the backend's native result order — reranking never makes a search fail.
When `rerank_bm25` is enabled, the candidate text for memory, tool, and skill retrieval (memory name/description/content and tool/skill names + descriptions) is also sent to the rerank endpoint — a self-hosted endpoint (vLLM/TEI/llama.cpp) keeps it on your infrastructure, a hosted provider (Cohere/Jina/Voyage) sends it off-box.
**Serving a Qwen3-Reranker with vLLM.** The model is instruction-aware, so vLLM **must** apply its chat template — pass `--chat-template` explicitly. Without it the bare query produces near-random scores and reranking actively *hurts* retrieval (verified: an irrelevant passage outscored the correct one):
```bash
vllm serve /models/Qwen3-Reranker-0.6B \
--runner pooling \
--hf-overrides '{"architectures":["Qwen3ForSequenceClassification"],"classifier_from_token":["no","yes"],"is_original_qwen3_reranker":true}' \
--chat-template /models/Qwen3-Reranker-0.6B/chat_template.jinja \
--served-model-name qwen3-reranker --port 8000
```
Then add a reranker model in the **Models** tab with `base_url` `http://vllm:8000/rerank` (model name `qwen3-reranker`) and select it under **Models → Roles → Reranker**.
For an endpoint that does *not* apply the model's template, set `rerank_instruction` instead — Turnstone then wraps each query as `<Instruct>: {instruction}` / `<Query>: {query}` (Qwen3's own default is `Given a web search query, retrieve relevant passages that answer the query`). Use the chat template **or** the instruction, not both (they double-wrap).
**Picking `rerank_bm25_threshold`.** The relevance floor that gates proactive memory injection is a probability in `[0, 1]`, but the right value differs per model (a sharp 0.6B reranker may want ~0.95; a broader 4B ~0.33). Calibrate it against your endpoint:
```bash
turnstone-admin rerank-calibrate # probe the endpoint, recommend a floor
turnstone-admin rerank-calibrate --apply # ...and write tools.rerank_bm25_threshold
```
It reports the score scale, whether the endpoint cleanly separates relevant from irrelevant probes (a **"no clean separation"** result flags a mis-served or weak reranker), and the suggested floor. Leave the threshold at `0` to rerank-without-filtering.
- **Agent availability**: `agent` and `task_agent`.
---
## Agent
The tool name uses the `_agent` suffix — bare `task` collides with
Tool names use the `_agent` suffix — bare `plan` / `task` collide with
chat-template channel names on some local models.
### task_agent
@@ -358,9 +368,23 @@ Delegate a general-purpose task to an autonomous sub-agent.
|-----------|--------|----------|-------------|
| `prompt` | string | yes | Complete task description for the sub-agent. |
- **What it does**: Spawns a sub-agent that inherits the `TASK_AGENT_TOOLS` set (read, write, edit, search, bash, web tools, memory tools). The sub-agent runs autonomously to completion. Use for work that requires file modifications or command execution.
- **What it does**: Spawns a sub-agent that inherits the `TASK_AGENT_TOOLS` set (read, write, edit, search, bash, math, man, web tools, memory tools). The sub-agent runs autonomously to completion. Use for work that requires file modifications or command execution.
- **Auto-approve**: No -- requires user confirmation.
- **Agent availability**: Top-level only.
- **Agent availability**: Not available to sub-agents (top-level only).
---
### plan_agent
Plan before implementing -- an autonomous agent explores the codebase and writes a structured plan.
| Parameter | Type | Required | Description |
|-----------|--------|----------|-------------|
| `prompt` | string | yes | What to plan -- the goal, constraints, and scope. |
- **What it does**: Spawns a planning sub-agent with `AGENT_TOOLS` (read-only tools: `read_file`, `search`, `math`, `man`, `web_fetch`, `web_search`). The agent explores the codebase and writes a structured plan to `.plan-<ws_id>.md` (unique per workstream, so concurrent workstreams never collide). If the `plan` tool has been called before in the same session, the prior plan is passed to the agent as context so it refines rather than restarts. After completion, the user is prompted to review and can accept, reject, or annotate the plan.
- **Auto-approve**: No -- requires user confirmation, plus post-execution review gate.
- **Agent availability**: Not available to sub-agents (top-level only).
---
@@ -421,7 +445,7 @@ Provide either `username` for user-based targeting or `channel_type` +
- **What it does**: Sends a notification via the channel gateway's HTTP endpoint (`POST /v1/api/notify`). The server queries the `services` table for healthy channel gateways, authenticates with a service JWT (`aud: turnstone-channel`), and delivers to the first healthy gateway. On failure, retries up to 2 additional times with backoff (1s, 3s). Rate-limited to 5 notifications per turn (counter only increments on success).
- **Auto-approve**: Yes — notifications are time-sensitive and auto-approved so the model can alert users urgently.
- **Agent availability**: `task_agent`.
- **Agent availability**: `agent` and `task_agent`.
> See [Channel Integrations: Notifications](channels.md#notifications)
> for the full delivery flow, service registry details, and security
@@ -497,7 +521,7 @@ data.get("mergedAt") is not None
- Duplicate names rejected within the same workstream.
- **Auto-approve**: `create` requires approval; `list` and `cancel` are auto-approved.
- **Agent availability**: Main session only — not available to task sub-agents.
- **Agent availability**: Main session only — not available to plan/task sub-agents.
> See [Watch Architecture](diagrams/png/18-watch-architecture.png) for the
> full poll → evaluate → dispatch flow.
@@ -530,30 +554,33 @@ pre-configure skills at workstream creation.
- **Auto-approve**: `load` requires approval (changes session behavior); `search`
is auto-approved (read-only).
- **Agent availability**: Main session only — not available to task sub-agents.
- **Agent availability**: Main session only — not available to plan/task sub-agents.
---
## Summary Table
| Tool | Category | Auto-approve | task_agent | primary_key |
|--------------|------------|--------------|------------|-------------|
| `bash` | File Ops | No | Yes | `command` |
| `read_file` | File Ops | Yes | Yes | `path` |
| `write_file` | File Ops | No | Yes | `content` |
| `edit_file` | File Ops | No | Yes | `old_string`|
| `search` | File Ops | Yes | Yes | `query` |
| `web_fetch` | Info | No | Yes | `url` |
| `web_search` | Info | No | Yes | `query` |
| `task_agent` | Agent | No | No | `prompt` |
| `memory` | Memory | Yes | No | `name` |
| `recall` | Memory | Yes | No | `query` |
| `notify` | Notify | Yes | Yes | `message` |
| `watch` | Monitor | No (create) | No | `command` |
| `read_resource`| MCP | No | Yes | `uri` |
| `use_prompt` | MCP | No | Yes | `name` |
| `skill` | Skills | No (load) | No | `name` |
| `tool_search`| Search | Yes | No | `query` |
| Tool | Category | Auto-approve | agent | task_agent | primary_key |
|--------------|------------|--------------|-------|------------|-------------|
| `bash` | File Ops | No | No | Yes | `command` |
| `read_file` | File Ops | Yes | Yes | Yes | `path` |
| `write_file` | File Ops | No | No | Yes | `content` |
| `edit_file` | File Ops | No | No | Yes | `old_string`|
| `search` | File Ops | Yes | Yes | Yes | `query` |
| `math` | Compute | No | Yes | Yes | `code` |
| `man` | Info | Yes | Yes | Yes | `page` |
| `web_fetch` | Info | No | Yes | Yes | `url` |
| `web_search` | Info | No | Yes | Yes | `query` |
| `task_agent` | Agent | No | No | No | `prompt` |
| `plan_agent` | Agent | No | No | No | `goal` |
| `memory` | Memory | Yes | No | No | `name` |
| `recall` | Memory | Yes | No | No | `query` |
| `notify` | Notify | Yes | Yes | Yes | `message` |
| `watch` | Monitor | No (create) | No | No | `command` |
| `read_resource`| MCP | No | Yes | Yes | `uri` |
| `use_prompt` | MCP | No | Yes | Yes | `name` |
| `skill` | Skills | No (load) | No | No | `name` |
| `tool_search`| Search | Yes | No | No | `query` |
---
@@ -605,9 +632,8 @@ CLI flags override the config file:
directly.
2. **Partitioning**: When active, tools are split into two sets:
- **Always-on** -- the built-in tools present in the current session
(interactive sessions currently have 16; `BUILTIN_TOOL_NAMES` is the
28-tool built-in union). These are always visible to the model.
- **Always-on** -- the 19 built-in tools (members of `BUILTIN_TOOL_NAMES`).
These are always visible to the model.
- **Deferred** -- all MCP tools. These are not sent in the tool list unless
the model searches for them.
@@ -621,10 +647,10 @@ CLI flags override the config file:
### Agent exemption
Task sub-agents do not use tool search. They operate on the scoped tool set
(`TASK_AGENT_TOOLS`) with MCP tools merged in. Tool search is only active for
the top-level session, where the model can interactively search for tools it
needs.
Plan and task sub-agents do not use tool search. They operate on scoped tool
sets (`AGENT_TOOLS` for plan agents, `TASK_AGENT_TOOLS` for task agents) with
MCP tools merged in. Tool search is only active for the top-level session,
where the model can interactively search for tools it needs.
---
@@ -649,7 +675,7 @@ MCP-compatible service.
3. **Schema conversion**: Each MCP tool's `inputSchema` is converted to OpenAI
function-calling format. The tool name is prefixed: `mcp__{server}__{tool}`.
4. **Merging**: MCP tools are appended after the 16 built-in tools via
4. **Merging**: MCP tools are appended after the 19 built-in tools via
`merge_mcp_tools()`. Built-in tools appear first, giving them natural LLM priority.
When dynamic tool search is active, MCP tools are deferred rather than directly
visible -- the model discovers them via search as needed (see
@@ -675,6 +701,7 @@ gives per-tool-type granularity (e.g., all `use_prompt` calls).
MCP tools are available to:
- **Main session** — full access
- **Task sub-agents** — via `self._task_tools` (merged list)
- **Plan sub-agents** — via `self._agent_tools` (merged list)
### Naming convention
@@ -747,7 +774,7 @@ MCP tool lists stay up-to-date without restart through two mechanisms:
When tools change, `MCPClientManager` rebuilds its merged tool list using copy-on-write
(new list/dict objects assigned atomically) and notifies all active `ChatSession`
instances via registered listener callbacks. Each session rebuilds its `_tools`,
`_task_tools`, and reconstructs its `ToolSearchManager` (if active),
`_task_tools`, `_agent_tools`, and reconstructs its `ToolSearchManager` (if active),
preserving the set of previously expanded (discovered) tools.
```
@@ -803,7 +830,7 @@ Use read_resource(uri='...') to access the resources listed above.
- **What it does**: Reads the resource from its MCP server via `MCPClientManager.read_resource_sync()`. Returns text content for text resources or base64-encoded data for binary resources. Output is truncated by the standard tool output limiter.
- **Auto-approve**: No -- requires user confirmation (reads external data).
- **Agent availability**: `task_agent`.
- **Agent availability**: `agent` and `task_agent`.
### Capability guards
@@ -845,7 +872,7 @@ the `initialize` handshake. Each prompt is stored with its prefixed name
- **What it does**: Invokes an MCP prompt by name via `MCPClientManager.get_prompt_sync()`, expanding it into messages. Returns the expanded prompt content formatted as `[role]: content` blocks joined with blank lines. The prompt catalog is listed in the system message so the model knows which prompts are available. Output is truncated by the standard tool output limiter.
- **Auto-approve**: No -- requires user confirmation (invokes external prompt servers).
- **Agent availability**: `task_agent`.
- **Agent availability**: `agent` and `task_agent`.
### Invocation
-9
View File
@@ -1,9 +0,0 @@
__pycache__/
.venv/
*.db
*.db-wal
*.db-shm
.ruff_cache/
.pytest_cache/
.mypy_cache/
uv.lock
-282
View File
@@ -1,282 +0,0 @@
# Understone
A small, multiplayer, BBS-style **ANSI door game** served over the Model
Context Protocol (MCP). It is a text RPG in the spirit of *Legend of the Red
Dragon* — explore an overworld of box-drawing maps, fight wandering monsters,
shop and rest in town, and descend a dungeon — except the "door" is an MCP
server and the player drives it by talking to an AI assistant.
The server is the rules engine and the single source of truth. Players share
**one persistent world**: your assistant calls tools, the server returns
authoritative frames and facts, and the assistant narrates the story around
them.
This is a self-contained reference example. It depends only on `mcp` — there
is no dependency on Turnstone itself — so it runs against any MCP client.
## How to play
There is **no prompt to paste and no persona to configure**. The tool schema
is the whole interface. Once the server is registered with your assistant:
1. Tell your assistant you'd like to play an ANSI door game / text dungeon
RPG (it can discover the tools by name and description).
2. The assistant calls `door_help` to learn how to run the world, then
`door_join` with your adventurer's name.
3. Play unfolds as a conversation: "head east", "fight it", "rest at the inn".
Everything the assistant needs to run the game well is returned by
`door_help`.
## Gameplay
A run is a little RPG loop, played a bit each day:
- **Explore** the overworld of box-drawing maps. Walking is free, but the wild
country has texture — a step may turn up a wandering monster, a purse of
gold, a healing spring, a small trap (which can never kill you), or a scrap
of old Vale lore. Only one such find happens per move, and the non-combat
ones don't interrupt your walk.
- **Fight, shop, and heal** in and around town. Fighting and descending one
rung of the dungeon each spend one of your daily turns; resting, shopping and
moving do not.
- **Delve the deep, a rung at a time.** The dungeon is a ladder of guardians:
each `descend` faces the next one past your deepest and either advances your
depth or bounces you home (your depth persists either way). Carry a few
**potions in your satchel**`quaff` the strongest when you choose, and if a
fight would kill you the satchel saves you automatically, the elixir burning
down your throat at death's edge. Clearing a rung also yields **forge ore**,
which rides the satchel (a won forest fight sometimes turns up a little, too).
- **Forge an edge — with gold AND ore.** At the shop's **forge** you can add a
+1 edge to your equipped weapon or armour, up to a cap, each step dearer than
the last. A step costs gold *and* the ore you won in the deep — so the forge is
fed by descending, not just by a fat purse. Watch, too, for the **rare beasts**
that prowl the forest: felling one is Herald news and always drops a draught.
- **Win the game** by slaying **the Wyrm Below**. Once your hero is seasoned
enough AND has plumbed the deep to its floor, `challenge` it at the dungeon. A
victory frees the Vale, carves your run into the **Hall of Legends**, and — in
the tradition of the classic BBS door games — begins a new life: your
character resets to first-day gear and stats but keeps a permanent ★ for every
Wyrm slain, ready to do it all again.
- **Read the news.** `door_log` is the **Understone Herald**, a shared
broadsheet of notable deeds across the whole world — who joined, who rose a
level, who was dragged home by a goblin, and who freed the Vale.
- **Make it social.** It is a shared world, so you can touch other players.
`ambush` a rival who has not yet acted today — a classic
style player-kill that robs a sleeping foe of some gold, except the surest
defence is simply to take your own turn (an active player is awake and can't
be caught). Lose the ambush and *you* are the one who flees, shamed on the
feed. `post` a private note another player reads on their next visit (it
never reaches the public Herald). Or `gamble` a little gold at the inn's dice
against the house. Ambush spends a turn; mail and dice do not.
- **Bank your coin.** The inn keeps a strongbox: `deposit` gold into the
**vault** and `withdraw` it later (no turn either way). Banked gold is **safe
from ambush** — a sleeping-robber only ever lifts what you carry — and it is
the one thing that **survives a Wyrm-win reset**, carrying wealth across runs.
## Installation
This example uses [`uv`](https://docs.astral.sh/uv/). From the example
directory:
```bash
cd examples/door-game
uv venv
uv pip install -e .
```
That installs the `understone` entry point into the environment.
To run the tests and quality gates:
```bash
uv pip install -e ".[test,dev]"
uv run pytest
uv run ruff check .
uv run ruff format --check .
uv run mypy understone/
```
## Running the server
By default the server speaks the **stdio** transport, which is how MCP clients
launch a per-session subprocess:
```bash
understone
```
To host one shared world over HTTP for several clients, run the
**streamable-http** transport as a single long-lived process:
```bash
UNDERSTONE_TRANSPORT=streamable-http understone
```
### Environment variables
| Variable | Default | Description |
|----------|---------|-------------|
| `UNDERSTONE_DB` | `./understone.db` | SQLite database file for the world's state. |
| `UNDERSTONE_WORLD` | _(packaged pack)_ | Directory of a content pack to load instead of the bundled Vale of Understone. |
| `UNDERSTONE_TRANSPORT` | `stdio` | `stdio` or `streamable-http`. |
| `UNDERSTONE_HOST` | `127.0.0.1` | Bind host (streamable-http only). |
| `UNDERSTONE_PORT` | `8077` | Bind port (streamable-http only). |
| `UNDERSTONE_PATH` | `/mcp` | HTTP path for the MCP endpoint (streamable-http only). |
## The Watch — a live spectator view
When the server runs under the **streamable-http** transport, it also serves a
read-only **Watch** page: the lobby TV of the Vale. Point a browser at
```
http://127.0.0.1:8077/watch
```
(the host and port follow `UNDERSTONE_HOST` / `UNDERSTONE_PORT`). It is a
period **CRT spectator console** — a green-and-amber phosphor map of the whole
world with every adventurer's `☻` marker, a live **Understone Herald** feed, the
**Hall of Legends**, and a roster of who is currently abroad. It refreshes every
couple of seconds; if it loses contact it dims and reads `SIGNAL LOST` until the
server returns. The console's palette follows the pack: a world may pick its own
CRT colour with `settings.watch_theme` (`phosphor` green, `amber` gold, `ice`
blue, `ember` red), defaulting to the Vale's green if it says nothing.
The Watch is **strictly read-only**. Input never flows through it — there are no
controls, no forms, nothing that can change the world. It reads the same shared
state the tools do and paints it; that is all. There is no authentication, in
keeping with the rest of this easter-egg server (see the safety note below), so
treat the page as you would the MCP endpoint itself.
> _Screenshot: the Watch console — a phosphor-green overworld map with amber
> `☻` markers, the Herald feed and Hall of Legends down the right-hand rail.
> (Image placeholder; run the server and open the URL to see it live.)_
When the Watch is up, the `door_join` welcome and the `door_help` manual both
print its URL so players (and the assistant narrating for them) know it exists.
If you bind to `0.0.0.0` to share the world across a network, advertise a host
that browsers can actually reach (your machine's LAN address or hostname) rather
than `0.0.0.0` itself — the link is composed from `UNDERSTONE_HOST`.
## Authoring worlds
The Vale of Understone is just the *bundled* world. The whole game — its map,
monsters, economy, and endgame — is a **content pack**: a directory of six JSON
files the server loads at start. Nothing about the Vale is privileged; point
the server at another pack and it runs that world instead. This is the seam
where the game becomes its own authoring target: a pack is plain data, so a
person *or an LLM* can write one, and the same zero-setup philosophy that makes
the game playable with no prompt makes it **authorable with no code**.
The loop has these commands:
```bash
understone newpack mypack # scaffold a pack (copies the Vale as a template)
# ...edit or LLM-generate the JSON in mypack/ to describe your world...
understone validate mypack # check it; prints a report or names what's wrong
understone simulate mypack # play a greedy bot through it and measure the balance
UNDERSTONE_WORLD=mypack understone # serve your world
understone worlds # list the bundled worlds and whether each is sound
```
`newpack` writes a starting template plus an `AUTHORING.md` manual — the
file-by-file schema, the enforced limits, and design guidance — written to be
followed cold by a model. `validate` loads the pack through exactly the same
hardened loader the server uses and either prints a summary ending **"This pack
is sound. The door stands open."** or fails with one precise line naming the
file, the row, and the field at fault.
`simulate` is the **balance instrument**: it drives a deliberately simple,
greedy bot through the *real* game — the same `join`/`move`/`action` calls the
tools make — over a seeded RNG and an injected clock, then prints a report
(final level, gold earned, fights fought, rungs cleared, whether and when the
Wyrm fell). It is a tuning probe, not a player to admire: it answers "is this
world *shaped* right, and is it *winnable*?". Pass `--days N`, `--seed S`, or
`--seeds K` for a multi-seed sweep with means and spreads. `worlds` lists every
bundled world — the default Vale plus any alternate packs shipped under
`understone/world/packs/` — loading each so it can report it as sound or flawed.
**A second bundled world: The Cinder Wastes.** Understone ships a second world
alongside the Vale, in `understone/world/packs/cinder-wastes/` — a volcanic
ash-and-slag map whose Watch page glows ember-red instead of the Vale's green
phosphor. It is the pipeline's own dogfood: it was authored **by an LLM working
only from `AUTHORING.md` and the `validate` loop**, with no engine code touched,
then bundled verbatim. `understone worlds` lists it as sound, and
`understone simulate understone/world/packs/cinder-wastes --days 50 --seeds 3`
shows the greedy bot taking its Magma Wyrm — the end-to-end proof that a world
described purely as data, from the manual alone, is genuinely playable to
victory. Serve it with
`UNDERSTONE_WORLD=understone/world/packs/cinder-wastes understone`.
Packs are validated **hard** at load: every map glyph must render as exactly
one terminal column (no fullwidth runes, no emoji, no combining marks — the
frames are box-drawing rectangles) and may not collide with the frame's
box-drawing lines or the player markers, dimensions and counts are bounded,
display names are length-checked, and every cross-reference (a legend
character, a starting item, the boss monster, a dungeon tier) must resolve. The
loader also pins the rules that keep the endgame coherent: a world has exactly
one boss, and a dungeon tier's lead monster (its fixed rung guardian) may not be
a rare. Because packs are now routinely untrusted, generated output, those error
messages are not a nuisance — they are the **feedback loop**. Iterate against
them until the door stands open.
## Registering with Turnstone
Understone is an ordinary MCP server, so it plugs into Turnstone's MCP client
config two ways.
**Stdio (per-session subprocess).** Turnstone launches the `understone`
command for each session. Each session gets its own subprocess, so for a
truly shared world prefer the HTTP form below; stdio is simplest for solo
play.
```toml
[mcp.servers.understone]
command = "understone"
[mcp.servers.understone.env]
UNDERSTONE_DB = "/var/lib/understone/world.db"
```
**Streamable-HTTP (one shared world).** Run a single Understone process with
`UNDERSTONE_TRANSPORT=streamable-http` and point every client at its URL. This
is the right setup for multiplayer: one process, one database, one world that
all adventurers share.
```toml
[mcp.servers.understone]
url = "http://localhost:8077/mcp"
```
> **Operator note.** For multiplayer, start exactly one shared process —
> `UNDERSTONE_TRANSPORT=streamable-http understone` — and have all clients use
> the url form. The world lives in a single SQLite file written by that one
> process.
## The tools
| Tool | What it does |
|------|--------------|
| `door_help` | The game-master manual. Start here. |
| `door_join` | Create or resume an adventurer; returns the opening map. |
| `door_status` | The character sheet (read-only). |
| `door_look` | Redraw the current view — overworld map or location menu. |
| `door_move` | Walk the overworld (free; no daily turn spent). |
| `door_action` | Context verbs: fight, flee, ambush (a rival), rest, deposit/withdraw (the inn vault), buy, sell, forge (a +1 edge, gold + ore), heal, gamble (inn dice), descend (one rung), challenge (the Wyrm), post (mail another player), quaff (a carried potion), leave. |
| `door_log` | The Understone Herald — the shared feed of notable deeds. |
| `door_rank` | The leaderboard, plus the Hall of Legends (★ marks Wyrm kills). |
| `door_bestow` | Game-master grant of a little gold/healing for a story beat. |
## A note on identity and safety
This example is an **easter egg**, not a hardened service. Identity is
**self-asserted**: a "player" is just a name passed to the tools, and there is
**no authentication** — anyone who can reach the server can act as any name.
That is fine for a shared toy world among people who trust each other, and
deliberately out of scope for a game. Do not store anything sensitive in it,
and if you expose the HTTP transport beyond localhost, put it behind whatever
access control your environment already provides.
The game master's `door_bestow` channel can only grant small, capped amounts
of in-game gold and healing — never items, never turns — and every grant is
written to the public in-world log, so its reach is bounded by design.
-55
View File
@@ -1,55 +0,0 @@
[build-system]
requires = ["hatchling>=1.29"]
build-backend = "hatchling.build"
[project]
name = "understone"
version = "0.10.0"
description = "Understone — a BBS-style ANSI door game served over MCP."
requires-python = ">=3.11"
license = "Apache-2.0"
dependencies = [
"mcp>=1.27,<2",
]
[project.scripts]
understone = "understone.server:main"
[project.optional-dependencies]
test = ["pytest>=9.0"]
dev = ["ruff>=0.9", "mypy>=1.14"]
[tool.hatch.build.targets.wheel]
packages = ["understone"]
[tool.pytest.ini_options]
testpaths = ["tests"]
[tool.ruff]
target-version = "py311"
line-length = 100
[tool.ruff.lint]
select = ["E", "F", "W", "I", "N", "UP", "B", "A", "SIM", "TCH"]
ignore = ["E501"]
[tool.ruff.format]
quote-style = "double"
[tool.mypy]
python_version = "3.11"
strict = true
warn_return_any = true
warn_unused_configs = true
disallow_untyped_defs = true
disallow_incomplete_defs = true
check_untyped_defs = true
no_implicit_optional = true
[[tool.mypy.overrides]]
module = ["mcp", "mcp.*"]
ignore_missing_imports = true
[[tool.mypy.overrides]]
module = "tests.*"
disallow_untyped_defs = false
-256
View File
@@ -1,256 +0,0 @@
"""Shared test fixtures and builders.
These builders construct engine objects directly (no JSON loader) so the
engine tests stay independent of the content pack. Later chunks add
fixtures that load the shipped world and build the game façade.
"""
from __future__ import annotations
from collections import Counter
from datetime import UTC, datetime
from typing import TYPE_CHECKING
import pytest
from understone.engine.models import (
Item,
LocationDef,
Mode,
Monster,
Player,
Settings,
Slot,
TerrainDef,
WorldEvent,
Zone,
)
from understone.engine.world import World
if TYPE_CHECKING:
from collections.abc import Callable
from understone.game import Game
# ---------------------------------------------------------------------------
# Terrain kinds for synthetic test worlds
# ---------------------------------------------------------------------------
GRASS = TerrainDef(key="grass", glyph=".", walkable=True, encounter_rate=0.0, color="floor")
WALL = TerrainDef(key="wall", glyph="", walkable=False, encounter_rate=0.0, color="wall")
WATER = TerrainDef(key="water", glyph="~", walkable=False, encounter_rate=0.0, color="water")
FOREST = TerrainDef(key="forest", glyph="", walkable=True, encounter_rate=1.0, color="tree")
SAFE_FOREST = TerrainDef(key="forest", glyph="", walkable=True, encounter_rate=0.0, color="tree")
DEFAULT_SETTINGS = Settings(
daily_turns=10,
rest_cost=15,
heal_cost_per_hp=2,
starting_gold=20,
starting_weapon="rusty_dagger",
starting_armor="cloth_tunic",
start_hp=20,
start_atk=3,
start_def=0,
xp_base=100,
growth_max_hp=6,
growth_atk=2,
growth_def=1,
bestow_daily_budget=25,
dungeon_tiers=(4, 5),
boss_monster="wyrm_below",
wyrm_min_level=6,
ambush_min_level=3,
ambush_level_band=2,
ambush_gold_pct=25,
post_daily_cap=5,
gamble_max_bet=50,
gamble_daily_cap=5,
satchel_max=3,
forge_base_cost=60,
forge_max_plus=3,
rare_drop_item="minor_potion",
forge_ore_item="iron_ore",
forge_ore_per_plus=1,
ore_dungeon_drop=2,
ore_forest_chance=0.2,
watch_theme="phosphor",
)
def make_settings(**overrides: object) -> Settings:
"""Return DEFAULT_SETTINGS with field overrides for band testing."""
base = {
"daily_turns": DEFAULT_SETTINGS.daily_turns,
"rest_cost": DEFAULT_SETTINGS.rest_cost,
"heal_cost_per_hp": DEFAULT_SETTINGS.heal_cost_per_hp,
"starting_gold": DEFAULT_SETTINGS.starting_gold,
"starting_weapon": DEFAULT_SETTINGS.starting_weapon,
"starting_armor": DEFAULT_SETTINGS.starting_armor,
"start_hp": DEFAULT_SETTINGS.start_hp,
"start_atk": DEFAULT_SETTINGS.start_atk,
"start_def": DEFAULT_SETTINGS.start_def,
"xp_base": DEFAULT_SETTINGS.xp_base,
"growth_max_hp": DEFAULT_SETTINGS.growth_max_hp,
"growth_atk": DEFAULT_SETTINGS.growth_atk,
"growth_def": DEFAULT_SETTINGS.growth_def,
"bestow_daily_budget": DEFAULT_SETTINGS.bestow_daily_budget,
"dungeon_tiers": DEFAULT_SETTINGS.dungeon_tiers,
"boss_monster": DEFAULT_SETTINGS.boss_monster,
"wyrm_min_level": DEFAULT_SETTINGS.wyrm_min_level,
"ambush_min_level": DEFAULT_SETTINGS.ambush_min_level,
"ambush_level_band": DEFAULT_SETTINGS.ambush_level_band,
"ambush_gold_pct": DEFAULT_SETTINGS.ambush_gold_pct,
"post_daily_cap": DEFAULT_SETTINGS.post_daily_cap,
"gamble_max_bet": DEFAULT_SETTINGS.gamble_max_bet,
"gamble_daily_cap": DEFAULT_SETTINGS.gamble_daily_cap,
"satchel_max": DEFAULT_SETTINGS.satchel_max,
"forge_base_cost": DEFAULT_SETTINGS.forge_base_cost,
"forge_max_plus": DEFAULT_SETTINGS.forge_max_plus,
"rare_drop_item": DEFAULT_SETTINGS.rare_drop_item,
"forge_ore_item": DEFAULT_SETTINGS.forge_ore_item,
"forge_ore_per_plus": DEFAULT_SETTINGS.forge_ore_per_plus,
"ore_dungeon_drop": DEFAULT_SETTINGS.ore_dungeon_drop,
"ore_forest_chance": DEFAULT_SETTINGS.ore_forest_chance,
"watch_theme": DEFAULT_SETTINGS.watch_theme,
}
base.update(overrides)
return Settings(**base) # type: ignore[arg-type]
def make_player(**overrides: object) -> Player:
"""Build a Player at sane defaults; override any field by keyword."""
fields = {
"name": "Tester",
"x": 5,
"y": 5,
"hp": 20,
"max_hp": 20,
"level": 1,
"xp": 0,
"gold": 50,
"atk": 5,
"def_": 1,
"weapon_id": "rusty_dagger",
"armor_id": "cloth_tunic",
"turns_left": 10,
"turn_day": 0,
"mode": Mode.TILE,
"at_location": "",
"created_at": "2026-01-01T00:00:00+00:00",
"last_seen": "2026-01-01T00:00:00+00:00",
"log_cursor": 0,
"bestow_spent": 0,
"bestow_day": 0,
"wins": 0,
"posts_sent": 0,
"post_day": 0,
"gambles": 0,
"gamble_day": 0,
}
fields.update(overrides)
return Player(**fields) # type: ignore[arg-type]
def make_monster(**overrides: object) -> Monster:
"""Build a Monster at tier-1 defaults."""
fields = {
"tier": 1,
"name": "Field Rat",
"hp": 6,
"atk": 3,
"def_": 0,
"xp": 8,
"gold": 3,
"monster_id": "",
"boss": False,
}
fields.update(overrides)
return Monster(**fields) # type: ignore[arg-type]
def make_world(
*,
grid: list[list[TerrainDef]] | None = None,
width: int = 11,
height: int = 11,
spawn: tuple[int, int] = (5, 5),
locations: list[LocationDef] | None = None,
zones: list[Zone] | None = None,
monsters: list[Monster] | None = None,
items: list[Item] | None = None,
settings: Settings | None = None,
events: list[WorldEvent] | None = None,
) -> World:
"""Build a small synthetic World (all-grass by default)."""
if grid is None:
grid = [[GRASS for _ in range(width)] for _ in range(height)]
return World(
name="Test Vale",
width=width,
height=height,
spawn=spawn,
terrain=grid,
locations=locations or [],
zones=zones or [],
monsters=monsters or [make_monster()],
items=items or _default_items(),
settings=settings or DEFAULT_SETTINGS,
events=events,
)
def _default_items() -> list[Item]:
return [
Item("rusty_dagger", "Rusty Dagger", Slot.WEAPON, 2, 0, 0, 0),
Item("short_sword", "Short Sword", Slot.WEAPON, 5, 0, 0, 40),
Item("cloth_tunic", "Cloth Tunic", Slot.ARMOR, 0, 1, 0, 0),
Item("leather_armor", "Leather Armor", Slot.ARMOR, 0, 3, 0, 50),
Item("minor_potion", "Minor Potion", Slot.CONSUMABLE, 0, 0, 15, 12),
Item("iron_ore", "Iron Ore", Slot.MATERIAL, 0, 0, 0, 0),
]
def fixed_clock(moment: datetime) -> Callable[[], datetime]:
"""Return a clock callable that always reports *moment*."""
def _clock() -> datetime:
return moment
return _clock
def utc(year: int, month: int, day: int, hour: int = 0, minute: int = 0) -> datetime:
"""Construct a tz-aware UTC datetime."""
return datetime(year, month, day, hour, minute, tzinfo=UTC)
# ---------------------------------------------------------------------------
# Satchel test helpers (the v0.10 stack encoding)
# ---------------------------------------------------------------------------
# The satchel is stack-based ("id:qty"); these wrap the game façade's stack
# helpers so a test can seed/read a bag as a flat id list (duplicate ids
# collapse to one stack), keeping the assertions readable. Shared by the
# descend and Wyrm suites.
def set_satchel(game: Game, player: object, ids: list[str]) -> None:
"""Seed *player*'s satchel from a flat id list (duplicates -> one stack qty)."""
counts = Counter(ids)
stacks = [(item_id, counts[item_id]) for item_id in dict.fromkeys(ids)]
game._satchel_set_stacks(player, stacks) # type: ignore[arg-type]
def satchel_ids(game: Game, player: object) -> list[str]:
"""Return the satchel as a flat id list, each stack expanded by its qty."""
out: list[str] = []
for item_id, qty in game._satchel_stacks(player): # type: ignore[arg-type]
out.extend([item_id] * qty)
return out
@pytest.fixture
def small_world() -> World:
"""An 11x11 all-grass world with the default content tables."""
return make_world()
@@ -1,7 +0,0 @@
┌── The Sleeping Drake ───┐
│ A warm hearth crackles. │
│ A bed costs 15 gold. │
│ │
│ (R)est (L)eave │
└─────────────────────────┘
[ status ]
@@ -1,8 +0,0 @@
┌─ Vale ──┐
│@........│
│.........│
│.........│
│.........│
│.........│
└─────────┘
[ status ]
@@ -1,8 +0,0 @@
┌─ Vale ──┐
│.........│
│.........│
│....@....│
│.........│
│.........│
└─────────┘
[ status ]
-374
View File
@@ -1,374 +0,0 @@
"""Tests for the pack-authoring command surface.
Covers the validate/newpack functions directly (sound and broken packs, the
scaffold round-trip, AUTHORING.md generation from the live loader bands, and
the refuse-non-empty guard), the ``server.main`` argv dispatch (validate routes
through and bare invocation still reaches serve without binding a port), and
one end-to-end subprocess smoke of ``python -m understone validate``.
"""
from __future__ import annotations
import json
import shutil
import subprocess
import sys
from io import StringIO
from pathlib import Path
from typing import TYPE_CHECKING, Any
import pytest
from understone import cli, server
from understone.world import loader
if TYPE_CHECKING:
from collections.abc import Callable
EXAMPLE_DIR = Path(__file__).resolve().parents[1]
SHIPPED = EXAMPLE_DIR / "understone" / "world" / "data"
# The six content files a scaffolded pack must carry, plus the manual.
_PACK_JSONS = {
"terrain.json",
"monsters.json",
"items.json",
"locations.json",
"events.json",
"world.json",
}
# ---------------------------------------------------------------------------
# cli_validate
# ---------------------------------------------------------------------------
def test_cli_validate_sound_pack_reports_and_returns_zero() -> None:
out, err = StringIO(), StringIO()
rc = cli.cli_validate(SHIPPED, out=out, err=err)
assert rc == 0
report = out.getvalue()
assert "This pack is sound. The door stands open." in report
# The report surfaces the headline facts the brief calls for.
assert "The Vale of Understone" in report
assert "96x48" in report
assert "1 boss" in report
assert "% fight" in report
assert err.getvalue() == ""
def test_cli_validate_broken_pack_names_field_and_returns_two(tmp_path: Path) -> None:
# A pack whose daily_turns is out of band: the loader names the field.
pack = _clone_shipped(tmp_path)
_patch_world(pack, _break_daily_turns)
out, err = StringIO(), StringIO()
rc = cli.cli_validate(pack, out=out, err=err)
assert rc == 2
message = err.getvalue()
assert message.startswith("The pack is flawed:")
assert "daily_turns" in message # the offending field is named
assert out.getvalue() == ""
def test_cli_validate_missing_directory_returns_two(tmp_path: Path) -> None:
out, err = StringIO(), StringIO()
rc = cli.cli_validate(tmp_path / "nope", out=out, err=err)
assert rc == 2
assert "The pack is flawed:" in err.getvalue()
# ---------------------------------------------------------------------------
# cli_newpack
# ---------------------------------------------------------------------------
def test_cli_newpack_writes_template_and_manual(tmp_path: Path) -> None:
dest = tmp_path / "mypack"
out, err = StringIO(), StringIO()
rc = cli.cli_newpack(dest, out=out, err=err)
assert rc == 0
present = {p.name for p in dest.iterdir()}
assert present >= _PACK_JSONS # the six content files are all there
assert "AUTHORING.md" in present
# Next-steps guidance points the author at the validate verb.
assert "understone validate" in out.getvalue()
def test_cli_newpack_scaffold_validates(tmp_path: Path) -> None:
"""The load-bearing test: a freshly scaffolded pack loads cleanly.
newpack -> load_world round-trip. If the template the scaffolder copies
ever drifts out of the loader's bands, this fails immediately.
"""
dest = tmp_path / "mypack"
assert cli.cli_newpack(dest, out=StringIO(), err=StringIO()) == 0
world = loader.load_world(dest)
assert world.name == "The Vale of Understone"
assert world.width == 96
def test_cli_newpack_authoring_md_renders_live_band(tmp_path: Path) -> None:
"""AUTHORING.md's bands are generated from the loader, not hand-copied.
The daily_turns band is read straight from the live loader table and must
appear verbatim in the scaffolded manual proving generation from source.
"""
dest = tmp_path / "mypack"
cli.cli_newpack(dest, out=StringIO(), err=StringIO())
manual = (dest / "AUTHORING.md").read_text(encoding="utf-8")
lo, hi = loader.SETTINGS_BANDS["daily_turns"]
assert lo is not None and hi is not None
assert f"`{lo}..{hi}`" in manual
assert "daily_turns" in manual
def test_cli_newpack_authoring_md_has_width_rule_and_live_palette(tmp_path: Path) -> None:
"""AUTHORING.md documents the one-column rule and renders the live palette.
The width section states the Western-monospace assumption, and the safe
palette is generated from ``textwidth.SAFE_PALETTE`` (same can't-drift
pattern as the bands table) every glyph appears, in a backticked cell.
"""
from understone.engine.textwidth import SAFE_PALETTE
dest = tmp_path / "mypack"
cli.cli_newpack(dest, out=StringIO(), err=StringIO())
manual = (dest / "AUTHORING.md").read_text(encoding="utf-8")
assert "## Glyph width" in manual
assert "exactly one terminal column" in manual
assert "Western monospace" in manual # the stated assumption
assert "Safe glyph palette" in manual
for glyph in SAFE_PALETTE:
assert f"`{glyph}`" in manual, f"palette glyph {glyph!r} missing from manual"
def test_cli_newpack_authoring_md_documents_action_sets(tmp_path: Path) -> None:
"""AUTHORING.md documents each building's real verb menu.
The per-building menus are an explicit table: the inn's `gamble` (v0.8) and
the v0.10 vault verbs `deposit`/`withdraw`, the shop's `forge`, and so on.
This pins the table rows and the "quaff anywhere" note so a doc regression
trips.
"""
dest = tmp_path / "mypack"
cli.cli_newpack(dest, out=StringIO(), err=StringIO())
manual = (dest / "AUTHORING.md").read_text(encoding="utf-8")
assert "| `inn` | `rest`, `deposit`, `withdraw`, `gamble`, `leave` |" in manual
assert "| `shop` | `buy`, `sell`, `forge`, `leave` |" in manual
assert "| `healer` | `heal`, `leave` |" in manual
assert "| `dungeon` | `descend`, `challenge`, `leave` |" in manual
assert "`quaff`" in manual and "legal **anywhere**" in manual
# The vault is described where its verbs are listed.
assert "VAULT" in manual and "SAFE from ambush" in manual
def test_cli_newpack_authoring_md_documents_ore_forge(tmp_path: Path) -> None:
"""AUTHORING.md documents the v0.10 ore-gated forge: material slot + settings.
The forge ore is a `material` item earned in combat; the four ore settings
(item, per-plus, dungeon drop, forest chance) are documented, and the band
figures are generated from the live loader so they cannot drift.
"""
dest = tmp_path / "mypack"
cli.cli_newpack(dest, out=StringIO(), err=StringIO())
manual = (dest / "AUTHORING.md").read_text(encoding="utf-8")
assert "`material`" in manual # the new slot
assert "forge_ore_item" in manual
assert "ore_forest_chance" in manual # the float setting (prose, not the band table)
# The two banded ore settings carry their LIVE bands.
lo, hi = loader.SETTINGS_BANDS["ore_dungeon_drop"]
assert f"`{lo}..{hi}`" in manual
assert "earns in combat" in manual or "earned in combat" in manual
def test_cli_newpack_authoring_md_states_color_advisory_and_spawn_walkable(
tmp_path: Path,
) -> None:
"""AUTHORING.md states color is advisory (loader does not validate it) and
that spawn must be on walkable terrain both v0.8 honesty fixes."""
dest = tmp_path / "mypack"
cli.cli_newpack(dest, out=StringIO(), err=StringIO())
manual = (dest / "AUTHORING.md").read_text(encoding="utf-8")
# color is documented as advisory / not validated (it matches loader behaviour).
assert "advisory and not validated" in manual
# spawn's walkability requirement is now stated where spawn is introduced.
assert "must be on walkable terrain" in manual
def test_cli_newpack_authoring_md_color_roles_generated_from_enum(tmp_path: Path) -> None:
"""AUTHORING.md's colour-role vocabulary is generated from the Color enum.
The v0.9 fix: the assignable roles were hand-listed (and went stale road
and the per-building roles were missing). They are now generated from
``Color.assignable()`` the single source for the overlay-vs-assignable
split so the manual lists exactly what the Watch can paint and cannot
drift. This asserts the NEW roles appear, that every assignable enum role
appears, and that the non-assignable roles (overlays + DEFAULT) are NOT
offered as author-assignable.
"""
from understone.screen.palette import Color
dest = tmp_path / "mypack"
cli.cli_newpack(dest, out=StringIO(), err=StringIO())
manual = (dest / "AUTHORING.md").read_text(encoding="utf-8")
# A sampling of the new v0.9 roles is offered in the manual, backticked.
for role in ("road", "forest", "lava", "barren", "inn", "shop", "healer"):
assert f"`{role}`" in manual, f"new colour role {role!r} missing from manual"
# EVERY assignable enum role appears (generated, so the full set is present).
color_section = manual[manual.index("`color` — a palette role string") :].split("###", 1)[0]
for role in Color.assignable():
assert f"`{role.value}`" in manual, f"assignable role {role.value!r} missing from manual"
# The non-assignable roles (runtime overlays + the DEFAULT fallback) are NOT
# offered as terrain/location colours.
non_assignable = {c for c in Color} - set(Color.assignable())
assert Color.DEFAULT in non_assignable # the fallback is not author-pickable
for role in non_assignable:
assert f"`{role.value}`" not in color_section, (
f"non-assignable role {role.value!r} wrongly offered as author-assignable"
)
def test_cli_newpack_authoring_md_has_validate_coverage_split(tmp_path: Path) -> None:
"""AUTHORING.md honestly separates machine-enforced rules from eyeball-only.
The v0.8 subsection lists what `validate` DOES catch (including the two new
enforcements rare-as-guardian and single-boss) and what it does NOT (chief
among them: location menu `actions` contents are unvalidated).
"""
dest = tmp_path / "mypack"
cli.cli_newpack(dest, out=StringIO(), err=StringIO())
manual = (dest / "AUTHORING.md").read_text(encoding="utf-8")
assert "What `validate` checks, and what it cannot" in manual
# The newly-enforced rules are named in the DOES-catch list.
assert "Exactly one boss" in manual
assert "fixed rung guardian) must" in manual # rare-as-guardian enforcement
# The eyeball-only short list names the actions gap and the flavour caveat.
assert "Location menu `actions` contents" in manual
assert "Flavour and narration quality" in manual
def test_cli_newpack_refuses_non_empty_dir(tmp_path: Path) -> None:
dest = tmp_path / "occupied"
dest.mkdir()
(dest / "keep.txt").write_text("mine", encoding="utf-8")
out, err = StringIO(), StringIO()
rc = cli.cli_newpack(dest, out=out, err=err)
assert rc == 2
assert "non-empty" in err.getvalue()
# The pre-existing file is untouched (nothing was scaffolded over it).
assert (dest / "keep.txt").read_text(encoding="utf-8") == "mine"
assert not (dest / "AUTHORING.md").exists()
def test_cli_newpack_into_empty_existing_dir_succeeds(tmp_path: Path) -> None:
"""An existing but empty directory is a fine scaffold target."""
dest = tmp_path / "empty"
dest.mkdir()
assert cli.cli_newpack(dest, out=StringIO(), err=StringIO()) == 0
assert (dest / "AUTHORING.md").exists()
# ---------------------------------------------------------------------------
# server.main argv dispatch
# ---------------------------------------------------------------------------
def test_main_validate_dispatch_returns_status(
tmp_path: Path, capsys: pytest.CaptureFixture
) -> None:
# A broken pack routed through main exits 2; a sound one exits 0.
pack = _clone_shipped(tmp_path)
_patch_world(pack, _break_daily_turns)
with pytest.raises(SystemExit) as broken:
server.main(["validate", str(pack)])
assert broken.value.code == 2
with pytest.raises(SystemExit) as sound:
server.main(["validate", str(SHIPPED)])
assert sound.value.code == 0
assert "The door stands open." in capsys.readouterr().out
def test_main_newpack_dispatch(tmp_path: Path) -> None:
dest = tmp_path / "viamain"
with pytest.raises(SystemExit) as exc:
server.main(["newpack", str(dest)])
assert exc.value.code == 0
assert (dest / "AUTHORING.md").exists()
def test_main_worlds_dispatch(capsys: pytest.CaptureFixture) -> None:
"""`understone worlds` routes through main, exits 0, and lists the Vale."""
with pytest.raises(SystemExit) as exc:
server.main(["worlds"])
assert exc.value.code == 0
out = capsys.readouterr().out
assert "vale" in out
assert "The Vale of Understone" in out
assert "UNDERSTONE_WORLD=" in out
def test_bare_invocation_resolves_to_serve_without_side_effects() -> None:
"""Parsing no argv yields the serve path, and parsing has no side effects.
The transport launch (_serve) is reachable, but argument parsing neither
loads a world nor binds a port so this asserts the resolved command
without ever calling _serve.
"""
args = server._build_parser().parse_args([])
assert args.cmd is None # None => the serve branch in main()
assert callable(server._serve)
def test_subprocess_validate_packaged_world_exits_zero() -> None:
"""End-to-end smoke: `python -m understone validate <packaged dir>` exits 0."""
result = subprocess.run(
[sys.executable, "-m", "understone", "validate", str(SHIPPED)],
cwd=EXAMPLE_DIR,
capture_output=True,
text=True,
timeout=60,
)
assert result.returncode == 0, result.stderr
assert "The door stands open." in result.stdout
# ---------------------------------------------------------------------------
# helpers
# ---------------------------------------------------------------------------
def _clone_shipped(tmp_path: Path) -> Path:
dest = tmp_path / "pack"
shutil.copytree(SHIPPED, dest)
return dest
def _patch_world(pack: Path, mutate: Callable[[dict[str, Any]], None]) -> None:
path = pack / "world.json"
data = json.loads(path.read_text(encoding="utf-8"))
mutate(data)
path.write_text(json.dumps(data), encoding="utf-8")
def _break_daily_turns(data: dict[str, Any]) -> None:
"""Set daily_turns out of its 1..100 band so the pack fails to load."""
data["settings"]["daily_turns"] = 0
-123
View File
@@ -1,123 +0,0 @@
"""Combat resolution tests.
Pins determinism (a fixed seed yields identical results twice, log and
deltas), each outcome (win/lose/flee), xp/gold crediting on victory, and
the defeat contract: the result flags a spawn bounce with no xp/gold and a
zero hp delta (the façade applies hp=1 and the move).
"""
from __future__ import annotations
from tests.conftest import make_monster, make_player
from understone.engine.combat import Outcome, resolve_fight, resolve_flee
from understone.engine.rng import GameRNG
# A strong adventurer vs a Field Rat wins on every probed seed.
_WIN_SEED = 1
# A fragile adventurer vs a Stone Wyrm loses on every probed seed.
_LOSE_SEED = 0
# Flee outcomes (probed): seed 1 escapes clean, seed 0 is caught.
_FLEE_CLEAN_SEED = 1
_FLEE_CAUGHT_SEED = 0
def _strong_player() -> object:
return make_player(hp=20, max_hp=20, atk=5, def_=1, xp=0, gold=50)
def _wyrm() -> object:
return make_monster(tier=5, name="Stone Wyrm", hp=60, atk=18, def_=6, xp=140, gold=60)
def test_fight_is_deterministic_under_fixed_seed() -> None:
r1 = resolve_fight(GameRNG(seed=7), make_player(), make_monster())
r2 = resolve_fight(GameRNG(seed=7), make_player(), make_monster())
assert r1.log == r2.log
assert (r1.outcome, r1.xp_delta, r1.gold_delta, r1.hp_delta) == (
r2.outcome,
r2.xp_delta,
r2.gold_delta,
r2.hp_delta,
)
def test_win_credits_xp_and_gold() -> None:
player = make_player(hp=20, max_hp=20, atk=5, def_=1)
monster = make_monster(hp=6, atk=3, def_=0, xp=8, gold=3)
result = resolve_fight(GameRNG(seed=_WIN_SEED), player, monster)
assert result.outcome is Outcome.WIN
assert result.xp_delta == 8
assert result.gold_delta == 3
# hp_delta is non-positive (you may take a scratch) and never fatal here.
assert result.hp_delta <= 0
assert not result.bounce_to_spawn
def test_win_deltas_are_exact_for_pinned_seed() -> None:
player = make_player(hp=20, max_hp=20, atk=5, def_=1)
monster = make_monster(hp=6, atk=3, def_=0, xp=8, gold=3)
result = resolve_fight(GameRNG(seed=_WIN_SEED), player, monster)
# Pinned from a determinism probe; guards against silent damage drift.
assert result.hp_delta == -1
# The engine no longer emits a "falls + reward" line — that sentence is
# composed by the game façade where the xp/gold are actually banked — so
# the WIN log is one line shorter than before and ends on the kill blow.
assert len(result.log) == 4
assert result.log[-1] == "You strike for 6. (Field Rat: 0 HP)"
def test_win_log_does_not_claim_rewards() -> None:
"""The engine narrates the kill blow only; it never claims xp/gold itself.
Reward ownership lives in the façade (so the Wyrm-win legacy reset, which
keeps no xp/gold, narrates no reward). The deltas are still carried on the
result for the caller to apply.
"""
player = make_player(hp=20, max_hp=20, atk=5, def_=1)
monster = make_monster(hp=6, atk=3, def_=0, xp=8, gold=3)
result = resolve_fight(GameRNG(seed=_WIN_SEED), player, monster)
assert result.outcome is Outcome.WIN
assert result.xp_delta == 8 and result.gold_delta == 3 # deltas still set
joined = "\n".join(result.log)
assert "falls" not in joined # no kill/reward sentence in the engine log
assert "XP" not in joined and "gold" not in joined
def test_loss_flags_bounce_without_rewards() -> None:
result = resolve_fight(GameRNG(seed=_LOSE_SEED), _strong_player_loses(), _wyrm())
assert result.outcome is Outcome.LOSE
assert result.bounce_to_spawn is True
assert result.xp_delta == 0
assert result.gold_delta == 0
# Combat does not set hp to 1 itself — that is the façade's job.
assert result.hp_delta == 0
def _strong_player_loses() -> object:
return make_player(hp=12, max_hp=12, atk=4, def_=0)
def test_flee_can_escape_clean() -> None:
player = make_player(hp=20, max_hp=20, def_=1)
monster = make_monster(atk=8, def_=2)
result = resolve_flee(GameRNG(seed=_FLEE_CLEAN_SEED), player, monster)
assert result.outcome is Outcome.FLED
assert result.hp_delta == 0
def test_flee_caught_costs_hp_but_never_kills() -> None:
player = make_player(hp=20, max_hp=20, def_=1)
monster = make_monster(atk=8, def_=2)
result = resolve_flee(GameRNG(seed=_FLEE_CAUGHT_SEED), player, monster)
assert result.outcome is Outcome.FLED
assert result.hp_delta < 0
# A caught flight cannot drop the player to or below zero.
assert player.hp + result.hp_delta >= 1
def test_flee_caught_never_kills_at_low_hp() -> None:
player = make_player(hp=1, max_hp=20, def_=0)
monster = make_monster(atk=40, def_=0)
result = resolve_flee(GameRNG(seed=_FLEE_CAUGHT_SEED), player, monster)
# At 1 HP the most a failed flee can cost is 0 (cannot go below 1).
assert result.hp_delta == 0
File diff suppressed because it is too large Load Diff
-858
View File
@@ -1,858 +0,0 @@
"""Game façade integration tests over the shipped world.
Drives a full session against a temp store, a frozen clock, and a seeded
RNG: join -> status -> look -> move -> action(buy/rest/fight) -> log ->
rank -> bestow. Persistence is exercised by reopening the store.
Negative-test discipline (turn guard and bestow cap):
Two guards are pinned by assertions here. To confirm each assertion has
teeth, the implementer temporarily reverted the guard line and observed
the matching test FAIL, then restored it:
* Turn guard (engine/turns.py spend_turn): replacing
``if player.turns_left <= 0: return False`` with ``return True``
let fighting continue past the daily budget ``test_turn_budget_blocks``
then failed on the "spent for today" assertion. Restored.
* Bestow cap (game.py bestow): removing the ``if cost > remaining``
refusal let an over-budget bestowal through ``test_bestow_cap_refuses``
then failed on the unchanged-gold assertion. Restored.
* Sanitizer control-char guard (game.py _sanitize): disabling the
``not cleaned.isprintable()`` clause let a newline-injected name create a
player row and a public event ``test_join_rejects_control_char_name``
then failed. Restored. (See the comment block above the hygiene tests.)
"""
from __future__ import annotations
import unicodedata
from pathlib import Path
import pytest
from tests.conftest import fixed_clock, utc
from understone.engine.models import Mode
from understone.engine.rng import GameRNG
from understone.game import Game
from understone.persistence import Store
from understone.world.loader import load_world
PACK = Path(__file__).resolve().parents[1] / "understone" / "world" / "data"
@pytest.fixture
def clock() -> object:
return fixed_clock(utc(2026, 6, 12, 10, 0))
def _game(tmp_path: Path, clock: object, seed: int = 7) -> Game:
world = load_world(PACK)
store = Store(tmp_path / "game.db")
return Game(world, store, clock=clock, rng=GameRNG(seed=seed)) # type: ignore[arg-type]
# ---------------------------------------------------------------------------
# Join / status / look
# ---------------------------------------------------------------------------
def test_join_creates_player_at_spawn(tmp_path: Path, clock: object) -> None:
game = _game(tmp_path, clock)
out = game.join("Brandr")
player = game.players["Brandr"]
assert (player.x, player.y) == game.world.spawn
assert player.gold == game.world.settings.starting_gold
assert "@" in out
assert game.world.name in out
def test_join_resumes_existing(tmp_path: Path, clock: object) -> None:
game = _game(tmp_path, clock)
game.join("Brandr")
game.players["Brandr"].gold = 123
out = game.join("Brandr")
assert "Welcome back" in out
assert game.players["Brandr"].gold == 123
def test_status_unknown_player_is_friendly(tmp_path: Path, clock: object) -> None:
game = _game(tmp_path, clock)
out = game.status("Nobody")
assert "has signed the ledger" in out
assert "door_join" in out
def test_look_overworld_has_frame(tmp_path: Path, clock: object) -> None:
game = _game(tmp_path, clock)
game.join("Brandr")
out = game.look("Brandr")
assert "@" in out
assert "" in out and "" in out
assert len(out) < 2048
def test_overworld_frame_textured_borders_intact(tmp_path: Path, clock: object) -> None:
"""The textured overworld frame keeps square borders and a single player marker.
Structural discipline for the v0.6 texture: variants change the GLYPHS but
must never change the geometry. The box rows are uniform width, exactly one
'@' is painted, and the grass field shows more than one variant in a row
(the deterministic stipple, not a flat sheet of '.').
"""
game = _game(tmp_path, clock)
game.join("Brandr")
frame = game.look("Brandr")
lines = frame.split("\n")
# Box rows: top border + VIEW_H grid rows + bottom border, all equal width.
box = [ln for ln in lines if ln and ln[0] in "┌│└"]
widths = {len(ln) for ln in box}
assert len(widths) == 1, f"textured frame rows ragged: {widths}"
# Exactly one player marker, regardless of the surrounding texture.
assert frame.count("@") == 1
# The grass texture varies: a body row carries at least two of . , '
body = [ln for ln in lines if ln.startswith("")]
assert any(len({ch for ch in ln if ch in ".,'"}) >= 2 for ln in body)
def test_look_in_menu_shows_location(tmp_path: Path, clock: object) -> None:
game = _game(tmp_path, clock)
game.join("Brandr")
# Shop is two cells east of spawn along the road.
game.move("Brandr", "", "east", 2)
assert game.players["Brandr"].mode is Mode.MENU
out = game.look("Brandr")
assert "(B)uy" in out and "(L)eave" in out
# ---------------------------------------------------------------------------
# Move
# ---------------------------------------------------------------------------
def test_move_blocked_in_menu(tmp_path: Path, clock: object) -> None:
game = _game(tmp_path, clock)
game.join("Brandr")
game.move("Brandr", "", "east", 2) # into the shop menu
out = game.move("Brandr", "", "east", 2)
assert "inside" in out.lower()
def test_move_enters_location(tmp_path: Path, clock: object) -> None:
game = _game(tmp_path, clock)
game.join("Brandr")
out = game.move("Brandr", "", "west", 2) # inn is two cells west
assert game.players["Brandr"].at_location == "inn"
assert "step inside" in out.lower()
# ---------------------------------------------------------------------------
# Actions: rest, fight, turn budget
# ---------------------------------------------------------------------------
def test_rest_heals_and_charges(tmp_path: Path, clock: object) -> None:
game = _game(tmp_path, clock)
game.join("Brandr")
player = game.players["Brandr"]
player.hp = 5
game.move("Brandr", "", "west", 2) # inn
out = game.action("Brandr", "rest", "", "")
assert player.hp == player.max_hp
assert player.gold == game.world.settings.starting_gold - game.world.settings.rest_cost
assert "full health" in out.lower()
def test_rest_when_spent_restores_a_fresh_days_turns(tmp_path: Path, clock: object) -> None:
"""Sleeping at the inn with no turns left rolls into a fresh day's allowance."""
game = _game(tmp_path, clock)
game.join("Brandr")
player = game.players["Brandr"]
daily = game.world.settings.daily_turns
player.turns_left = 0 # spent for the day
player.hp = 5
game.move("Brandr", "", "west", 2) # step into the inn
out = game.action("Brandr", "rest", "", "")
assert player.turns_left == daily # a fresh day's turns restored
assert player.hp == player.max_hp # and fully mended
assert f"/{daily} ]" in out # footer reflects the refreshed budget
def test_rest_with_turns_in_hand_never_inflates_the_budget(tmp_path: Path, clock: object) -> None:
"""Resting mid-day mends but adds no turns — the top-up only fires at zero."""
game = _game(tmp_path, clock)
game.join("Brandr")
player = game.players["Brandr"]
daily = game.world.settings.daily_turns
player.turns_left = daily - 3 # turns still in hand
player.hp = 5
game.move("Brandr", "", "west", 2) # step into the inn
game.action("Brandr", "rest", "", "")
assert player.turns_left == daily - 3 # unchanged: no farming past the cap
assert player.hp == player.max_hp # but the heal still lands
def test_fight_spends_a_turn_and_credits(tmp_path: Path, clock: object) -> None:
game = _game(tmp_path, clock)
game.join("Brandr")
player = game.players["Brandr"]
# Drop into the forest_near zone so an encounter is available.
player.x, player.y = 35, 25
before_turns = player.turns_left
out = game.action("Brandr", "fight", "", "")
assert player.turns_left == before_turns - 1
assert player.xp > 0
assert "XP" in out
def test_turn_budget_blocks(tmp_path: Path, clock: object) -> None:
"""Pins the spend_turn guard: at 0 turns, fighting is refused.
See the module docstring for the revert-and-observe-failure check that
proves this assertion has teeth.
"""
game = _game(tmp_path, clock)
game.join("Brandr")
player = game.players["Brandr"]
player.x, player.y = 35, 25
player.turns_left = 0
out = game.action("Brandr", "fight", "", "")
assert "spent for today" in out.lower()
# No turn was consumed past zero, and no XP was gained.
assert player.turns_left == 0
assert player.xp == 0
# ---------------------------------------------------------------------------
# Log / rank
# ---------------------------------------------------------------------------
def test_log_reports_then_advances(tmp_path: Path, clock: object) -> None:
game = _game(tmp_path, clock)
game.join("Brandr")
# A second player acting creates a public event Brandr has not yet seen.
game.join("Sigrun")
first = game.log("Brandr")
assert "Sigrun" in first or "Brandr" in first
assert "The Understone Herald" in first # dressed as the broadsheet
# The cursor advanced; a second read with no new events is quiet.
second = game.log("Brandr")
assert "The Understone Herald" in second # the masthead still prints
assert "still" in second.lower() # the herald-flavoured "all quiet" line
def test_rank_marks_caller(tmp_path: Path, clock: object) -> None:
game = _game(tmp_path, clock)
game.join("Brandr")
game.join("Sigrun")
game.players["Sigrun"].level = 5
out = game.rank("Brandr")
assert "Brandr" in out and "Sigrun" in out
assert "*" in out # the caller's row is marked
assert "" in out # box-drawing table
# ---------------------------------------------------------------------------
# Rank ★ column: stars live in their own column, so a long name keeps them
# ---------------------------------------------------------------------------
def test_win_stars_column_formats() -> None:
"""Zero is blank, 1..5 render as ★ runs, and >5 collapses to ★xN."""
from understone.game import _win_stars
assert _win_stars(0) == ""
assert _win_stars(1) == ""
assert _win_stars(5) == "★★★★★"
assert _win_stars(7) == "★x7"
def test_long_name_with_one_win_keeps_its_star() -> None:
"""A full 24-char name no longer eats its own ★ (the v0.1 truncation bug).
The name occupied the whole 20-wide field before, clipping the star away;
with a separate stars column the survives beside a maximal name.
"""
from understone.engine.rank import RankEntry
from understone.game import _render_rank_table
name = "X" * 24
rows = _render_rank_table([RankEntry(name=name, level=5, xp=100, gold=50, wins=1)], caller="")
body = "\n".join(rows)
assert name in body # the full name is present
assert "" in body # and so is its star
def test_high_win_count_renders_compact_marker() -> None:
"""Seven wins render as the compact ``★x7`` rather than seven glyphs."""
from understone.engine.rank import RankEntry
from understone.game import _render_rank_table
rows = _render_rank_table([RankEntry(name="Champ", level=9, xp=9, gold=9, wins=7)], caller="")
body = "\n".join(rows)
assert "★x7" in body
assert "★★★★★★★" not in body # not seven literal stars
def test_shared_world_other_player_marker(tmp_path: Path, clock: object) -> None:
game = _game(tmp_path, clock)
game.join("Brandr")
game.join("Sigrun")
# Stand Sigrun one cell east of Brandr's spawn so she lands in the view.
sig = game.players["Sigrun"]
brandr = game.players["Brandr"]
sig.x, sig.y = brandr.x + 1, brandr.y
out = game.look("Brandr")
assert "" in out # the other player shows as '☻'
# ---------------------------------------------------------------------------
# Bestow (+ cap negative test)
# ---------------------------------------------------------------------------
def test_bestow_grants_gold(tmp_path: Path, clock: object) -> None:
game = _game(tmp_path, clock)
game.join("Brandr")
player = game.players["Brandr"]
before = player.gold
out = game.bestow("Brandr", "a daring rescue", 10, 0)
assert player.gold == before + 10
assert player.bestow_spent == 10
assert "bestowal" in out.lower()
def test_bestow_heal_charges_only_applied(tmp_path: Path, clock: object) -> None:
game = _game(tmp_path, clock)
game.join("Brandr")
player = game.players["Brandr"]
player.hp = player.max_hp - 3 # only 3 missing
game.bestow("Brandr", "mercy after a hard fight", 0, 10)
assert player.hp == player.max_hp
# Charged for 3 HP at heal_cost_per_hp, not the requested 10.
assert player.bestow_spent == 3 * game.world.settings.heal_cost_per_hp
def test_bestow_requires_reason(tmp_path: Path, clock: object) -> None:
game = _game(tmp_path, clock)
game.join("Brandr")
out = game.bestow("Brandr", " ", 10, 0)
assert "reason" in out.lower()
assert game.players["Brandr"].gold == game.world.settings.starting_gold
def test_bestow_requires_nonzero(tmp_path: Path, clock: object) -> None:
game = _game(tmp_path, clock)
game.join("Brandr")
out = game.bestow("Brandr", "nothing at all", 0, 0)
assert "at least" in out.lower()
def test_bestow_cap_refuses(tmp_path: Path, clock: object) -> None:
"""Pins the bestow cap: an over-budget grant is refused without mutation.
See the module docstring for the revert-and-observe-failure check that
proves this assertion has teeth.
"""
game = _game(tmp_path, clock)
game.join("Brandr")
player = game.players["Brandr"]
budget = game.world.settings.bestow_daily_budget
before_gold = player.gold
out = game.bestow("Brandr", "an absurd windfall", budget + 100, 0)
assert "the fates allow" in out.lower()
# Refused cleanly: no gold moved and no pool spent.
assert player.gold == before_gold
assert player.bestow_spent == 0
def test_bestow_pool_resets_next_day(tmp_path: Path, clock: object) -> None:
game = _game(tmp_path, clock)
game.join("Brandr")
player = game.players["Brandr"]
game.bestow("Brandr", "first blessing", 20, 0)
assert player.bestow_spent == 20
# Advance the clock past UTC midnight; the next bestow sees a fresh pool.
game.clock = fixed_clock(utc(2026, 6, 13, 0, 5)) # type: ignore[assignment]
game.bestow("Brandr", "a new day's fortune", 20, 0)
assert player.bestow_spent == 20 # reset to 0 then +20, not 40
# ---------------------------------------------------------------------------
# Persistence round-trip through the façade
# ---------------------------------------------------------------------------
def test_state_survives_store_reopen(tmp_path: Path, clock: object) -> None:
game = _game(tmp_path, clock)
game.join("Brandr")
game.players["Brandr"].x, game.players["Brandr"].y = 35, 25
game.action("Brandr", "fight", "", "")
xp_after = game.players["Brandr"].xp
gold_after = game.players["Brandr"].gold
game.store.close()
world = load_world(PACK)
reopened = Store(tmp_path / "game.db")
revived = Game(world, reopened, clock=clock) # type: ignore[arg-type]
assert revived.players["Brandr"].xp == xp_after
assert revived.players["Brandr"].gold == gold_after
# ---------------------------------------------------------------------------
# Day rollover applies to fight/descend, not just join/bestow
# ---------------------------------------------------------------------------
class _MutableClock:
"""A clock whose reported moment can be advanced between calls."""
def __init__(self, moment: object) -> None:
self.moment = moment
def __call__(self) -> object:
return self.moment
def test_fight_refreshes_budget_across_midnight(tmp_path: Path) -> None:
"""A fight on a new UTC day must reset the budget without re-joining.
Before the fix, _resolve_encounter spent a turn without calling
_ensure_day, so an exhausted player who returned the next day was still
blocked until they happened to re-join.
"""
clk = _MutableClock(utc(2026, 6, 12, 23, 0))
world = load_world(PACK)
store = Store(tmp_path / "game.db")
game = Game(world, store, clock=clk, rng=GameRNG(seed=7)) # type: ignore[arg-type]
game.join("Brandr")
player = game.players["Brandr"]
player.x, player.y = 35, 25 # forest_near zone: an encounter is available
player.turns_left = 0 # spent for the day
daily = game.world.settings.daily_turns
clk.moment = utc(2026, 6, 13, 0, 5) # cross UTC midnight, no re-join
out = game.action("Brandr", "fight", "", "")
assert "spent for today" not in out.lower() # the fresh day let the fight run
assert player.turns_left == daily - 1 # reset to full, then one spent
assert player.xp > 0
assert f"/{daily} ]" in out # footer shows the refreshed budget
def test_descend_refreshes_budget_across_midnight(tmp_path: Path) -> None:
"""Descending on a new UTC day resets the budget without re-joining."""
clk = _MutableClock(utc(2026, 6, 12, 23, 0))
world = load_world(PACK)
store = Store(tmp_path / "game.db")
game = Game(world, store, clock=clk, rng=GameRNG(seed=7)) # type: ignore[arg-type]
game.join("Hero")
player = game.players["Hero"]
# Overwhelming stats so the gauntlet itself never bounces the player.
player.level, player.atk, player.def_ = 20, 200, 100
player.hp = player.max_hp = 500
player.mode = Mode.MENU
player.at_location = "dungeon"
player.turns_left = 0
daily = game.world.settings.daily_turns
clk.moment = utc(2026, 6, 13, 0, 5)
out = game.action("Hero", "descend", "", "")
assert "too weary" not in out.lower()
assert player.turns_left == daily - 1
# ---------------------------------------------------------------------------
# Input hygiene chokepoint (the _sanitize helper)
# ---------------------------------------------------------------------------
#
# Negative-test discipline (security invariant): to prove the control-char
# rejection in Game._sanitize has teeth, the implementer temporarily replaced
# its ``not cleaned.isprintable()`` clause with ``False`` (disabling the
# check) and confirmed test_join_rejects_control_char_name FAILED — the
# injected name created a player row and a public event. The clause was then
# restored. The newline-injection test below is the standing regression for
# that invariant.
def test_join_rejects_control_char_name(tmp_path: Path, clock: object) -> None:
"""A bell/control character in a name is refused with the runes line."""
game = _game(tmp_path, clock)
out = game.join("Bra\x07ndr")
assert "strange runes" in out
assert game.players == {} # no row created
assert game.events == [] # nothing persisted
def test_join_rejects_newline_name_no_persist(tmp_path: Path, clock: object) -> None:
"""An embedded newline (log-injection vector) is refused, nothing written.
The name is kept short so it is the control-char clause not the length
clause that rejects it; this is the standing regression for the
isprintable security invariant documented in the module docstring.
"""
game = _game(tmp_path, clock)
out = game.join("Bra\nndr") # 7 chars: well under the 24 limit
assert "strange runes" in out # the runes (bad-character) refusal, not length
# The security invariant: no player row and no event row escaped the guard.
assert game.players == {}
assert game.events == []
def test_join_rejects_overlong_name(tmp_path: Path, clock: object) -> None:
"""A 25-character name is refused with the narrow-ledger line."""
game = _game(tmp_path, clock)
out = game.join("X" * 25)
assert "ledger is narrow" in out
assert game.players == {}
def test_join_accepts_max_length_name(tmp_path: Path, clock: object) -> None:
"""A 24-character name is exactly at the limit and accepted."""
game = _game(tmp_path, clock)
name = "X" * 24
game.join(name)
assert name in game.players
# ---------------------------------------------------------------------------
# Narrow-ledger width rule (the _sanitize one-column clause, v0.6)
#
# Names/reasons/mail render inside fixed-width frames and tables, so a glyph
# that does not fit a single column would shove a column out of true. The
# sanitizer rejects wide runes and combining marks; a printable-but-wide name
# gets the dedicated narrow-ledger refusal, not the control-char "runes" line.
# ---------------------------------------------------------------------------
def test_join_rejects_wide_cjk_name(tmp_path: Path, clock: object) -> None:
"""A CJK ideograph name is refused with the narrow-ledger line; nothing written."""
game = _game(tmp_path, clock)
out = game.join("")
assert "columns are narrow" in out
assert game.players == {}
assert game.events == []
def test_join_rejects_emoji_name(tmp_path: Path, clock: object) -> None:
"""An emoji in a name (🌲x) is wide and refused with the narrow-ledger line."""
game = _game(tmp_path, clock)
out = game.join("🌲x")
assert "columns are narrow" in out
assert game.players == {}
def test_join_rejects_fullwidth_name(tmp_path: Path, clock: object) -> None:
"""A fullwidth Latin letter () is two columns and refused."""
game = _game(tmp_path, clock)
out = game.join("")
assert "columns are narrow" in out
assert game.players == {}
def test_join_rejects_combining_mark_name(tmp_path: Path, clock: object) -> None:
"""A name with a combining mark (decomposed accent) is refused as wide.
The name is normalised to NFD so the 'o' carries a separate U+0308
combining diaeresis a zero-width code point that desynchronises the
column count. Built explicitly so the source encoding cannot mask it.
"""
game = _game(tmp_path, clock)
decomposed = unicodedata.normalize("NFD", "Bj\u00f6rn")
assert any(unicodedata.combining(ch) for ch in decomposed) # genuinely NFD
out = game.join(decomposed)
assert "columns are narrow" in out
assert game.players == {}
def test_join_accepts_composed_latin_name(tmp_path: Path, clock: object) -> None:
"""A precomposed Latin accent (NFC name) is all single-column and accepted."""
game = _game(tmp_path, clock)
composed = unicodedata.normalize("NFC", "Bj\u00f6rn")
game.join(composed)
assert composed in game.players
def _seed_wide_named_player(db: Path, clock: object, wide_name: str) -> None:
"""Write a stored adventurer whose name is a now-illegal wide rune.
Bypasses ``join`` (which would refuse a wide name at creation) by upserting
a Player row straight through the Store, so the fixture stands in for a save
that predates the narrow-ledger rule. Built by renaming a legitimately-
created hero so every other field stays valid.
"""
from dataclasses import replace
world = load_world(PACK)
seed = Store(db)
game = Game(world, seed, clock=clock, rng=GameRNG(seed=7)) # type: ignore[arg-type]
game.join("Brandr")
base = game.players["Brandr"]
seed.upsert_player(replace(base, name=wide_name))
seed.commit()
seed.close()
def test_join_resumes_stored_wide_name(tmp_path: Path, clock: object) -> None:
"""An existing adventurer with a wide-rune name resumes \u2014 identity is never re-gated.
Resume keys off the exact stored name BEFORE the sanitizer, so a character
whose name predates the narrow-ledger rule is welcomed back rather than
locked out. This is the resume-by-exact-name invariant.
"""
db = tmp_path / "game.db"
wide = "\u9f8d"
_seed_wide_named_player(db, clock, wide)
world = load_world(PACK)
game = Game(world, Store(db), clock=clock, rng=GameRNG(seed=7)) # type: ignore[arg-type]
out = game.join(wide)
assert "Welcome back" in out # resumed, not refused
assert "columns are narrow" not in out
assert wide in game.players
def test_join_still_refuses_new_wide_name(tmp_path: Path, clock: object) -> None:
"""Creation is still gated: a NEW wide name with no stored row is refused.
The resume bypass is exact-name only; a wide name that matches no stored
adventurer falls through to the creation gate and gets the narrow-ledger
refusal, with nothing written.
"""
db = tmp_path / "game.db"
# Seed one wide-named save, then try to CREATE a different wide name.
_seed_wide_named_player(db, clock, "\u9f8d")
world = load_world(PACK)
game = Game(world, Store(db), clock=clock, rng=GameRNG(seed=7)) # type: ignore[arg-type]
out = game.join("\u7363") # a different wide rune \u2014 no stored row for it
assert "columns are narrow" in out
assert "\u7363" not in game.players
def test_bestow_rejects_newline_reason_no_persist(tmp_path: Path, clock: object) -> None:
"""A newline-embedded bestow reason is refused; no event, pool unchanged."""
game = _game(tmp_path, clock)
game.join("Brandr")
player = game.players["Brandr"]
events_before = len(game.events)
out = game.bestow("Brandr", "heroics\nand a forged log line", 10, 0)
assert "plainly-spoken" in out
assert len(game.events) == events_before # no bestow event appended
assert player.bestow_spent == 0 # pool untouched
# ---------------------------------------------------------------------------
# Bestow: heal-only at full HP grants nothing (no empty grant persisted)
# ---------------------------------------------------------------------------
def test_bestow_heal_only_at_full_hp_refused(tmp_path: Path, clock: object) -> None:
"""A heal-only bestow at full HP applies nothing and must not persist."""
game = _game(tmp_path, clock)
game.join("Brandr")
player = game.players["Brandr"]
assert player.hp == player.max_hp # join starts at full health
events_before = len(game.events)
out = game.bestow("Brandr", "a quiet blessing", 0, 10)
assert "already hale" in out
assert len(game.events) == events_before # no "Fortune favours" line written
assert player.bestow_spent == 0 # nothing charged
# ---------------------------------------------------------------------------
# Descend the deep: one rung per descent (see test_descend.py for the ladder)
# ---------------------------------------------------------------------------
def test_descend_fights_one_rung_and_advances(tmp_path: Path, clock: object) -> None:
"""A strong player clears the next rung: one foe fought, rewards banked, depth +1."""
game = _game(tmp_path, clock)
game.join("Hero")
player = game.players["Hero"]
player.level, player.atk, player.def_ = 20, 200, 100
player.hp = player.max_hp = 500
player.mode = Mode.MENU
player.at_location = "dungeon"
before_turns, before_gold, before_xp = player.turns_left, player.gold, player.xp
out = game.action("Hero", "descend", "", "")
# The first rung is the tier-3 guardian (Forest Wolf); deeper rungs do NOT
# appear in one descent — the deep is fought a rung at a time now.
assert "Forest Wolf" in out
assert "Cave Troll" not in out
assert player.deepest_rung == 1
assert player.turns_left == before_turns - 1
assert player.gold > before_gold
assert player.xp > before_xp
def test_descend_bounces_weak_player_to_spawn(tmp_path: Path, clock: object) -> None:
"""A fresh weak player falls on the first rung and wakes at the spawn.
Depth is NOT advanced by a loss, but it persists at whatever it was (here 0).
"""
game = _game(tmp_path, clock)
game.join("Weakling")
player = game.players["Weakling"]
player.mode = Mode.MENU
player.at_location = "dungeon"
out = game.action("Weakling", "descend", "", "")
assert player.hp == 1
assert player.mode is Mode.TILE
assert player.at_location == ""
assert (player.x, player.y) == game.world.spawn
assert player.deepest_rung == 0 # a loss never advances the deep
# Felled by the first rung (the tier-3 Forest Wolf).
assert "Forest Wolf" in out
# ---------------------------------------------------------------------------
# Shop façade: buy / upgrade / sell / heal stat arithmetic
# ---------------------------------------------------------------------------
def test_shop_buy_upgrade_sell_heal_cycle(tmp_path: Path, clock: object) -> None:
"""Equip deltas apply once on buy/upgrade and unwind cleanly on sell."""
game = _game(tmp_path, clock)
game.join("Brandr")
player = game.players["Brandr"]
player.gold = 1000
player.mode = Mode.MENU
player.at_location = "shop"
short_sword = game.world.item_by_id("short_sword")
war_axe = game.world.item_by_id("war_axe")
starter = game.world.item_by_id(game.world.settings.starting_weapon)
assert short_sword is not None and war_axe is not None and starter is not None
starter_atk = player.atk # 3 base + rusty dagger bonus
# Buy the short sword: gold falls by its price, atk rises by the delta.
gold0 = player.gold
game.action("Brandr", "buy", "", "short_sword")
assert player.gold == gold0 - short_sword.price
assert player.atk == starter_atk + (short_sword.atk - starter.atk)
atk_with_sword = player.atk
# Upgrade to the war axe: atk reflects the difference, not a double-add.
gold1 = player.gold
game.action("Brandr", "buy", "", "war_axe")
assert player.gold == gold1 - war_axe.price
assert player.atk == atk_with_sword + (war_axe.atk - short_sword.atk)
# Sell the war axe: half-price refund, atk falls back to the starter bonus.
gold2 = player.gold
game.action("Brandr", "sell", "", "")
assert player.gold == gold2 + war_axe.price // 2
assert player.atk == starter_atk
# Heal at the shrine: HP restored, gold debited per missing point.
player.mode = Mode.MENU
player.at_location = "healer"
player.hp = player.max_hp - 5
per_hp = game.world.settings.heal_cost_per_hp
gold3 = player.gold
game.action("Brandr", "heal", "", "")
assert player.hp == player.max_hp
assert player.gold == gold3 - 5 * per_hp
def test_sell_starter_weapon_refused(tmp_path: Path, clock: object) -> None:
"""The starter blade is unsellable regardless of price (no free-gold loop)."""
game = _game(tmp_path, clock)
game.join("Brandr")
player = game.players["Brandr"]
assert player.weapon_id == game.world.settings.starting_weapon
player.mode = Mode.MENU
player.at_location = "shop"
gold_before = player.gold
out = game.action("Brandr", "sell", "", "")
assert "nothing worth selling" in out.lower()
assert player.gold == gold_before
# ---------------------------------------------------------------------------
# Bounded in-memory event tail (full history stays in SQLite)
# ---------------------------------------------------------------------------
def test_event_tail_is_capped_but_log_still_works(tmp_path: Path, clock: object) -> None:
"""Loading caps the resident tail; door_log still serves recent events."""
from understone.engine.log import since
from understone.game import EVENT_TAIL_KEEP
db = tmp_path / "game.db"
seed_store = Store(db)
last_id = 0
for i in range(EVENT_TAIL_KEEP + 50):
last_id = seed_store.insert_event("t", "sys", "note", f"event {i}")
seed_store.commit()
seed_store.close()
world = load_world(PACK)
game = Game(world, Store(db), clock=clock, rng=GameRNG(seed=7)) # type: ignore[arg-type]
# Only the most recent EVENT_TAIL_KEEP events are resident in memory.
assert len(game.events) == EVENT_TAIL_KEEP
assert game.events[-1].event_id == last_id
# door_log still reports events after a recent cursor.
recent_cursor = game.events[-3].event_id
game.join("Brandr")
game.players["Brandr"].log_cursor = recent_cursor
out = game.log("Brandr")
assert "The Understone Herald" in out # broadsheet masthead
assert "since your last visit" in out
fresh, new_cursor = since(game.events, recent_cursor)
assert fresh # there are events past the cursor
assert new_cursor == game.events[-1].event_id
def test_private_mail_survives_tail_eviction(tmp_path: Path, clock: object) -> None:
"""A private note older than the resident tail is still delivered (durable mail).
Public history that falls off the in-memory tail is gone by design (the
broadsheet does not keep), but mail must not be: a note left while the
recipient was away has to surface however many public events have since
pushed it out of the tail. A third player whose cursor also predates the
note must still never see it, because it was never theirs.
"""
from understone.persistence import EVENT_TAIL_KEEP
db = tmp_path / "game.db"
store = Store(db)
game = Game(load_world(PACK), store, clock=clock, rng=GameRNG(seed=7)) # type: ignore[arg-type]
game.join("Scribe")
game.join("Reader")
game.join("Bystander")
# Scribe leaves Reader a private note; neither Reader nor Bystander reads it.
secret = "the cellar key is under the third barrel"
game.action("Scribe", "post", "Reader", "", secret)
# Flood the feed past the tail bound so the note is evicted from memory.
for i in range(EVENT_TAIL_KEEP + 20):
store.insert_event("t", "sys", "note", f"broadsheet filler {i}")
store.commit()
store.close()
# Reopen: only the newest tail is resident, so the note now lives in the gap.
reopened = Store(db)
revived = Game(load_world(PACK), reopened, clock=clock, rng=GameRNG(seed=7)) # type: ignore[arg-type]
note_id = next(
e.event_id
for e in reopened.targeted_events_since("Reader", 0) # note: from SQLite, not the tail
if secret in e.text
)
assert note_id < revived.events[0].event_id # the note really is past the tail
# The recipient still sees the note, backfilled from SQLite...
reader_log = revived.log("Reader")
assert secret in reader_log
assert "While you were away" in reader_log
# ...but a third player never does, even though their cursor predates it too.
third_log = revived.log("Bystander")
assert secret not in third_log
reopened.close()
-118
View File
@@ -1,118 +0,0 @@
"""XP curve, level-up, and restorative-maths tests.
Pins the threshold edges (at / just below / just above), a multi-level
jump from a single award, the exact growth table, the inn's flat-rate
full heal with affordability gating, and the healer's per-HP cost maths.
"""
from __future__ import annotations
from tests.conftest import DEFAULT_SETTINGS, make_player, make_settings
from understone.engine.leveling import apply_xp, heal, rest, xp_for_level
# Default curve is 100 * (n-1)*n/2 cumulative:
# L2 = 100, L3 = 300, L4 = 600, L5 = 1000.
def test_xp_curve_thresholds() -> None:
assert xp_for_level(1, DEFAULT_SETTINGS) == 0
assert xp_for_level(2, DEFAULT_SETTINGS) == 100
assert xp_for_level(3, DEFAULT_SETTINGS) == 300
assert xp_for_level(4, DEFAULT_SETTINGS) == 600
assert xp_for_level(5, DEFAULT_SETTINGS) == 1000
def test_just_below_threshold_does_not_level() -> None:
player = make_player(level=1, xp=0, hp=20, max_hp=20)
gains = apply_xp(player, 99, DEFAULT_SETTINGS)
assert gains == []
assert player.level == 1
def test_exact_threshold_levels_once() -> None:
player = make_player(level=1, xp=0, hp=10, max_hp=20, atk=5, def_=1)
gains = apply_xp(player, 100, DEFAULT_SETTINGS)
assert len(gains) == 1
assert player.level == 2
# Growth table applied and a full heal granted on level-up.
assert player.max_hp == 26
assert player.atk == 7
assert player.def_ == 2
assert player.hp == player.max_hp
def test_just_above_threshold_levels_once() -> None:
player = make_player(level=1, xp=0)
gains = apply_xp(player, 101, DEFAULT_SETTINGS)
assert len(gains) == 1
assert player.level == 2
assert player.xp == 101
def test_single_award_can_jump_multiple_levels() -> None:
player = make_player(level=1, xp=0, max_hp=20, atk=5, def_=1)
gains = apply_xp(player, 600, DEFAULT_SETTINGS)
# 600 cumulative reaches level 4 (L2=100, L3=300, L4=600).
assert player.level == 4
assert [g.new_level for g in gains] == [2, 3, 4]
# Three levels of growth stacked.
assert player.max_hp == 20 + 3 * 6
assert player.atk == 5 + 3 * 2
assert player.def_ == 1 + 3 * 1
def test_growth_table_respects_settings() -> None:
settings = make_settings(growth_max_hp=10, growth_atk=3, growth_def=2, xp_base=50)
player = make_player(level=1, xp=0, max_hp=20, atk=5, def_=1)
apply_xp(player, 50, settings) # L2 at 50 with xp_base=50
assert player.level == 2
assert player.max_hp == 30
assert player.atk == 8
assert player.def_ == 3
# ---------------------------------------------------------------------------
# rest (inn) and heal (healer)
# ---------------------------------------------------------------------------
def test_rest_full_heals_and_charges() -> None:
player = make_player(hp=5, max_hp=20, gold=50)
assert rest(player, cost=15) is True
assert player.hp == 20
assert player.gold == 35
def test_rest_refused_when_unaffordable() -> None:
player = make_player(hp=5, max_hp=20, gold=10)
assert rest(player, cost=15) is False
assert player.hp == 5
assert player.gold == 10
def test_heal_charges_only_for_hp_restored() -> None:
player = make_player(hp=15, max_hp=20, gold=100)
result = heal(player, amount=10, cost_per_hp=2)
# Only 5 HP were missing.
assert result.healed == 5
assert result.cost == 10
assert player.hp == 20
assert player.gold == 90
def test_heal_bounded_by_affordability() -> None:
player = make_player(hp=2, max_hp=20, gold=7)
result = heal(player, amount=10, cost_per_hp=2)
# 7 gold buys 3 HP at 2/hp.
assert result.healed == 3
assert result.cost == 6
assert player.hp == 5
assert player.gold == 1
def test_heal_noop_when_full() -> None:
player = make_player(hp=20, max_hp=20, gold=100)
result = heal(player, amount=10, cost_per_hp=2)
assert result.healed == 0
assert result.cost == 0
assert player.gold == 100
@@ -1,288 +0,0 @@
"""End-to-end MCP integration test — the only test that touches the network.
Boots the real Understone FastMCP app (backed by a temp DB) in a uvicorn
thread, then drives it over the real streamable-HTTP wire with the real MCP
client: initialize, list_tools (all nine door_* names), join, look. A second
client session joins a second adventurer in the SAME process and world, and
the first player's view then shows the '&' other-player marker — proving the
shared-world, single-process contract over a real wire.
A second test drives the read-only Watch routes that ride inside the same app:
GET /watch (the HTML page), /watch/world.json (the static map), and
/watch/state.json (the live snapshot) confirming the spectator endpoints
serve real world data alongside a working /mcp without breaking either.
"""
from __future__ import annotations
import asyncio
import socket
import threading
import time
from typing import TYPE_CHECKING, Any
import httpx
import pytest
import uvicorn
from mcp import ClientSession
from mcp.client.streamable_http import streamable_http_client
from understone import server as understone_server
if TYPE_CHECKING:
from pathlib import Path
PACK = str(understone_server.PACKAGED_WORLD_DIR)
def _find_free_port() -> int:
s = socket.socket()
s.bind(("127.0.0.1", 0))
port = s.getsockname()[1]
s.close()
return int(port)
def _build_server(port: int, db_path: str) -> uvicorn.Server:
app = understone_server.create_app(db_path, PACK)
config = uvicorn.Config(app, host="127.0.0.1", port=port, log_level="warning", access_log=False)
return uvicorn.Server(config)
def _wait_ready(port: int, timeout: float = 5.0) -> None:
deadline = time.monotonic() + timeout
while time.monotonic() < deadline:
try:
with socket.create_connection(("127.0.0.1", port), timeout=0.5):
return
except OSError:
time.sleep(0.05)
raise TimeoutError(f"understone server at 127.0.0.1:{port} not ready after {timeout}s")
@pytest.fixture
def live_server(tmp_path: Path) -> Any:
"""Boot the real Understone app in a background uvicorn thread."""
port = _find_free_port()
db_path = str(tmp_path / "wire.db")
server = _build_server(port, db_path)
def _run() -> None:
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
loop.run_until_complete(server.serve())
thread = threading.Thread(target=_run, daemon=True, name="understone-itest")
thread.start()
try:
_wait_ready(port)
yield f"http://127.0.0.1:{port}/mcp"
finally:
server.should_exit = True
thread.join(timeout=5)
# create_app installed a module-level game whose Store holds an open
# SQLite connection; close it and clear the singleton so the next test
# builds its own rather than inheriting this temp DB.
if understone_server._GAME is not None:
understone_server._GAME.store.close()
understone_server._GAME = None
# FastMCP caches a StreamableHTTPSessionManager on the module-level mcp
# singleton and refuses a second lifespan .run() on the same instance.
# Reset it so each fixture instance boots a fresh session manager (the
# production server only ever runs one). Without this, a second
# fixture-using test fails on "run() can only be called once".
understone_server.mcp._session_manager = None
async def _call_text(session: ClientSession, name: str, arguments: dict[str, Any]) -> str:
result = await session.call_tool(name, arguments)
chunks = [block.text for block in result.content if getattr(block, "type", None) == "text"]
return "\n".join(chunks)
async def _drive(url: str) -> dict[str, Any]:
"""Run the full client conversation and return observations."""
observations: dict[str, Any] = {}
async with (
streamable_http_client(url) as (read, write, _get_session_id),
ClientSession(read, write) as session,
):
await session.initialize()
tools = await session.list_tools()
observations["tool_names"] = sorted(t.name for t in tools.tools)
observations["join_one"] = await _call_text(session, "door_join", {"player": "Brandr"})
observations["look_one_before"] = await _call_text(
session, "door_look", {"player": "Brandr"}
)
# A SECOND, independent session joins a second adventurer in the same world.
async with (
streamable_http_client(url) as (read, write, _get_session_id),
ClientSession(read, write) as session,
):
await session.initialize()
# Place player two adjacent to player one so they share the view.
await _call_text(session, "door_join", {"player": "Sigrun"})
await _call_text(
session, "door_move", {"player": "Sigrun", "heading": "east", "distance": 1}
)
# Back as player one: the shared world now shows the other adventurer.
async with (
streamable_http_client(url) as (read, write, _get_session_id),
ClientSession(read, write) as session,
):
await session.initialize()
observations["look_one_after"] = await _call_text(
session, "door_look", {"player": "Brandr"}
)
observations["rank"] = await _call_text(session, "door_rank", {"player": "Brandr"})
return observations
def test_mcp_end_to_end(live_server: str) -> None:
obs = asyncio.run(_drive(live_server))
# All nine tools are advertised over the wire.
expected = {
"door_help",
"door_join",
"door_status",
"door_look",
"door_move",
"door_action",
"door_log",
"door_rank",
"door_bestow",
}
assert set(obs["tool_names"]) == expected
# The join + look frames are real ASCII map frames.
assert "@" in obs["join_one"]
look_before = obs["look_one_before"]
assert "@" in look_before
assert "" in look_before and "" in look_before
# Shared-world proof: after player two joins next door, player one sees '☻'.
assert "" in obs["look_one_after"]
# And the leaderboard lists both adventurers (one process, one world).
assert "Brandr" in obs["rank"]
assert "Sigrun" in obs["rank"]
def _watch_base(mcp_url: str) -> str:
"""Derive the app root (where /watch lives) from the /mcp endpoint URL."""
return mcp_url[: -len("/mcp")] if mcp_url.endswith("/mcp") else mcp_url
async def _join_over_mcp(mcp_url: str, name: str) -> None:
"""Sign one adventurer in over the real MCP wire (so state.json sees them)."""
async with (
streamable_http_client(mcp_url) as (read, write, _get_session_id),
ClientSession(read, write) as session,
):
await session.initialize()
await _call_text(session, "door_join", {"player": name})
def test_watch_routes_serve_world_state(live_server: str) -> None:
base = _watch_base(live_server)
# The MCP join writes the player into the shared world the routes read.
asyncio.run(_join_over_mcp(live_server, "Watcher"))
with httpx.Client(timeout=5.0) as client:
page = client.get(f"{base}/watch")
world = client.get(f"{base}/watch/world.json")
state = client.get(f"{base}/watch/state.json")
# The page is real HTML carrying the static masthead.
assert page.status_code == 200
assert page.headers["content-type"].startswith("text/html")
assert "Understone — Live Watch" in page.text
# The static world payload matches the loaded world.
assert world.status_code == 200
world_body = world.json()
assert world_body["width"] == 96
assert world_body["height"] == 48
assert len(world_body["glyph_rows"]) == world_body["height"]
assert all(len(row) == world_body["width"] for row in world_body["glyph_rows"])
# The live snapshot lists the adventurer who joined over MCP.
assert state.status_code == 200
state_body = state.json()
names = {p["name"] for p in state_body["players"]}
assert "Watcher" in names
def test_watch_routes_coexist_with_mcp(live_server: str) -> None:
"""The custom routes don't shadow /mcp: tool calls still work alongside them."""
base = _watch_base(live_server)
async def _drive_both() -> tuple[str, int]:
async with (
streamable_http_client(live_server) as (read, write, _get_session_id),
ClientSession(read, write) as session,
):
await session.initialize()
joined = await _call_text(session, "door_join", {"player": "Coexist"})
with httpx.Client(timeout=5.0) as client:
status = client.get(f"{base}/watch/state.json").status_code
return joined, status
joined, watch_status = asyncio.run(_drive_both())
assert "@" in joined # the MCP tool still returns a real frame
assert watch_status == 200 # and the watch route still answers
def test_streamable_http_host_gate_off_localhost() -> None:
"""A non-localhost bind must accept remote `Host` headers on /mcp.
REGRESSION: FastMCP freezes DNS-rebinding protection (a localhost-only Host
allowlist) at CONSTRUCTION, and ``server`` builds its FastMCP at import with
the default 127.0.0.1 host. A 0.0.0.0/LAN bind therefore answered TCP and
`/watch` but 421'd `/mcp` for every remote node ("Invalid Host header").
``_serve`` drops the allowlist when bound off localhost; this pins the
mechanism a default instance rejects a foreign Host, a protection-disabled
one accepts it (a 421 in the second case is the bug returning).
Uses fresh FastMCP instances (not the module singleton) so there is no
shared-state or app-cache coupling with the live-server tests above.
"""
from mcp.server.fastmcp import FastMCP
from mcp.server.transport_security import TransportSecuritySettings
from starlette.testclient import TestClient
foreign = {
"Host": "192.168.0.239:8077",
"Accept": "application/json, text/event-stream",
"Content-Type": "application/json",
}
init = {
"jsonrpc": "2.0",
"id": 1,
"method": "initialize",
"params": {
"protocolVersion": "2025-03-26",
"capabilities": {},
"clientInfo": {"name": "probe", "version": "0"},
},
}
# Default (localhost-baked allowlist) — a remote Host is refused.
locked = FastMCP("hostgate-locked")
with TestClient(locked.streamable_http_app()) as client:
assert client.post("/mcp", headers=foreign, json=init).status_code == 421
# Protection disabled (what _serve does off localhost) — remote Host accepted.
opened = FastMCP("hostgate-open")
opened.settings.transport_security = TransportSecuritySettings(
enable_dns_rebinding_protection=False
)
with TestClient(opened.streamable_http_app()) as client:
resp = client.post("/mcp", headers=foreign, json=init)
assert resp.status_code != 421, f"remote Host still rejected: {resp.status_code} {resp.text}"
-336
View File
@@ -1,336 +0,0 @@
"""Movement resolution tests.
Covers edge clipping on all four sides, blocking terrain, the two input
forms (``"NNEE"`` vs heading+distance) and their equivalence, location
entry flipping to MENU, the MAX_STEPS cap, and a stubbed always-encounter
RNG interrupting a walk with a pending fight.
"""
from __future__ import annotations
from tests.conftest import (
FOREST,
GRASS,
WALL,
WATER,
LocationDef,
Zone,
make_player,
make_world,
)
from understone.engine.models import Mode, WorldEvent
from understone.engine.movement import MAX_STEPS, parse_directions, resolve_move
from understone.engine.rng import GameRNG
class _NeverRNG(GameRNG):
"""An RNG whose chance() never fires (no wandering encounters)."""
def __init__(self) -> None:
super().__init__(seed=0)
def chance(self, probability: float) -> bool: # noqa: ARG002
return False
class _AlwaysRNG(GameRNG):
"""An RNG whose chance() always fires (forces an encounter).
The seed still drives ``weighted_index``/``randint``, so different seeds
select different event rows while every encounter roll fires.
"""
def __init__(self, seed: int = 0) -> None:
super().__init__(seed=seed)
def chance(self, probability: float) -> bool: # noqa: ARG002
return True
# ---------------------------------------------------------------------------
# parse_directions
# ---------------------------------------------------------------------------
def test_parse_steps_string() -> None:
assert parse_directions("NNEE", "", 1) == ["N", "N", "E", "E"]
def test_parse_heading_distance() -> None:
assert parse_directions("", "east", 3) == ["E", "E", "E"]
def test_parse_clamps_to_max_steps() -> None:
assert parse_directions("NNNNNNNNNNNN", "", 1) == ["N"] * MAX_STEPS
assert parse_directions("", "north", 99) == ["N"] * MAX_STEPS
def test_parse_rejects_unknown_direction() -> None:
try:
parse_directions("NQ", "", 1)
except ValueError as exc:
assert "Q" in str(exc)
else: # pragma: no cover - failure path
raise AssertionError("expected ValueError")
# ---------------------------------------------------------------------------
# Edge clipping (all four sides)
# ---------------------------------------------------------------------------
def test_clip_north_edge() -> None:
world = make_world()
player = make_player(x=5, y=0)
result = resolve_move(world, player, _NeverRNG(), heading="north", distance=3)
assert player.y == 0
assert result.steps_taken == 0
assert result.blocked
def test_clip_south_edge() -> None:
world = make_world()
player = make_player(x=5, y=10)
result = resolve_move(world, player, _NeverRNG(), heading="south", distance=3)
assert player.y == 10
assert result.blocked
def test_clip_west_edge() -> None:
world = make_world()
player = make_player(x=0, y=5)
result = resolve_move(world, player, _NeverRNG(), heading="west", distance=3)
assert player.x == 0
assert result.blocked
def test_clip_east_edge() -> None:
world = make_world()
player = make_player(x=10, y=5)
result = resolve_move(world, player, _NeverRNG(), heading="east", distance=3)
assert player.x == 10
assert result.blocked
def test_partial_move_then_clip() -> None:
world = make_world()
player = make_player(x=8, y=5)
result = resolve_move(world, player, _NeverRNG(), heading="east", distance=5)
# 8 -> 9 -> 10, then edge.
assert player.x == 10
assert result.steps_taken == 2
assert result.blocked
# ---------------------------------------------------------------------------
# Blocking terrain
# ---------------------------------------------------------------------------
def test_blocked_by_wall() -> None:
grid = [[GRASS for _ in range(11)] for _ in range(11)]
grid[5][6] = WALL
world = make_world(grid=grid)
player = make_player(x=5, y=5)
result = resolve_move(world, player, _NeverRNG(), heading="east", distance=2)
assert player.x == 5
assert result.blocked
assert "wall" in result.blocked_reason
def test_blocked_by_water() -> None:
grid = [[GRASS for _ in range(11)] for _ in range(11)]
grid[4][5] = WATER
world = make_world(grid=grid)
player = make_player(x=5, y=5)
result = resolve_move(world, player, _NeverRNG(), heading="north", distance=2)
assert player.y == 5
assert result.blocked
assert "water" in result.blocked_reason
# ---------------------------------------------------------------------------
# Input-form equivalence and direction correctness
# ---------------------------------------------------------------------------
def test_nnee_lands_at_expected_cell() -> None:
world = make_world()
player = make_player(x=5, y=5)
resolve_move(world, player, _NeverRNG(), steps="NNEE")
# Two north (y-2), two east (x+2).
assert (player.x, player.y) == (7, 3)
def test_heading_equivalent_to_steps() -> None:
world_a = make_world()
player_a = make_player(x=5, y=5)
resolve_move(world_a, player_a, _NeverRNG(), steps="EEE")
world_b = make_world()
player_b = make_player(x=5, y=5)
resolve_move(world_b, player_b, _NeverRNG(), heading="east", distance=3)
assert (player_a.x, player_a.y) == (player_b.x, player_b.y)
def test_max_steps_truncates_long_walk() -> None:
world = make_world(width=40, height=11)
player = make_player(x=0, y=5)
result = resolve_move(world, player, _NeverRNG(), heading="east", distance=99)
assert result.steps_taken == MAX_STEPS
assert player.x == MAX_STEPS
# ---------------------------------------------------------------------------
# Location entry flips to MENU
# ---------------------------------------------------------------------------
def test_entering_location_flips_menu_mode() -> None:
loc = LocationDef(
key="inn",
kind="inn",
name="The Sleeping Drake",
x=7,
y=5,
glyph="I",
color="town",
actions=("rest", "leave"),
)
world = make_world(locations=[loc])
player = make_player(x=5, y=5)
result = resolve_move(world, player, _NeverRNG(), heading="east", distance=4)
assert player.mode is Mode.MENU
assert player.at_location == "inn"
assert result.entered_location == "inn"
# Stopped on the door at x=7 even though distance asked for 4.
assert (player.x, player.y) == (7, 5)
# ---------------------------------------------------------------------------
# Encounter interrupt
# ---------------------------------------------------------------------------
def test_always_encounter_stops_with_pending_fight() -> None:
grid = [[FOREST for _ in range(11)] for _ in range(11)]
zone = Zone(key="wood", x0=0, y0=0, x1=10, y1=10, tier_lo=1, tier_hi=2)
world = make_world(grid=grid, zones=[zone])
player = make_player(x=5, y=5)
result = resolve_move(world, player, _AlwaysRNG(), heading="east", distance=5)
assert result.pending_fight == (1, 2)
# The encounter fires on the first entered cell.
assert result.steps_taken == 1
assert player.x == 6
def test_no_zone_means_no_encounter() -> None:
grid = [[FOREST for _ in range(11)] for _ in range(11)]
world = make_world(grid=grid, zones=[])
player = make_player(x=5, y=5)
result = resolve_move(world, player, _AlwaysRNG(), heading="east", distance=3)
assert result.pending_fight is None
assert result.steps_taken == 3
# ---------------------------------------------------------------------------
# Weighted non-combat overworld events (v0.2)
# ---------------------------------------------------------------------------
def _event_world(*events: WorldEvent) -> object:
"""An all-forest, fully-zoned world carrying a crafted event table."""
grid = [[FOREST for _ in range(11)] for _ in range(11)]
zone = Zone(key="wood", x0=0, y0=0, x1=10, y1=10, tier_lo=1, tier_hi=2)
return make_world(grid=grid, zones=[zone], events=list(events))
def test_event_fight_stops_the_walk() -> None:
"""A fight-kind event sets pending_fight and halts the walk like v0.1."""
world = _event_world(WorldEvent("fight", 1, "", 0, 0))
player = make_player(x=5, y=5)
result = resolve_move(world, player, _AlwaysRNG(), heading="east", distance=5)
assert result.pending_fight == (1, 2)
assert result.event is None
assert result.steps_taken == 1 # stopped on the first triggering cell
def test_event_gold_credits_and_continues() -> None:
"""A gold event credits the rolled amount and does NOT stop the walk."""
world = _event_world(WorldEvent("gold", 1, "a coin-purse", 5, 5))
player = make_player(x=5, y=5, gold=10)
result = resolve_move(world, player, _AlwaysRNG(), heading="east", distance=3)
assert result.event is not None
assert result.event.kind == "gold"
assert result.event.amount == 5 # min == max == 5, so deterministic
assert player.gold == 15
assert result.pending_fight is None
assert result.steps_taken == 3 # the walk ran to completion
def test_event_heal_caps_at_max_hp() -> None:
"""A heal event never overfills: hp is clamped to max_hp."""
world = _event_world(WorldEvent("heal", 1, "a spring", 50, 50))
player = make_player(x=5, y=5, hp=18, max_hp=20)
result = resolve_move(world, player, _AlwaysRNG(), heading="east", distance=1)
assert player.hp == 20 # +50 requested, capped at the 2 missing
assert result.event is not None and result.event.amount == 2
def test_event_trap_floors_hp_at_one_and_spares_gold() -> None:
"""A trap event never kills (floors at 1 HP) and never touches gold."""
world = _event_world(WorldEvent("trap", 1, "old briars", 500, 500))
player = make_player(x=5, y=5, hp=10, max_hp=20, gold=42)
result = resolve_move(world, player, _AlwaysRNG(), heading="east", distance=1)
assert player.hp == 1 # huge trap, but floored
assert player.gold == 42 # gold untouched
assert result.event is not None and result.event.amount == 9 # only 9 could be taken
def test_event_lore_mutates_nothing() -> None:
"""A lore event changes no state and reports a zero amount."""
world = _event_world(WorldEvent("lore", 1, "an old waystone", 0, 0))
player = make_player(x=5, y=5, hp=15, max_hp=20, gold=7)
before = (player.hp, player.gold)
result = resolve_move(world, player, _AlwaysRNG(), heading="east", distance=2)
assert (player.hp, player.gold) == before
assert result.event is not None and result.event.kind == "lore"
assert result.event.amount == 0
assert result.steps_taken == 2
def test_at_most_one_event_per_walk() -> None:
"""Once any event fires, no further cells roll for the rest of the walk.
Two distinct gold rolls would credit 2 gold (1 each); a single fired event
credits exactly 1, proving the walk stops rolling after the first trigger.
"""
world = _event_world(WorldEvent("gold", 1, "a coin", 1, 1))
player = make_player(x=5, y=5, gold=0)
resolve_move(world, player, _AlwaysRNG(), heading="east", distance=5)
assert player.gold == 1 # exactly one event, not five
def test_each_event_kind_reachable_with_crafted_table() -> None:
"""Equal weights make every kind in a crafted table reachable from movement."""
table = [
WorldEvent("fight", 1, "", 0, 0),
WorldEvent("gold", 1, "g", 1, 1),
WorldEvent("heal", 1, "h", 1, 1),
WorldEvent("trap", 1, "t", 1, 1),
WorldEvent("lore", 1, "l", 0, 0),
]
zone = Zone(key="wood", x0=0, y0=0, x1=0, y1=0, tier_lo=1, tier_hi=2)
grid = [[FOREST for _ in range(11)] for _ in range(11)]
world = make_world(grid=grid, zones=[zone], events=table)
seen: set[str] = set()
for seed in range(60):
player = make_player(x=0, y=1, hp=10, max_hp=20) # one step north into the zone cell
result = resolve_move(world, player, _AlwaysRNG(seed), steps="N")
if result.pending_fight is not None:
seen.add("fight")
elif result.event is not None:
seen.add(result.event.kind)
assert seen == {"fight", "gold", "heal", "trap", "lore"}
-9
View File
@@ -1,9 +0,0 @@
"""Smoke test for the packaging skeleton."""
from __future__ import annotations
import understone
def test_version_present() -> None:
assert understone.__version__ == "0.10.0"
@@ -1,269 +0,0 @@
"""SQLite persistence tests.
Covers idempotent schema init, a full player round-trip through every
column (including ``def_``, ``turn_day``, ``log_cursor`` and the bestow
fields), event append with cursor-based catch-up, leaderboard tie-breaks,
and that WAL journaling is active.
"""
from __future__ import annotations
from typing import TYPE_CHECKING
from tests.conftest import make_player
from understone.engine.log import since
from understone.engine.models import Mode
from understone.persistence import Store
if TYPE_CHECKING:
from pathlib import Path
def _store(tmp_path: Path) -> Store:
return Store(tmp_path / "understone.db")
def test_schema_init_is_idempotent(tmp_path: Path) -> None:
db = tmp_path / "understone.db"
Store(db).close()
# Re-opening the same file must not error or duplicate schema.
second = Store(db)
assert second.get_meta("schema_version") == "1"
second.close()
def test_wal_mode_active(tmp_path: Path) -> None:
store = _store(tmp_path)
assert store.journal_mode().lower() == "wal"
store.close()
def test_player_round_trip_all_columns(tmp_path: Path) -> None:
store = _store(tmp_path)
player = make_player(
name="Brandr",
x=12,
y=7,
hp=18,
max_hp=26,
level=3,
xp=305,
gold=88,
atk=9,
def_=4,
weapon_id="short_sword",
armor_id="leather_armor",
turns_left=6,
turn_day=739_400,
mode=Mode.MENU,
at_location="inn",
log_cursor=42,
bestow_spent=15,
bestow_day=739_400,
posts_sent=3,
post_day=739_400,
gambles=2,
gamble_day=739_400,
banked=420,
)
store.upsert_player(player)
store.commit()
store.close()
reopened = _store(tmp_path)
players, _ = reopened.load_all()
loaded = players["Brandr"]
assert loaded == player
assert loaded.banked == 420
# Spot-check the fields most prone to silent drop.
assert loaded.def_ == 4
assert loaded.turn_day == 739_400
assert loaded.log_cursor == 42
assert loaded.bestow_spent == 15
assert loaded.bestow_day == 739_400
assert loaded.mode is Mode.MENU
# The v0.5 social columns survive the round-trip too.
assert loaded.posts_sent == 3
assert loaded.post_day == 739_400
assert loaded.gambles == 2
assert loaded.gamble_day == 739_400
reopened.close()
def test_event_target_round_trips(tmp_path: Path) -> None:
"""A targeted (private) event keeps its target across a reopen; public is ''."""
store = _store(tmp_path)
pub = store.insert_event("t1", "Brandr", "join", "set out")
priv = store.insert_event("t2", "Sigrun", "ambushed", "robbed in your sleep", "Brandr")
store.commit()
store.close()
reopened = _store(tmp_path)
_, events = reopened.load_all()
by_id = {e.event_id: e for e in events}
assert by_id[pub].target == "" # public stays empty
assert by_id[priv].target == "Brandr" # private keeps its recipient
reopened.close()
def test_ambush_table_per_day_uniqueness(tmp_path: Path) -> None:
"""The ambushes PK is (attacker, target, day): one row per pair per day."""
store = _store(tmp_path)
day = 739_400
assert store.has_ambushed("Brandr", "Sigrun", day) is False
store.record_ambush("Brandr", "Sigrun", day)
store.commit()
assert store.has_ambushed("Brandr", "Sigrun", day) is True
# A second record for the same pair/day is a no-op (INSERT OR IGNORE):
# the duplicate must not raise and must not add a row.
store.record_ambush("Brandr", "Sigrun", day)
store.commit()
rows = store._conn.execute(
"SELECT COUNT(*) AS n FROM ambushes WHERE attacker=? AND target=? AND day=?",
("Brandr", "Sigrun", day),
).fetchone()
assert rows["n"] == 1
# A new day is a fresh attempt; the old day stays recorded.
assert store.has_ambushed("Brandr", "Sigrun", day + 1) is False
store.record_ambush("Brandr", "Sigrun", day + 1)
store.commit()
assert store.has_ambushed("Brandr", "Sigrun", day) is True
assert store.has_ambushed("Brandr", "Sigrun", day + 1) is True
store.close()
def test_upsert_updates_existing_row(tmp_path: Path) -> None:
store = _store(tmp_path)
player = make_player(name="Sigrun", gold=10)
store.upsert_player(player)
store.commit()
player.gold = 999
store.upsert_player(player)
store.commit()
store.close()
reopened = _store(tmp_path)
players, _ = reopened.load_all()
assert players["Sigrun"].gold == 999
assert len(players) == 1
reopened.close()
def test_event_append_and_since_cursor(tmp_path: Path) -> None:
store = _store(tmp_path)
id1 = store.insert_event("t1", "Brandr", "fight", "slew a rat")
id2 = store.insert_event("t2", "Sigrun", "bestow", "blessed with gold")
store.commit()
store.close()
reopened = _store(tmp_path)
_, events = reopened.load_all()
assert [e.event_id for e in events] == [id1, id2]
# Catch up from a cursor before both, then advance past the first.
fresh, cursor = since(events, 0)
assert len(fresh) == 2
assert cursor == id2
after_first, cursor2 = since(events, id1)
assert [e.event_id for e in after_first] == [id2]
assert cursor2 == id2
nothing, cursor3 = since(events, id2)
assert nothing == []
assert cursor3 == id2
reopened.close()
def test_top_ranks_tie_breaks(tmp_path: Path) -> None:
store = _store(tmp_path)
# Same level: higher XP ranks first; equal XP breaks by name ascending.
store.upsert_player(make_player(name="Carol", level=5, xp=1200, gold=10))
store.upsert_player(make_player(name="Alice", level=5, xp=1500, gold=10))
store.upsert_player(make_player(name="Bob", level=5, xp=1500, gold=10))
store.upsert_player(make_player(name="Dave", level=4, xp=9999, gold=10))
store.commit()
ranks = store.top_ranks(limit=10)
assert [r.name for r in ranks] == ["Alice", "Bob", "Carol", "Dave"]
store.close()
def test_top_ranks_honours_limit(tmp_path: Path) -> None:
store = _store(tmp_path)
for i in range(15):
store.upsert_player(make_player(name=f"P{i:02d}", level=i, xp=i * 10))
store.commit()
ranks = store.top_ranks(limit=10)
assert len(ranks) == 10
# Highest level first.
assert ranks[0].name == "P14"
store.close()
def test_meta_round_trip(tmp_path: Path) -> None:
store = _store(tmp_path)
store.set_meta("world_name", "The Vale of Understone")
assert store.get_meta("world_name") == "The Vale of Understone"
assert store.get_meta("missing") is None
store.close()
def test_retention_columns_round_trip(tmp_path: Path) -> None:
"""The retention columns survive a reopen: depth, the v0.10 stack-encoded
satchel, the two forged plusses, and the v0.10 banked vault gold."""
store = _store(tmp_path)
player = make_player(
name="Delver",
deepest_rung=2,
satchel="minor_potion:3,iron_ore:5", # v0.10 "id:qty" stack encoding
weapon_plus=2,
armor_plus=1,
banked=300,
)
store.upsert_player(player)
store.commit()
store.close()
reopened = _store(tmp_path)
players, _ = reopened.load_all()
loaded = players["Delver"]
assert loaded == player # full equality across every column
assert loaded.deepest_rung == 2
assert loaded.satchel == "minor_potion:3,iron_ore:5"
assert loaded.weapon_plus == 2
assert loaded.armor_plus == 1
assert loaded.banked == 300
reopened.close()
def test_v0_7_depth_columns_default_for_legacy_rows(tmp_path: Path) -> None:
"""A row written without the new columns loads them at their defaults.
The schema mutates in place (no migration, stamp stays 1), so the new
columns carry DB-side defaults: a pre-v0.7 player row (inserted with the
legacy column set) must read back deepest_rung 0, an empty satchel, and
zero plusses rather than erroring.
"""
store = _store(tmp_path)
store._conn.execute(
"INSERT INTO players "
"(name, x, y, hp, max_hp, level, xp, gold, atk, def_, weapon_id, armor_id, "
" turns_left, turn_day, mode, at_location, created_at, last_seen, log_cursor, "
" bestow_spent, bestow_day) "
"VALUES ('Old', 5, 5, 20, 20, 1, 0, 20, 5, 1, 'rusty_dagger', 'cloth_tunic', "
" 10, 0, 'tile', '', 't0', 't0', 0, 0, 0)",
)
store.commit()
store.close()
reopened = _store(tmp_path)
players, _ = reopened.load_all()
old = players["Old"]
assert old.deepest_rung == 0
assert old.satchel == ""
assert old.weapon_plus == 0
assert old.armor_plus == 0
assert old.banked == 0 # the v0.10 vault column defaults too
assert reopened.get_meta("schema_version") == "1" # stamp unchanged
reopened.close()
-47
View File
@@ -1,47 +0,0 @@
"""GameRNG tests — the deterministic randomness seam.
Covers the v0.2 ``weighted_index`` helper: that a fixed seed reproduces the
same stream, that the cumulative-sum mapping honours the weights' proportions,
and that every index of a crafted table is reachable.
"""
from __future__ import annotations
from collections import Counter
from understone.engine.rng import GameRNG
def test_weighted_index_is_deterministic_under_seed() -> None:
"""Two RNGs at the same seed yield the identical weighted-index stream."""
weights = [55, 8, 7, 5, 5, 5, 5, 3, 3, 4]
a = GameRNG(seed=2026)
b = GameRNG(seed=2026)
draws_a = [a.weighted_index(weights) for _ in range(50)]
draws_b = [b.weighted_index(weights) for _ in range(50)]
assert draws_a == draws_b
def test_weighted_index_every_index_reachable() -> None:
"""With equal weights, a crafted table sees every index appear."""
weights = [1, 1, 1, 1, 1]
rng = GameRNG(seed=7)
seen = {rng.weighted_index(weights) for _ in range(500)}
assert seen == set(range(len(weights)))
def test_weighted_index_single_entry_always_zero() -> None:
"""A one-row table can only ever pick index 0."""
rng = GameRNG(seed=1)
assert all(rng.weighted_index([9]) == 0 for _ in range(20))
def test_weighted_index_respects_proportions() -> None:
"""A heavily-weighted index dominates the empirical distribution."""
weights = [90, 5, 5]
rng = GameRNG(seed=99)
counts = Counter(rng.weighted_index(weights) for _ in range(4000))
# Index 0 carries 90% of the mass; it must be by far the most common.
assert counts[0] > counts[1] + counts[2]
# And the rare indices still occur (no off-by-one swallowing the tail).
assert counts[1] > 0 and counts[2] > 0
-63
View File
@@ -1,63 +0,0 @@
"""The satchel "id:qty" wire codec (understone.engine.satchel).
Pins the single-source codec the game façade, the Watch payload, and the
balance simulator all decode through. The format is comma-joined ``id:qty``
stacks; this proves a clean round-trip, the defensive bare-id => qty-1 rule, the
malformed/zero/empty fragments that are skipped, and that the encoder never
emits a zero-or-negative stack.
"""
from __future__ import annotations
import pytest
from understone.engine.satchel import decode_satchel, encode_satchel
def test_round_trips_id_qty_stacks() -> None:
"""The canonical "id:qty,id:qty" data decodes and re-encodes unchanged."""
encoded = "minor_potion:3,iron_ore:5"
stacks = decode_satchel(encoded)
assert stacks == [("minor_potion", 3), ("iron_ore", 5)]
assert encode_satchel(stacks) == encoded
def test_bare_id_decodes_as_qty_one() -> None:
"""A colonless chunk is a single item (defensive — never silently dropped)."""
assert decode_satchel("minor_potion") == [("minor_potion", 1)]
# Mixed with a normal stack, order preserved.
assert decode_satchel("minor_potion,iron_ore:5") == [
("minor_potion", 1),
("iron_ore", 5),
]
@pytest.mark.parametrize(
("encoded", "reason"),
[
("id:0", "zero quantity"),
("id:-1", "negative quantity"),
("id:abc", "non-integer quantity"),
(":5", "empty id"),
("", "empty string"),
("minor_potion:3,", "trailing comma yields an empty chunk"),
(",minor_potion:3", "leading comma yields an empty chunk"),
],
)
def test_skips_malformed_or_zero_fragments(encoded: str, reason: str) -> None:
"""A present-but-invalid or non-positive fragment is skipped; valid ones survive."""
stacks = decode_satchel(encoded)
assert all(item_id and qty > 0 for item_id, qty in stacks), reason
# The only valid stack in the trailing/leading-comma cases is the potion.
if "minor_potion:3" in encoded:
assert stacks == [("minor_potion", 3)]
else:
assert stacks == []
def test_encode_drops_non_positive_stacks() -> None:
"""The encoder never emits "id:0" or a negative quantity."""
assert encode_satchel([("minor_potion", 0)]) == ""
assert encode_satchel([("minor_potion", -2)]) == ""
assert encode_satchel([("minor_potion", 2), ("iron_ore", 0)]) == "minor_potion:2"
assert encode_satchel([]) == ""
-169
View File
@@ -1,169 +0,0 @@
"""Screen-layer tests: viewport maths, frame rendering, menu rendering.
Golden discipline: the golden files under ``tests/golden`` are authored by
hand (correct borders/centring, eyeballed) and are NOT machine-dumped
renderer output. Every golden comparison is paired with structural asserts
that hold independent of the exact golden bytes, so a renderer regression
that happens to match a stale golden still trips a structural check.
"""
from __future__ import annotations
from pathlib import Path
from understone.screen.grid import Cell, CellGrid
from understone.screen.menus import render_menu
from understone.screen.palette import Color
from understone.screen.text_renderer import render_frame
from understone.screen.viewport import compute_window
GOLDEN = Path(__file__).parent / "golden"
# ---------------------------------------------------------------------------
# viewport.compute_window
# ---------------------------------------------------------------------------
def test_window_centers_when_interior() -> None:
# 100x100 map, 48x16 view, focus at (50, 50): centred.
x0, y0 = compute_window(100, 100, 48, 16, 50, 50)
assert x0 == 50 - 48 // 2
assert y0 == 50 - 16 // 2
def test_window_clamps_nw_corner() -> None:
x0, y0 = compute_window(100, 100, 48, 16, 0, 0)
assert (x0, y0) == (0, 0)
def test_window_clamps_ne_corner() -> None:
x0, y0 = compute_window(100, 100, 48, 16, 99, 0)
assert x0 == 100 - 48
assert y0 == 0
def test_window_clamps_sw_corner() -> None:
x0, y0 = compute_window(100, 100, 48, 16, 0, 99)
assert x0 == 0
assert y0 == 100 - 16
def test_window_clamps_se_corner() -> None:
x0, y0 = compute_window(100, 100, 48, 16, 99, 99)
assert x0 == 100 - 48
assert y0 == 100 - 16
def test_window_view_larger_than_map_pins_origin() -> None:
x0, y0 = compute_window(10, 8, 48, 16, 5, 4)
assert (x0, y0) == (0, 0)
# ---------------------------------------------------------------------------
# Shared small-grid builders for the golden frames
# ---------------------------------------------------------------------------
_FLOOR = Cell(".", Color.FLOOR)
_PLAYER = Cell("@", Color.PLAYER)
def _floor_grid(rows: int, cols: int) -> CellGrid:
grid = CellGrid(rows, cols)
for r in range(rows):
for c in range(cols):
grid.set(r, c, _FLOOR)
return grid
def _spawn_grid() -> CellGrid:
"""9x5 floor with the player centred at (row 2, col 4)."""
grid = _floor_grid(5, 9)
grid.set(2, 4, _PLAYER)
return grid
def _edge_nw_grid() -> CellGrid:
"""9x5 floor with the player pinned to the NW corner (row 0, col 0)."""
grid = _floor_grid(5, 9)
grid.set(0, 0, _PLAYER)
return grid
# ---------------------------------------------------------------------------
# text_renderer.render_frame
# ---------------------------------------------------------------------------
def test_render_frame_matches_golden_spawn() -> None:
frame = render_frame(_spawn_grid(), title="Vale", status="[ status ]")
expected = (GOLDEN / "viewport_spawn.txt").read_text(encoding="utf-8")
assert frame == expected.rstrip("\n")
def test_render_frame_matches_golden_edge_nw() -> None:
frame = render_frame(_edge_nw_grid(), title="Vale", status="[ status ]")
expected = (GOLDEN / "viewport_edge_nw.txt").read_text(encoding="utf-8")
assert frame == expected.rstrip("\n")
def test_render_frame_structural_invariants() -> None:
frame = render_frame(_spawn_grid(), title="Vale", status="[ status ]")
lines = frame.split("\n")
# Top border, 5 grid rows, bottom border, status = 8 lines.
assert len(lines) == 8
# Title substring lives in the top border.
assert "Vale" in lines[0]
# Uniform width across the box (top border through bottom border).
box_lines = lines[:-1]
widths = {len(line) for line in box_lines}
assert len(widths) == 1, f"box rows ragged: {widths}"
# Exactly one '@' and it sits at the centre column of the interior.
body = lines[1:-2]
at_positions = [(r, line.index("@")) for r, line in enumerate(body) if "@" in line]
assert len(at_positions) == 1
_, col = at_positions[0]
# Interior centre: 1 (left border) + cols//2 = 1 + 4 = 5.
assert col == 1 + 9 // 2
# Status line is preserved verbatim as the last line.
assert lines[-1] == "[ status ]"
def test_render_frame_under_size_budget() -> None:
grid = _floor_grid(16, 48)
grid.set(8, 24, _PLAYER)
frame = render_frame(grid, title="The Vale of Understone", status="[ a long status line here ]")
assert len(frame) < 2048
# ---------------------------------------------------------------------------
# menus.render_menu
# ---------------------------------------------------------------------------
def test_render_menu_matches_golden_inn() -> None:
menu = render_menu(
"The Sleeping Drake",
["A warm hearth crackles.", "A bed costs 15 gold."],
["(R)est", "(L)eave"],
"[ status ]",
)
expected = (GOLDEN / "menu_inn.txt").read_text(encoding="utf-8")
assert menu == expected.rstrip("\n")
def test_render_menu_structural_invariants() -> None:
menu = render_menu(
"The Sleeping Drake",
["A warm hearth crackles.", "A bed costs 15 gold."],
["(R)est", "(L)eave"],
"[ status ]",
)
lines = menu.split("\n")
assert "The Sleeping Drake" in lines[0]
assert lines[-1] == "[ status ]"
box = lines[:-1]
widths = {len(line) for line in box}
assert len(widths) == 1, f"menu box ragged: {widths}"
# Option line is present inside the body.
assert any("(R)est" in line and "(L)eave" in line for line in lines)
-306
View File
@@ -1,306 +0,0 @@
"""Tests for the balance instrument (the greedy bot simulator).
These run the REAL game façade end-to-end, so they double as the fiercest
integration test in the suite: determinism (same inputs identical report),
that the greedy bot makes genuine progress over a Vale run, that its realized
fight share lands in a sane band, that a multi-seed sweep aggregates and the
report renders and the single best end-to-end assertion, that a short seed
sweep actually SLAYS THE WYRM, proving the whole v0.1v0.7 loop is winnable by
an unclever bot.
"""
from __future__ import annotations
from io import StringIO
from pathlib import Path
from typing import TYPE_CHECKING
from understone import sim
from understone.engine.models import LocationDef, Mode, Zone
from understone.engine.rng import GameRNG
from understone.game import Game
from understone.persistence import Store
from understone.sim import BalanceReport, simulate
from .conftest import make_monster, make_world
if TYPE_CHECKING:
import pytest
PACK = Path(__file__).resolve().parents[1] / "understone" / "world" / "data"
# ---------------------------------------------------------------------------
# determinism
# ---------------------------------------------------------------------------
def test_same_inputs_give_identical_report() -> None:
"""Same (pack, days, seed) → byte-identical BalanceReport (frozen + seeded)."""
a = simulate(PACK, 20, 5)
b = simulate(PACK, 20, 5)
assert a == b
assert isinstance(a, BalanceReport)
def test_different_seeds_diverge() -> None:
"""Different seeds produce different runs (the RNG actually threads through)."""
a = simulate(PACK, 20, 1)
b = simulate(PACK, 20, 2)
# The runs are not identical (some headline measure differs).
assert (a.fights_fought, a.total_gold_earned, a.day_of_first_wyrm_kill) != (
b.fights_fought,
b.total_gold_earned,
b.day_of_first_wyrm_kill,
)
# ---------------------------------------------------------------------------
# progress
# ---------------------------------------------------------------------------
def test_bot_makes_progress_over_thirty_days() -> None:
"""A 30-day Vale run climbs past level 1 and actually fights."""
r = simulate(PACK, 30, 1)
assert r.final_level > 1
assert r.fights_fought > 0
assert r.total_gold_earned > 0
# It also plumbs the deep — the rung ladder is reachable for a geared bot.
assert r.rungs_cleared > 0
def test_realized_fight_share_in_sane_band() -> None:
"""The bot's fight share is a real fraction and forest-fight dominant.
A greedy XP grinder spends most of its turns fighting the wood (the rest are
the handful of descents and the Wyrm bout), so the share is high but it is
a genuine fraction in (0, 1], never a degenerate 0 or a value out of range.
"""
r = simulate(PACK, 30, 3)
assert 0.0 < r.realized_fight_share <= 1.0
# Fights dominate the turn-spend, but descents/challenges exist too, so the
# share is below a hard 1.0 floor only loosely — assert the sane half-band.
assert r.realized_fight_share >= 0.5
# ---------------------------------------------------------------------------
# reporting & sweep
# ---------------------------------------------------------------------------
def test_report_renders_without_crashing() -> None:
r = simulate(PACK, 15, 1)
text = sim._render_report("The Vale of Understone", r)
assert "greedy bot" in text
assert "final level" in text
assert "Wyrm slain" in text
def test_cli_simulate_single_seed_renders(tmp_path: Path) -> None:
out = StringIO()
rc = sim.cli_simulate(PACK, 15, 1, out=out)
assert rc == 0
assert "The Vale of Understone" in out.getvalue()
assert "fight share" in out.getvalue()
def test_cli_simulate_sweep_aggregates() -> None:
"""A --seeds sweep prints per-seed lines plus an aggregate with spreads."""
out = StringIO()
rc = sim.cli_simulate(PACK, 20, 1, out=out, seeds=3)
assert rc == 0
text = out.getvalue()
assert "3 seeds" in text
assert "aggregate" in text
# Per-seed lines for each of the three seeds.
for seed in (1, 2, 3):
assert f"seed {seed:>3}" in text or f"seed {seed}" in text
# The aggregate carries a mean [min..max] spread.
assert "[" in text and "]" in text
def test_sweep_reports_are_each_deterministic() -> None:
"""Each seed in a sweep is independently reproducible by single simulate."""
seed = 4
swept = simulate(PACK, 20, seed)
again = simulate(PACK, 20, seed)
assert swept == again
# ---------------------------------------------------------------------------
# the load-bearing assertion: the world is winnable
# ---------------------------------------------------------------------------
def test_greedy_bot_slays_the_wyrm() -> None:
"""The single best end-to-end check: a short seed sweep KILLS THE WYRM.
If a greedy, unclever bot can take the Wyrm Below playing through the real
façade, then the whole authored loop movement, the zone-banded forest, the
economy, the rung ladder, the satchel death-save, the forge, and the endgame
gate composes into a *winnable* game. A run that ever stops winning trips
here. A small sweep (not one lucky seed) so the proof is robust.
"""
reports = [simulate(PACK, 40, seed) for seed in (1, 2, 3)]
kills = [r for r in reports if r.wyrm_killed]
assert kills, "the greedy bot never slew the Wyrm across the seed sweep"
# Every kill records the day it first happened, within the run window.
for r in kills:
assert r.day_of_first_wyrm_kill is not None
assert 1 <= r.day_of_first_wyrm_kill <= 40
# ---------------------------------------------------------------------------
# the bundled ALTERNATE world: The Cinder Wastes (LLM-authored from the manual)
#
# The Vale assertions above are the primary proof. These mirror them against the
# real bundled second world, so the dogfood pack — authored cold from AUTHORING.md
# — is held to the same bar: the bot must make genuine progress through it, and a
# short seed sweep must actually slay its Magma Wyrm. If the authored world ever
# stops being winnable, this trips.
# ---------------------------------------------------------------------------
CINDER = Path(__file__).resolve().parents[1] / "understone" / "world" / "packs" / "cinder-wastes"
def test_cinder_wastes_bot_makes_progress() -> None:
"""A short Cinder Wastes run climbs past level 1 and genuinely plays.
Fifteen days lands before the bot's first Wyrm kill (~day 24), so the level
is still climbing rather than reset post-win a stable "the world plays"
signal across the durable measures (level, fights, gold, the rung ladder).
"""
r = simulate(CINDER, 15, 1)
assert r.final_level > 1
assert r.fights_fought > 0
assert r.total_gold_earned > 0
assert r.rungs_cleared > 0 # the caldera rung ladder is reachable
def test_cinder_wastes_is_winnable() -> None:
"""The dogfood proof: a greedy bot SLAYS THE MAGMA WYRM in the authored world.
The Cinder Wastes was written by an LLM working only from AUTHORING.md and
the validator. This is the end-to-end demonstration that the manual plus the
loader produce not merely a *valid* pack but a *playable-to-victory* one a
short seed sweep takes the Magma Wyrm. (It is harder than the Vale: the kill
lands later, so the window is wider than the Vale's.)
"""
reports = [simulate(CINDER, 50, seed) for seed in (1, 2, 3)]
kills = [r for r in reports if r.wyrm_killed]
assert kills, "the greedy bot never slew the Magma Wyrm across the seed sweep"
for r in kills:
assert r.day_of_first_wyrm_kill is not None
assert 1 <= r.day_of_first_wyrm_kill <= 50
# ---------------------------------------------------------------------------
# robustness on non-shipped pack shapes: location doors inside hunt zones
#
# The bot runs arbitrary authored packs, not just the two bundled worlds, so a
# zone may overlap a location door. A door cell is "walkable" (you can step onto
# it) but standing on it flips the bot into that location's MENU — useless ground
# for a forest fight, and a "fight" issued from a MENU is rejected by the engine
# WITHOUT spending a turn. These pin the two guards that keep that from spinning
# the per-day loop or over-counting fights.
# ---------------------------------------------------------------------------
def _door(x: int, y: int) -> LocationDef:
"""A bare location door placed at ``(x, y)`` (an inn, for concreteness)."""
return LocationDef(
key="inn",
kind="inn",
name="Wayhouse",
x=x,
y=y,
glyph="",
color="town",
actions=("rest", "leave"),
)
def test_nearest_in_zone_skips_a_door_cell() -> None:
"""A door is never returned as a zone's hunt cell, even when it is nearest.
The zone here spans a column running away from the spawn; its closest-to-spawn
walkable cell IS a location door, with open ground one step further. The
helper must skip the door (it would only trap the bot in a menu) and return
the open cell beyond it the FIX-2 filter, mirroring ``_adjacent_open``.
"""
# 11x11 grass; spawn (5, 5). A door at (5, 6) is the nearest cell inside the
# zone (Manhattan 1); the nearest OPEN in-zone cell is (5, 7) (Manhattan 2).
world = make_world(
locations=[_door(5, 6)],
zones=[Zone(key="wood", x0=5, y0=6, x1=5, y1=9, tier_lo=1, tier_hi=1)],
)
walkable = sim._reachable(world)
assert (5, 6) in walkable # the door cell is walkable...
cell = sim._nearest_in_zone(world, walkable, world.zones[0])
assert cell is not None
assert cell != (5, 6) # ...but the helper does not pick it
assert world.location_at(*cell) is None # the returned cell is open ground
assert cell == (5, 7) # the nearest open in-zone cell beyond the door
def test_zone_hunt_spots_drops_a_zone_with_no_fightable_foe() -> None:
"""A zone whose tier band holds no foe is dropped, not appended with None.
FIX-4: the fallback in ``_best_hunt_spot`` (``ranked[-1]``) must never land on
a zone where no monster can roll. A zone banded to a tier with no monster is
simply not a hunting ground, so it never enters the spot list.
"""
# One zone banded to tier 9 (no monster lives there); the only monster is a
# tier-1 rat. The empty-band zone must be dropped entirely.
world = make_world(
monsters=[make_monster(tier=1)],
zones=[Zone(key="void", x0=4, y0=4, x1=6, y1=6, tier_lo=9, tier_hi=9)],
)
spots = sim._zone_hunt_spots(world, sim._reachable(world))
assert spots == [] # the foe-less zone is not a spot
def test_hunt_yields_the_turn_when_stuck_in_a_menu(
monkeypatch: pytest.MonkeyPatch, tmp_path: Path
) -> None:
"""A hunt that ends in a MENU yields the turn instead of over-counting.
The defence-in-depth for FIX-1: should the bot ever reach the fight moment
still inside a location MENU (a door swallowed the walk), the engine would
REJECT the "fight" without spending a turn and the old string-only check
misread that reject as a won bout, over-counting and spinning the loop. The
new mode pre-check must instead leave the menu and return False (yield), so no
phantom fight is recorded and the day loop makes honest progress.
"""
# A door at (5, 4) inside a tier-1 zone. We inject this door cell as the hunt
# spot directly — the pre-FIX-2 state where a door WAS the nearest in-zone
# cell — so the guard, not the spot-selection filter, is what is under test.
world = make_world(
locations=[_door(5, 4)],
zones=[Zone(key="wood", x0=4, y0=3, x1=6, y1=5, tier_lo=1, tier_hi=1)],
monsters=[make_monster(tier=1)],
)
clock = sim._Clock(sim._SIM_START)
game = Game(world, Store(tmp_path / "g.db"), clock=clock, rng=GameRNG(seed=1)) # type: ignore[arg-type]
bot = sim._Bot(game, world, clock)
game.join(bot.name)
bot._hunt_spots = [(1, (5, 4), make_monster(tier=1))]
player = game.players[bot.name]
# Model "a location door swallowed the walk": every navigation step ends with
# the bot back inside the door's menu, so the hunt reaches its fight decision
# still in MENU mode no matter how many times it tries to step clear — exactly
# the trap the guard exists for (a single un-menu + re-walk cannot escape it).
def _walk_into_door(_goal: tuple[int, int]) -> None:
player.mode = Mode.MENU
player.at_location = "inn"
monkeypatch.setattr(bot, "_goto_xy", _walk_into_door)
_walk_into_door((5, 4)) # start the hunt already inside the menu
fought = bot._hunt()
assert fought is False # the turn is yielded, not spent on a menu-reject
assert bot.fights_fought == 0 # no phantom fight recorded
assert game.players[bot.name].mode is Mode.TILE # and the menu was left behind
-862
View File
@@ -1,862 +0,0 @@
"""The v0.5 social slice — ambush (async PvP), inn mail, and inn dice.
Drives the game façade over the shipped world with a frozen clock and a seeded
RNG. Three feature areas:
* AMBUSH the full eligibility matrix (every refusal branch), the win path
(exact gold transfer, victim bounced to spawn at 1 HP, private mail visible
only to the victim, public news), the lose path (attacker bounced, no
transfer), the flee stalemate, per-day once-per-pair, and next-day retry.
* MAIL ``post`` delivers a private note to the target's log once, the sender
is confirmed, the daily cap refuses the overflow, the sanitizer rejects a
newline body, and the Watch state payload NEVER carries a targeted row.
* DICE win/lose/push under a seeded RNG, the bet band, affordability, the
daily cap (a push still counts), and the Herald firing only on a big win.
Negative-test discipline (the SLEEP RULE has teeth):
``test_sleep_rule_guard_has_teeth`` documents the revert-and-observe check.
Disabling the ``target.turn_day >= today`` clause in Game._ambush_refusal
let an ALREADY-AWAKE target be ambushed ``test_ambush_refused_target_awake``
then failed (the attempt resolved instead of being refused). The clause was
restored; that refusal test is the standing regression for the invariant.
"""
from __future__ import annotations
from pathlib import Path
import pytest
from tests.conftest import fixed_clock, utc
from understone.engine.models import Mode
from understone.engine.rng import GameRNG
from understone.game import Game
from understone.persistence import Store
from understone.watch import build_state_payload
from understone.world.loader import load_world
PACK = Path(__file__).resolve().parents[1] / "understone" / "world" / "data"
# The frozen "today" all these tests run on; the sleep rule keys off its ordinal.
_NOW = utc(2026, 6, 12, 10, 0)
_TODAY = _NOW.toordinal()
@pytest.fixture
def clock() -> object:
return fixed_clock(_NOW)
def _game(tmp_path: Path, clock: object, seed: int = 7) -> Game:
world = load_world(PACK)
store = Store(tmp_path / "social.db")
return Game(world, store, clock=clock, rng=GameRNG(seed=seed)) # type: ignore[arg-type]
def _arm_ambush(
game: Game,
*,
attacker_level: int = 5,
target_level: int = 5,
target_asleep: bool = True,
target_gold: int = 100,
) -> tuple[object, object]:
"""Join an attacker + target and tune their sheets for an ambush.
The attacker is overworld and seasoned; the target sits at *target_level*
with *target_gold*, and ``target_asleep`` controls the sleep rule (a
sleeping target has not acted today). Returns ``(attacker, target)``.
"""
game.join("Raider")
game.join("Sleeper")
attacker = game.players["Raider"]
target = game.players["Sleeper"]
attacker.level = attacker_level
target.level = target_level
target.gold = target_gold
target.turn_day = _TODAY - 1 if target_asleep else _TODAY
return attacker, target
# ---------------------------------------------------------------------------
# Ambush — eligibility matrix (each refusal is a distinct in-fiction line)
# ---------------------------------------------------------------------------
def test_ambush_refused_unknown_target(tmp_path: Path, clock: object) -> None:
game = _game(tmp_path, clock)
game.join("Raider")
game.players["Raider"].level = 5
out = game.action("Raider", "ambush", "Ghost", "")
assert "signed the ledger" in out # the unknown-player refusal
# No turn spent on an unresolvable target.
assert game.players["Raider"].turns_left == game.world.settings.daily_turns
def test_ambush_refused_self(tmp_path: Path, clock: object) -> None:
game = _game(tmp_path, clock)
game.join("Raider")
game.players["Raider"].level = 5
out = game.action("Raider", "ambush", "Raider", "")
assert "yourself" in out.lower()
def test_ambush_refused_young_attacker(tmp_path: Path, clock: object) -> None:
game = _game(tmp_path, clock)
floor = game.world.settings.ambush_min_level
_arm_ambush(game, attacker_level=floor - 1, target_level=floor + 1)
out = game.action("Raider", "ambush", "Sleeper", "")
assert "shields the young" in out
assert game.players["Raider"].turns_left == game.world.settings.daily_turns
def test_ambush_refused_young_target(tmp_path: Path, clock: object) -> None:
game = _game(tmp_path, clock)
floor = game.world.settings.ambush_min_level
# Attacker is seasoned but the target is below the floor: still shielded.
_arm_ambush(game, attacker_level=floor + 1, target_level=floor - 1)
out = game.action("Raider", "ambush", "Sleeper", "")
assert "shields the young" in out
def test_ambush_refused_out_of_band(tmp_path: Path, clock: object) -> None:
game = _game(tmp_path, clock)
band = game.world.settings.ambush_level_band
floor = game.world.settings.ambush_min_level
_arm_ambush(
game,
attacker_level=floor + band + 5,
target_level=floor,
)
out = game.action("Raider", "ambush", "Sleeper", "")
assert "far from your measure" in out
def test_ambush_band_beats_awake_in_refusal_order(tmp_path: Path, clock: object) -> None:
"""PRECEDENCE: the band gate is checked before the sleep rule.
A target who is BOTH out of band AND awake must report the band message,
not the watchful one pinning the documented order (level gates before the
live-play sleep defence).
"""
game = _game(tmp_path, clock)
band = game.world.settings.ambush_level_band
floor = game.world.settings.ambush_min_level
_arm_ambush(
game,
attacker_level=floor + band + 1, # one past the band...
target_level=floor,
target_asleep=False, # ...and also awake
)
out = game.action("Raider", "ambush", "Sleeper", "")
assert "far from your measure" in out # the band gate wins
assert "watchful today" not in out
def test_ambush_band_boundary_exact_is_allowed(tmp_path: Path, clock: object) -> None:
"""Exactly ``ambush_level_band`` apart clears the band gate (it is inclusive).
Armed awake so the very next gate the sleep rule is what speaks: a
'watchful today' refusal proves the band gate let this pair through.
"""
game = _game(tmp_path, clock)
band = game.world.settings.ambush_level_band
floor = game.world.settings.ambush_min_level
_arm_ambush(
game,
attacker_level=floor + band, # exactly band levels above the floor
target_level=floor,
target_asleep=False,
)
out = game.action("Raider", "ambush", "Sleeper", "")
assert "far from your measure" not in out # past the band gate
assert "watchful today" in out # stopped by the next gate instead
def test_ambush_band_boundary_one_over_is_refused(tmp_path: Path, clock: object) -> None:
"""One level past ``ambush_level_band`` is refused with the band message."""
game = _game(tmp_path, clock)
band = game.world.settings.ambush_level_band
floor = game.world.settings.ambush_min_level
_arm_ambush(
game,
attacker_level=floor + band + 1, # just over the band
target_level=floor,
)
out = game.action("Raider", "ambush", "Sleeper", "")
assert "far from your measure" in out
assert game.store.has_ambushed("Raider", "Sleeper", _TODAY) is False
def test_ambush_refused_target_awake(tmp_path: Path, clock: object) -> None:
"""The SLEEP RULE: a target who has already acted today is un-ambushable.
See the module docstring for the revert-and-observe check proving this
refusal has teeth.
"""
game = _game(tmp_path, clock)
_arm_ambush(game, target_asleep=False)
out = game.action("Raider", "ambush", "Sleeper", "")
assert "watchful today" in out
# Refused without resolving: no turn spent, no ambush recorded.
assert game.players["Raider"].turns_left == game.world.settings.daily_turns
assert game.store.has_ambushed("Raider", "Sleeper", _TODAY) is False
def test_ambush_refused_repeat_same_pair_same_day(tmp_path: Path, clock: object) -> None:
game = _game(tmp_path, clock)
attacker, target = _arm_ambush(game)
# First attempt resolves (attacker overwhelming -> a clean win).
attacker.atk = 200
target.hp = 5
game.action("Raider", "ambush", "Sleeper", "")
assert game.store.has_ambushed("Raider", "Sleeper", _TODAY) is True
# Re-arm the target as sleeping AND healed above 1 HP (so the mercy rule
# does not intercept first); the SAME pair is still barred for the day.
target.turn_day = _TODAY - 1
target.hp = 20
out = game.action("Raider", "ambush", "Sleeper", "")
assert "already lain in wait" in out
def test_ambush_refused_pile_on_downed_victim(tmp_path: Path, clock: object) -> None:
"""MERCY RULE: a second, DIFFERENT attacker cannot kick a just-bounced sleeper.
The first ambush leaves the victim at 1 HP (still asleep being robbed does
not start their day). A fresh raider then finds them battered in the ditch;
even bandits have standards, so the pile-on is refused outright no turn
spent, no pair-row written for the second attacker.
"""
game = _game(tmp_path, clock)
first, target = _arm_ambush(game, target_gold=100)
first.atk = 200 # one-shot: leaves the victim at 1 HP
target.hp = 5
game.action("Raider", "ambush", "Sleeper", "")
assert target.hp == 1 # downed and still asleep
# A second, seasoned raider tries to finish the job.
game.join("Marauder")
second = game.players["Marauder"]
second.level = 5
turns_before = second.turns_left
out = game.action("Marauder", "ambush", "Sleeper", "")
assert "battered in the ditch" in out
# No turn spent and no attempt recorded for the second attacker.
assert second.turns_left == turns_before
assert game.store.has_ambushed("Marauder", "Sleeper", _TODAY) is False
def test_ambush_healed_victim_is_ambushable_again(tmp_path: Path, clock: object) -> None:
"""The mercy rule lifts once the victim mends: healed above 1 HP (and still
asleep), a fresh attacker may strike."""
game = _game(tmp_path, clock)
first, target = _arm_ambush(game, target_gold=100)
first.atk = 200
target.hp = 5
game.action("Raider", "ambush", "Sleeper", "")
assert target.hp == 1
# The victim is tended back above the floor (still asleep this day).
target.hp = 18
game.join("Marauder")
second = game.players["Marauder"]
second.level = 5
second.atk = 200 # one-shot again
out = game.action("Marauder", "ambush", "Sleeper", "")
assert "battered in the ditch" not in out
# The fresh ambush resolved: recorded, and the victim is bounced anew.
assert game.store.has_ambushed("Marauder", "Sleeper", _TODAY) is True
assert target.hp == 1
def test_ambush_refused_zero_turns(tmp_path: Path, clock: object) -> None:
game = _game(tmp_path, clock)
attacker, _ = _arm_ambush(game)
attacker.turns_left = 0
out = game.action("Raider", "ambush", "Sleeper", "")
assert "spent for today" in out.lower()
# Eligible but exhausted: nothing recorded (the attempt never landed).
assert game.store.has_ambushed("Raider", "Sleeper", _TODAY) is False
# ---------------------------------------------------------------------------
# Ambush — outcomes (win / lose / flee) and the records they leave
# ---------------------------------------------------------------------------
def test_ambush_win_transfers_gold_and_bounces_victim(tmp_path: Path, clock: object) -> None:
game = _game(tmp_path, clock)
attacker, target = _arm_ambush(game, target_gold=100)
attacker.atk = 200 # one-shot the sleeper
target.hp = 5
pct = game.world.settings.ambush_gold_pct
steal = 100 * pct // 100 # 25 gold at the shipped 25%
raider_gold_before = attacker.gold
out = game.action("Raider", "ambush", "Sleeper", "")
# Exact transfer: attacker up by steal, victim down by the same.
assert attacker.gold == raider_gold_before + steal
assert target.gold == 100 - steal
# The victim wakes at the spawn at 1 HP, knocked out of any menu.
assert target.hp == 1
assert (target.x, target.y) == game.world.spawn
assert target.mode is Mode.TILE
assert target.at_location == ""
assert f"{steal} gold" in out
# The attempt is recorded.
assert game.store.has_ambushed("Raider", "Sleeper", _TODAY) is True
def test_ambush_steals_only_carried_gold_not_the_vault(tmp_path: Path, clock: object) -> None:
"""A winning ambush robs carried gold only — banked vault gold is untouched.
The steal is a slice of ``target.gold`` (gold in hand); the strongbox
(``banked``) is safe by design. This pins the vault's whole point: bank your
coin before you sleep and a sleeping-robber cannot lift it.
"""
game = _game(tmp_path, clock)
attacker, target = _arm_ambush(game, target_gold=40)
target.banked = 1000 # a fat vault the raider must not be able to touch
attacker.atk = 200 # one-shot the sleeper
target.hp = 5
pct = game.world.settings.ambush_gold_pct
steal = 40 * pct // 100 # a slice of the CARRIED 40, not the banked 1000
game.action("Raider", "ambush", "Sleeper", "")
assert target.gold == 40 - steal # carried gold robbed
assert target.banked == 1000 # the vault is wholly untouched
assert attacker.gold == game.world.settings.starting_gold + steal
def test_ambush_win_applies_attacker_wear(tmp_path: Path, clock: object) -> None:
"""A multi-round win banks the attacker's wear: the log narrates the
sleeper's counter-blows, so the sheet must show the HP they cost.
The one-shot win above leaves the attacker untouched, which would mask a
WIN branch that drops ``hp_delta`` on the floor. Here the sleeper is tanky
enough to trade blows before falling (and the attacker still wins), so the
attacker must end below full HP. Stats and seed are tuned so the win is
decisive but not instant.
"""
game = _game(tmp_path, clock)
attacker, target = _arm_ambush(game, target_gold=100)
attacker.atk, attacker.def_ = 8, 2
attacker.hp = attacker.max_hp = 30
target.atk, target.def_, target.hp = 5, 1, 25
out = game.action("Raider", "ambush", "Sleeper", "")
# The win lands (victim robbed and bounced to 1 HP)...
assert target.hp == 1
assert (
any(crow in out for crow in ("made off", "robbed the sleeping", "lifted")) or "rob" in out
)
# ...but the sleeper's counter-blows cost the attacker real HP this time.
assert attacker.hp < attacker.max_hp
assert attacker.hp >= 1 # never below the floor
def test_ambush_win_news_is_public_and_mail_is_private(tmp_path: Path, clock: object) -> None:
"""The victory crows on the public feed; the victim gets a PRIVATE note.
A THIRD player must see the public ambush line but never the private one.
"""
game = _game(tmp_path, clock)
attacker, target = _arm_ambush(game, target_gold=80)
attacker.atk = 200
target.hp = 5
game.join("Bystander") # a third player who must never see the private note
game.action("Raider", "ambush", "Sleeper", "")
# The victim reads the private "While you slept" note in their own log.
victim_log = game.log("Sleeper")
assert "While you slept" in victim_log
assert "ambushed you" in victim_log
# The bystander sees the public crow but NOT the private note.
third_log = game.log("Bystander")
assert (
"made off with" in third_log
or "robbed the sleeping" in third_log
or ("lifted" in third_log)
)
assert "While you slept" not in third_log
def test_ambush_win_on_pauper_steals_nothing_but_still_lands(tmp_path: Path, clock: object) -> None:
"""A win over a penniless sleeper: steal is 0, but the beat still plays.
The victim is bounced to the spawn at 1 HP all the same, the public herald
crows the robbery, and the private 'while you slept' note still reaches the
victim the gold transfer being empty changes none of that.
"""
game = _game(tmp_path, clock)
attacker, target = _arm_ambush(game, target_gold=0)
attacker.atk = 200 # one-shot the sleeper
target.hp = 5
game.join("Bystander")
raider_gold_before = attacker.gold
out = game.action("Raider", "ambush", "Sleeper", "")
# Nothing to steal: both purses are unchanged by the transfer.
assert attacker.gold == raider_gold_before
assert target.gold == 0
assert "0 gold" in out
# The victim is still bounced to the spawn at 1 HP.
assert target.hp == 1
assert (target.x, target.y) == game.world.spawn
assert target.mode is Mode.TILE
assert target.at_location == ""
# Public herald fires (a bystander reads the crow)...
third_log = game.log("Bystander")
assert any(crow in third_log for crow in ("made off", "robbed the sleeping", "lifted"))
# ...and the private mail still reaches the victim.
victim_log = game.log("Sleeper")
assert "While you slept" in victim_log
assert "ambushed you" in victim_log
def test_ambush_lose_bounces_attacker_no_transfer(tmp_path: Path, clock: object) -> None:
game = _game(tmp_path, clock)
attacker, target = _arm_ambush(game, target_gold=100)
# The sleeper is deadly: the ambush rebounds onto the attacker.
target.atk = 200
target.def_ = 100
target.hp = 200
attacker_gold_before = attacker.gold
out = game.action("Raider", "ambush", "Sleeper", "")
# No gold moved; the ATTACKER is the one bounced to spawn at 1 HP.
assert attacker.gold == attacker_gold_before
assert target.gold == 100
assert attacker.hp == 1
assert (attacker.x, attacker.y) == game.world.spawn
assert "flee" in out.lower() or "wakes" in out.lower()
# The attempt is still spent.
assert game.store.has_ambushed("Raider", "Sleeper", _TODAY) is True
def test_ambush_records_attempt_on_every_outcome(tmp_path: Path, clock: object) -> None:
"""Win, lose, or flee — the (attacker, target, day) row is always written."""
game = _game(tmp_path, clock)
attacker, target = _arm_ambush(game)
# Tune a flee: when neither side can meaningfully dent the other, the fight
# grinds to the 50-round stalemate guard, which resolves as FLED with no
# transfer. Both deal the 1-damage floor (atk << def), and both carry far
# more HP than 50 rounds can drain, so neither drops first.
attacker.atk, attacker.def_ = 1, 200
attacker.hp = attacker.max_hp = 500
target.atk, target.def_, target.hp = 1, 200, 500
gold_before = attacker.gold
out = game.action("Raider", "ambush", "Sleeper", "")
assert game.store.has_ambushed("Raider", "Sleeper", _TODAY) is True
assert attacker.gold == gold_before # a flee moves no gold
assert "slip away" in out.lower() or "nerve" in out.lower()
def test_ambush_next_day_retry_allowed(tmp_path: Path, clock: object) -> None:
"""A new UTC day clears the once-per-pair lock (advance the injected clock)."""
game = _game(tmp_path, clock)
attacker, target = _arm_ambush(game)
attacker.atk = 200
target.hp = 5
game.action("Raider", "ambush", "Sleeper", "")
assert game.store.has_ambushed("Raider", "Sleeper", _TODAY) is True
# Advance past UTC midnight; re-arm the sleeper for the new day.
tomorrow = utc(2026, 6, 13, 9, 0)
game.clock = fixed_clock(tomorrow) # type: ignore[assignment]
target.turn_day = tomorrow.toordinal() - 1 # asleep again
target.hp = 5
out = game.action("Raider", "ambush", "Sleeper", "")
assert "already lain in wait" not in out # the new day permits a fresh attempt
assert game.store.has_ambushed("Raider", "Sleeper", tomorrow.toordinal()) is True
def test_sleep_rule_guard_has_teeth(tmp_path: Path, clock: object) -> None:
"""Pin the sleep rule on a single-field divergence.
The un-ambushable case and the ambushable case differ ONLY in ``turn_day``:
with the target awake the action is refused, and flipping that one field to
asleep makes the very same attempt resolve and record.
"""
game = _game(tmp_path, clock)
attacker, target = _arm_ambush(game, target_asleep=False)
attacker.atk = 200
target.hp = 5
refused = game.action("Raider", "ambush", "Sleeper", "")
assert "watchful today" in refused
assert game.store.has_ambushed("Raider", "Sleeper", _TODAY) is False
# Flip ONLY the sleep field; now the very same attempt lands.
target.turn_day = _TODAY - 1
resolved = game.action("Raider", "ambush", "Sleeper", "")
assert "watchful today" not in resolved
assert game.store.has_ambushed("Raider", "Sleeper", _TODAY) is True
def test_ambush_both_rows_persist_in_one_transaction(tmp_path: Path, clock: object) -> None:
"""A win commits BOTH fighters' rows; a store reopen sees the transfer."""
game = _game(tmp_path, clock)
attacker, target = _arm_ambush(game, target_gold=100)
attacker.atk = 200
target.hp = 5
game.action("Raider", "ambush", "Sleeper", "")
raider_gold = attacker.gold
sleeper_gold = target.gold
game.store.close()
world = load_world(PACK)
reopened = Store(tmp_path / "social.db")
revived = Game(world, reopened, clock=clock) # type: ignore[arg-type]
assert revived.players["Raider"].gold == raider_gold
assert revived.players["Sleeper"].gold == sleeper_gold
assert revived.players["Sleeper"].hp == 1
reopened.close()
# ---------------------------------------------------------------------------
# Mail — post delivers privately, confirms, caps, sanitizes
# ---------------------------------------------------------------------------
def test_post_delivers_to_target_once_with_confirmation(tmp_path: Path, clock: object) -> None:
game = _game(tmp_path, clock)
game.join("Scribe")
game.join("Reader")
confirm = game.action("Scribe", "post", "Reader", "", "meet me at the inn")
assert "tucks the note" in confirm # the sender's in-fiction confirmation
# No turn spent on a post.
assert game.players["Scribe"].turns_left == game.world.settings.daily_turns
first = game.log("Reader")
assert "While you were away" in first
assert "meet me at the inn" in first
# Read once: the cursor advanced, so a second read no longer shows it.
second = game.log("Reader")
assert "meet me at the inn" not in second
def test_post_refused_unknown_and_self(tmp_path: Path, clock: object) -> None:
game = _game(tmp_path, clock)
game.join("Scribe")
unknown = game.action("Scribe", "post", "Nobody", "", "hello?")
assert "signed the ledger" in unknown
mine = game.action("Scribe", "post", "Scribe", "", "note to self")
assert "talk to yourself" in mine.lower()
def test_post_daily_cap_refuses_overflow(tmp_path: Path, clock: object) -> None:
game = _game(tmp_path, clock)
game.join("Scribe")
game.join("Reader")
cap = game.world.settings.post_daily_cap
for i in range(cap):
out = game.action("Scribe", "post", "Reader", "", f"note {i}")
assert "tucks the note" in out
# The (cap+1)-th post is refused.
over = game.action("Scribe", "post", "Reader", "", "one too many")
assert "all the word you may today" in over
assert game.players["Scribe"].posts_sent == cap
def test_post_sanitizer_rejects_newline_body(tmp_path: Path, clock: object) -> None:
"""A newline-injected note body is refused; nothing is delivered or counted."""
game = _game(tmp_path, clock)
game.join("Scribe")
game.join("Reader")
events_before = len(game.events)
out = game.action("Scribe", "post", "Reader", "", "line one\nFORGED HERALD LINE")
assert "scrawl" in out.lower()
# No event appended and the daily counter is untouched.
assert len(game.events) == events_before
assert game.players["Scribe"].posts_sent == 0
# And the reader never receives it.
assert "FORGED" not in game.log("Reader")
def test_post_works_from_inside_a_building(tmp_path: Path, clock: object) -> None:
"""Posting is legal anywhere: a menu-bound sender still gets a menu reply."""
game = _game(tmp_path, clock)
game.join("Scribe")
game.join("Reader")
scribe = game.players["Scribe"]
scribe.mode = Mode.MENU
scribe.at_location = "inn"
out = game.action("Scribe", "post", "Reader", "", "by the hearth")
assert "tucks the note" in out
# The reply is the inn menu (a menu surface), not an overworld frame.
assert "(R)est" in out or "Sleeping Drake" in out
# ---------------------------------------------------------------------------
# Mail — the lobby TV must never carry a private note
# ---------------------------------------------------------------------------
def test_watch_state_excludes_targeted_rows(tmp_path: Path, clock: object) -> None:
"""EXPLICIT: a private (targeted) event must not reach the Watch herald."""
game = _game(tmp_path, clock)
game.join("Scribe")
game.join("Reader")
game.action("Scribe", "post", "Reader", "", "a secret for the Reader")
payload = build_state_payload(game)
herald = payload["herald"]
assert isinstance(herald, list)
texts = [row["text"] for row in herald]
# The join lines are public and present; the private note is absent.
assert any("Scribe" in t or "Reader" in t for t in texts) # public joins show
assert all("a secret for the Reader" not in t for t in texts)
def test_watch_state_excludes_private_ambush_note(tmp_path: Path, clock: object) -> None:
"""The ambush victim's private alert is filtered from the lobby TV too."""
game = _game(tmp_path, clock)
attacker, target = _arm_ambush(game, target_gold=80)
attacker.atk = 200
target.hp = 5
game.action("Raider", "ambush", "Sleeper", "")
herald_texts = [row["text"] for row in build_state_payload(game)["herald"]] # type: ignore[union-attr]
# The PUBLIC ambush crow is on the feed...
assert any(
"Sleeper" in t and ("made off" in t or "robbed" in t or "lifted" in t) for t in herald_texts
)
# ...but the PRIVATE "While you slept" note never is.
assert all("While you slept" not in t for t in herald_texts)
# ---------------------------------------------------------------------------
# Dice — win / lose / push under a seeded RNG, bands, cap, herald gate
# ---------------------------------------------------------------------------
def _at_inn(game: Game, name: str) -> object:
"""Join *name* and seat them at the inn (MENU surface)."""
game.join(name)
player = game.players[name]
player.mode = Mode.MENU
player.at_location = "inn"
return player
def test_gamble_win_under_seeded_rng(tmp_path: Path, clock: object) -> None:
game = _game(tmp_path, clock)
player = _at_inn(game, "Gambler")
player.gold = 100
# Seed 2 makes the gamble child roll 11 (you) vs 9 (house) -> a win.
game.rng = GameRNG(seed=2)
out = game.action("Gambler", "gamble", "", "", "", 10)
assert player.gold == 110 # stake doubled back
assert "win" in out.lower()
# No turn spent; one game counted.
assert player.turns_left == game.world.settings.daily_turns
assert player.gambles == 1
def test_gamble_lose_under_seeded_rng(tmp_path: Path, clock: object) -> None:
game = _game(tmp_path, clock)
player = _at_inn(game, "Gambler")
player.gold = 100
# Seed 0 rolls 4 (you) vs 9 (house) -> a loss.
game.rng = GameRNG(seed=0)
out = game.action("Gambler", "gamble", "", "", "", 10)
assert player.gold == 90
assert "lose" in out.lower()
assert player.gambles == 1
def test_gamble_push_under_seeded_rng(tmp_path: Path, clock: object) -> None:
game = _game(tmp_path, clock)
player = _at_inn(game, "Gambler")
player.gold = 100
# Seed 1 rolls 6 vs 6 -> a push: no gold change, but it still counts.
game.rng = GameRNG(seed=1)
out = game.action("Gambler", "gamble", "", "", "", 10)
assert player.gold == 100
assert "push" in out.lower()
assert player.gambles == 1 # a push still consumes a daily game
def test_gamble_bet_band_refused(tmp_path: Path, clock: object) -> None:
game = _game(tmp_path, clock)
player = _at_inn(game, "Gambler")
player.gold = 100_000
max_bet = game.world.settings.gamble_max_bet
low = game.action("Gambler", "gamble", "", "", "", 0)
assert f"1 to {max_bet}" in low
high = game.action("Gambler", "gamble", "", "", "", max_bet + 1)
assert f"1 to {max_bet}" in high
# A rejected bet neither moves gold nor counts toward the cap.
assert player.gold == 100_000
assert player.gambles == 0
def test_gamble_unaffordable_refused(tmp_path: Path, clock: object) -> None:
game = _game(tmp_path, clock)
player = _at_inn(game, "Gambler")
player.gold = 5
out = game.action("Gambler", "gamble", "", "", "", 10) # within band, can't cover
assert "can't cover" in out.lower()
assert player.gold == 5
assert player.gambles == 0
def test_gamble_daily_cap_refused(tmp_path: Path, clock: object) -> None:
game = _game(tmp_path, clock)
player = _at_inn(game, "Gambler")
player.gold = 100_000
cap = game.world.settings.gamble_daily_cap
player.gambles = cap # already at the cap
out = game.action("Gambler", "gamble", "", "", "", 5)
assert "enough for one day" in out
assert player.gambles == cap # not incremented past the cap
def test_gamble_outside_inn_refused(tmp_path: Path, clock: object) -> None:
"""The dice live at the inn: the verb is illegal in another building."""
game = _game(tmp_path, clock)
player = _at_inn(game, "Gambler")
player.at_location = "shop" # the shop has no 'gamble' action
player.gold = 100
out = game.action("Gambler", "gamble", "", "", "", 10)
assert "can't 'gamble' here" in out.lower()
assert player.gold == 100
def test_gamble_big_win_heralds(tmp_path: Path, clock: object) -> None:
"""A win of >= 25 gold reaches the public Herald; a small one does not."""
game = _game(tmp_path, clock)
player = _at_inn(game, "Gambler")
player.gold = 1000
# A 50-gold win (>= the 25 threshold) writes a public dice line.
game.rng = GameRNG(seed=2) # a winning roll
events_before = len(game.events)
game.action("Gambler", "gamble", "", "", "", 50)
new = game.events[events_before:]
assert any(e.kind == "gamble" and e.target == "" for e in new)
assert player.gold == 1050
def test_gamble_small_win_is_quiet(tmp_path: Path, clock: object) -> None:
game = _game(tmp_path, clock)
player = _at_inn(game, "Gambler")
player.gold = 1000
# A 10-gold win is below the 25-gold Herald threshold: no public line.
game.rng = GameRNG(seed=2)
events_before = len(game.events)
game.action("Gambler", "gamble", "", "", "", 10)
new = game.events[events_before:]
assert all(e.kind != "gamble" for e in new)
assert player.gold == 1010
# ---------------------------------------------------------------------------
# The Vault — deposit/withdraw at the inn (no turn; banked gold is safe)
# ---------------------------------------------------------------------------
def test_deposit_moves_gold_to_the_vault_no_turn(tmp_path: Path, clock: object) -> None:
"""Deposit moves coin from hand to vault, costs no turn, and is friendly."""
game = _game(tmp_path, clock)
player = _at_inn(game, "Saver")
player.gold = 100
turns_before = player.turns_left
out = game.action("Saver", "deposit", "", "", "", 60)
assert player.gold == 40
assert player.banked == 60
assert player.turns_left == turns_before # banking spends no turn
assert "strongbox" in out.lower()
def test_withdraw_moves_gold_back_to_hand(tmp_path: Path, clock: object) -> None:
"""Withdraw moves coin from vault to hand."""
game = _game(tmp_path, clock)
player = _at_inn(game, "Saver")
player.gold = 10
player.banked = 90
game.action("Saver", "withdraw", "", "", "", 50)
assert player.gold == 60
assert player.banked == 40
def test_deposit_amount_exceeding_holdings_refused(tmp_path: Path, clock: object) -> None:
"""Depositing more than you carry is refused without mutation."""
game = _game(tmp_path, clock)
player = _at_inn(game, "Saver")
player.gold = 30
player.banked = 0
out = game.action("Saver", "deposit", "", "", "", 50)
assert player.gold == 30 # unchanged
assert player.banked == 0
assert "1 to 30" in out
def test_deposit_with_nothing_in_hand_refused(tmp_path: Path, clock: object) -> None:
"""Depositing with an empty hand is a friendly refusal."""
game = _game(tmp_path, clock)
player = _at_inn(game, "Saver")
player.gold = 0
out = game.action("Saver", "deposit", "", "", "", 10)
assert player.banked == 0
assert "no coin" in out.lower()
def test_withdraw_amount_exceeding_vault_refused(tmp_path: Path, clock: object) -> None:
"""Withdrawing more than is banked is refused without mutation."""
game = _game(tmp_path, clock)
player = _at_inn(game, "Saver")
player.gold = 0
player.banked = 20
out = game.action("Saver", "withdraw", "", "", "", 50)
assert player.gold == 0
assert player.banked == 20 # unchanged
assert "1 to 20" in out
def test_withdraw_empty_vault_refused(tmp_path: Path, clock: object) -> None:
"""Withdrawing from an empty vault is a friendly refusal."""
game = _game(tmp_path, clock)
player = _at_inn(game, "Saver")
player.banked = 0
out = game.action("Saver", "withdraw", "", "", "", 10)
assert player.gold == game.world.settings.starting_gold # unchanged
assert "empty" in out.lower()
def test_status_shows_carried_and_vault_gold(tmp_path: Path, clock: object) -> None:
"""door_status reports gold as carried-on-hand plus banked-in-the-vault."""
game = _game(tmp_path, clock)
game.join("Saver")
player = game.players["Saver"]
player.gold = 75
player.banked = 250
out = game.status("Saver")
assert "75 on hand" in out
assert "250 in the vault" in out
-60
View File
@@ -1,60 +0,0 @@
"""Deterministic terrain texturing (understone.screen.texture).
Pins the contract the Watch JS mirrors: a textured glyph is a pure function of
its cell coordinate (stable per cell), an un-listed glyph is returned
untouched, and the selection formula is ``(x * _HASH_X + y * _HASH_Y) % n``
derived from the module's hash constants. The formula is asserted against those
constants so a retune moves the test with it and a drift is caught.
"""
from __future__ import annotations
from understone.screen.texture import _HASH_X, _HASH_Y, VARIANTS, textured
def test_untextured_glyph_is_unchanged() -> None:
"""A glyph with no VARIANTS row passes through verbatim (actors, walls)."""
for ch in "█@☻⌂$":
assert textured(ch, 3, 7) == ch
def test_same_coord_same_variant() -> None:
"""Texturing is position-only and stable: one cell always picks one glyph."""
first = textured(".", 12, 5)
for _ in range(5):
assert textured(".", 12, 5) == first
def test_variant_is_always_in_the_row() -> None:
"""Every selected glyph is one of the declared variants for its base."""
choices = VARIANTS["."]
for x in range(20):
for y in range(20):
assert textured(".", x, y) in choices
def test_a_row_uses_more_than_one_variant() -> None:
"""Across a row the hash spreads — the texture is not a single repeated glyph."""
seen = {textured(".", x, 0) for x in range(len(VARIANTS["."]) * 4)}
assert len(seen) > 1
def test_formula_matches_the_hash_constants() -> None:
"""The selection index is (x * _HASH_X + y * _HASH_Y) % len — the JS twin's formula.
Derived from the live ``_HASH_X`` / ``_HASH_Y`` constants (not the literal
31/17) and checked against the live VARIANTS rows, so it stays a formula
test that tracks a retune rather than a snapshot a table or constant edit
could silently invalidate.
"""
for base, choices in VARIANTS.items():
n = len(choices)
for x, y in [(0, 0), (1, 0), (0, 1), (12, 5), (7, 13), (255, 255)]:
assert textured(base, x, y) == choices[(x * _HASH_X + y * _HASH_Y) % n]
def test_origin_cell_is_the_base_glyph() -> None:
"""Cell (0,0) hashes to index 0, which is the base glyph (variants[0])."""
for base, choices in VARIANTS.items():
assert textured(base, 0, 0) == choices[0]
assert choices[0] == base
@@ -1,81 +0,0 @@
"""The one-glyph-one-column grid contract (understone.engine.textwidth).
Pins the accept/reject boundary of :func:`is_grid_safe` and proves every
:data:`SAFE_PALETTE` entry clears it. The acceptances include the
East-Asian-Width *Ambiguous* CP437 glyphs the game leans on (`` ``),
which render single-column under the Western monospace our surfaces use; the
rejections are the genuinely double-width and zero-width classes that tear a
frame.
"""
from __future__ import annotations
import unicodedata
import pytest
from understone.engine.textwidth import SAFE_PALETTE, is_grid_safe
from understone.world.loader import RESERVED_GLYPHS
# Single-column glyphs that must be admitted: plain ASCII, a Latin accent that
# is one composed code point, and the Ambiguous-width CP437 set the re-skin uses.
_ACCEPTED = ["a", "Z", "ö", "", "", "", "", "", "", "", ".", "$", " "]
# Must be rejected, with the reason each one trips the gate.
_REJECTED = {
"": "wide CJK ideograph (EAW=W) — two columns",
"🌲": "emoji (EAW=W) — two columns",
"": "fullwidth Latin A (EAW=F) — two columns",
"": "decomposed e + combining acute — two code points",
"́": "a lone combining acute — zero width",
"👨‍👩": "ZWJ sequence — multiple code points",
"ab": "two characters",
"": "empty string",
"\t": "a control character",
}
@pytest.mark.parametrize("ch", _ACCEPTED)
def test_is_grid_safe_accepts(ch: str) -> None:
assert is_grid_safe(ch) is True
@pytest.mark.parametrize("text", list(_REJECTED), ids=list(_REJECTED.values()))
def test_is_grid_safe_rejects(text: str) -> None:
assert is_grid_safe(text) is False
def test_safe_palette_is_all_grid_safe() -> None:
"""Every curated palette glyph clears the gate — the appendix can't ship a dud."""
bad = [g for g in SAFE_PALETTE if not is_grid_safe(g)]
assert bad == [], f"palette has non-grid-safe glyphs: {bad}"
def test_safe_palette_has_no_reserved_glyphs() -> None:
"""No palette glyph is a loader-reserved marker — the 'author-usable' promise.
The appendix tells a pack author to pull any palette glyph for terrain,
structures, or actors, but the loader rejects the box-drawing frame lines
and the '@'/'' player markers (``loader.RESERVED_GLYPHS``). A palette entry
that is also reserved would hand the author a glyph that load-fails the
exact doc-vs-enforcement trap. Guarding the intersection keeps "all tested
safe AND author-usable" enforced, not merely asserted on width.
"""
collisions = set(SAFE_PALETTE) & RESERVED_GLYPHS
assert collisions == set(), f"palette offers loader-reserved glyphs: {sorted(collisions)}"
def test_safe_palette_has_no_duplicates() -> None:
"""The palette is a set in spirit; a dupe would be an authoring slip."""
assert len(SAFE_PALETTE) == len(set(SAFE_PALETTE))
def test_ambiguous_width_glyphs_are_accepted() -> None:
"""Document the load-bearing call: EAW=Ambiguous is admitted, not barred.
These are the CP437 glyphs the game depends on; if a future tightening
barred Ambiguous, the whole re-skin would vanish from the map.
"""
for ch in "█♣↑∩≈★":
assert unicodedata.east_asian_width(ch) == "A"
assert is_grid_safe(ch) is True
-94
View File
@@ -1,94 +0,0 @@
"""Daily-turn budget and UTC rollover tests.
Covers spend/refuse semantics, the lazy reset when the UTC day advances
(including a 23:59 -> 00:01 crossing on the same Player instance), and the
shared rollover of the bestow pool.
"""
from __future__ import annotations
from tests.conftest import fixed_clock, make_player, utc
from understone.engine.turns import ensure_day, spend_turn
def test_spend_decrements() -> None:
player = make_player(turns_left=3)
assert spend_turn(player) is True
assert player.turns_left == 2
def test_spend_refuses_at_zero_without_mutation() -> None:
player = make_player(turns_left=0)
before = player.turns_left
assert spend_turn(player) is False
assert player.turns_left == before
def test_ensure_day_resets_on_new_day() -> None:
day = utc(2026, 6, 12).toordinal()
player = make_player(turns_left=0, turn_day=day - 1, bestow_spent=20, bestow_day=day - 1)
reset = ensure_day(player, fixed_clock(utc(2026, 6, 12, 9, 0)), daily_turns=10)
assert reset is True
assert player.turns_left == 10
assert player.turn_day == day
assert player.bestow_spent == 0
assert player.bestow_day == day
def test_ensure_day_noop_within_same_day() -> None:
day = utc(2026, 6, 12).toordinal()
# Every day marker is already today, so no allowance (turns, bestow, posts,
# dice) is touched — the rollover is a pure no-op.
player = make_player(
turns_left=4,
turn_day=day,
bestow_spent=10,
bestow_day=day,
post_day=day,
gamble_day=day,
)
reset = ensure_day(player, fixed_clock(utc(2026, 6, 12, 23, 0)), daily_turns=10)
assert reset is False
assert player.turns_left == 4
assert player.bestow_spent == 10
def test_midnight_crossing_refreshes_on_same_instance() -> None:
# Evening of day one: spend down to a low budget.
player = make_player(turns_left=10, turn_day=0, bestow_spent=0, bestow_day=0)
evening = utc(2026, 6, 12, 23, 59)
ensure_day(player, fixed_clock(evening), daily_turns=10)
for _ in range(8):
spend_turn(player)
assert player.turns_left == 2
# Just past midnight (UTC) the next action refreshes the budget.
after_midnight = utc(2026, 6, 13, 0, 1)
reset = ensure_day(player, fixed_clock(after_midnight), daily_turns=10)
assert reset is True
assert player.turns_left == 10
assert player.turn_day == after_midnight.toordinal()
def test_bestow_pool_resets_on_the_same_boundary() -> None:
player = make_player(bestow_spent=25, bestow_day=utc(2026, 6, 12).toordinal())
ensure_day(player, fixed_clock(utc(2026, 6, 13, 0, 1)), daily_turns=10)
assert player.bestow_spent == 0
assert player.bestow_day == utc(2026, 6, 13).toordinal()
def test_social_caps_reset_on_the_same_boundary() -> None:
"""Posts and dice counts ride the same UTC rollover as turns and bestow."""
yesterday = utc(2026, 6, 12).toordinal()
player = make_player(
posts_sent=5,
post_day=yesterday,
gambles=5,
gamble_day=yesterday,
)
reset = ensure_day(player, fixed_clock(utc(2026, 6, 13, 0, 1)), daily_turns=10)
assert reset is True
assert player.posts_sent == 0
assert player.post_day == utc(2026, 6, 13).toordinal()
assert player.gambles == 0
assert player.gamble_day == utc(2026, 6, 13).toordinal()
-591
View File
@@ -1,591 +0,0 @@
"""Watch-page payload builders and the watch-URL advertisement.
These are pure-unit tests of :mod:`understone.watch` (no network): the static
world payload's shape and legend completeness, the dynamic state payload's
player/herald/hall content under a frozen clock, and the join/help "Watch the
Vale live" line that appears only when a Game carries a watch URL.
"""
from __future__ import annotations
import re
from pathlib import Path
from typing import TYPE_CHECKING
import pytest
from tests.conftest import fixed_clock, utc
from understone import server as understone_server
from understone import watch
from understone.engine.log import Event
from understone.engine.rng import GameRNG
from understone.game import Game
from understone.persistence import Store
from understone.screen.palette import Color
from understone.world.loader import load_world
if TYPE_CHECKING:
from understone.engine.world import World
PACK = Path(__file__).resolve().parents[1] / "understone" / "world" / "data"
@pytest.fixture
def world() -> World:
return load_world(PACK)
@pytest.fixture
def clock() -> object:
return fixed_clock(utc(2026, 6, 12, 10, 30))
def _game(tmp_path: Path, clock: object, watch_url: str | None = None) -> Game:
world = load_world(PACK)
store = Store(tmp_path / "watch.db")
return Game( # type: ignore[arg-type]
world, store, clock=clock, rng=GameRNG(seed=7), watch_url=watch_url
)
# ---------------------------------------------------------------------------
# World payload (static)
# ---------------------------------------------------------------------------
def test_world_payload_shape(world: World) -> None:
payload = watch.build_world_payload(world)
assert payload["name"] == world.name
assert payload["width"] == world.width
assert payload["height"] == world.height
rows = payload["glyph_rows"]
assert isinstance(rows, list)
assert len(rows) == world.height
assert all(isinstance(r, str) and len(r) == world.width for r in rows)
def test_world_payload_legend_is_complete(world: World) -> None:
payload = watch.build_world_payload(world)
rows = payload["glyph_rows"]
legend = payload["legend"]
assert isinstance(rows, list)
assert isinstance(legend, dict)
# Contract: every glyph that appears in the rows has a colour in the legend.
glyphs = {ch for row in rows for ch in row}
assert glyphs <= set(legend)
# And every legend colour is a real palette colour name (no stray roles).
valid = {c.value for c in Color}
assert set(legend.values()) <= valid
def test_world_payload_locations_present(world: World) -> None:
payload = watch.build_world_payload(world)
locations = payload["locations"]
assert isinstance(locations, list)
assert len(locations) == len(world.locations)
by_name = {loc["name"]: loc for loc in locations}
# The dungeon mouth rides in the locations overlay with its glyph + colour.
deep = by_name["The Understone Deep"]
assert deep["glyph"] == ""
assert deep["color"] == "dungeon"
assert (deep["x"], deep["y"]) == (70, 12)
def test_world_payload_carries_reskinned_glyphs(world: World) -> None:
"""The v0.6 re-skin reaches the Watch: ≋ water in the rows, ⌂/✚/∩ buildings.
Water rides the base terrain (glyph_rows + legend); the buildings ride the
locations overlay. If a glyph reverts, the live map drifts from the frames.
"""
payload = watch.build_world_payload(world)
rows = payload["glyph_rows"]
assert isinstance(rows, list)
glyphs = {ch for row in rows for ch in row}
assert "" in glyphs # water in the base map
assert "~" not in glyphs # the old water glyph is gone
legend = payload["legend"]
assert isinstance(legend, dict)
assert "" in legend
by_name = {loc["name"]: loc["glyph"] for loc in payload["locations"]} # type: ignore[index,union-attr]
assert by_name["The Sleeping Drake"] == ""
assert by_name["The Quiet Shrine"] == ""
assert by_name["The Understone Deep"] == ""
# ---------------------------------------------------------------------------
# v0.9 colour-role split — the payload now carries the EXPANDED vocabulary, so
# distinct terrain/building types read by hue on the Watch and not just by glyph.
# These pin the literal fixes: road no longer shares grass's colour, forest no
# longer shares tree's, the town buildings each carry their own role, and the
# Cinder slag is lava (orange), no longer water (blue).
# ---------------------------------------------------------------------------
CINDER = Path(__file__).resolve().parents[1] / "understone" / "world" / "packs" / "cinder-wastes"
def _terrain_kinds(world: World) -> dict[str, str]:
"""Return the distinct terrain kinds in *world* as ``{key: colour role}``.
``world.terrain`` is the painted 2-D grid (one ``TerrainDef`` per cell); the
distinct kinds are recovered by deduplicating it on ``key``. Every kind in a
shipped world appears on the map, so this sees all of them.
"""
kinds: dict[str, str] = {}
for row in world.terrain:
for cell in row:
kinds[cell.key] = cell.color
return kinds
def _legend_for_terrain_key(world: World, key: str) -> str:
"""Return the legend colour the payload carries for terrain ``key``.
Resolves the terrain key to its glyph, then reads that glyph's colour out of
the built payload's legend — so the assertion is on what the Watch receives,
not on the raw JSON.
"""
payload = watch.build_world_payload(world)
legend = payload["legend"]
assert isinstance(legend, dict)
glyph = next(cell.glyph for row in world.terrain for cell in row if cell.key == key)
return legend[glyph]
def test_vale_payload_road_is_not_floor(world: World) -> None:
"""REGRESSION (the literal bug the slice fixes): road has its OWN colour.
Before v0.9 the Vale road shared ``floor`` with grass, so a path was
indistinguishable from open ground on the Watch. The road now carries
``road``; grass keeps ``floor``; they must differ.
"""
road = _legend_for_terrain_key(world, "road")
grass = _legend_for_terrain_key(world, "grass")
assert road == "road"
assert grass == "floor"
assert road != grass
def test_vale_payload_forest_is_not_tree(world: World) -> None:
"""REGRESSION: forest has its OWN colour, no longer shared with tree.
Dense forest scrub used to share ``tree`` with the tree wall, so the two
read identically. Forest now carries ``forest``; tree keeps ``tree``.
"""
forest = _legend_for_terrain_key(world, "forest")
tree = _legend_for_terrain_key(world, "tree")
assert forest == "forest"
assert tree == "tree"
assert forest != tree
def test_vale_payload_buildings_carry_distinct_roles(world: World) -> None:
"""Each Vale town building rides its own role (inn/shop/healer), not ``town``."""
payload = watch.build_world_payload(world)
by_name = {loc["name"]: loc["color"] for loc in payload["locations"]} # type: ignore[index,union-attr]
assert by_name["The Sleeping Drake"] == "inn"
assert by_name["Gravel & Sons Outfitters"] == "shop"
assert by_name["The Quiet Shrine"] == "healer"
assert by_name["The Understone Deep"] == "dungeon"
# No two distinct buildings share a colour role.
roles = list(by_name.values())
assert len(set(roles)) == len(roles)
def test_cinder_payload_slag_is_lava_not_water() -> None:
"""The Cinder slag carries ``lava`` (orange), never ``water`` (blue) again.
This is the Cinder half of the bug: molten slag shared ``water``, so the
lava rendered BLUE on the Watch. After the remap the legend carries ``lava``
and ``water`` appears NOWHERE in the Cinder payload (no water in this world).
"""
cinder = load_world(CINDER)
payload = watch.build_world_payload(cinder)
legend = payload["legend"]
assert isinstance(legend, dict)
assert _legend_for_terrain_key(cinder, "slag") == "lava"
assert "water" not in legend.values()
def test_cinder_payload_carries_expanded_roles() -> None:
"""The Cinder terrain reads by hue: ash→barren, basalt→road, cinder→scrub.
Cinder-fields use ``scrub`` (dusky ember-brown), NOT ``forest`` (green)
a volcanic waste must not render as lush woods. ``forest`` is for green
worlds; ``scrub`` is its barren counterpart.
"""
cinder = load_world(CINDER)
assert _legend_for_terrain_key(cinder, "ash") == "barren"
assert _legend_for_terrain_key(cinder, "basalt") == "road"
assert _legend_for_terrain_key(cinder, "cinder") == "scrub"
legend = watch.build_world_payload(cinder)["legend"]
assert isinstance(legend, dict)
assert "forest" not in legend.values() # no green woods in a volcanic waste
# Obsidian spire reuses the wall role (a rock barrier), same as caldera.
assert _legend_for_terrain_key(cinder, "spire") == "wall"
assert _legend_for_terrain_key(cinder, "caldera") == "wall"
def test_both_worlds_terrain_roles_are_distinct_per_world() -> None:
"""No two DISTINCT terrain types share a colour role within a world.
The point of the slice: after the remap each terrain kind reads by its own
hue. (A role MAY be shared by two types that are deliberately the same
barrier spire/caldera both ``wall`` in Cinder so this checks distinct
KEYS that map to the same role are only the intended wall pair.)
"""
for world_dir, allowed_shared in (
(PACK, set()),
(CINDER, {("caldera", "spire")}),
):
w = load_world(world_dir)
by_role: dict[str, list[str]] = {}
for key, role in _terrain_kinds(w).items():
by_role.setdefault(role, []).append(key)
for role, keys in by_role.items():
if len(keys) > 1:
pair = tuple(sorted(keys))
assert pair in allowed_shared, f"unexpected shared role {role!r}: {keys}"
# ---------------------------------------------------------------------------
# State payload (dynamic)
# ---------------------------------------------------------------------------
def test_state_payload_includes_joined_player(tmp_path: Path, clock: object) -> None:
game = _game(tmp_path, clock)
game.join("Brandr")
payload = watch.build_state_payload(game)
players = payload["players"]
assert isinstance(players, list)
brandr = next(p for p in players if p["name"] == "Brandr")
assert brandr["level"] == 1
assert brandr["wins"] == 0
assert brandr["hp"] == brandr["max_hp"]
assert brandr["mode"] == "tile"
assert (brandr["x"], brandr["y"]) == game.world.spawn
# v0.10: a fresh hero shows their starting gold on hand, nothing banked, and
# an empty satchel.
assert brandr["gold"] == game.world.settings.starting_gold
assert brandr["banked"] == 0
assert brandr["satchel"] == []
def test_state_payload_surfaces_gold_banked_and_satchel(tmp_path: Path, clock: object) -> None:
"""A joined hero with a stocked satchel and banked gold shows the right values.
The lobby TV surfaces the whole shared world, so each player's purse (gold
on hand + vault) and satchel stacks (name + qty, resolved via the pack) ride
the state payload.
"""
game = _game(tmp_path, clock)
game.join("Brandr")
player = game.players["Brandr"]
player.gold = 120
player.banked = 300
game._satchel_set_stacks(player, [("iron_ore", 5), ("minor_potion", 2)])
payload = watch.build_state_payload(game)
brandr = next(p for p in payload["players"] if p["name"] == "Brandr") # type: ignore[union-attr]
assert brandr["gold"] == 120
assert brandr["banked"] == 300
# Stacks resolve their display name from the pack, preserving stow order.
assert brandr["satchel"] == [
{"name": "Iron Ore", "qty": 5},
{"name": "Minor Potion", "qty": 2},
]
def test_state_payload_satchel_unknown_id_falls_back_to_raw(tmp_path: Path, clock: object) -> None:
"""A satchel id no longer in the pack falls back to the raw id, never blank."""
game = _game(tmp_path, clock)
game.join("Brandr")
game.players["Brandr"].satchel = "ghost_item:2" # not in the pack
payload = watch.build_state_payload(game)
brandr = next(p for p in payload["players"] if p["name"] == "Brandr") # type: ignore[union-attr]
assert brandr["satchel"] == [{"name": "ghost_item", "qty": 2}]
def test_state_payload_reports_all_players_including_menu(tmp_path: Path, clock: object) -> None:
game = _game(tmp_path, clock)
game.join("Brandr")
game.join("Sigrun")
# Put Sigrun in a MENU surface; the Watch still shows her on the board.
sigrun = game.players["Sigrun"]
from understone.engine.models import Mode
sigrun.mode = Mode.MENU
sigrun.at_location = "inn"
payload = watch.build_state_payload(game)
names = {p["name"] for p in payload["players"]} # type: ignore[union-attr]
assert names == {"Brandr", "Sigrun"}
menu = next(p for p in payload["players"] if p["name"] == "Sigrun") # type: ignore[union-attr]
assert menu["mode"] == "menu"
def test_state_payload_ts_comes_from_clock(tmp_path: Path, clock: object) -> None:
game = _game(tmp_path, clock)
payload = watch.build_state_payload(game)
assert payload["ts"] == "2026-06-12T10:30:00+00:00"
def test_state_payload_herald_is_last_15_oldest_first(tmp_path: Path, clock: object) -> None:
game = _game(tmp_path, clock)
# Replace the resident feed with 20 synthetic events in ascending id order.
game.events = [
Event(
event_id=i,
ts=f"2026-06-12T10:{i:02d}:00+00:00",
kind="join",
actor=f"Hero{i}",
text=f"event {i}",
)
for i in range(1, 21)
]
payload = watch.build_state_payload(game)
herald = payload["herald"]
assert isinstance(herald, list)
assert len(herald) == 15
# Oldest-first: the window is events 6..20, in ascending order.
assert herald[0]["text"] == "event 6"
assert herald[-1]["text"] == "event 20"
def test_state_payload_herald_full_window_despite_sparse_ids(tmp_path: Path, clock: object) -> None:
"""Id gaps must not shrink the feed (regression: the window is a list
tail, not id arithmetic AUTOINCREMENT ids may be non-contiguous)."""
game = _game(tmp_path, clock)
game.events = [
Event(
event_id=i * 7, # sparse, non-contiguous ids
ts=f"2026-06-12T10:{i:02d}:00+00:00",
kind="join",
actor=f"Hero{i}",
text=f"event {i}",
)
for i in range(1, 21)
]
herald = watch.build_state_payload(game)["herald"]
assert len(herald) == 15
assert herald[0]["text"] == "event 6"
assert herald[-1]["text"] == "event 20"
def test_state_payload_herald_handles_short_feed(tmp_path: Path, clock: object) -> None:
game = _game(tmp_path, clock)
game.events = [
Event(
event_id=1,
ts="2026-06-12T10:00:00+00:00",
kind="join",
actor="Solo",
text="only one",
)
]
payload = watch.build_state_payload(game)
herald = payload["herald"]
assert isinstance(herald, list)
assert [e["text"] for e in herald] == ["only one"]
def test_state_payload_hall_capped_at_five(tmp_path: Path, clock: object) -> None:
game = _game(tmp_path, clock)
# Seven immortalised runs; the Watch shows only the five most recent.
for i in range(7):
game.store.insert_hall_row(f"Hero{i}", f"2026-06-{10 + i:02d}T12:00:00+00:00", i, 6 + i)
game.store.commit()
payload = watch.build_state_payload(game)
hall = payload["hall"]
assert isinstance(hall, list)
assert len(hall) == 5
# Newest first (store ordering): Hero6 leads.
assert hall[0]["name"] == "Hero6"
assert hall[0]["level_at_win"] == 12
# ---------------------------------------------------------------------------
# Watch-URL advertisement (join banner + help manual)
# ---------------------------------------------------------------------------
def test_join_advertises_watch_url_when_set(tmp_path: Path, clock: object) -> None:
game = _game(tmp_path, clock, watch_url="http://127.0.0.1:8077/watch")
out = game.join("Brandr")
assert "Watch the Vale live: http://127.0.0.1:8077/watch" in out
def test_join_omits_watch_line_when_unset(tmp_path: Path, clock: object) -> None:
game = _game(tmp_path, clock)
out = game.join("Brandr")
assert "Watch the Vale live" not in out
def test_resume_advertises_watch_url_when_set(tmp_path: Path, clock: object) -> None:
game = _game(tmp_path, clock, watch_url="http://127.0.0.1:8077/watch")
game.join("Brandr")
again = game.join("Brandr")
assert "Welcome back" in again
assert "Watch the Vale live: http://127.0.0.1:8077/watch" in again
def test_help_advertises_watch_url_when_set(tmp_path: Path) -> None:
# door_help reads the module game; install one carrying a watch URL.
world = load_world(PACK)
store = Store(tmp_path / "help.db")
understone_server._set_game(Game(world, store, watch_url="http://127.0.0.1:8077/watch"))
try:
manual = understone_server.door_help()
assert "Watch the Vale live: http://127.0.0.1:8077/watch" in manual
finally:
understone_server._GAME.store.close() # type: ignore[union-attr]
understone_server._GAME = None
def test_help_omits_watch_line_when_unset(tmp_path: Path) -> None:
world = load_world(PACK)
store = Store(tmp_path / "help.db")
understone_server._set_game(Game(world, store))
try:
manual = understone_server.door_help()
assert "Watch the Vale live" not in manual
finally:
understone_server._GAME.store.close() # type: ignore[union-attr]
understone_server._GAME = None
# ---------------------------------------------------------------------------
# WATCH_HTML lockstep guards (the JS twin of texture.py + the v0.6 glow-up)
#
# The inline page reproduces logic that lives in Python; these guard the two
# invariants most prone to silent drift — the texture selection formula and the
# other-player marker — plus the presence of the day-phase machinery.
# ---------------------------------------------------------------------------
def test_watch_html_derives_texture_formula_from_constants() -> None:
"""The page's JS index string is DERIVED from texture._HASH_X / _HASH_Y.
Not a hard-coded "x * 31 + y * 17" snapshot: the expected substring is built
from the live constants, so a Python-side retune that the watch builder
fails to track trips here instead of silently shipping a stale formula.
"""
from understone.screen import texture
expected = f"x * {texture._HASH_X} + y * {texture._HASH_Y}"
assert expected in watch.WATCH_HTML
def test_watch_html_js_selection_agrees_with_textured() -> None:
"""The JS selection arithmetic, replayed in Python, matches ``textured``.
The page computes ``variants[(x * _HASH_X + y * _HASH_Y) % len]``. Replaying
that exact formula here from the SAME constants and the SAME VARIANTS rows
and asserting it equals ``texture.textured`` over a full screen grid proves
both implementations select identically a stronger lockstep than a string
match, since it pins the result, not the source text.
"""
from understone.screen import texture
for base, choices in texture.VARIANTS.items():
for x in range(24):
for y in range(16):
js_pick = choices[(x * texture._HASH_X + y * texture._HASH_Y) % len(choices)]
assert texture.textured(base, x, y) == js_pick
def test_watch_html_variants_match_texture_table() -> None:
"""Every base->variants row in texture.VARIANTS appears in the JS VARIANTS map.
Glyphs ride into the inline JS as ``\\uXXXX`` escapes, so compare against the
escaped form. A new variant added to Python but not the page trips this.
"""
from understone.screen import texture
html = watch.WATCH_HTML
for base, choices in texture.VARIANTS.items():
for glyph in {base, *choices}:
token = glyph if glyph.isascii() else f"\\u{ord(glyph):04x}"
assert token in html, f"variant glyph {glyph!r} missing from WATCH_HTML"
def test_watch_html_uses_other_player_marker() -> None:
"""Players on the lobby TV wear the ☻ marker (escaped) — no bare '@' marker paint."""
assert "\\u263b" in watch.WATCH_HTML
def test_watch_html_renders_gold_banked_and_satchel() -> None:
"""The Adventurers panel JS references each player's gold, vault, and satchel."""
html = watch.WATCH_HTML
# The roster sub-lines read these state fields by name.
assert "p.gold" in html
assert "p.banked" in html
assert "p.satchel" in html
# The satchel line has a dedicated renderer with an empty-bag note.
assert "satchelText" in html
assert "satchel empty" in html
assert "vault" in html
def test_watch_html_has_day_phase_machinery() -> None:
"""The dusk/dawn glow-up is wired: the tint classes and the UTC-hour read."""
html = watch.WATCH_HTML
assert "applyDayPhase" in html
assert "getUTCHours" in html
assert ".map-frame.night" in html
assert ".map-frame.twilight" in html
assert "Noto Sans Mono" in html
# ---------------------------------------------------------------------------
# PALETTE completeness — the v0.9 invariant that kills the "silent fallback"
# bug class. The road bug existed because a Color role with no hex in the JS
# PALETTE map fell back to default; this pins that EVERY role has a hex.
# ---------------------------------------------------------------------------
def _watch_palette_keys() -> set[str]:
"""Parse the JS ``var PALETTE = { ... }`` map out of WATCH_HTML, return its keys.
The map uses bare (unquoted) JS identifier keys ``road: "#b89a6a",`` so
this slices the object literal and collects every ``key:`` token. Keeping the
parse here (not a hard-coded list) means the test reads whatever the page
actually ships, so a typo'd or dropped key surfaces as a missing role.
"""
html = watch.WATCH_HTML
start = html.index("var PALETTE = {")
body = html[start : html.index("};", start)]
# Each entry is `<ident>: "<hex>"`; capture the identifier before the colon.
return set(re.findall(r"(\w+)\s*:\s*\"#", body))
def test_watch_palette_covers_every_color_role() -> None:
"""EVERY Color enum value has an entry in the JS PALETTE map — no fallbacks.
This is the literal fix for the road bug: a shipped role with no hex paints
as ``default`` silently. Asserting ``{c.value} <= palette_keys`` means adding
a Color without a Watch hex trips here instead of shipping a grey/green road.
"""
palette_keys = _watch_palette_keys()
roles = {c.value for c in Color}
missing = roles - palette_keys
assert not missing, f"Color roles with no PALETTE hex (silent fallback): {sorted(missing)}"
def test_watch_palette_distinct_new_terrain_hexes() -> None:
"""The expanded terrain roles carry DISTINCT hexes (the point of the slice).
A guard that the seven new roles didn't accidentally collapse onto one hex
(which would re-introduce the very "two types, one colour" bug v0.9 fixes).
Parsed straight from the shipped map.
"""
html = watch.WATCH_HTML
start = html.index("var PALETTE = {")
body = html[start : html.index("};", start)]
pairs = dict(re.findall(r"(\w+)\s*:\s*\"(#[0-9a-fA-F]{6})\"", body))
new_roles = ["road", "forest", "lava", "barren", "inn", "shop", "healer"]
hexes = [pairs[r] for r in new_roles]
assert all(r in pairs for r in new_roles), "a new v0.9 role is missing its hex"
assert len(set(hexes)) == len(hexes), f"new roles share a hex: {hexes}"
# The molten role must NOT reuse water's blue (the Cinder slag bug).
assert pairs["lava"] != pairs["water"]
@@ -1,143 +0,0 @@
"""Tests for the per-pack Watch CRT theme (v0.8).
Covers the loader band (each of the four legal themes loads; an unknown theme
is rejected naming the legal set; an omitted theme defaults to phosphor), the
state-payload carrying the theme, and the WATCH_HTML page's JS THEME table —
including the load-bearing guard that the "phosphor" values byte-match the
original ``:root`` CSS, so the bundled Vale stays visually identical.
"""
from __future__ import annotations
import json
import shutil
from pathlib import Path
from typing import Any
import pytest
from understone import watch
from understone.errors import WorldLoadError
from understone.world.loader import (
DEFAULT_WATCH_THEME,
WATCH_THEMES,
load_world,
)
SHIPPED = Path(__file__).resolve().parents[1] / "understone" / "world" / "data"
# The original :root CRT custom-property values (pre-v0.8). The "phosphor" theme
# MUST reproduce these byte-for-byte so the default Vale is pixel-identical.
_ORIGINAL_ROOT = {
"--phosphor": "#7dffa0",
"--phosphor-dim": "#2f7a46",
"--amber": "#ffb44d",
"--bg": "#050a06",
"--panel": "#0a140d",
"--edge": "#163a22",
}
def _pack_with_theme(tmp_path: Path, theme: Any) -> Path:
"""Clone the Vale into a temp pack with ``settings.watch_theme`` set/removed.
``theme`` set to a string writes that value; set to the sentinel ``...``
DELETES the key entirely (to exercise the omitted-defaults path).
"""
dest = tmp_path / "themed"
shutil.copytree(SHIPPED, dest)
world_json = dest / "world.json"
data = json.loads(world_json.read_text(encoding="utf-8"))
if theme is ...:
data["settings"].pop("watch_theme", None)
else:
data["settings"]["watch_theme"] = theme
world_json.write_text(json.dumps(data), encoding="utf-8")
return dest
# ---------------------------------------------------------------------------
# loader band
# ---------------------------------------------------------------------------
@pytest.mark.parametrize("theme", sorted(WATCH_THEMES))
def test_each_legal_theme_loads(tmp_path: Path, theme: str) -> None:
pack = _pack_with_theme(tmp_path, theme)
world = load_world(pack)
assert world.settings.watch_theme == theme
def test_unknown_theme_rejected_naming_the_set(tmp_path: Path) -> None:
pack = _pack_with_theme(tmp_path, "ultraviolet")
with pytest.raises(WorldLoadError) as exc:
load_world(pack)
message = str(exc.value)
assert "watch_theme" in message
assert "ultraviolet" in message
# The friendly message lists every legal theme so the author can fix it.
for name in WATCH_THEMES:
assert name in message
def test_omitted_theme_defaults_to_phosphor(tmp_path: Path) -> None:
pack = _pack_with_theme(tmp_path, ...) # delete the key entirely
world = load_world(pack)
assert world.settings.watch_theme == DEFAULT_WATCH_THEME == "phosphor"
def test_shipped_vale_is_phosphor() -> None:
"""The bundled Vale ships the phosphor theme (its green is unchanged)."""
world = load_world(SHIPPED)
assert world.settings.watch_theme == "phosphor"
# ---------------------------------------------------------------------------
# payload + WATCH_HTML
# ---------------------------------------------------------------------------
def test_world_payload_carries_theme(tmp_path: Path) -> None:
pack = _pack_with_theme(tmp_path, "ice")
world = load_world(pack)
payload = watch.build_world_payload(world)
assert payload["theme"] == "ice"
def test_shipped_payload_theme_is_phosphor() -> None:
world = load_world(SHIPPED)
payload = watch.build_world_payload(world)
assert payload["theme"] == "phosphor"
def test_watch_html_has_theme_table_and_all_names() -> None:
"""The page carries a JS THEME table keyed by every legal theme name."""
html = watch.WATCH_HTML
assert "var THEMES" in html
assert "applyTheme" in html
for name in WATCH_THEMES:
# Each theme is a JS object key, e.g. ``phosphor: {``.
assert f"{name}: {{" in html, f"theme {name!r} missing from THEME table"
def test_watch_html_phosphor_values_byte_match_original_root() -> None:
"""The "phosphor" theme reproduces the original :root values exactly.
This is the load-bearing guard for "the Vale looks identical": every
original custom-property value still appears in the page (in the :root block
AND the THEME table), so swapping in the phosphor theme is a no-op repaint.
"""
html = watch.WATCH_HTML
for prop, value in _ORIGINAL_ROOT.items():
# The value lives both in the :root CSS and the phosphor theme entry.
assert html.count(value) >= 2, f"{prop} value {value} not byte-matched twice"
# And the phosphor theme maps the property to exactly that value.
assert f'"{prop}": "{value}"' in html, f"phosphor {prop} != {value}"
def test_watch_html_applies_theme_on_world_fetch() -> None:
"""The page applies the theme when world.json arrives (in paintMap)."""
html = watch.WATCH_HTML
assert "applyTheme(world.theme)" in html
# It swaps CSS custom properties on the document root.
assert "documentElement.style.setProperty" in html
@@ -1,851 +0,0 @@
"""Content-pack loader tests.
Asserts the shipped pack loads, and that representative malformed packs
each raise :class:`WorldLoadError` with a readable message: a bad legend
character, a location placed on non-walkable terrain, a row-width / height
mismatch, and an economy setting outside its sanity band.
"""
from __future__ import annotations
import json
import shutil
from pathlib import Path
from typing import Any
import pytest
from understone.errors import WorldLoadError
from understone.world.loader import load_world
SHIPPED = Path(__file__).resolve().parents[1] / "understone" / "world" / "data"
def test_shipped_pack_loads() -> None:
world = load_world(SHIPPED)
assert world.name == "The Vale of Understone"
assert world.width == 96
assert world.height == 48
assert world.is_walkable(*world.spawn)
assert len(world.locations) == 4
assert len(world.zones) == 2
# Tiers 1..5 are the random foes; tier 6 is the boss (the Wyrm Below).
assert {m.tier for m in world.monsters} == {1, 2, 3, 4, 5, 6}
boss = world.monster_by_id(world.settings.boss_monster)
assert boss is not None and boss.boss and boss.name == "the Wyrm Below"
def _clone_pack(tmp_path: Path) -> Path:
dest = tmp_path / "pack"
shutil.copytree(SHIPPED, dest)
return dest
def _rewrite(path: Path, mutate: Any) -> None:
data = json.loads(path.read_text(encoding="utf-8"))
mutate(data)
path.write_text(json.dumps(data), encoding="utf-8")
def test_bad_legend_char_rejected(tmp_path: Path) -> None:
pack = _clone_pack(tmp_path)
def mutate(data: dict[str, Any]) -> None:
# Splice an unknown glyph into the middle of a terrain row.
row = list(data["terrain_rows"][24])
row[40] = "Z"
data["terrain_rows"][24] = "".join(row)
_rewrite(pack / "world.json", mutate)
with pytest.raises(WorldLoadError, match="not in the legend"):
load_world(pack)
def test_location_on_non_walkable_rejected(tmp_path: Path) -> None:
pack = _clone_pack(tmp_path)
def mutate(data: dict[str, Any]) -> None:
# Move the inn onto a tree-border tile (col 0 is the tree frame).
for loc in data["locations"]:
if loc["key"] == "inn":
loc["x"] = 0
loc["y"] = 24
_rewrite(pack / "world.json", mutate)
with pytest.raises(WorldLoadError, match="non-walkable"):
load_world(pack)
def test_dimension_mismatch_rejected(tmp_path: Path) -> None:
pack = _clone_pack(tmp_path)
def mutate(data: dict[str, Any]) -> None:
# Truncate one row so its width no longer matches the declared width.
data["terrain_rows"][10] = data["terrain_rows"][10][:-5]
_rewrite(pack / "world.json", mutate)
with pytest.raises(WorldLoadError, match="wide but width is"):
load_world(pack)
def test_height_mismatch_rejected(tmp_path: Path) -> None:
pack = _clone_pack(tmp_path)
def mutate(data: dict[str, Any]) -> None:
data["terrain_rows"] = data["terrain_rows"][:-1]
_rewrite(pack / "world.json", mutate)
with pytest.raises(WorldLoadError, match="rows but height is"):
load_world(pack)
def test_settings_out_of_band_rejected(tmp_path: Path) -> None:
pack = _clone_pack(tmp_path)
def mutate(data: dict[str, Any]) -> None:
data["settings"]["daily_turns"] = 0 # band is 1..100
_rewrite(pack / "world.json", mutate)
with pytest.raises(WorldLoadError, match="daily_turns"):
load_world(pack)
def test_start_hp_zero_rejected(tmp_path: Path) -> None:
"""A starting HP of 0 is out of band (1..500): a hero must begin alive."""
pack = _clone_pack(tmp_path)
def mutate(data: dict[str, Any]) -> None:
data["settings"]["start_hp"] = 0 # band is 1..500
_rewrite(pack / "world.json", mutate)
with pytest.raises(WorldLoadError, match="start_hp"):
load_world(pack)
def test_unknown_starting_item_rejected(tmp_path: Path) -> None:
pack = _clone_pack(tmp_path)
def mutate(data: dict[str, Any]) -> None:
data["settings"]["starting_weapon"] = "no_such_blade"
_rewrite(pack / "world.json", mutate)
with pytest.raises(WorldLoadError, match="not a known item id"):
load_world(pack)
def test_missing_pack_file_rejected(tmp_path: Path) -> None:
pack = _clone_pack(tmp_path)
(pack / "monsters.json").unlink()
with pytest.raises(WorldLoadError, match="missing pack file"):
load_world(pack)
def test_monster_nonpositive_hp_rejected(tmp_path: Path) -> None:
pack = _clone_pack(tmp_path)
def mutate(data: list[dict[str, Any]]) -> None:
data[0]["hp"] = 0 # a monster with no hit points is unkillable nonsense
_rewrite(pack / "monsters.json", mutate)
with pytest.raises(WorldLoadError, match=r"monsters\.json\[0\] hp must be >= 1"):
load_world(pack)
def test_monster_negative_stat_rejected(tmp_path: Path) -> None:
pack = _clone_pack(tmp_path)
def mutate(data: list[dict[str, Any]]) -> None:
data[1]["gold"] = -5
_rewrite(pack / "monsters.json", mutate)
with pytest.raises(WorldLoadError, match=r"monsters\.json\[1\] gold must be >= 0"):
load_world(pack)
def test_item_negative_price_rejected(tmp_path: Path) -> None:
pack = _clone_pack(tmp_path)
def mutate(data: list[dict[str, Any]]) -> None:
data[1]["price"] = -10 # a negative price would pay the player to take it
_rewrite(pack / "items.json", mutate)
with pytest.raises(WorldLoadError, match=r"items\.json\[1\] price must be >= 0"):
load_world(pack)
def test_dungeon_tier_without_monster_rejected(tmp_path: Path) -> None:
pack = _clone_pack(tmp_path)
def mutate(data: dict[str, Any]) -> None:
# Tier 9 has no monster in the pack, so the gauntlet rung is unfillable.
data["settings"]["dungeon_tiers"] = [4, 9]
_rewrite(pack / "world.json", mutate)
with pytest.raises(WorldLoadError, match=r"dungeon_tiers\[1\] = 9 has no non-boss monster"):
load_world(pack)
def test_dungeon_tiers_empty_rejected(tmp_path: Path) -> None:
pack = _clone_pack(tmp_path)
def mutate(data: dict[str, Any]) -> None:
data["settings"]["dungeon_tiers"] = []
_rewrite(pack / "world.json", mutate)
with pytest.raises(WorldLoadError, match="dungeon_tiers must be a non-empty list"):
load_world(pack)
def test_dungeon_tier_backed_only_by_boss_rejected(tmp_path: Path) -> None:
"""A boss-only tier is unfillable: the gauntlet excludes boss monsters.
Tier 6 in the shipped pack holds only the Wyrm Below (a boss). A gauntlet
rung at tier 6 would draw from monsters_for_tier_band, which filters bosses
out, so the rung silently does nothing the loader must reject it instead.
The message says "no NON-boss monster" (not merely "no monster"): the boss
is present at that tier, it just cannot fill a rung, and the wording must
point the author at exactly that.
"""
pack = _clone_pack(tmp_path)
def mutate(data: dict[str, Any]) -> None:
data["settings"]["dungeon_tiers"] = [4, 6] # 6 is the boss-only tier
_rewrite(pack / "world.json", mutate)
with pytest.raises(WorldLoadError, match=r"dungeon_tiers\[1\] = 6 has no non-boss monster"):
load_world(pack)
# ---------------------------------------------------------------------------
# v0.2 loader rejections: the event table and the Wyrm settings
# ---------------------------------------------------------------------------
def test_events_without_fight_rejected(tmp_path: Path) -> None:
pack = _clone_pack(tmp_path)
def mutate(data: dict[str, Any]) -> None:
# Strip every fight row; a walk could then never spawn a monster.
data["events"] = [e for e in data["events"] if e["kind"] != "fight"]
_rewrite(pack / "events.json", mutate)
with pytest.raises(WorldLoadError, match="at least one 'fight' entry"):
load_world(pack)
def test_event_zero_weight_rejected(tmp_path: Path) -> None:
pack = _clone_pack(tmp_path)
def mutate(data: dict[str, Any]) -> None:
data["events"][0]["weight"] = 0
_rewrite(pack / "events.json", mutate)
with pytest.raises(WorldLoadError, match="weight must be > 0"):
load_world(pack)
def test_event_min_exceeds_max_rejected(tmp_path: Path) -> None:
pack = _clone_pack(tmp_path)
def mutate(data: dict[str, Any]) -> None:
# Find a value-bearing row and invert its band.
for event in data["events"]:
if event["kind"] == "gold":
event["min"], event["max"] = 9, 2
break
_rewrite(pack / "events.json", mutate)
with pytest.raises(WorldLoadError, match="min 9 exceeds max 2"):
load_world(pack)
def test_event_amount_out_of_band_rejected(tmp_path: Path) -> None:
pack = _clone_pack(tmp_path)
def mutate(data: dict[str, Any]) -> None:
for event in data["events"]:
if event["kind"] == "heal":
event["max"] = 500 # heal band is 1..100
break
_rewrite(pack / "events.json", mutate)
with pytest.raises(WorldLoadError, match=r"heal amount .* is out of band"):
load_world(pack)
def test_event_nonfight_blank_text_rejected(tmp_path: Path) -> None:
pack = _clone_pack(tmp_path)
def mutate(data: dict[str, Any]) -> None:
for event in data["events"]:
if event["kind"] == "lore":
event["text"] = " "
break
_rewrite(pack / "events.json", mutate)
with pytest.raises(WorldLoadError, match="requires non-empty 'text'"):
load_world(pack)
def test_boss_monster_unknown_rejected(tmp_path: Path) -> None:
pack = _clone_pack(tmp_path)
def mutate(data: dict[str, Any]) -> None:
data["settings"]["boss_monster"] = "no_such_wyrm"
_rewrite(pack / "world.json", mutate)
with pytest.raises(WorldLoadError, match="is not a known monster id"):
load_world(pack)
def test_boss_monster_not_flagged_boss_rejected(tmp_path: Path) -> None:
pack = _clone_pack(tmp_path)
def mutate(data: list[dict[str, Any]]) -> None:
# Give a plain monster an id and point boss_monster at it; it lacks the
# boss flag, so it must be rejected as the endgame foe.
data[0]["id"] = "field_rat"
_rewrite(pack / "monsters.json", mutate)
def point(data: dict[str, Any]) -> None:
data["settings"]["boss_monster"] = "field_rat"
_rewrite(pack / "world.json", point)
with pytest.raises(WorldLoadError, match='must be flagged "boss": true'):
load_world(pack)
def test_wyrm_min_level_out_of_band_rejected(tmp_path: Path) -> None:
pack = _clone_pack(tmp_path)
def mutate(data: dict[str, Any]) -> None:
data["settings"]["wyrm_min_level"] = 0 # band is 1..50
_rewrite(pack / "world.json", mutate)
with pytest.raises(WorldLoadError, match="wyrm_min_level"):
load_world(pack)
# ---------------------------------------------------------------------------
# v0.5 social settings: ambush / post / gamble economy bands
# ---------------------------------------------------------------------------
def test_ambush_gold_pct_out_of_band_rejected(tmp_path: Path) -> None:
"""The steal percentage is a 0..100 band; 101 is rejected by name."""
pack = _clone_pack(tmp_path)
def mutate(data: dict[str, Any]) -> None:
data["settings"]["ambush_gold_pct"] = 101 # band is 0..100
_rewrite(pack / "world.json", mutate)
with pytest.raises(WorldLoadError, match="ambush_gold_pct"):
load_world(pack)
def test_ambush_level_band_out_of_band_rejected(tmp_path: Path) -> None:
pack = _clone_pack(tmp_path)
def mutate(data: dict[str, Any]) -> None:
data["settings"]["ambush_level_band"] = 11 # band is 0..10
_rewrite(pack / "world.json", mutate)
with pytest.raises(WorldLoadError, match="ambush_level_band"):
load_world(pack)
def test_gamble_max_bet_out_of_band_rejected(tmp_path: Path) -> None:
"""A max bet of 0 is below the 1..10000 floor: the house needs a real stake."""
pack = _clone_pack(tmp_path)
def mutate(data: dict[str, Any]) -> None:
data["settings"]["gamble_max_bet"] = 0 # band is 1..10000
_rewrite(pack / "world.json", mutate)
with pytest.raises(WorldLoadError, match="gamble_max_bet"):
load_world(pack)
def test_post_daily_cap_out_of_band_rejected(tmp_path: Path) -> None:
pack = _clone_pack(tmp_path)
def mutate(data: dict[str, Any]) -> None:
data["settings"]["post_daily_cap"] = 51 # band is 0..50
_rewrite(pack / "world.json", mutate)
with pytest.raises(WorldLoadError, match="post_daily_cap"):
load_world(pack)
def test_missing_social_setting_rejected(tmp_path: Path) -> None:
"""A pack that predates the social settings fails loudly (no silent default)."""
pack = _clone_pack(tmp_path)
def mutate(data: dict[str, Any]) -> None:
del data["settings"]["ambush_min_level"]
_rewrite(pack / "world.json", mutate)
with pytest.raises(WorldLoadError, match="ambush_min_level"):
load_world(pack)
# ---------------------------------------------------------------------------
# v0.4 loader hardening: glyphs, map size, count caps, and name lengths
#
# Packs are now routinely untrusted LLM output, so the loader bands the shapes
# that could tear a frame, balloon memory, or impersonate a player. Each
# rejection still names the file and field at fault.
# ---------------------------------------------------------------------------
def test_box_drawing_terrain_glyph_rejected(tmp_path: Path) -> None:
"""A terrain glyph may not be a frame box-drawing line (it would tear borders)."""
pack = _clone_pack(tmp_path)
def mutate(data: dict[str, Any]) -> None:
data["."]["glyph"] = "" # the horizontal frame run
_rewrite(pack / "terrain.json", mutate)
with pytest.raises(WorldLoadError, match=r"terrain\.json.* box-drawing"):
load_world(pack)
def test_player_marker_terrain_glyph_rejected(tmp_path: Path) -> None:
"""A terrain glyph may not be '@' — that is the player's own marker."""
pack = _clone_pack(tmp_path)
def mutate(data: dict[str, Any]) -> None:
data["."]["glyph"] = "@"
_rewrite(pack / "terrain.json", mutate)
with pytest.raises(WorldLoadError, match=r"terrain\.json.* reserved for player markers"):
load_world(pack)
def test_other_player_marker_terrain_glyph_rejected(tmp_path: Path) -> None:
"""A terrain glyph may not be '' — the v0.6 other-player marker."""
pack = _clone_pack(tmp_path)
def mutate(data: dict[str, Any]) -> None:
data["."]["glyph"] = ""
_rewrite(pack / "terrain.json", mutate)
with pytest.raises(WorldLoadError, match=r"terrain\.json.* reserved for player markers"):
load_world(pack)
def test_ampersand_terrain_glyph_now_accepted(tmp_path: Path) -> None:
"""'&' is no longer an actor marker (☻ took that role), so it is pack-legal.
The load itself is the assertion it must not raise the actor-marker
rejection. A grass cell then carries the new glyph.
"""
pack = _clone_pack(tmp_path)
def mutate(data: dict[str, Any]) -> None:
data["."]["glyph"] = "&"
_rewrite(pack / "terrain.json", mutate)
world = load_world(pack) # no WorldLoadError: '&' is admitted
grass = next(
world.terrain_at(x, y)
for y in range(world.height)
for x in range(world.width)
if world.terrain_at(x, y).key == "grass"
)
assert grass.glyph == "&"
def test_wide_cjk_terrain_glyph_rejected(tmp_path: Path) -> None:
"""A Wide (EAW=W) ideograph would render two columns and tear the frame."""
pack = _clone_pack(tmp_path)
def mutate(data: dict[str, Any]) -> None:
data["."]["glyph"] = ""
_rewrite(pack / "terrain.json", mutate)
with pytest.raises(WorldLoadError, match=r"terrain\.json.* exactly one column"):
load_world(pack)
def test_fullwidth_terrain_glyph_rejected(tmp_path: Path) -> None:
"""A Fullwidth (EAW=F) Latin letter is two columns and is rejected."""
pack = _clone_pack(tmp_path)
def mutate(data: dict[str, Any]) -> None:
data["."]["glyph"] = "" # U+FF21 FULLWIDTH LATIN CAPITAL LETTER A
_rewrite(pack / "terrain.json", mutate)
with pytest.raises(WorldLoadError, match=r"terrain\.json.* exactly one column"):
load_world(pack)
def test_reskinned_shipped_pack_glyphs() -> None:
"""The shipped pack carries the v0.6 re-skin and still loads cleanly.
The load-bearing guard for the re-skin: water is and the three lettered
buildings became //. If a data edit reverts a glyph, this trips.
"""
world = load_world(SHIPPED)
waters = {
world.terrain_at(x, y).glyph
for y in range(world.height)
for x in range(world.width)
if world.terrain_at(x, y).key == "water"
}
assert waters == {""}
by_key = {loc.key: loc.glyph for loc in world.locations}
assert by_key["inn"] == ""
assert by_key["healer"] == ""
assert by_key["dungeon"] == ""
assert by_key["shop"] == "$" # the shop glyph is unchanged
def test_multichar_location_glyph_rejected(tmp_path: Path) -> None:
"""A location glyph must be exactly one character."""
pack = _clone_pack(tmp_path)
def mutate(data: dict[str, Any]) -> None:
data["inn"]["glyph"] = "In" # two characters
_rewrite(pack / "locations.json", mutate)
with pytest.raises(WorldLoadError, match=r"locations\.json.* single character"):
load_world(pack)
def test_oversized_map_rejected(tmp_path: Path) -> None:
"""A 300x300 map is past the dimension ceiling (8..256)."""
pack = _clone_pack(tmp_path)
def mutate(data: dict[str, Any]) -> None:
data["width"] = 300
data["height"] = 300
_rewrite(pack / "world.json", mutate)
with pytest.raises(WorldLoadError, match=r"world\.json width = 300 is out of band"):
load_world(pack)
def test_too_many_events_rejected(tmp_path: Path) -> None:
"""An event table over the 500-row cap is rejected before it is decoded."""
pack = _clone_pack(tmp_path)
def mutate(data: dict[str, Any]) -> None:
filler = {"kind": "lore", "weight": 1, "text": "filler"}
data["events"] = [filler.copy() for _ in range(501)]
_rewrite(pack / "events.json", mutate)
with pytest.raises(WorldLoadError, match=r"events\.json defines 501 events; the limit is 500"):
load_world(pack)
def test_overlong_monster_name_rejected(tmp_path: Path) -> None:
"""A 49-character monster name is one past the 48-char display limit."""
pack = _clone_pack(tmp_path)
def mutate(data: list[dict[str, Any]]) -> None:
data[0]["name"] = "x" * 49
_rewrite(pack / "monsters.json", mutate)
with pytest.raises(WorldLoadError, match=r"monsters\.json\[0\] name is 49 characters"):
load_world(pack)
# ---------------------------------------------------------------------------
# v0.7 loader rejections: the satchel/forge bands, rare_drop_item, monster weight
# ---------------------------------------------------------------------------
def test_rare_drop_item_unknown_rejected(tmp_path: Path) -> None:
"""A rare_drop_item that names no item is rejected with the item-id message."""
pack = _clone_pack(tmp_path)
def mutate(data: dict[str, Any]) -> None:
data["settings"]["rare_drop_item"] = "no_such_draught"
_rewrite(pack / "world.json", mutate)
with pytest.raises(WorldLoadError, match="rare_drop_item = 'no_such_draught' is not a known"):
load_world(pack)
def test_rare_drop_item_non_consumable_rejected(tmp_path: Path) -> None:
"""A rare_drop_item that names a weapon (not a consumable) is rejected.
The drop goes straight into the satchel to be quaffed, so a weapon or
armour id is incoherent the loader pins the slot.
"""
pack = _clone_pack(tmp_path)
def mutate(data: dict[str, Any]) -> None:
data["settings"]["rare_drop_item"] = "iron_sword" # a weapon, not a draught
_rewrite(pack / "world.json", mutate)
with pytest.raises(WorldLoadError, match="rare_drop_item = 'iron_sword' must be a consumable"):
load_world(pack)
def test_satchel_max_out_of_band_rejected(tmp_path: Path) -> None:
"""satchel_max above its 1..10 band is a load error."""
pack = _clone_pack(tmp_path)
def mutate(data: dict[str, Any]) -> None:
data["settings"]["satchel_max"] = 11
_rewrite(pack / "world.json", mutate)
with pytest.raises(WorldLoadError, match=r"satchel_max = 11 is out of band \(1\.\.10\)"):
load_world(pack)
def test_forge_max_plus_out_of_band_rejected(tmp_path: Path) -> None:
"""forge_max_plus above its 0..10 band is a load error."""
pack = _clone_pack(tmp_path)
def mutate(data: dict[str, Any]) -> None:
data["settings"]["forge_max_plus"] = 11
_rewrite(pack / "world.json", mutate)
with pytest.raises(WorldLoadError, match=r"forge_max_plus = 11 is out of band \(0\.\.10\)"):
load_world(pack)
def test_forge_base_cost_out_of_band_rejected(tmp_path: Path) -> None:
"""forge_base_cost below its floor of 1 is a load error."""
pack = _clone_pack(tmp_path)
def mutate(data: dict[str, Any]) -> None:
data["settings"]["forge_base_cost"] = 0
_rewrite(pack / "world.json", mutate)
with pytest.raises(WorldLoadError, match=r"forge_base_cost = 0 is out of band \(1\.\.10000\)"):
load_world(pack)
def test_forge_ore_item_unknown_rejected(tmp_path: Path) -> None:
"""A forge_ore_item that names no item is rejected with the item-id message."""
pack = _clone_pack(tmp_path)
def mutate(data: dict[str, Any]) -> None:
data["settings"]["forge_ore_item"] = "no_such_ore"
_rewrite(pack / "world.json", mutate)
with pytest.raises(WorldLoadError, match="forge_ore_item = 'no_such_ore' is not a known"):
load_world(pack)
def test_forge_ore_item_non_material_rejected(tmp_path: Path) -> None:
"""A forge_ore_item that names a non-material (a potion) is rejected.
Ore is carried in the satchel and spent at the forge, never equipped or
quaffed, so a consumable/weapon/armour id is incoherent the loader pins
the slot to ``material`` (mirroring the rare_drop_item consumable check).
"""
pack = _clone_pack(tmp_path)
def mutate(data: dict[str, Any]) -> None:
data["settings"]["forge_ore_item"] = "greater_potion" # a draught, not ore
_rewrite(pack / "world.json", mutate)
with pytest.raises(
WorldLoadError, match="forge_ore_item = 'greater_potion' must be a material"
):
load_world(pack)
def test_forge_ore_per_plus_out_of_band_rejected(tmp_path: Path) -> None:
"""forge_ore_per_plus above its 0..10 band is a load error."""
pack = _clone_pack(tmp_path)
def mutate(data: dict[str, Any]) -> None:
data["settings"]["forge_ore_per_plus"] = 11
_rewrite(pack / "world.json", mutate)
with pytest.raises(WorldLoadError, match=r"forge_ore_per_plus = 11 is out of band \(0\.\.10\)"):
load_world(pack)
def test_ore_dungeon_drop_out_of_band_rejected(tmp_path: Path) -> None:
"""ore_dungeon_drop above its 0..20 band is a load error."""
pack = _clone_pack(tmp_path)
def mutate(data: dict[str, Any]) -> None:
data["settings"]["ore_dungeon_drop"] = 21
_rewrite(pack / "world.json", mutate)
with pytest.raises(WorldLoadError, match=r"ore_dungeon_drop = 21 is out of band \(0\.\.20\)"):
load_world(pack)
def test_ore_forest_chance_out_of_band_rejected(tmp_path: Path) -> None:
"""ore_forest_chance outside 0.0..1.0 is a load error (it is a probability)."""
pack = _clone_pack(tmp_path)
def mutate(data: dict[str, Any]) -> None:
data["settings"]["ore_forest_chance"] = 1.5
_rewrite(pack / "world.json", mutate)
with pytest.raises(
WorldLoadError, match=r"ore_forest_chance = 1.5 is out of band \(0.0..1.0\)"
):
load_world(pack)
def test_monster_zero_weight_rejected(tmp_path: Path) -> None:
"""A monster weight of 0 is rejected (the weighted pick needs a positive total)."""
pack = _clone_pack(tmp_path)
def mutate(data: list[dict[str, Any]]) -> None:
data[0]["weight"] = 0
_rewrite(pack / "monsters.json", mutate)
with pytest.raises(WorldLoadError, match=r"monsters\.json\[0\] weight must be > 0"):
load_world(pack)
def test_shipped_pack_carries_rares_and_weights() -> None:
"""The shipped pack parses the v0.7 rare beasts with their low weights."""
world = load_world(SHIPPED)
rares = [m for m in world.monsters if m.rare]
names = {m.name for m in rares}
assert names == {"the Gilded Stag", "the Hollow Knight"}
assert all(m.weight == 1 for m in rares) # rares surface seldom
# The rare_drop_item resolves to a consumable.
drop = world.item_by_id(world.settings.rare_drop_item)
assert drop is not None and drop.slot.value == "consumable"
# The new economy settings land on their shipped values.
assert world.settings.satchel_max == 3
assert world.settings.forge_base_cost == 60
assert world.settings.forge_max_plus == 3
assert world.settings.dungeon_tiers == (3, 4, 5)
# v0.10 ore-forge settings resolve, and the forge ore is a material item.
assert world.settings.forge_ore_item == "iron_ore"
ore = world.item_by_id(world.settings.forge_ore_item)
assert ore is not None and ore.slot.value == "material"
assert world.settings.forge_ore_per_plus == 1
assert world.settings.ore_dungeon_drop == 2
assert world.settings.ore_forest_chance == 0.2
def test_monster_weight_and_rare_default_when_omitted(tmp_path: Path) -> None:
"""A monster spec without weight/rare loads as weight 10, rare False.
Both fields are optional with defaults, so an unannotated common monster
(the shipped Field Rat) parses to the default weight and the non-rare flag.
"""
world = load_world(SHIPPED)
rat = next(m for m in world.monsters if m.name == "Field Rat")
assert rat.weight == 10 # the default biasing weight
assert rat.rare is False
# ---------------------------------------------------------------------------
# v0.8 loader hardening: rare-as-rung-guardian and the single-boss invariant
#
# AUTHORING states both as rules; v0.8 makes them machine-checked. A rare in
# the lead slot of a dungeon tier would be silently promoted to a fixed rung
# guardian (and pulled from the rare pool); a stray second boss would validate
# clean yet make "the one endgame foe" a lie.
# ---------------------------------------------------------------------------
def test_rare_as_first_dungeon_tier_monster_rejected(tmp_path: Path) -> None:
"""A rare in the FIRST slot of a dungeon tier becomes a fixed guardian — rejected.
Tier 3 backs a ``dungeon_tiers`` rung and its first monster (the Forest
Wolf) is the rung guardian (``band[0]``). Flagging that lead monster rare
would quietly turn the rare into the fixed, repeatable guardian and remove
it from the weighted rare roll, so the loader rejects it by name.
"""
pack = _clone_pack(tmp_path)
def mutate(data: list[dict[str, Any]]) -> None:
wolf = next(m for m in data if m["name"] == "Forest Wolf") # first tier-3
wolf["rare"] = True
_rewrite(pack / "monsters.json", mutate)
with pytest.raises(
WorldLoadError,
match=r"'Forest Wolf' is rare but is the first tier-3 monster.*fixed guardian",
):
load_world(pack)
def test_rare_after_guardian_in_dungeon_tier_accepted(tmp_path: Path) -> None:
"""A rare placed AFTER the guardian in the same dungeon tier loads cleanly.
The shipped pack already does exactly this (the Hollow Knight is the third
tier-3 entry, behind the Forest Wolf guardian). Inserting another rare also
after the guardian must not trip the new check only the LEAD slot of a
dungeon tier is constrained.
"""
pack = _clone_pack(tmp_path)
def mutate(data: list[dict[str, Any]]) -> None:
# Splice a second tier-3 rare in just before the boss (well after the
# tier-3 guardian), so the tier's first non-boss monster is unchanged.
extra = {
"tier": 3,
"name": "the Ashen Stalker",
"hp": 26,
"atk": 10,
"def": 3,
"xp": 55,
"gold": 75,
"weight": 1,
"rare": True,
}
data.insert(len(data) - 1, extra)
_rewrite(pack / "monsters.json", mutate)
world = load_world(pack) # no WorldLoadError: the rare is not the lead foe
tier3 = world.monsters_for_tier_band(3, 3)
assert tier3[0].name == "Forest Wolf" # the guardian is still the non-rare lead
assert any(m.name == "the Ashen Stalker" and m.rare for m in tier3)
def test_two_bosses_rejected(tmp_path: Path) -> None:
"""Two ``boss``-flagged monsters are rejected: a world has exactly one boss."""
pack = _clone_pack(tmp_path)
def mutate(data: list[dict[str, Any]]) -> None:
# Give the Field Rat the boss flag too; now two monsters claim the role.
rat = next(m for m in data if m["name"] == "Field Rat")
rat["boss"] = True
rat["id"] = "field_rat"
_rewrite(pack / "monsters.json", mutate)
with pytest.raises(WorldLoadError, match=r"flags 2 monsters as .boss.* true"):
load_world(pack)
def test_single_boss_accepted() -> None:
"""The shipped pack carries exactly one boss and loads — the single-boss path.
The positive half of the invariant: the Wyrm Below is the only boss, so the
load succeeds and the boss count is exactly one.
"""
world = load_world(SHIPPED)
bosses = [m for m in world.monsters if m.boss]
assert len(bosses) == 1
assert bosses[0].name == "the Wyrm Below"
def test_overlapping_zones_rejected(tmp_path: Path) -> None:
"""Overlapping zone rectangles are a load error.
``zone_for`` returns the FIRST matching zone, so two zones sharing any cell
would silently shadow one tier band there exactly the bug a cold-authored
pack shipped (a 1-column caldera-edge strip dropped to the low band). Pull
the deep zone west so its rect overlaps the near zone and confirm the loader
refuses it rather than loading the ambiguity.
"""
pack = _clone_pack(tmp_path)
def mutate(data: dict[str, Any]) -> None:
for zone in data["zones"]:
if zone["key"] == "dungeon_deep":
zone["rect"][0] = 50 # now overlaps forest_near's x30..60 strip
_rewrite(pack / "world.json", mutate)
with pytest.raises(WorldLoadError, match="overlap"):
load_world(pack)
-206
View File
@@ -1,206 +0,0 @@
"""Tests for bundled-world discovery and the ``worlds`` listing.
Covers the discovery helper (the Vale leads, alternate packs follow
alphabetically, non-pack directories are skipped) and the ``cli_worlds``
listing it backs: a sound fixture pack reports "sound", a deliberately-flawed
fixture pack reports "flawed", and the Vale is always listed first. The
``packs/`` directory is monkeypatched to a temp fixture tree so these tests
never depend on the real (separately-authored) second world.
"""
from __future__ import annotations
import json
import shutil
from io import StringIO
from pathlib import Path
from typing import TYPE_CHECKING, Any
from understone import cli
from understone import world as world_pkg
from understone.world import VALE_SLUG, bundled_world_dirs
if TYPE_CHECKING:
import pytest
SHIPPED = Path(__file__).resolve().parents[1] / "understone" / "world" / "data"
def _make_packs(tmp_path: Path, *, sound: list[str], flawed: dict[str, Any]) -> Path:
"""Build a temp ``packs/`` tree: sound slugs plus flawed-world slugs.
Each sound slug is a verbatim copy of the shipped Vale; each flawed slug is
a copy whose ``world.json`` is patched with the given settings overrides so
it fails to load. Returns the packs root to monkeypatch ``PACKS_DIR`` onto.
"""
packs = tmp_path / "packs"
packs.mkdir()
for slug in sound:
shutil.copytree(SHIPPED, packs / slug)
for slug, overrides in flawed.items():
dest = packs / slug
shutil.copytree(SHIPPED, dest)
world_json = dest / "world.json"
data = json.loads(world_json.read_text(encoding="utf-8"))
data["settings"].update(overrides)
world_json.write_text(json.dumps(data), encoding="utf-8")
return packs
# ---------------------------------------------------------------------------
# bundled_world_dirs discovery
# ---------------------------------------------------------------------------
def test_bundled_world_dirs_vale_leads_then_alpha(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
packs = _make_packs(tmp_path, sound=["zephyr", "ashfall"], flawed={})
monkeypatch.setattr(world_pkg, "PACKS_DIR", packs)
found = bundled_world_dirs()
slugs = [slug for slug, _ in found]
# The Vale is always first; alternates follow alphabetically.
assert slugs == [VALE_SLUG, "ashfall", "zephyr"]
# The Vale entry points at the packaged data dir, not a packs subdir.
assert found[0][1] == world_pkg.PACKAGED_WORLD_DIR
def test_bundled_world_dirs_skips_non_pack_entries(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
packs = _make_packs(tmp_path, sound=["real"], flawed={})
# A README placeholder and a directory with no world.json are NOT worlds.
(packs / "README.md").write_text("placeholder", encoding="utf-8")
(packs / "empty_dir").mkdir()
monkeypatch.setattr(world_pkg, "PACKS_DIR", packs)
slugs = [slug for slug, _ in bundled_world_dirs()]
assert slugs == [VALE_SLUG, "real"]
def test_bundled_world_dirs_handles_absent_packs_dir(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
"""A missing packs/ directory yields just the Vale (never raises)."""
monkeypatch.setattr(world_pkg, "PACKS_DIR", tmp_path / "does_not_exist")
found = bundled_world_dirs()
assert [slug for slug, _ in found] == [VALE_SLUG]
# ---------------------------------------------------------------------------
# cli_worlds listing
# ---------------------------------------------------------------------------
def test_cli_worlds_lists_vale_sound_first(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.setattr(world_pkg, "PACKS_DIR", tmp_path / "empty")
out, err = StringIO(), StringIO()
rc = cli.cli_worlds(out=out, err=err)
assert rc == 0
text = out.getvalue()
lines = [ln for ln in text.splitlines() if ln.strip()]
# The very first listing line is the Vale, reported sound, with its size.
assert lines[0].split()[0] == VALE_SLUG
assert "The Vale of Understone" in lines[0]
assert "96x48" in lines[0]
assert "sound" in lines[0]
# The serve hint closes the listing.
assert "UNDERSTONE_WORLD=" in text
assert "the default Vale needs no setting" in text
def test_cli_worlds_reports_sound_alternate(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
packs = _make_packs(tmp_path, sound=["mirefen"], flawed={})
monkeypatch.setattr(world_pkg, "PACKS_DIR", packs)
out = StringIO()
cli.cli_worlds(out=out)
text = out.getvalue()
line = next(ln for ln in text.splitlines() if ln.strip().startswith("mirefen"))
assert "sound" in line
assert "flawed" not in line
def test_cli_worlds_flags_flawed_alternate(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
# daily_turns 0 is out of its 1..100 band: the pack fails to load.
packs = _make_packs(tmp_path, sound=["sound_one"], flawed={"broken": {"daily_turns": 0}})
monkeypatch.setattr(world_pkg, "PACKS_DIR", packs)
out = StringIO()
rc = cli.cli_worlds(out=out)
assert rc == 0 # a flawed pack is reported, never fatal
text = out.getvalue()
broken_line = next(ln for ln in text.splitlines() if ln.strip().startswith("broken"))
assert "flawed:" in broken_line
assert "daily_turns" in broken_line # the offending field surfaces
# The sound pack alongside it still reports sound — one bad pack doesn't
# poison the survey.
sound_line = next(ln for ln in text.splitlines() if ln.strip().startswith("sound_one"))
assert "sound" in sound_line
def test_cli_worlds_vale_sorts_before_flawed_alternate(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
"""Even with an alphabetically-earlier flawed pack, the Vale leads."""
packs = _make_packs(tmp_path, sound=[], flawed={"aaa_broken": {"start_hp": 0}})
monkeypatch.setattr(world_pkg, "PACKS_DIR", packs)
out = StringIO()
cli.cli_worlds(out=out)
lines = [ln for ln in out.getvalue().splitlines() if ln.strip()]
assert lines[0].split()[0] == VALE_SLUG
assert lines[1].strip().startswith("aaa_broken")
assert "flawed:" in lines[1]
# ---------------------------------------------------------------------------
# the REAL bundled alternate world (no monkeypatch): The Cinder Wastes
#
# The tests above stub PACKS_DIR to a fixture tree so they never depend on the
# separately-authored pack. These two exercise the actual shipped packs/ — the
# bundled Cinder Wastes must discover, load, validate, and appear in the listing
# as sound, so a broken or unbundled alternate trips here.
# ---------------------------------------------------------------------------
CINDER = Path(__file__).resolve().parents[1] / "understone" / "world" / "packs" / "cinder-wastes"
def test_bundled_cinder_wastes_loads_and_validates() -> None:
"""The bundled Cinder Wastes loads through the (strict v0.8) loader cleanly.
It is LLM-authored from AUTHORING.md alone, so this is the dogfood proof
that the manual + validator produce a pack the real loader accepts and,
after v0.8, one that passes the stricter rare-as-guardian and single-boss
checks (its rares sit after their guardians; it has exactly one boss).
"""
from understone.world.loader import load_world
world = load_world(CINDER)
assert world.name == "The Cinder Wastes"
assert world.settings.watch_theme == "ember" # the thematic ember CRT palette
bosses = [m for m in world.monsters if m.boss]
assert len(bosses) == 1 and bosses[0].name == "the Magma Wyrm"
# The boss id resolves and is the declared endgame foe.
assert world.settings.boss_monster == "magma_wyrm"
def test_cli_worlds_lists_bundled_cinder_wastes_sound() -> None:
"""`understone worlds` discovers the real bundled Cinder Wastes as sound.
No monkeypatch: this runs against the actual packs/ directory, so it asserts
the genuinely-shipped second world appears in the listing (alongside the
fixture-based listing tests above, which stay).
"""
out = StringIO()
rc = cli.cli_worlds(out=out)
assert rc == 0
line = next(ln for ln in out.getvalue().splitlines() if ln.strip().startswith("cinder-wastes"))
assert "The Cinder Wastes" in line
assert "sound" in line
assert "flawed" not in line
-600
View File
@@ -1,600 +0,0 @@
"""The Wyrm Below — the v0.2 endgame, legacy reset, and the Herald feed.
Drives the challenge verb against the shipped pack: the level gate, the win
path (Hall of Legends + reincarnation), defeat, and the stalemate flight, plus
the run-days bookkeeping. Also pins the boss exclusion from random selection
and proves the new level_up / defeat beats reach OTHER players' Herald.
Negative-test discipline (the level gate):
The challenge gate is pinned by ``test_challenge_under_level_refused``. To
confirm the assertion has teeth, the implementer temporarily removed the
``if player.level < min_level`` refusal in Game._challenge (letting an
under-level hero spend a turn and fight the Wyrm); the test then FAILED on
the unchanged-turns assertion (a turn was consumed and the refusal line was
absent). The guard was restored. This test is the standing regression.
"""
from __future__ import annotations
from pathlib import Path
import pytest
from tests.conftest import (
fixed_clock,
satchel_ids,
set_satchel,
utc,
)
from understone.engine.models import Mode
from understone.engine.rng import GameRNG
from understone.game import Game
from understone.persistence import Store
from understone.world.loader import load_world
PACK = Path(__file__).resolve().parents[1] / "understone" / "world" / "data"
# Module-local aliases for the shared satchel helpers, keeping the existing
# call sites (_set_satchel / _satchel_ids) unchanged.
_set_satchel = set_satchel
_satchel_ids = satchel_ids
@pytest.fixture
def clock() -> object:
return fixed_clock(utc(2026, 6, 12, 10, 0))
def _game(tmp_path: Path, clock: object, seed: int = 7) -> Game:
world = load_world(PACK)
store = Store(tmp_path / "game.db")
return Game(world, store, clock=clock, rng=GameRNG(seed=seed)) # type: ignore[arg-type]
# The flat-id-list satchel helpers (_set_satchel / _satchel_ids) live in
# tests/conftest.py now, shared with the descend suite; they are imported above.
def _at_dungeon(game: Game, name: str) -> object:
"""Place an already-joined player inside the dungeon menu, at the deep floor.
The challenge verb now gates on depth as well as level: the Wyrm will not
stir until the hero has plumbed the deep to its floor. These challenge
tests exercise the win/lose/flee paths, not the gate, so the helper puts
the hero at the bottom (deepest_rung == the rung count). The depth gate
itself is exercised by the dedicated tests in test_descend.py.
"""
player = game.players[name]
player.mode = Mode.MENU
player.at_location = "dungeon"
player.deepest_rung = len(game.world.settings.dungeon_tiers)
return player
# ---------------------------------------------------------------------------
# Boss exclusion from random selection
# ---------------------------------------------------------------------------
def test_boss_never_in_any_tier_band(tmp_path: Path, clock: object) -> None:
"""The Wyrm Below is never returned by monsters_for_tier_band, any band."""
game = _game(tmp_path, clock)
world = game.world
tiers = [m.tier for m in world.monsters]
lo, hi = min(tiers), max(tiers)
for band_lo in range(lo, hi + 2):
for band_hi in range(band_lo, hi + 2):
band = world.monsters_for_tier_band(band_lo, band_hi)
assert all(not m.boss for m in band)
assert all(m.monster_id != "wyrm_below" for m in band)
# ---------------------------------------------------------------------------
# The level gate (negative-tested; see module docstring)
# ---------------------------------------------------------------------------
def test_challenge_under_level_refused(tmp_path: Path, clock: object) -> None:
"""An under-level hero is turned away in-fiction, spending no turn.
See the module docstring for the revert-and-observe-failure check proving
the gate has teeth.
"""
game = _game(tmp_path, clock)
game.join("Brak")
player = _at_dungeon(game, "Brak")
assert player.level < game.world.settings.wyrm_min_level
before_turns = player.turns_left
before_events = len(game.events)
out = game.action("Brak", "challenge", "", "")
assert "sixth circle" in out.lower() # names the threshold in-fiction
assert player.turns_left == before_turns # no turn spent
assert player.level == 1 # nothing reset
assert len(game.events) == before_events # no public news
assert player.mode is Mode.MENU # still standing at the dungeon
def test_challenge_at_level_threshold_is_allowed(tmp_path: Path, clock: object) -> None:
"""Exactly at the threshold the challenge proceeds (spends a turn)."""
game = _game(tmp_path, clock)
game.join("Brak")
player = _at_dungeon(game, "Brak")
player.level = game.world.settings.wyrm_min_level
player.atk, player.def_, player.hp, player.max_hp = 500, 100, 500, 500
before_turns = player.turns_left
out = game.action("Brak", "challenge", "", "")
assert "sixth circle" not in out.lower() # not refused
assert player.turns_left == before_turns - 1 # a turn was spent
def test_challenge_at_zero_turns_refused_clean(tmp_path: Path) -> None:
"""At the level gate but out of turns, the challenge is refused with no effect.
A wyrm-eligible hero with an empty daily budget (and no day-roll to refill
it) is turned away in-fiction: no turn drops below zero, no Hall row is
cut, no public beat is written, wins are untouched and the no-op player
row is still committed (the refusal branch upserts + commits), so a store
reopen sees the unchanged hero.
"""
clk = _MutableClock(utc(2026, 6, 12, 10, 0))
world = load_world(PACK)
store = Store(tmp_path / "game.db")
game = Game(world, store, clock=clk, rng=GameRNG(seed=7)) # type: ignore[arg-type]
game.join("Brak")
player = _at_dungeon(game, "Brak")
player.level = game.world.settings.wyrm_min_level # eligible
player.turns_left = 0 # but spent for the day (same day: no refill)
events_before = len(game.events)
hall_before = len(game.store.top_hall(50))
out = game.action("Brak", "challenge", "", "")
assert "tomorrow" in out.lower() # the "too spent ... today" refusal
assert "sixth circle" not in out.lower() # not the level gate
assert player.turns_left == 0 # never spent below zero
assert player.wins == 0 # no win recorded
assert len(game.events) == events_before # no public feed beat
assert len(game.store.top_hall(50)) == hall_before # no Hall row
assert player.mode is Mode.MENU # still standing at the dungeon
# The refusal branch commits the (unchanged) row: a reopen sees the hero.
game.store.close()
reopened = Game(world, Store(tmp_path / "game.db"), clock=clk) # type: ignore[arg-type]
assert reopened.players["Brak"].turns_left == 0
assert reopened.players["Brak"].wins == 0
# ---------------------------------------------------------------------------
# Win path: Hall of Legends + legacy reset
# ---------------------------------------------------------------------------
def test_challenge_win_resets_with_legacy(tmp_path: Path, clock: object) -> None:
"""A win records the run, heralds it, and reincarnates the hero with a ★."""
game = _game(tmp_path, clock)
game.join("Brak")
player = _at_dungeon(game, "Brak")
# Mid-run state that must be wiped by the reset.
player.level, player.xp = 12, 5000
player.atk, player.def_, player.hp, player.max_hp = 500, 100, 500, 500
player.gold = 999
player.weapon_id, player.armor_id = "war_axe", "chainmail"
# State that must SURVIVE the reset.
player.turns_left = 4
player.log_cursor = 1
player.bestow_spent = 7
events_before = len(game.events)
settings = game.world.settings
out = game.action("Brak", "challenge", "", "")
# Win narration and the immortalised run.
assert "freed the vale" in out.lower()
assert "hall of legends" in out.lower()
# The legacy reset wipes xp/gold, so the Wyrm win must NOT narrate a reward
# the hero never keeps (the old engine appended "+400 XP, +250 gold." to the
# kill line, which _wyrm_won echoed verbatim). The boss's reward never lands.
boss = game.world.monster_by_id(game.world.settings.boss_monster)
assert boss is not None
assert f"+{boss.xp} XP" not in out # i.e. "+400 XP"
assert f"+{boss.gold} gold" not in out # i.e. "+250 gold"
assert "+400 XP" not in out and "+250 gold" not in out
hall = game.store.top_hall(5)
assert len(hall) == 1
assert hall[0].name == "Brak"
assert hall[0].level_at_win == 12 # the level at the moment of the kill
assert hall[0].run_days == 0 # same UTC day as the join under the frozen clock
# A public news beat was written (all-caps herald moment).
assert len(game.events) == events_before + 1
assert game.events[-1].kind == "wyrm_win"
assert "WYRM" in game.events[-1].text
# Reincarnation: stats/gold/gear/position back to first-day values.
assert player.wins == 1
assert player.level == 1
assert player.xp == 0
assert player.gold == settings.starting_gold
assert player.weapon_id == settings.starting_weapon
assert player.armor_id == settings.starting_armor
assert player.hp == player.max_hp
assert (player.x, player.y) == game.world.spawn
assert player.mode is Mode.TILE
assert player.at_location == ""
# The daily clock and the log cursor were deliberately left alone.
assert player.turns_left == 4 - 1 # only the one challenge turn was spent
assert player.log_cursor == 1
assert player.bestow_spent == 7
def test_challenge_win_legacy_reset_spares_the_vault(tmp_path: Path, clock: object) -> None:
"""The vault SURVIVES a Wyrm-win rebirth; carried gold resets to starting.
Banked gold is the one wealth (besides the ) a legacy reset does not clear:
the strongbox is the inn's, not the reborn hero's. This deposits gold into
the vault through the inn, drives a Wyrm WIN, and asserts ``banked`` is
UNCHANGED while ``gold`` drops back to ``starting_gold``.
Negative-check (the revert-and-observe-failure discipline of this module):
the implementer temporarily added ``player.banked = 0`` to
Game._reset_with_legacy; this test then FAILED on the unchanged-``banked``
assertion (the vault was wiped by the rebirth). The line was restored, so
this test is the standing regression that the vault outlives the reset.
"""
game = _game(tmp_path, clock)
game.join("Brak")
player = game.players["Brak"]
# Bank some gold through the real inn path, then stand at the dungeon floor.
player.gold = 200
player.mode = Mode.MENU
player.at_location = "inn"
game.action("Brak", "deposit", "", "", amount=120)
assert player.banked == 120 and player.gold == 80 # vault holds; hand drained
player = _at_dungeon(game, "Brak")
player.level = game.world.settings.wyrm_min_level
player.atk, player.def_, player.hp, player.max_hp = 500, 100, 500, 500
out = game.action("Brak", "challenge", "", "")
assert "freed the vale" in out.lower() # a genuine win drove the reset
assert player.wins == 1
assert player.banked == 120 # the vault is untouched by the rebirth
assert player.gold == game.world.settings.starting_gold # carried wealth resets
def test_challenge_win_star_in_rank_and_hall(tmp_path: Path, clock: object) -> None:
"""After a win, door_rank shows the ★ and renders the Hall of Legends."""
game = _game(tmp_path, clock)
game.join("Brak")
player = _at_dungeon(game, "Brak")
player.level = game.world.settings.wyrm_min_level
player.atk, player.def_, player.hp, player.max_hp = 500, 100, 500, 500
game.action("Brak", "challenge", "", "")
out = game.rank("Brak")
assert "" in out
assert "Hall of Legends" in out
assert "Brak" in out
def test_two_wins_render_two_stars(tmp_path: Path, clock: object) -> None:
"""A second Wyrm kill stacks a second ★ on the leaderboard name."""
game = _game(tmp_path, clock)
game.join("Brak")
for _ in range(2):
player = _at_dungeon(game, "Brak")
player.level = game.world.settings.wyrm_min_level
player.atk, player.def_, player.hp, player.max_hp = 500, 100, 500, 500
game.action("Brak", "challenge", "", "")
assert game.players["Brak"].wins == 2
assert "★★" in game.rank("Brak")
# ---------------------------------------------------------------------------
# Lose path and flight
# ---------------------------------------------------------------------------
def test_challenge_loss_bounces_and_heralds(tmp_path: Path, clock: object) -> None:
"""A defeat drops the hero to 1 HP at the spawn and heralds the devouring."""
game = _game(tmp_path, clock)
game.join("Brak")
player = _at_dungeon(game, "Brak")
player.level = game.world.settings.wyrm_min_level
player.atk, player.def_, player.hp, player.max_hp = 5, 1, 20, 20 # outmatched
events_before = len(game.events)
out = game.action("Brak", "challenge", "", "")
assert player.hp == 1
assert (player.x, player.y) == game.world.spawn
assert player.mode is Mode.TILE
assert player.at_location == ""
assert player.wins == 0 # a loss is not a win
assert len(game.events) == events_before + 1
devoured = game.events[-1]
assert devoured.kind == "wyrm_lose"
# Either phrasing of the devouring names the hero and the Wyrm.
assert "Brak" in devoured.text and "Wyrm" in devoured.text
assert "lays you low" in out.lower() or "wyrm" in out.lower()
def _doomed_wyrm_challenger(game: Game, name: str) -> object:
"""Stand *name* at the floor, wyrm-eligible, and doomed to a GRINDING loss.
The stats modest atk and def, hp 50 below max_hp 80, well off the spawn
make the Wyrm bout a genuine multi-round lethal loss (not a one-shot where
no blow lands before the save). hp 50 is none of the potion heal values
(15/40/70), so a death-save that sets hp to the potion's heal is unmistakable.
"""
player = _at_dungeon(game, name)
player.level = game.world.settings.wyrm_min_level
player.x, player.y = 35, 25 # away from the spawn (a save never moves them)
player.atk, player.def_, player.hp, player.max_hp = 6, 12, 50, 80
return player
def test_challenge_loss_with_potion_survives_no_legacy_reset(tmp_path: Path, clock: object) -> None:
"""A lethal Wyrm bout with a potion is SURVIVED — no bounce, no legacy reset.
The universal death-save reaches the Wyrm: a carried draught is drunk instead
of the devouring. A save is NOT a win, so NOTHING resets level, gold, and
``deepest_rung`` all stand and it is NOT the devouring either, so the hero
keeps their place at the dungeon. The PUBLIC beat is the survival one
(``wyrm_flee``, "driven back, alive but unproven"), NEVER "devoured". The
turn is still spent and the draught is consumed.
"""
game = _game(tmp_path, clock)
game.join("Brak")
player = _doomed_wyrm_challenger(game, "Brak")
potion = game.world.item_by_id("greater_potion")
assert potion is not None
_set_satchel(game, player, ["greater_potion"])
floor = len(game.world.settings.dungeon_tiers)
spawn = game.world.spawn
before_turns = player.turns_left
before_level, before_gold = player.level, player.gold
events_before = len(game.events)
out = game.action("Brak", "challenge", "", "")
# Survived standing: hp at the potion's value, no bounce, draught spent.
assert player.hp == min(player.max_hp, potion.heal)
assert (player.x, player.y) != spawn # NOT bounced to the spawn
assert player.mode is Mode.MENU # still standing at the dungeon
assert _satchel_ids(game, player) == [] # the draught was spent
assert "death's edge" in out.lower() # the spliced survival line
assert player.turns_left == before_turns - 1 # the challenge still cost a turn
# No win, so NO legacy reset: level, gold, and depth all stand.
assert player.wins == 0
assert player.level == before_level
assert player.gold == before_gold
assert player.deepest_rung == floor # depth untouched (no reset to 0)
# The PUBLIC beat is the survival one, NOT the devouring.
assert len(game.events) == events_before + 1
beat = game.events[-1]
assert beat.kind == "wyrm_flee"
assert beat.kind != "wyrm_lose"
assert "fled" in beat.text.lower() or "ran" in beat.text.lower()
def test_challenge_loss_potion_negative_without_save_devours(
tmp_path: Path, clock: object, monkeypatch: pytest.MonkeyPatch
) -> None:
"""NEGATIVE TEST: with the death-save disabled, the same potion-carrier is devoured.
The mechanical equivalent of reverting the added ``_death_save`` call in
``_wyrm_lost``: we stub ``_death_save`` to always decline, then run the exact
scenario of the survival test. The potion-carrier must now bounce to the
spawn at 1 HP with the draught UNSPENT and the PUBLIC beat back to
``wyrm_lose`` (devoured) proving the death-save (not some other path) is
what saves them at the Wyrm. Restoring the real method (automatic when the
patch lifts) restores the survival behaviour.
"""
game = _game(tmp_path, clock)
game.join("Brak")
player = _doomed_wyrm_challenger(game, "Brak")
_set_satchel(game, player, ["greater_potion"])
floor = len(game.world.settings.dungeon_tiers)
spawn = game.world.spawn
monkeypatch.setattr(Game, "_death_save", lambda self, pl, lines: False)
out = game.action("Brak", "challenge", "", "")
assert player.hp == 1 # devoured, not saved
assert (player.x, player.y) == spawn
assert player.mode is Mode.TILE
assert player.deepest_rung == floor # a defeat keeps depth (no reset, no advance)
assert _satchel_ids(game, player) == ["greater_potion"] # the draught is UNSPENT
assert "death's edge" not in out.lower() # no save, no dramatic line
assert game.events[-1].kind == "wyrm_lose" # the devouring beat, not the survival one
def test_challenge_stalemate_counts_as_flight(tmp_path: Path, clock: object) -> None:
"""A 50-round stalemate resolves as a flight: a wyrm_flee news beat.
With atk == boss def (no kill possible in the round cap) and enough HP to
outlast the boss's chip damage, resolve_fight returns FLED deterministically.
"""
game = _game(tmp_path, clock)
game.join("Brak")
player = _at_dungeon(game, "Brak")
player.level = game.world.settings.wyrm_min_level
player.atk, player.def_, player.hp, player.max_hp = 8, 24, 200, 200
events_before = len(game.events)
game.action("Brak", "challenge", "", "")
assert player.wins == 0
assert player.hp >= 1 # never killed by a flight
assert len(game.events) == events_before + 1
assert game.events[-1].kind == "wyrm_flee"
assert "fled" in game.events[-1].text.lower() or "ran" in game.events[-1].text.lower()
# ---------------------------------------------------------------------------
# run_days from a frozen, advanced clock
# ---------------------------------------------------------------------------
class _MutableClock:
"""A clock whose reported moment can be advanced between calls."""
def __init__(self, moment: object) -> None:
self.moment = moment
def __call__(self) -> object:
return self.moment
def test_run_days_counts_whole_days(tmp_path: Path) -> None:
"""Joining, advancing the clock three days, then winning records run_days==3."""
clk = _MutableClock(utc(2026, 6, 12, 10, 0))
world = load_world(PACK)
store = Store(tmp_path / "game.db")
game = Game(world, store, clock=clk, rng=GameRNG(seed=7)) # type: ignore[arg-type]
game.join("Brak")
player = _at_dungeon(game, "Brak")
player.level = game.world.settings.wyrm_min_level
player.atk, player.def_, player.hp, player.max_hp = 500, 100, 500, 500
clk.moment = utc(2026, 6, 15, 12, 0) # three days (and a couple hours) later
game.action("Brak", "challenge", "", "")
hall = game.store.top_hall(1)
assert hall[0].run_days == 3
def test_top_hall_orders_most_recent_first(tmp_path: Path) -> None:
"""Two heroes slay the Wyrm at advancing times; the latest tops the Hall.
Pins ``ORDER BY id DESC`` in ``Store.top_hall`` the most recently cut
run is at index 0, regardless of name or level-at-win order.
"""
clk = _MutableClock(utc(2026, 6, 12, 10, 0))
world = load_world(PACK)
store = Store(tmp_path / "game.db")
game = Game(world, store, clock=clk, rng=GameRNG(seed=7)) # type: ignore[arg-type]
def _win(name: str) -> None:
game.join(name)
hero = _at_dungeon(game, name)
hero.level = game.world.settings.wyrm_min_level
hero.atk, hero.def_, hero.hp, hero.max_hp = 500, 100, 500, 500
game.action(name, "challenge", "", "")
_win("Early")
clk.moment = utc(2026, 6, 13, 10, 0) # a day later
_win("Later")
hall = game.store.top_hall(5)
assert len(hall) == 2
assert hall[0].name == "Later" # most recent run is first
assert hall[1].name == "Early"
# ---------------------------------------------------------------------------
# Shared-feed proof: level_up and defeat reach ANOTHER player's Herald
# ---------------------------------------------------------------------------
def test_multi_level_jump_is_one_feed_beat_naming_final_level(
tmp_path: Path, clock: object
) -> None:
"""A single award crossing two thresholds posts ONE level_up beat, at the top.
With xp parked just under the level-3 line while still level 1, one forest
kill vaults the hero past both the level-2 and level-3 thresholds. The
public feed must carry exactly one level_up beat a multi-level jump is one
notable moment, not a flood and that beat must name the FINAL level (3),
not the intermediate one.
"""
game = _game(tmp_path, clock)
game.join("Climber")
climber = game.players["Climber"]
climber.x, climber.y = 35, 25 # forest_near zone
climber.atk, climber.def_, climber.hp, climber.max_hp = 100, 50, 100, 100
# Level 1 but xp just under L3 (300): the smallest forest reward (8) crosses
# both L2 (100) and L3 (300) in this one award.
climber.level, climber.xp = 1, 295
events_before = len(game.events)
game.action("Climber", "fight", "", "")
assert climber.level == 3 # vaulted two levels on the single kill
new_events = game.events[events_before:]
level_ups = [e for e in new_events if e.kind == "level_up"]
assert len(level_ups) == 1 # one beat, not one per level crossed
assert "level 3" in level_ups[0].text.lower() # names the final level
assert "level 2" not in level_ups[0].text.lower() # not the intermediate
def test_level_up_appears_in_other_players_herald(tmp_path: Path, clock: object) -> None:
"""A level-up by one hero is news in another hero's Herald."""
game = _game(tmp_path, clock)
game.join("Riser")
game.join("Watcher")
watcher = game.players["Watcher"]
watcher.log_cursor = game._latest_event_id() # start Watcher caught up
riser = game.players["Riser"]
riser.x, riser.y = 35, 25 # forest_near zone
riser.atk, riser.def_, riser.hp, riser.max_hp = 100, 50, 100, 100
riser.xp = 95 # one win (>= 8 xp) crosses the level-2 threshold of 100
game.action("Riser", "fight", "", "")
assert riser.level >= 2 # the fight pushed Riser over the line
out = game.log("Watcher")
assert "Riser" in out
assert "level 2" in out.lower()
def test_defeat_appears_in_other_players_herald(tmp_path: Path, clock: object) -> None:
"""A defeat by a regular monster is news in another hero's Herald."""
game = _game(tmp_path, clock)
game.join("Faller")
game.join("Watcher")
watcher = game.players["Watcher"]
watcher.log_cursor = game._latest_event_id()
faller = game.players["Faller"]
faller.x, faller.y = 35, 25 # forest_near zone
faller.atk, faller.def_, faller.hp, faller.max_hp = 1, 0, 2, 20 # certain to fall
game.action("Faller", "fight", "", "")
assert faller.hp == 1 # bounced
out = game.log("Watcher")
assert "Faller" in out
assert "dragged back" in out.lower() or "fell to" in out.lower() or "bested" in out.lower()
# ---------------------------------------------------------------------------
# Movement events at the façade: no turn, no public feed
# ---------------------------------------------------------------------------
def test_move_events_cost_no_turn_and_write_no_feed(tmp_path: Path, clock: object) -> None:
"""A walk that fires non-combat events spends no turn and posts no Herald news.
Walks Brak back and forth across the forest_near zone (encounter_rate 0.25)
enough that some non-fight event almost certainly fires; whatever happens,
no turn is consumed and no public event is appended.
"""
game = _game(tmp_path, clock)
game.join("Brak")
player = game.players["Brak"]
player.x, player.y = 35, 25 # inside forest_near
before_turns = player.turns_left
before_events = len(game.events)
for _ in range(12):
game.move("Brak", "", "east", 1)
game.move("Brak", "", "west", 1)
assert player.turns_left == before_turns # movement never costs a turn
assert len(game.events) == before_events # walk texture is private
@@ -1,3 +0,0 @@
"""Understone — a BBS-style ANSI door game served over MCP."""
__version__ = "0.10.0"
@@ -1,4 +0,0 @@
from understone.server import main
if __name__ == "__main__":
main()
-724
View File
@@ -1,724 +0,0 @@
"""The pack-authoring command surface — validate a pack and scaffold a new one.
This module is deliberately pure: it imports only the loader and the standard
library, takes no part in argument parsing (``server.main`` owns the argparse
front end), and writes to the streams it is handed. That keeps the authoring
loop ``newpack`` then ``validate`` testable as plain function calls.
Three entry points back the three verbs:
* :func:`cli_validate` loads a pack and, on success, prints a human-readable
report; on failure it prints the loader's author-facing message and returns
a non-zero code. This is the feedback half of the loop.
* :func:`cli_newpack` scaffolds a new pack: it copies the bundled world as a
starting template and writes an ``AUTHORING.md`` manual whose bands table is
generated from the loader's own band data, so the documented limits can
never drift from the enforced ones.
* :func:`cli_worlds` lists the bundled worlds the default Vale plus every
alternate pack shipped under ``world/packs/`` loading each so it can report
whether it is sound or flawed, the discovery seam for "worlds without authors".
"""
from __future__ import annotations
import shutil
import sys
from typing import TYPE_CHECKING, TextIO
from understone.engine.textwidth import SAFE_PALETTE
from understone.errors import WorldLoadError
from understone.world import PACKAGED_WORLD_DIR, bundled_world_dirs, loader
if TYPE_CHECKING:
from pathlib import Path
from understone.engine.world import World
# The six packaged content files copied verbatim as a new pack's template.
_PACK_FILES = (
"terrain.json",
"monsters.json",
"items.json",
"locations.json",
"events.json",
"world.json",
)
def cli_validate(pack_dir: Path, out: TextIO | None = None, err: TextIO | None = None) -> int:
"""Load *pack_dir* and report; return 0 if sound, 2 if it fails to load.
On success a pack report is written to *out* and the function returns 0.
On any :class:`WorldLoadError` the loader's message — which names the
file, index, and field at fault is written to *err* and the function
returns 2. The author iterates against that message until the pack loads.
*out*/*err* default to the live ``sys.stdout``/``sys.stderr`` resolved at
call time, so a caller (or pytest's capture) may redirect them.
"""
out = out if out is not None else sys.stdout
err = err if err is not None else sys.stderr
try:
world = loader.load_world(pack_dir)
except WorldLoadError as exc:
print(f"The pack is flawed: {exc}", file=err)
return 2
print(_pack_report(world), file=out)
return 0
def cli_newpack(dest: Path, out: TextIO | None = None, err: TextIO | None = None) -> int:
"""Scaffold a new content pack at *dest*; return 0, or 2 if *dest* is taken.
Refuses to write into an existing non-empty directory (so an author never
clobbers work in progress). Otherwise it creates *dest*, copies the six
packaged content files as a starting template, and writes an
``AUTHORING.md`` manual generated from the live loader bands. The author
then edits or regenerates the JSON and runs ``validate``.
*out*/*err* default to the live ``sys.stdout``/``sys.stderr`` resolved at
call time, so a caller (or pytest's capture) may redirect them.
"""
out = out if out is not None else sys.stdout
err = err if err is not None else sys.stderr
if dest.exists() and dest.is_dir() and any(dest.iterdir()):
print(f"refusing to scaffold into non-empty directory: {dest}", file=err)
return 2
if dest.exists() and not dest.is_dir():
print(f"refusing to scaffold over a file: {dest}", file=err)
return 2
dest.mkdir(parents=True, exist_ok=True)
for name in _PACK_FILES:
shutil.copyfile(PACKAGED_WORLD_DIR / name, dest / name)
(dest / "AUTHORING.md").write_text(build_authoring_md(), encoding="utf-8")
print(f"Scaffolded a new pack at {dest}.", file=out)
print("Six content files plus AUTHORING.md are in place; the template is the", file=out)
print("shipped Vale of Understone, ready to edit or regenerate.", file=out)
print(f"Next: edit or regenerate the JSON, then: understone validate {dest}", file=out)
return 0
def cli_worlds(out: TextIO | None = None, err: TextIO | None = None) -> int:
"""List every bundled world, reporting each as sound or flawed; return 0.
Discovers the worlds through :func:`~understone.world.bundled_world_dirs`
(the default Vale first, then the alternate packs alphabetically) and loads
each one. Each world is one line its slug, name, ``WxH``, and either
``sound`` or ``flawed: <short reason>`` so a shipped pack that has gone
out of band is visible at a glance rather than only failing at serve time.
A flawed world is reported, not fatal: the listing always returns 0 and
always ends with the hint for serving an alternate. *err* is accepted for a
uniform signature with the other verbs; the listing writes only to *out*.
"""
out = out if out is not None else sys.stdout
for slug, world_dir in bundled_world_dirs():
print(_world_line(slug, world_dir), file=out)
print("", file=out)
print(
"Serve one with UNDERSTONE_WORLD=<path> (or the default Vale needs no setting).",
file=out,
)
return 0
def _world_line(slug: str, world_dir: Path) -> str:
"""Render one ``worlds`` listing line for the world at *world_dir*.
Loads the world to report its real name, dimensions, and soundness. A pack
that fails to load is summarised as ``flawed: <reason>`` using the loader's
own author-facing message (truncated to keep the listing to one line per
world), never raised the listing surveys every bundled world even when one
is broken.
"""
try:
world = loader.load_world(world_dir)
except WorldLoadError as exc:
return f" {slug:<10} flawed: {_short_reason(str(exc))}"
return f" {slug:<10} {world.name}{world.width}x{world.height} — sound"
# How much of a loader error message the one-line ``worlds`` summary keeps.
_FLAW_REASON_MAX = 70
def _short_reason(message: str) -> str:
"""Trim a loader error to a single readable clause for the worlds listing."""
flattened = " ".join(message.split())
if len(flattened) <= _FLAW_REASON_MAX:
return flattened
return flattened[: _FLAW_REASON_MAX - 1].rstrip() + ""
def _pack_report(world: World) -> str:
"""Render the success report for a loaded *world*.
Counts and shares are computed from the runtime world so the figures match
what the engine will actually run, not what the JSON nominally declares.
"""
settings = world.settings
boss_count = sum(1 for m in world.monsters if m.boss)
fight_share = _fight_share_pct(world)
lines = [
f"{world.name}{world.width}x{world.height}",
f" monsters : {len(world.monsters)} ({boss_count} boss)",
f" items : {len(world.items)}",
f" zones : {len(world.zones)}",
f" events : {len(world.events)} ({fight_share}% fight by weight)",
(
" settings : "
f"{settings.daily_turns} turns/day, "
f"bestow budget {settings.bestow_daily_budget}, "
f"Wyrm gate level {settings.wyrm_min_level}"
),
"",
"This pack is sound. The door stands open.",
]
return "\n".join(lines)
def _fight_share_pct(world: World) -> int:
"""Return the share of overworld encounter weight that is a ``fight``.
Reported by weight, not row count, because weight is the draw probability
the engine actually rolls against it is the number an author tunes to hit
the ~55% fight feel.
"""
total = sum(e.weight for e in world.events)
if total == 0:
return 0
fight = sum(e.weight for e in world.events if e.kind == "fight")
return round(100 * fight / total)
def build_authoring_md() -> str:
"""Build the AUTHORING.md manual, bands table and glyph palette included.
Both the bands section and the safe-glyph palette are generated from live
source the loader's own band tables and ``textwidth.SAFE_PALETTE`` — so
the documented limits and the suggested glyphs are exactly what the loader
enforces and admits, and cannot silently drift from it.
"""
md = _AUTHORING_TEMPLATE.replace("{{BANDS}}", _render_bands())
md = md.replace("{{PALETTE}}", _render_palette())
md = md.replace("{{COLOR_ROLES}}", _render_color_roles())
return md.replace("{{VALIDATE_COVERAGE}}", _render_validate_coverage())
def _render_bands() -> str:
"""Render the bands reference straight from the loader's band data."""
parts: list[str] = []
parts.append("### Map and counts\n")
parts.append(
f"* Map width and height: each `{loader.MAP_DIM_MIN}`..`{loader.MAP_DIM_MAX}` cells."
)
# monsters/items/events are their own files; locations and zones are lists
# inside world.json, so name each cap's real source.
count_source = {
"monsters": "`monsters.json`",
"items": "`items.json`",
"events": "`events.json`",
"locations": "`world.json` → `locations`",
"zones": "`world.json` → `zones`",
}
for name, cap in loader.MAX_COUNTS.items():
parts.append(f"* {count_source[name]}: at most `{cap}` entries.")
parts.append(
f"* Display names (monster, item, location): at most "
f"`{loader.MAX_NAME_LEN}` printable characters."
)
parts.append(
"* Map glyphs (terrain, location, legend keys): exactly one terminal "
"column (one printable code point, no fullwidth runes, no combining "
"marks — see the width rule above), and never one of "
+ ", ".join(f"`{g}`" for g in _reserved_glyph_list())
+ " (the frame box-drawing lines and the `@`/`☻` player markers)."
)
parts.append("")
parts.append("### Economy and progression settings (`world.json` → `settings`)\n")
parts.append("| field | allowed range |")
parts.append("| --- | --- |")
for field_name, (lo, hi) in loader.SETTINGS_BANDS.items():
rng = f"{lo}..{hi}" if hi is not None else f"{lo} or more"
parts.append(f"| `{field_name}` | `{rng}` |")
parts.append("")
parts.append("### Overworld event amounts (`events.json`, per kind)\n")
parts.append("| kind | min..max amount |")
parts.append("| --- | --- |")
for kind, (lo, hi) in loader.EVENT_AMOUNT_BANDS.items():
parts.append(f"| `{kind}` | `{lo}..{hi}` |")
parts.append(
"\n(`fight` and `lore` carry no amount; `fight` draws its foe from the "
"zone tier band, `lore` is pure flavour text.)\n"
)
parts.append("### Watch theme (`world.json` → `settings.watch_theme`)\n")
legal = ", ".join(f"`{name}`" for name in sorted(loader.WATCH_THEMES))
parts.append(
f"OPTIONAL. The CRT palette the live Watch page paints your world in, "
f"one of: {legal}. It defaults to `{loader.DEFAULT_WATCH_THEME}` (the "
f"original green phosphor), so you may leave it out entirely — a pack "
f"that omits it looks exactly as the bundled Vale always has. Set it to "
f"give your world its own colour: `amber` is a warm gold monitor, `ice` "
f"a cold pale blue, `ember` a hot red/orange. An unknown name is a load "
f"error naming the legal set."
)
parts.append("\n### The ore-gated forge (`world.json` → `settings`)\n")
ore_per = loader.SETTINGS_BANDS["forge_ore_per_plus"]
dungeon = loader.SETTINGS_BANDS["ore_dungeon_drop"]
parts.append(
"Forging a +1 edge now costs both GOLD and ORE — a `material` item the "
"hero earns in combat, never buys. Four settings bind it:"
)
parts.append(
"* `forge_ore_item` — REQUIRED. The item id of your world's forge ore; "
"it must name an `items.json` entry whose `slot` is `material` (an "
"unknown id or a non-material slot is a load error). The Vale uses "
"`iron_ore`."
)
parts.append(
f"* `forge_ore_per_plus` — band `{ore_per[0]}..{ore_per[1]}`. Ore per +1 "
f"step: a +N forge costs `(current_plus + 1) * forge_ore_per_plus` ore. "
f"{_forge_ore_worked_example()}"
)
parts.append(
f"* `ore_dungeon_drop` — band `{dungeon[0]}..{dungeon[1]}`. Ore granted "
f"on every WON dungeon rung — the reliable source. The Vale drops 2."
)
parts.append(
"* `ore_forest_chance` — a `0.0`..`1.0` probability (a float, validated "
"outside the integer band table). The chance a WON forest fight yields "
"one ore — the occasional bonus source. The Vale uses `0.2`."
)
parts.append(
"\nOre rides the satchel as a stack, so it shares the `satchel_max` "
"DISTINCT-stack budget with potions (per-stack quantity is unbounded). "
"Tune the two sources so a hero who descends steadily earns enough ore "
"to forge without grinding — the `simulate` bot will tell you if the "
"gate stalls a winnable run."
)
return "\n".join(parts)
def _forge_ore_worked_example() -> str:
"""Render the per-step ore costs from the bundled Vale's live forge settings.
The starter template :func:`cli_newpack` copies IS the bundled Vale, so the
worked figures are computed from its actual ``forge_ore_per_plus`` and
``forge_max_plus`` rather than hardcoded a retune of the template moves
the manual with it. The steps are ``per_plus * (i + 1)`` for each ``i`` in
``range(forge_max_plus)``; the total is what it costs to max one slot.
"""
settings = loader.load_world(PACKAGED_WORLD_DIR).settings
per_plus = settings.forge_ore_per_plus
max_plus = settings.forge_max_plus
steps = [per_plus * (i + 1) for i in range(max_plus)]
if not steps:
return (
f"At the template's value of {per_plus}, slots cannot be forged (`forge_max_plus` 0)."
)
ladder = ", ".join(str(cost) for cost in steps)
total = sum(steps)
return (
f"At the template's value of {per_plus}, the steps cost {ladder} ore "
f"({total} ore to max a slot at `forge_max_plus` {max_plus})."
)
def _reserved_glyph_list() -> list[str]:
"""Return the reserved glyphs in a stable, readable order for the manual."""
box = [g for g in "┌┐└┘─│═" if g in loader.RESERVED_GLYPHS]
actors = [g for g in "@☻" if g in loader.RESERVED_GLYPHS]
return box + actors
def _render_palette() -> str:
"""Render the safe-glyph appendix straight from ``textwidth.SAFE_PALETTE``.
The glyphs are emitted in their declared order, wrapped in backticks so the
monospace renders them as discrete cells. Generated from the live constant,
so the suggested palette is exactly the set the loader's width gate admits.
"""
glyphs = " ".join(f"`{g}`" for g in SAFE_PALETTE)
return (
"Any single-column glyph the loader accepts is fair game, but these "
"carry the period BBS / CP437 flavour and are all guaranteed safe:\n\n"
f"{glyphs}"
)
def _render_color_roles() -> str:
"""Render the author-assignable colour roles, generated from the Color enum.
The Watch knows how to paint exactly the roles in ``screen.palette.Color``;
``Color.assignable()`` is the single source for which of those an author may
put on terrain or a location (the runtime overlay roles an actor/item wears,
and the DEFAULT fallback, are filtered out there). Generated from the enum,
so the documented vocabulary can never drift from what the Watch can
actually colour the same can't-drift discipline as the bands and the
safe-glyph palette. ``color`` itself stays advisory: the loader does not
validate it, so a typo is harmless and an unknown role just paints as the
default; these are simply the roles the Watch recognises.
"""
from understone.screen.palette import Color
return ", ".join(f"`{role.value}`" for role in Color.assignable())
def _render_validate_coverage() -> str:
"""Render the list of rules the loader actually enforces, generated from it.
The figures that can drift (the number of banded settings, the name-length
cap, the reserved glyphs) are read from the live loader so the list cannot
fall out of step with what `validate` does; the prose names each family of
check. This is the machine-enforced half of the honesty split in the manual
the eyeball-only half is hand-written below it, because "is the fiction
any good" is exactly what the loader can never see.
"""
settings_count = len(loader.SETTINGS_BANDS)
reserved = ", ".join(f"`{g}`" for g in _reserved_glyph_list())
bullets = [
f"* **Economy and progression bands** — every one of the {settings_count} `settings` fields must sit in its allowed range (the table above), and `growth` must be present and non-negative.",
f"* **Glyph safety** — every terrain, location, and legend glyph must render exactly one column and must not be a reserved marker ({reserved}).",
"* **Map integrity** — `width`/`height` in band, every `terrain_rows` row exactly `width` long with `height` rows, and every row character in the `legend`.",
"* **Walkability** — `spawn` and every placed location must sit on walkable terrain (and no two locations share a cell).",
f"* **Display-name length** — every monster, item, and location name within `{loader.MAX_NAME_LEN}` printable characters; content lists within their caps.",
"* **The fight row** — `events.json` must hold at least one `fight` entry, with weights `> 0`, `min <= max`, and amounts in their per-kind band.",
'* **Cross-references** — `legend` → terrain key, location placements → `locations.json` keys, `starting_weapon`/`starting_armor` → item ids, `boss_monster` → a monster flagged `"boss": true`, `rare_drop_item` → a consumable item id, and `forge_ore_item` → a `material` item id.',
"* **Zone tiers** — every zone's tier band must overlap at least one monster tier.",
"* **Dungeon ladder** — every `dungeon_tiers` tier must have a non-boss monster, and that tier's FIRST monster (its fixed rung guardian) must not be `rare`.",
'* **Exactly one boss** — at most one monster may carry `"boss": true`.',
]
return "\n".join(bullets)
_AUTHORING_TEMPLATE = """\
# Authoring a world pack for Understone
A *world pack* is a directory of six JSON files that the server loads at start
to become the entire game world its map, its monsters, its economy, its
endgame. There is no code to write: you describe a world as data, the loader
validates it hard, and the server runs it. This file is the manual; you can
follow it cold, by hand or with an LLM.
The loop is short:
1. `understone newpack mypack` scaffold this template (you are reading the
copy it wrote into `mypack/AUTHORING.md`).
2. Edit or regenerate the JSON files to describe your world.
3. `understone validate mypack` the loader checks the pack and either prints
a report ending **"This pack is sound. The door stands open."** or tells you
exactly which file, row, and field is wrong.
4. Repeat step 2 until it is sound, then serve it:
`UNDERSTONE_WORLD=mypack understone`.
The loader's error messages are written FOR you: every failure names the file,
the index, and the field, and says what was expected. Treat them as the
feedback loop iterate until the report says the door stands open.
---
## The six files and how they fit together
| file | shape | holds |
| --- | --- | --- |
| `terrain.json` | object keyed by legend char | terrain kinds: glyph, walkability, encounter rate |
| `monsters.json` | list | monster stat blocks, tiered; one flagged the boss |
| `items.json` | list | weapons, armour, consumables for the shop |
| `locations.json` | object keyed by location key | building kinds: name, glyph, menu actions, flavour |
| `events.json` | object with an `events` list | the weighted overworld encounter table |
| `world.json` | object | the map, placements, zones, and `settings` that bind it all |
The cross-references the loader enforces:
* every character in `world.json` `legend` must name a terrain `key` from
`terrain.json`; every character in `terrain_rows` must be in that legend;
* every placement in `world.json` `locations` must name a key defined in
`locations.json`, and must sit on walkable terrain;
* `settings.starting_weapon` / `starting_armor` must be ids from
`items.json`; `settings.boss_monster` must be an id from `monsters.json`
that is flagged `"boss": true`; `settings.rare_drop_item` must be an id from
`items.json` whose `slot` is `consumable`; `settings.forge_ore_item` must be
an id from `items.json` whose `slot` is `material`;
* every tier in `settings.dungeon_tiers` must be backed by a NON-boss monster;
* every zone's tier band must overlap at least one monster tier.
---
## File-by-file schema
### `terrain.json`
An object whose keys are the single-character legend symbols used in the map.
```json
{
".": {"key": "grass", "glyph": ".", "walkable": true, "encounter_rate": 0.1, "color": "floor"}
}
```
* `key` internal name the map legend resolves to.
* `glyph` the single character drawn on the map (see glyph rules below).
* `walkable` may a player stand here.
* `encounter_rate` `0.0`..`1.0`, the per-step chance a walk rolls the event
table on this terrain.
* `color` a palette role string. It is **advisory and not validated**: the
loader stores it but the text frame draws glyphs only (it is monochrome), so
any string loads and an unrecognised role simply maps to the default at render
time. Where colour DOES show is the live Watch page, which paints each role a
distinct hue. The roles the Watch knows how to paint pick the closest fit
are: {{COLOR_ROLES}}. A typo here is harmless, not a load error; it just
paints as the default. The four runtime overlay colours (the hero, rival
players, monsters, dropped items) are set by the engine, not assignable here.
### `monsters.json`
A list of stat blocks. `tier` groups foes by difficulty; zones and the dungeon
gauntlet draw from tiers. Exactly one monster should be the boss.
```json
{"tier": 2, "name": "Goblin", "hp": 12, "atk": 5, "def": 1, "xp": 18, "gold": 7}
```
The boss adds an `id` and `"boss": true`, and is referenced by
`settings.boss_monster`:
```json
{"tier": 6, "name": "the Wyrm Below", "hp": 120, "atk": 24, "def": 8,
"xp": 400, "gold": 250, "boss": true, "id": "wyrm_below"}
```
Two optional fields tune random forest encounters. `weight` (default `10`,
must be `> 0`) biases the weighted draw within a zone band a low weight
surfaces seldom and `rare` (default `false`) marks a named beast that, on
its kill, fires a public Herald flash and drops the pack's `rare_drop_item`
into the slayer's satchel. Rung guardians ignore both (a rung always takes the
FIRST monster of its tier, never a weighted roll), so a rare should not be the
first entry of a tier that backs a `dungeon_tiers` rung.
```json
{"tier": 2, "name": "the Gilded Stag", "hp": 16, "atk": 6, "def": 2,
"xp": 40, "gold": 60, "weight": 1, "rare": true}
```
### `items.json`
A list of equipment, consumables, and crafting materials. `slot` is `weapon`,
`armor`, `consumable`, or `material`. Weapons add `atk`, armour adds `def`,
consumables `heal`; a `material` carries none of these it is the forge ORE,
carried in the satchel and spent at the forge.
```json
{"id": "short_sword", "name": "Short Sword", "slot": "weapon", "atk": 5, "price": 40}
```
The forge ore is a `material` item the player EARNS in combat (not the shop):
price it `0` ore is never bought or sold and point `settings.forge_ore_item`
at its id. A won dungeon rung always drops `settings.ore_dungeon_drop` of it, and
a won forest fight has a `settings.ore_forest_chance` chance of one.
```json
{"id": "iron_ore", "name": "Iron Ore", "slot": "material", "price": 0}
```
### `locations.json`
An object keyed by location key. Each entry is a building kind with a menu of
`actions` the player may take inside it.
```json
{
"inn": {"kind": "inn", "name": "The Sleeping Drake", "glyph": "I",
"color": "town", "actions": ["rest", "gamble", "leave"],
"flavor": ["Lamplight pools on worn oak tables."]}
}
```
Give each building the menu that matches its role. The four building kinds and
the verbs the engine honours inside each are:
| `kind` | actions the engine understands |
| --- | --- |
| `inn` | `rest`, `deposit`, `withdraw`, `gamble`, `leave` |
| `shop` | `buy`, `sell`, `forge`, `leave` |
| `healer` | `heal`, `leave` |
| `dungeon` | `descend`, `challenge`, `leave` |
The inn's `deposit`/`withdraw` are the VAULT: a player banks gold into the inn
strongbox (`deposit amount=<gold>`) and draws it back (`withdraw amount=<gold>`).
Banked gold is SAFE from ambush a sleeping-robber only ever lifts gold in hand
and it SURVIVES the Wyrm-win legacy reset, so it is the one store of wealth
that carries across runs. Both cost no turn.
`quaff` (drink a satchel tonic) is legal **anywhere** and needs no menu entry.
The `actions` list is advisory it is the menu the narrator offers, NOT a
validated whitelist (see "What `validate` checks" below): a verb the engine does
not back simply confuses the narrator, so give each building only the verbs from
its row above.
### `events.json`
An object with an `events` list the weighted overworld encounter table the
server rolls as a player walks.
```json
{"events": [
{"kind": "fight", "weight": 82, "text": "Something snarls out of the brush."},
{"kind": "gold", "weight": 8, "text": "a rotted coin-purse", "min": 4, "max": 12}
]}
```
* `kind` `fight`, `gold`, `heal`, `trap`, or `lore`.
* `weight` relative draw weight (`> 0`).
* `text` required (non-empty) for every kind except `fight`.
* `min`/`max` required for the value-bearing kinds (`gold`, `heal`, `trap`).
There MUST be at least one `fight` row, or a walk could never find a monster.
### `world.json`
The binding file: `name`, `width`, `height`, `spawn` `[x, y]` (the hero's start
cell, which must be on walkable terrain), a `legend` mapping characters to
terrain keys, `terrain_rows` (one string per row, each exactly `width` long), a
`locations` list of `{"key", "x", "y"}` placements (each also on walkable
terrain), a `zones` list (rectangles that bias monster tiers), and a `settings`
object.
```json
{"key": "forest_near", "rect": [30, 18, 60, 36], "tier_lo": 1, "tier_hi": 2}
```
---
## The bands — the limits the loader enforces
These are generated from the loader's own tables, so they are exactly what
`validate` checks. A value outside its band is a load error.
{{BANDS}}
---
## Glyph width — the one-column rule
Every glyph drawn on the map must occupy **exactly one terminal column**. The
frames are box-drawing rectangles; a glyph that renders two columns (a CJK
ideograph like ``, an emoji like `🌲`, a fullwidth ``) shoves its row right
and tears the border, and a combining mark (a decomposed `é`, a lone accent)
stacks onto its neighbour and breaks the count the other way. The loader
rejects all of these at load.
What is admitted is judged for the **Western monospace** metrics every
Understone surface actually uses (the Watch's pinned font stack, a chat
client's code block): under those metrics the East-Asian "Ambiguous" width
class renders single-column, and that class is the CP437 heartland ``, ``,
``, ``, ``, `` all live there so the rule admits it and bars only the
genuinely double-width Wide and Fullwidth classes.
### Safe glyph palette
{{PALETTE}}
---
## Design guidance
**Turn economy.** `daily_turns` is the whole pacing lever: only fighting,
descending, and challenging the Wyrm spend a turn (moving, resting, shopping
are free). A small budget (the Vale uses 10) makes this a correspondence game
played a little each day. Set `rest_cost`, `heal_cost_per_hp`, and shop prices
so a day's gold roughly covers a day's recovery too cheap and there is no
tension, too dear and a hero stalls.
**Tier curve.** Lay monster tiers as a rising staircase: each tier should be a
real step up in `hp`/`atk` and a real step up in `xp`/`gold`, so the reward of
pushing into a harder zone pays for the risk. Keep two or three foes per tier
for variety. The boss should tower over the top random tier it is the climax.
**Encounter feel.** Aim for roughly 55% of overworld encounter WEIGHT on
`fight` rows; the rest is the texture of travel small gold finds, healing
springs, harmless traps, and lore that hints at the endgame. (The validate
report prints your actual fight share so you can tune it.)
**Glyphs.** Map glyphs must render as exactly one terminal column (see the
one-column rule above) and must never collide with the frame's box-drawing
lines or the `@`/`` player markers. Pick glyphs that read at a glance the
bundled Vale uses `.` open ground, `` water, `` tree, `` inn, `$` shop, ``
healer, `` dungeon and lean on the safe palette for period flavour.
**Boss rules.** Exactly one monster carries `"boss": true` and an `id`, and
`settings.boss_monster` points at it. The boss is the only win condition and is
faced only through the `challenge` verb, gated by `settings.wyrm_min_level`. A
boss tier must NOT appear in `settings.dungeon_tiers`: the gauntlet excludes
boss monsters, so a boss-only rung would be unfillable back every dungeon
tier with at least one ordinary monster.
**The deep, the satchel, and the forge.** `dungeon_tiers` is now a RUNG LADDER
fought one rung per `descend` list the tiers shallow-to-deep, and make it long
enough to feel like a journey (the Vale uses three). The Wyrm gates on reaching
the floor as well as on level. Size the satchel with `satchel_max` it caps the
DISTINCT stacks the bag holds (potions and ore each take a slot; per-stack
quantity is unbounded), and it is the death-save reserve, so keep it small (the
Vale carries 3). The forge is the late-game GOLD-AND-ORE sink: `forge_base_cost`
is the gold price of a +1 edge and scales up each tier (`base * (current_plus +
1)`), capped at `forge_max_plus`, and each step ALSO costs ore (see the ore-gated
forge above). Ore is won in the deep (and seldom in the forest), so the forge is
fed by descending price the gold so a fully-forged piece is a multi-day saving,
and set the ore sources so a steady delver can afford it without a grind.
**Rare beasts.** A rare monster is a small legend: give it a low `weight` so it
surfaces seldom, stats and rewards a clear notch above its tier, and remember it
always drops `rare_drop_item` (a consumable) into the satchel. Keep rares OFF
the first slot of any `dungeon_tiers` tier, or they would become a fixed rung
guardian instead of a rare roll `validate` now ENFORCES this, so a rare in a
dungeon tier's lead slot is a load error, not just bad form. Place the rare
anywhere after that tier's first ordinary monster.
**Location menus.** Give each building only the actions it can honour, drawn
from the per-kind table under `locations.json` above. An inn that offers `buy`
but no shop logic will confuse the narrator. This is the one major thing
`validate` does NOT check (see below): a wrong or invented verb loads fine and
only muddles the narration, so it is on you to match each menu to its building.
---
## The validate loop
Run `understone validate mypack` after every change. On success you get a
report name, size, monster/item/zone/event counts, fight share, and the key
settings ending in **"This pack is sound. The door stands open."** On
failure you get one precise line naming the file, the row, and the field.
The error messages are deliberately instructive: they are the authoring API.
Keep editing and re-validating until the door stands open, then point the
server at your pack with `UNDERSTONE_WORLD=mypack`.
### What `validate` checks, and what it cannot
`validate` runs your pack through the very loader the server uses, so a pack
that validates will load and serve. But the loader checks *structure and
references*, not *meaning* it cannot read your fiction. Keep the split honest:
**`validate` DOES catch (a load error if wrong):**
{{VALIDATE_COVERAGE}}
**`validate` does NOT catch (the eyeball-only short list):**
* **Location menu `actions` contents.** The list is the narrator's menu, not a
validated whitelist: a verb the engine does not back (a typo, or a fictional
`pray`) loads fine and only confuses the narration. Match each building's menu
to the per-kind table under `locations.json`.
* **Flavour and narration quality.** Names, `flavor` lines, event `text`, the
feel of the tier curve and the economy the loader checks they are present
and in band, never whether they are *good*. That judgement is yours; the
`simulate` bot can tell you a world is winnable and sanely paced, but only you
can tell whether it is worth playing.
"""
@@ -1,6 +0,0 @@
"""Game engine — pure stdlib mechanics with injectable clock and RNG.
This package has no knowledge of MCP, persistence, or rendering. Every
function takes its inputs explicitly (world, player, rng, clock) so the
mechanics are deterministic under test.
"""
@@ -1,117 +0,0 @@
"""Combat resolution — pure math over an injected RNG.
A fight runs deterministic rounds: both sides trade blows until one drops.
Damage is ``max(1, attacker_atk - defender_def)`` jittered by a small RNG
swing so identical stats still produce varied logs. The result is a value
object; turn accounting and the spawn-bounce on defeat are applied by the
caller (the game façade), keeping this module side-effect free.
"""
from __future__ import annotations
from dataclasses import dataclass, field
from enum import StrEnum
from typing import TYPE_CHECKING
if TYPE_CHECKING:
from understone.engine.models import Monster, Player
from understone.engine.rng import GameRNG
_MAX_ROUNDS = 50
class Outcome(StrEnum):
"""How a fight ended."""
WIN = "win"
LOSE = "lose"
FLED = "fled"
@dataclass(slots=True)
class FightResult:
"""The full outcome of a combat exchange.
Deltas are signed and meant to be applied to the player by the caller.
``bounce_to_spawn`` signals a defeat: the caller sets ``hp`` to 1 and
moves the player back to the spawn point.
"""
outcome: Outcome
log: list[str] = field(default_factory=list)
xp_delta: int = 0
gold_delta: int = 0
hp_delta: int = 0
bounce_to_spawn: bool = False
monster_name: str = ""
def _swing(rng: GameRNG, atk: int, def_: int) -> int:
"""Return one blow's damage: floor of 1, with a small RNG jitter."""
base = atk - def_
jitter = rng.randint(-1, 2)
return max(1, base + jitter)
def resolve_fight(rng: GameRNG, player: Player, monster: Monster) -> FightResult:
"""Run a full fight between *player* and *monster*.
The player strikes first each round. On victory the player banks the
monster's xp/gold and keeps any hp lost during the exchange. On defeat
the result flags a spawn bounce for the caller to apply.
"""
result = FightResult(outcome=Outcome.WIN, monster_name=monster.name)
player_hp = player.hp
monster_hp = monster.hp
result.log.append(f"You close with the {monster.name}.")
for _ in range(_MAX_ROUNDS):
dealt = _swing(rng, player.atk, monster.def_)
monster_hp -= dealt
result.log.append(f"You strike for {dealt}. ({monster.name}: {max(monster_hp, 0)} HP)")
if monster_hp <= 0:
result.outcome = Outcome.WIN
result.xp_delta = monster.xp
result.gold_delta = monster.gold
result.hp_delta = player_hp - player.hp
# The kill round (the strike line above) stays; the "falls + reward"
# sentence is composed by the caller at the moment it actually banks
# the xp/gold, so a reward is never narrated where none is applied
# (e.g. the Wyrm-win legacy reset, which keeps no xp/gold).
return result
taken = _swing(rng, monster.atk, player.def_)
player_hp -= taken
result.log.append(f"It hits back for {taken}. (You: {max(player_hp, 0)} HP)")
if player_hp <= 0:
result.outcome = Outcome.LOSE
result.bounce_to_spawn = True
result.log.append(
f"The {monster.name} lays you low. You wake at the spawn, barely alive."
)
return result
# Stalemate guard: treat an unresolved marathon as a flight to safety.
result.outcome = Outcome.FLED
result.hp_delta = player_hp - player.hp
result.log.append("The fight grinds on until you break away, winded.")
return result
def resolve_flee(rng: GameRNG, player: Player, monster: Monster) -> FightResult:
"""Attempt to flee a fight.
A successful flee escapes clean. A failed flee costs one free blow from
the monster but never drops the player below 1 HP (fleeing is a way out,
not a death trap).
"""
result = FightResult(outcome=Outcome.FLED, monster_name=monster.name)
if rng.chance(0.6):
result.log.append(f"You slip away from the {monster.name}.")
return result
taken = _swing(rng, monster.atk, player.def_)
taken = min(taken, max(player.hp - 1, 0))
result.hp_delta = -taken
result.log.append(f"You turn to run; the {monster.name} catches you for {taken} as you go.")
return result
@@ -1,108 +0,0 @@
"""Experience, level-ups, and the inn/healer restorative maths.
The XP curve and stat growth come from the content pack's settings, so no
progression constants live in this module. Level-ups loop (a single XP
award can cross several thresholds), grant flat stat growth, and fully
heal on each level gained.
"""
from __future__ import annotations
from dataclasses import dataclass
from typing import TYPE_CHECKING
if TYPE_CHECKING:
from understone.engine.models import Player, Settings
@dataclass(slots=True)
class LevelUp:
"""A record of a single level gained, for narration."""
new_level: int
hp_gain: int
atk_gain: int
def_gain: int
def xp_for_level(level: int, settings: Settings) -> int:
"""Return cumulative XP required to *reach* ``level``.
Level 1 needs 0. The default curve is ``base * n*(n+1)/2`` over the
completed levels, i.e. a triangular ramp scaled by ``xp_base``.
"""
if level <= 1:
return 0
completed = level - 1
return settings.xp_base * completed * (completed + 1) // 2
def apply_xp(player: Player, amount: int, settings: Settings) -> list[LevelUp]:
"""Award ``amount`` XP to *player*, applying every level-up it unlocks.
Returns one :class:`LevelUp` per level gained (empty when none). Each
level grants flat growth from settings and fully heals the player.
"""
player.xp += max(0, amount)
gains: list[LevelUp] = []
while player.xp >= xp_for_level(player.level + 1, settings):
player.level += 1
player.max_hp += settings.growth_max_hp
player.atk += settings.growth_atk
player.def_ += settings.growth_def
player.hp = player.max_hp
gains.append(
LevelUp(
new_level=player.level,
hp_gain=settings.growth_max_hp,
atk_gain=settings.growth_atk,
def_gain=settings.growth_def,
)
)
return gains
def rest(player: Player, cost: int) -> bool:
"""Fully heal *player* at the inn for a flat ``cost``.
Returns ``False`` without mutation when the player cannot afford it.
Resting when already at full HP still succeeds (and still charges),
matching the inn's flat-rate fiction.
"""
if player.gold < cost:
return False
player.gold -= cost
player.hp = player.max_hp
return True
@dataclass(slots=True)
class HealResult:
"""Outcome of a healer purchase: HP actually restored and gold spent."""
healed: int
cost: int
def heal(player: Player, amount: int, cost_per_hp: int) -> HealResult:
"""Restore up to ``amount`` HP at ``cost_per_hp`` gold each.
Heals only the missing portion, charges only for HP actually restored,
and is further bounded by what the player can afford. Returns the amount
healed and the gold spent (both zero when nothing could be done).
"""
missing = player.max_hp - player.hp
want = max(0, min(amount, missing))
if want <= 0 or cost_per_hp < 0:
return HealResult(healed=0, cost=0)
if cost_per_hp == 0:
player.hp += want
return HealResult(healed=want, cost=0)
affordable = player.gold // cost_per_hp
apply = min(want, affordable)
if apply <= 0:
return HealResult(healed=0, cost=0)
spent = apply * cost_per_hp
player.hp += apply
player.gold -= spent
return HealResult(healed=apply, cost=spent)
@@ -1,63 +0,0 @@
"""The shared event log — a world-wide feed players catch up on.
Events are append-only and ordered by insertion. Each player tracks a
cursor (the id of the last event they have seen); ``since`` returns the
slice after a cursor and the new cursor to persist.
An event carries a ``target``: empty means PUBLIC (the broadsheet and the
lobby TV), a player name means a PRIVATE note that only that player reads in
their own catch-up. Targeted rows ride the same id order as public ones, so
the cursor advances identically whether or not a private note was shown.
"""
from __future__ import annotations
from dataclasses import dataclass
@dataclass(frozen=True, slots=True)
class Event:
"""A single logged happening in the shared world.
``target`` is the empty string for public events (heralded to everyone)
or a player's name for a private note delivered only to that player.
"""
event_id: int
ts: str
kind: str
actor: str
text: str
target: str = ""
def since(events: list[Event], cursor: int) -> tuple[list[Event], int]:
"""Return events newer than ``cursor`` and the cursor to store next.
Events are kept in ascending id order (the store hydrates the newest tail
and reverses it to ascending; appends are monotonic), so the last fresh
event carries the highest id; that becomes the new cursor.
When nothing is new the input cursor is returned, so advancing is
idempotent.
"""
fresh = [e for e in events if e.event_id > cursor]
if not fresh:
return [], cursor
return fresh, fresh[-1].event_id
def since_visible(events: list[Event], cursor: int, viewer: str) -> tuple[list[Event], int]:
"""Like :func:`since`, but hide private notes not addressed to *viewer*.
Returns the events newer than ``cursor`` that *viewer* may read every
public event (empty ``target``) plus the private notes addressed to them
and the new cursor. The cursor advances to the highest id PAST the old
cursor regardless of visibility, so a private note for someone else is
consumed (never re-scanned) without ever being shown here.
"""
fresh = [e for e in events if e.event_id > cursor]
if not fresh:
return [], cursor
new_cursor = fresh[-1].event_id
visible = [e for e in fresh if not e.target or e.target == viewer]
return visible, new_cursor
@@ -1,232 +0,0 @@
"""Core data models for the game engine.
All models are plain dataclasses. ``Player`` is mutable (the engine applies
deltas in place); the static content models (``Monster``, ``Item``,
``TerrainDef``, ``LocationDef``, ``Zone``) are frozen.
"""
from __future__ import annotations
from dataclasses import dataclass, field
from enum import StrEnum
class Mode(StrEnum):
"""Which interaction surface the player is currently on."""
TILE = "tile"
MENU = "menu"
class Slot(StrEnum):
"""Equipment / item slot kinds."""
WEAPON = "weapon"
ARMOR = "armor"
CONSUMABLE = "consumable"
# v0.10 forge ore: a crafting MATERIAL carried in the satchel and spent at
# the forge. It is never equipped, never quaffed (no atk/def/heal), and
# never sold or bought — ore is earned in combat, not traded.
MATERIAL = "material"
@dataclass(slots=True)
class Player:
"""A single adventurer's durable state.
Coordinates are map cells; ``mode`` and ``at_location`` track whether
the player is on the overworld or inside a location menu. Turn fields
gate the daily action budget; bestow fields gate the daily fortune pool.
"""
name: str
x: int
y: int
hp: int
max_hp: int
level: int
xp: int
gold: int
atk: int
def_: int
weapon_id: str
armor_id: str
turns_left: int
turn_day: int
mode: Mode
at_location: str
created_at: str
last_seen: str
log_cursor: int
bestow_spent: int
bestow_day: int
wins: int = 0
posts_sent: int = 0
post_day: int = 0
gambles: int = 0
gamble_day: int = 0
# v0.7 "depth below" retention columns: how far the dungeon has been
# plumbed (0 = never descended; N = cleared rung N, 1-indexed), the
# carried satchel (see below), and the enhancement plus on whichever
# weapon/armour is CURRENTLY equipped in each slot.
deepest_rung: int = 0
# v0.10 STACK-BASED satchel: comma-joined "id:qty" stacks ('' = empty),
# e.g. "minor_potion:3,iron_ore:5". ``satchel_max`` caps DISTINCT stacks,
# not total items; per-stack qty is unbounded. Replaces the v0.7 flat id
# list. The "id:qty" wire format is owned by understone.engine.satchel
# (decode_satchel/encode_satchel); every reader goes through that codec.
satchel: str = ""
weapon_plus: int = 0
armor_plus: int = 0
# v0.10 the Vault: gold banked at the inn. SAFE from ambush (the steal only
# ever touches carried ``gold``) and SURVIVES the Wyrm-win legacy reset (a
# small persistent reward across runs, like a win ★).
banked: int = 0
@dataclass(frozen=True, slots=True)
class Monster:
"""A static monster definition from the content pack.
``boss`` monsters are the fixed endgame foe (the Wyrm Below): they are
excluded from random tier-band selection and only ever faced through the
deliberate ``challenge`` verb.
"""
tier: int
name: str
hp: int
atk: int
def_: int
xp: int
gold: int
monster_id: str = ""
boss: bool = False
# v0.7 weighted forest encounters: ``weight`` biases the random pick (a
# low weight surfaces seldom), ``rare`` marks a named beast that fires a
# public Herald flash and drops a guaranteed draught on the kill. Rung
# guardians ignore both (a rung is a fixed foe, never a weighted roll).
weight: int = 10
rare: bool = False
@dataclass(frozen=True, slots=True)
class Item:
"""A static item / equipment definition from the content pack."""
item_id: str
name: str
slot: Slot
atk: int
def_: int
heal: int
price: int
@dataclass(frozen=True, slots=True)
class TerrainDef:
"""A terrain kind: its glyph, walkability, encounter rate, colour role."""
key: str
glyph: str
walkable: bool
encounter_rate: float
color: str
@dataclass(frozen=True, slots=True)
class LocationDef:
"""A named location placed on the map (inn, shop, healer, dungeon)."""
key: str
kind: str
name: str
x: int
y: int
glyph: str
color: str
actions: tuple[str, ...]
flavor: tuple[str, ...] = field(default_factory=tuple)
@dataclass(frozen=True, slots=True)
class Zone:
"""A rectangular region that biases which monster tiers spawn."""
key: str
x0: int
y0: int
x1: int
y1: int
tier_lo: int
tier_hi: int
def contains(self, x: int, y: int) -> bool:
"""Return whether ``(x, y)`` falls inside this zone's rectangle."""
return self.x0 <= x <= self.x1 and self.y0 <= y <= self.y1
@dataclass(frozen=True, slots=True)
class WorldEvent:
"""One row of the weighted overworld encounter table.
``kind`` is one of ``fight``/``gold``/``heal``/``trap``/``lore``.
``weight`` biases random selection. ``lo``/``hi`` bound the rolled amount
for the value-bearing kinds (gold/heal/trap); they are unused for
``fight`` (the foe comes from the zone band) and ``lore`` (pure flavour).
"""
kind: str
weight: int
text: str
lo: int
hi: int
@dataclass(frozen=True, slots=True)
class Settings:
"""Economy and progression parameters sourced from the content pack."""
daily_turns: int
rest_cost: int
heal_cost_per_hp: int
starting_gold: int
starting_weapon: str
starting_armor: str
start_hp: int
start_atk: int
start_def: int
xp_base: int
growth_max_hp: int
growth_atk: int
growth_def: int
bestow_daily_budget: int
dungeon_tiers: tuple[int, ...]
boss_monster: str
wyrm_min_level: int
ambush_min_level: int
ambush_level_band: int
ambush_gold_pct: int
post_daily_cap: int
gamble_max_bet: int
gamble_daily_cap: int
# v0.7 "depth below": the carried-potion satchel size, the forge cost
# ladder (base * (current_plus + 1)) and its enhancement ceiling, and the
# consumable item a rare beast is guaranteed to drop on its kill.
satchel_max: int
forge_base_cost: int
forge_max_plus: int
rare_drop_item: str
# v0.10 the ore-gated forge: the world's forge MATERIAL item id (validated
# to slot=material), the ore each +1 step costs (need = (plus + 1) *
# per_plus), and the two ore sources — a guaranteed drop on a won dungeon
# rung and a chance of one ore on a won forest fight. Ore is combat-earned,
# never purchasable; the forge spends gold AND ore.
forge_ore_item: str
forge_ore_per_plus: int
ore_dungeon_drop: int
ore_forest_chance: float
# v0.8 "worlds without authors": the Watch's per-world CRT palette. One of
# the names in WATCH_THEMES; defaults to "phosphor" (the original green), so
# a pack that omits it looks exactly as the Vale always has.
watch_theme: str = "phosphor"
@@ -1,223 +0,0 @@
"""Overworld movement resolution.
Movement walks tile by tile so each intermediate cell is checked for
walls/edges and rolls an encounter. When a roll fires it weighted-picks one
row from the world's event table. A ``fight`` row STOPS the walk (a wandering
monster bars the path); the value-bearing rows (gold/heal/trap) and pure
``lore`` are applied immediately and the walk continues but only one event
fires per walk, so once any row has fired no further cells roll.
The walk stops early on the first of: running out of steps, hitting a blocked
cell, stepping onto a location door (flips to MENU), or a ``fight`` encounter.
Movement spends no daily turns only fighting does. Gold/heal/trap deltas are
applied straight to the player here (movement already mutates the player's
position), floored/capped so a trap never kills and a spring never overfills.
"""
from __future__ import annotations
from dataclasses import dataclass, field
from typing import TYPE_CHECKING
from understone.engine.models import Mode
if TYPE_CHECKING:
from understone.engine.models import Player, WorldEvent
from understone.engine.rng import GameRNG
from understone.engine.world import World
MAX_STEPS = 8
_DELTAS: dict[str, tuple[int, int]] = {
"N": (0, -1),
"S": (0, 1),
"E": (1, 0),
"W": (-1, 0),
}
_HEADINGS: dict[str, str] = {
"north": "N",
"south": "S",
"east": "E",
"west": "W",
"n": "N",
"s": "S",
"e": "E",
"w": "W",
}
@dataclass(slots=True)
class MoveEvent:
"""A non-fight overworld event already applied to the player.
``kind`` is ``gold``/``heal``/``trap``/``lore``; ``text`` is the pack's
flavour line; ``amount`` is the rolled magnitude (0 for ``lore``). The
player's hp/gold have already been mutated by ``resolve_move`` — this
record exists only so the façade can narrate what happened.
"""
kind: str
text: str
amount: int = 0
@dataclass(slots=True)
class MoveResult:
"""Outcome of a movement attempt.
``steps_taken`` counts cells actually entered. ``blocked`` is set when a
wall/edge stopped the walk. ``entered_location`` carries a location key
when the walk ended on a door. ``pending_fight`` carries an opponent
tier band when a ``fight`` encounter interrupted the walk. ``event``
carries a non-fight overworld event (already applied) when one fired.
"""
steps_taken: int
blocked: bool = False
blocked_reason: str = ""
entered_location: str | None = None
pending_fight: tuple[int, int] | None = None
event: MoveEvent | None = None
path_notes: list[str] = field(default_factory=list)
def parse_directions(steps: str, heading: str, distance: int) -> list[str]:
"""Translate either input form into a clamped list of cardinal steps.
The ``steps`` string (e.g. ``"NNEE"``) takes precedence when non-empty;
otherwise ``heading`` + ``distance`` is expanded. Either way the result
is clamped to ``MAX_STEPS``. Unknown direction characters are rejected.
"""
raw = steps.strip().upper()
if raw:
dirs: list[str] = []
for ch in raw:
if ch not in _DELTAS:
raise ValueError(f"unknown direction {ch!r} (use N/S/E/W)")
dirs.append(ch)
return dirs[:MAX_STEPS]
head = heading.strip().lower()
if not head:
return []
if head not in _HEADINGS:
raise ValueError(f"unknown heading {heading!r} (use north/south/east/west)")
count = max(0, min(distance, MAX_STEPS))
return [_HEADINGS[head]] * count
def resolve_move(
world: World,
player: Player,
rng: GameRNG,
*,
steps: str = "",
heading: str = "",
distance: int = 1,
max_steps: int = MAX_STEPS,
) -> MoveResult:
"""Walk *player* across *world* one cell at a time, mutating position.
Stops at the first blocking edge/wall, location door, or encounter.
Returns a :class:`MoveResult` describing where and why the walk ended.
"""
directions = parse_directions(steps, heading, distance)[:max_steps]
result = MoveResult(steps_taken=0)
fired = False # at most one overworld event per walk
for direction in directions:
dx, dy = _DELTAS[direction]
nx, ny = player.x + dx, player.y + dy
if not world.in_bounds(nx, ny):
result.blocked = True
result.blocked_reason = "the edge of the known world"
break
if not world.is_walkable(nx, ny):
terrain = world.terrain_at(nx, ny)
result.blocked = True
result.blocked_reason = _blocked_phrase(terrain.key)
break
player.x, player.y = nx, ny
result.steps_taken += 1
location = world.location_at(nx, ny)
if location is not None:
player.mode = Mode.MENU
player.at_location = location.key
result.entered_location = location.key
break
if fired:
continue
band = _encounter_band(world, nx, ny)
if band is None:
continue
terrain = world.terrain_at(nx, ny)
if not rng.chance(terrain.encounter_rate):
continue
fired = True
picked = _pick_event(world, rng)
if picked is None or picked.kind == "fight":
result.pending_fight = band
break
result.event = _apply_event(player, rng, picked)
return result
def _pick_event(world: World, rng: GameRNG) -> WorldEvent | None:
"""Weighted-pick one row from the world's event table, or ``None``.
Returns ``None`` only when the pack ships no event table at all, in which
case the caller falls back to the legacy always-a-fight behaviour.
"""
weights = world.event_weights()
if not weights:
return None
return world.events[rng.weighted_index(weights)]
def _apply_event(player: Player, rng: GameRNG, event: WorldEvent) -> MoveEvent:
"""Apply a non-fight event to *player* and return a record for narration.
``gold`` credits a rolled amount; ``heal`` adds hp capped at ``max_hp``;
``trap`` subtracts hp floored at 1 (a trap never kills, and never touches
gold); ``lore`` mutates nothing. Amounts roll over ``[lo, hi]``.
"""
if event.kind == "lore":
return MoveEvent(kind="lore", text=event.text)
amount = rng.randint(event.lo, event.hi)
if event.kind == "gold":
player.gold += amount
elif event.kind == "heal":
amount = min(amount, player.max_hp - player.hp)
player.hp += amount
elif event.kind == "trap":
amount = min(amount, max(player.hp - 1, 0))
player.hp -= amount
return MoveEvent(kind=event.kind, text=event.text, amount=amount)
def _encounter_band(world: World, x: int, y: int) -> tuple[int, int] | None:
"""Return the tier band for an encounter at ``(x, y)``, or ``None``.
Encounters only happen inside a zone; open terrain with no zone is safe.
"""
zone = world.zone_for(x, y)
if zone is None:
return None
return (zone.tier_lo, zone.tier_hi)
def _blocked_phrase(terrain_key: str) -> str:
"""Return an in-fiction phrase for being blocked by *terrain_key*."""
phrases = {
"water": "deep water",
"tree": "an impassable thicket",
"wall": "a sheer wall",
}
return phrases.get(terrain_key, "rough ground")
@@ -1,42 +0,0 @@
"""Leaderboard ordering.
Adventurers are ranked by level (desc), then XP (desc), then name (asc)
so ties break deterministically and alphabetically. The Hall of Legends is a
separate, append-only roll of completed runs (Wyrm kills), ordered newest
first by the store.
"""
from __future__ import annotations
from dataclasses import dataclass
@dataclass(frozen=True, slots=True)
class RankEntry:
"""One row of the leaderboard.
``wins`` is the number of times the adventurer has slain the Wyrm Below
(each shown as a beside the name); it does not affect ordering.
"""
name: str
level: int
xp: int
gold: int
wins: int = 0
@dataclass(frozen=True, slots=True)
class HallEntry:
"""One immortalised run in the Hall of Legends (a Wyrm slain)."""
name: str
win_ts: str
run_days: int
level_at_win: int
def leaderboard(entries: list[RankEntry], limit: int = 10) -> list[RankEntry]:
"""Return the top ``limit`` entries in leaderboard order."""
ordered = sorted(entries, key=lambda e: (-e.level, -e.xp, e.name))
return ordered[:limit]
@@ -1,58 +0,0 @@
"""Randomness with deterministic test injection.
A single master ``GameRNG`` is seeded once at startup (from ``os.urandom``
in production). Per-encounter child generators are derived from the master
so a fight's rolls are reproducible given the same child seed. No tool
argument ever carries a seed randomness is server-authoritative.
"""
from __future__ import annotations
import os
import random
class GameRNG:
"""A thin wrapper over ``random.Random`` with child-RNG derivation."""
def __init__(self, seed: int | None = None) -> None:
if seed is None:
seed = int.from_bytes(os.urandom(8), "big")
self._random = random.Random(seed)
def chance(self, probability: float) -> bool:
"""Return ``True`` with the given probability in ``[0.0, 1.0]``."""
if probability <= 0.0:
return False
if probability >= 1.0:
return True
return self._random.random() < probability
def randint(self, lo: int, hi: int) -> int:
"""Return a random integer in the inclusive range ``[lo, hi]``."""
return self._random.randint(lo, hi)
def choice_index(self, count: int) -> int:
"""Return a random index in ``[0, count)``."""
return self._random.randrange(count)
def weighted_index(self, weights: list[int]) -> int:
"""Return an index into ``weights`` chosen in proportion to them.
A single uniform draw is mapped through the cumulative sum, so the
result is deterministic under a fixed seed. ``weights`` must be
non-empty with a positive total (the loader guarantees this for the
content pack's event table).
"""
total = sum(weights)
roll = self._random.randrange(total)
cumulative = 0
for index, weight in enumerate(weights):
cumulative += weight
if roll < cumulative:
return index
return len(weights) - 1
def child(self) -> GameRNG:
"""Derive an independent child RNG seeded from the master stream."""
return GameRNG(self._random.getrandbits(64))

Some files were not shown because too many files have changed in this diff Show More