Compare commits

...

29 Commits

Author SHA1 Message Date
Patrick Buckley 4a38b835f5 chore: bump version to 1.5.18 2026-05-19 08:12:32 -07:00
Patrick Buckley 415be00149 docs(changelog): release 1.5.18 notes 2026-05-19 08:12:24 -07:00
Patrick Buckley 346ad2a6aa docs(changelog): note admin config.toml support + load_config perm warning 2026-05-19 08:12:06 -07:00
Patrick Buckley 8003fcbebe feat(admin): align turnstone-admin DB config with server (config.toml + env) (#531)
* feat(admin): align turnstone-admin DB config with server (config.toml + env)

turnstone-admin previously read TURNSTONE_DB_* env vars only, forcing
operators with credentials in config.toml to re-export them just to
run admin commands. Wire add_config_arg + apply_config(["database"])
into main() so admin honors the same precedence as turnstone-server:
CLI / config.toml [database] > TURNSTONE_DB_* env > hardcoded defaults.

Also exposes pool_size + sslmode/sslrootcert/sslcert/sslkey to admin,
which previously dropped any such config silently.

Hardening: load_config() now warns once when config.toml is group- or
world-readable, since DB password and TLS key paths live in [database].

Tests cover precedence (default / config / env / partial fallback /
empty-string-in-config-beats-env), the real init_storage boundary on
a tmp sqlite path, the sys.argv -> main() pre-parser path, and the
new permission check (mode 0644 warns, 0600 quiet).

* test(admin): unify config import style in test_admin_db_config

Use module alias (config_mod.apply_config) instead of mixing
'import turnstone.core.config as config_mod' with 'from
turnstone.core.config import apply_config'.  Addresses
github-code-quality bot feedback on PR #531.
2026-05-19 08:12:06 -07:00
Patrick Buckley d4f3711d63 chore: bump version to 1.5.17 2026-05-18 20:33:49 -07:00
Patrick Buckley cea229206b docs(changelog): release 1.5.17 notes 2026-05-18 20:33:49 -07:00
Patrick Buckley b7b4dcc0df fix(judge): UPSERT intent_verdicts so llm_fallback upgrades land
Async LLM-tier "llm_fallback" verdicts (judge.py:1073, judge.py:1131
via _deliver_fallbacks) deliberately reuse the heuristic verdict's
``verdict_id`` so the row gets "upgraded in place" from heuristic →
llm_fallback when the LLM judge times out, is cancelled, or returns
no content.  The consumer ``_persist_intent_verdict`` was doing a
plain INSERT via ``create_intent_verdict``, hitting the
``intent_verdicts_pkey`` constraint on every llm_fallback delivery.
Postgres logged the duplicate-key error; the application try/except
swallowed it at log.debug — so the row never actually got upgraded
and the LLM judge's annotation ("(LLM judge did not return a
verdict)") was lost.

The collision rate exploded on stable/1.5 smoke tests because
PR #527 (just merged) added two new heuristic-INSERT paths in the
auto-approve early-return branches of ``approve_tools`` — previously
those branches dropped heuristic verdicts on the floor, leaving no
row for the fallback to collide with.

Fix:
- New ``upsert_intent_verdict`` method on the storage protocol +
  sqlite + postgres impls, using dialect-specific
  ``insert(...).on_conflict_do_update(index_elements=["verdict_id"],
  set_={...})``.  Set_ clause updates ONLY the three fields that
  genuinely change between heuristic and llm_fallback: ``tier``,
  ``reasoning``, ``judge_model``.
- Every other column is excluded from set_: identity columns
  (verdict_id, ws_id, call_id, func_name, func_args), carried-
  verbatim columns (intent_summary, risk_level, confidence,
  recommendation, evidence, latency_ms), and ``user_decision``.
- ``user_decision`` exclusion is load-bearing: ``IntentVerdict
  .to_dict()`` doesn't project it, so a fallback verdict reaching
  ``_persist_intent_verdict`` carries the kwarg's ``"pending"``
  default.  If the operator already resolved the approval between
  heuristic INSERT and fallback delivery, the row's user_decision
  has been stamped to ``"approved"``/``"denied"``/``"timeout"`` (or
  an auto-approve reason at heuristic-INSERT time per PR #527).
  Including ``user_decision`` in set_ would silently clobber that
  back to ``"pending"``.
- ``_persist_intent_verdict`` switched from ``create_*`` to
  ``upsert_*``.  Bulk path ``create_intent_verdicts_bulk`` stays as
  plain INSERT — every heuristic ``verdict_id`` is freshly minted
  in ``judge.evaluate`` so in-turn dups can't happen.  The inverse
  race (daemon-judge verdict lands BEFORE the bulk write) IS
  reachable today but its observable behavior is unchanged by the
  per-row UPSERT switch; documented at the bulk site for a future
  hardening pass.

Test coverage:
- TestIntentVerdictUpsert × 4 — fresh-id insert, conflict-upgrade,
  user_decision preservation across heuristic→approved→fallback,
  identity + carried-field preservation.
- Existing tests in test_session_ui_base.py updated to mock the
  new upsert method instead of create_intent_verdict.
2026-05-18 20:32:02 -07:00
Patrick Buckley 97080e1df9 fix(coord): drop unused snip-threshold constants, name elision margin
Two dead module-level constants flagged by github-code-quality on
PR #529: ``_INSPECT_MSG_SNIP_THRESHOLD`` and
``_INSPECT_TOOL_ARG_SNIP_THRESHOLD`` lost their callers when the
content-snip logic moved into the ``_snip_head_tail`` helper.  The
helper now reads ``head + tail + _INSPECT_ELISION_MARGIN`` so the
"reserve bytes for the elision marker" rationale that the dead
constants documented stays named instead of becoming a bare ``64``.
2026-05-18 19:39:30 -07:00
Patrick Buckley af0cbaaec3 feat(coord): three-tier compression for inspect_workstream output
A coord doing a fan-out wave of inspect_workstream calls against
tool-heavy children could blow the context budget on raw output
alone (one child with a 100 KB bash result × N children).  The
previous safety net was ``_truncate_output``'s head+tail strategy,
which silently drops *middle* messages — exactly the wrong shape
for a coordinator trying to understand a child's trajectory (the
LAST message tells the model what the child concluded; the FIRST
sets the brief; the middle is the connective tissue).

Three-tier degradation modeled on the search tool's pattern at
``session.py:_format_search_results``:

  Tier 1 (full):    every message verbatim — used when size fits.
  Tier 2 (compact): per-message head/tail-snipped content (600/300
                    chars) plus snipped ``tool_calls.arguments``
                    (300/100 chars).  When content snipping alone
                    doesn't fit, fall through a message-list trim
                    ladder ((20,30) → (10,20) → (5,10)) that keeps
                    head + tail messages and elides the middle as
                    ``{"_omitted": N}``.
  Tier 3 (skeleton): no messages — counts + role distribution +
                    verdicts-by-risk + last assistant preview.

Budget 32 KB (matches ``_SEARCH_OUTPUT_BUDGET``).  First emission
whose JSON serialization fits the budget wins.  ``_tier`` lands on
every non-error emission so the coordinator LLM and audit readers
can see which compression rung was selected; ``_tier_note`` carries
actionable advice (re-call with a smaller ``message_limit`` etc.).
Error-shape results bypass tiering — they're already small.

Bug fixes caught during review:
- ``_compact_message`` now preserves the assistant-side ``tool_calls``
  list with snipped ``function.arguments``; the pre-fix shape left
  audit readers with tool-result orphans against invisible calls.
- The intermediate Tier-2 list-trim ladder fixes a size-monotonicity
  bug where Tier-2 with un-snippable content (per-message body
  under the 964-char threshold) plus the added ``_tier_note`` came
  out STRICTLY larger than Tier-1, falling through to skeleton
  when a head+tail trim would have preserved dozens of messages.
- ``_inspect_skeleton`` reads ``result["skill_id"]`` (production
  storage row key) with a ``skill`` fallback; pre-fix it read
  ``skill`` only and emitted ``null`` for every real workstream.
2026-05-18 19:39:30 -07:00
Patrick Buckley b9ce0d388e fix(coord): address PR review threads on spawn_workstream rename
Three Copilot threads from PR #526:

1. ``_exec_spawn_workstream`` success path emitted
   ``{"child_ws_id": null}`` when the upstream response unexpectedly
   omitted ``ws_id`` (200-shape with no error field, no id field).
   Adds the missing guard — mirrors ``_exec_spawn_batch`` which
   already surfaces ``"spawn returned no ws_id"`` as a denied row.
   The LLM now sees a tool error and can retry instead of chasing
   a null id through follow-up tools.

2. ``docs/coordinator-skills.md`` UI render note said "keep the
   ws_id as the click-through key" in a paragraph that had just
   introduced ``child_ws_id`` — readable as "the ws_id value" but
   confusable as a field-name claim.  Clarifies that the value
   class is the same regardless of which key carried it.

3. ``docs/bulk-endpoints.md`` ``spawn_batch`` example shows
   ``child_ws_id`` (coord-tool output shape).  The doc title and
   the "model tool" column label already disambiguate it from HTTP
   API responses, but a reader landing at the example section
   directly could miss the framing.  Adds one explicit sentence.
2026-05-18 19:39:30 -07:00
Patrick Buckley d07d2242aa fix(coord): rename ws_id->child_ws_id in spawn return JSON
Coordinator LLMs on large fan-outs recency-bias on seeing `ws_id`
in a `spawn_workstream` / `spawn_batch` return -- calling
`spawn_workstream(ws_id=...)` again instead of progressing to
`wait_for_workstream(ws_ids=[...])`. On 10+ child fan-outs this
cascades into self-inflicted re-spawn loops.

Rename to `child_ws_id` (already an existing project term -- see
`tasks` tool, `child_event_bus.py`) defuses the recency bias.
Scope is the LLM-facing JSON only -- the server HTTP API at the
spawn endpoint still returns `ws_id`, and the internal reads of
that HTTP response are unchanged.

Also updates the two tool descriptions, the operator-facing skill
doc, and the bulk-endpoints example so docs don't undo the rename.
2026-05-18 19:39:30 -07:00
Patrick Buckley 70514cc406 fix(coord): omit empty allowed_tools in list_skills + clarify semantics
A skill with `allowed_tools=[]` in the coordinator's `list_skills`
response read as "no tools are usable by this skill" to a model that
didn't know the semantics — but the actual meaning is "no tools are
pre-approved for auto-approval (auto-approve exemption list)".  Real
misdiagnosis incident: a code-review child appeared to have been
spawned with zero tool access when in fact the skill simply hadn't
declared an auto-approve allowlist.

Two-part fix:
- `coordinator_client.list_skills` omits the `allowed_tools` key from
  the per-skill dict when empty.  Absence now carries the unambiguous
  meaning "no tool is pre-approved for this skill"; presence (with a
  non-empty list) keeps the standard Claude Code skill-spec shape.
- `turnstone/tools/list_skills.json` description rewrites the field
  doc so the LLM sees: "tool names exempt from the operator approval
  gate ... the field is OMITTED when empty: a skill without
  `allowed_tools` still has access to every tool in its session's
  toolset; absence of the field means no tool is pre-approved for
  this skill, not that the skill has no tools."

Field name stays `allowed_tools` — matches the upstream Claude Code
skill frontmatter (`allowed-tools` hyphenated, stored as
`allowed_tools` internally per `skill_parser.py:241-242`).  Parser,
storage column, admin UI, and SDK unchanged.
2026-05-18 19:39:30 -07:00
Patrick Buckley 687a3367c0 fix(judge): explicit user_decision vocabulary (no more empty strings)
Auto-approved tool calls left intent_verdict rows with `user_decision=""`,
indistinguishable from rows still pending manual review.  Real misdiagnosis
incident: a coord with `recommendation="review"` and `user_decision=""` was
read as "stuck waiting for approval" when in fact the tools had been
auto-approved and the child was running normally.

New vocabulary at the storage API boundary (column server_default stays
`""` so pre-fix legacy rows are still distinguishable as such):

- `pending`           — at insert, before any resolution
- `approved` / `denied` — manual user resolution
- `timeout`           — approval-event timeout (split from `denied` so the
                        audit column alone tells them apart; the feedback
                        string used to carry this distinction)
- `policy` / `blanket` / `skill` / `always` / `auto_approve_tools` —
                        auto-approve reasons (mirror `AutoApproveReason`)

Heuristic verdicts on the two auto-approve early-return branches are now
persisted with `user_decision=<reason>` (previously dropped on the floor).
Late LLM verdicts for already-auto-approved call_ids look up the reason via
a TTL-pruned `_auto_approve_reasons` map (lazy 60s prune at write time, so
no fixed cap can silently regress the fix on the N+1th auto-approve; LLM-
disabled sessions don't leak entries because prune fires whenever auto-
approves happen).

Bug fixes caught during review:
- `on_intent_verdict` early-returns when the verdict already carries an
  auto_reason — without this, a manual `resolve_approval` on a mixed batch
  would overwrite the auto-stamped row with `approved`/`denied`.
- `_record_auto_approves` runs BEFORE `_persist_auto_approved_heuristic_*`
  so the lookup map is populated before any concurrent LLM verdict can
  fire and miss it.
- `resolve_approval(timeout=True, approved=True)` now raises ValueError
  to make the split-brain shape unrepresentable.
- Approval-timeout feedback string derives from `_APPROVAL_WAIT_TIMEOUT`
  rather than the hardcoded "1 hour".
2026-05-18 19:39:30 -07:00
renovate[bot] bfd99c6a81 chore(deps): lock file maintenance 2026-05-18 19:39:30 -07:00
github-actions[bot] 1a813c8130 chore: download vendored JS files 2026-05-18 19:39:30 -07:00
renovate[bot] d6aa85db6d chore(deps): update dependency katex to v0.16.47 2026-05-18 19:39:30 -07:00
renovate[bot] c0c7fda7f9 chore(deps): lock file maintenance 2026-05-18 19:39:29 -07:00
Patrick Buckley 7fae2698d7 docs(storage): clarify LIKE_ESCAPE contract with .like(escape=...)
The previous comment described "\\" as "non-default", which is
backwards — "\\" is the SQL standard escape character.  The
actually-non-default part is SQLAlchemy's ``.like()`` itself: it
defaults to no escape character, so ``escape_like``'s output is only
interpreted correctly when callers pass ``escape=LIKE_ESCAPE``
explicitly.  Reword to put the caller-side requirement first.
2026-05-18 19:39:29 -07:00
Patrick Buckley 4bbe64755e fix(watch): deliver terminal fires instead of dropping them silently
WatchRunner._poll_watch committed active=False to the row BEFORE
calling _dispatch_result for a terminal fire, and the dispatch closure
registered by ChatSession.set_watch_runner enqueued each reminder with
a valid_until=is_watch_active predicate that re-read the row at drain
time. Since the runner already flipped active to 0, the predicate
returned False for every dispatched fire and NudgeQueue.drain silently
dropped the entry — the model never saw a watch result. Then a
subsequent action=cancel call hit list_watches_for_ws (filters
active==1), the now-inactive row was invisible, and the cancel
returned 'Watch "X" not found.' regardless of whether the watch had
actually run.

Reorder _poll_watch to dispatch before the row write, drop the
valid_until predicate from the watch closure (its only effect was the
bug above), and add a _terminal_dispatched guard on the runner so a
transient storage failure between dispatch and row-write doesn't
re-fire the reminder on the next tick. Add WatchRunner.forget_terminal_dispatched
and call it from the cancel path so an out-of-band deactivate (next_poll='')
doesn't leak the watch_id from the runner's pending-retry set indefinitely.

Cancel-by-name now routes through a new find_watch_by_name storage
method that ignores the active filter and prefers active rows over
newer-inactive same-name siblings. The session.py cancel branch
distinguishes 'already completed (auto-cancelled)' from 'not found'
so the model can tell apart 'this watch ran and finished' from
'no such watch.' Consolidate the two byte-identical _escape_like
/ _escape_ilike helpers in the storage backends into a single
turnstone.core.storage._utils.escape_like and apply it to the new
find_watch_by_name LIKE pattern so a model-supplied watch name
containing % or _ can't redirect a cancel to a sibling watch.

NudgeQueue.drain previously dropped predicate-failed entries without
logging anything, which is what hid this bug for so long. Drain now
emits nudge_queue.predicate_dropped: info for reason=predicate_false
(the normal lifecycle case — idle_children when every active child
finished between enqueue and drain), warning with exc_info for
reason=predicate_raised (a misbehaving predicate).

Tests: new test_poll_watch_terminal_fire_survives_drain (parametrized
stop_on_fired + max_polls_reached) drives the real WatchRunner._poll_watch
against a real tmp_db row and confirmed to fail against pristine main.
test_poll_watch_retry_deactivate_after_update_watch_failure exercises
the _terminal_dispatched retry-deactivate branch end to end.
test_cancel_clears_pending_terminal_dispatched_entry covers the cancel-
path leak case. test_find_by_name_prefers_active_over_newer_inactive
catches the ordering regression. test_find_by_name_treats_percent_as_literal
+ test_find_by_name_treats_underscore_as_literal pin the LIKE escape.
2026-05-18 19:39:29 -07:00
Patrick Buckley c1281b9721 chore: bump version to 1.5.16 2026-05-13 16:13:01 -07:00
Patrick Buckley 3124dbe52f fix(vendor): widen update-vendored-js sweep to catch shared_static/ + .py
The shared_static exclude in scripts/update-vendored-js.sh was meant to
skip self-references inside vendored libraries, but it also hid
shared_static/renderer.js — which loads the vendored libs and pinned
mermaid-11.14.0 across every renovate bump since #426. Tests under
tests/test_web_helpers.py were similarly invisible because the include
list omitted *.py.

Replace the broad shared_static exclude with the specific old-versioned
vendor directory (about to be rm -rf'd next anyway), and add *.py to the
include list. Bump renderer.js to mermaid-11.15.0 to repair the live
404, and refresh the test fixtures to current vendor versions so they
stop drifting.
2026-05-13 16:12:40 -07:00
renovate[bot] a8a1e738ca chore(deps): lock file maintenance 2026-05-13 15:50:14 -07:00
github-actions[bot] 27768abecc chore: download vendored JS files 2026-05-13 15:50:14 -07:00
renovate[bot] 1e5017293a chore(deps): update vendored js 2026-05-13 15:50:14 -07:00
renovate[bot] b2c688f3b1 chore(deps): update ghcr.io/astral-sh/uv docker tag to v0.11.14 2026-05-13 15:50:14 -07:00
renovate[bot] 09b1f07e18 chore(deps): update dependency vitest to v4.1.6 2026-05-13 15:50:14 -07:00
Patrick Buckley 1de8f6b4c7 chore: bump version to 1.5.15 2026-05-12 21:21:15 -07:00
Patrick Buckley 4f89c1c3f3 docs(changelog): release 1.5.15 notes
Fixes the 1.5.14 admin.js syntax error that left the console UI
non-functional whenever an MCP server row had consented users.
2026-05-12 21:21:08 -07:00
Patrick Buckley 6898860f18 fix(mcp): admin.js syntax error in bulk-revoke button (Phase 9)
Line 3446 used double-quote string delimiters with an embedded ">
that terminated the string mid-attribute, leaving "bulk-revoke (" as
bare tokens.  The rest of the surrounding block uses single-quote
delimiters; switch the broken line to match so the embedded > and "
sit safely inside the string.

The parse error wiped out every global in admin.js, so showAdmin and
the rest of the admin entry points were undefined — the console was
non-functional whenever an MCP server row had consented_users_count > 0.
2026-05-12 21:20:45 -07:00
108 changed files with 6555 additions and 3823 deletions
+141
View File
@@ -14,6 +14,147 @@ Three release tracks are maintained:
## [Unreleased]
## [1.5.18]
Backports the `turnstone-admin` config-loading alignment from `main`
plus the accompanying `load_config` permission-warning hardening. No
schema changes.
### Added
- **`turnstone-admin` reads `config.toml`** — the admin CLI now honors
the same `[database]` section that `turnstone-server` does, with the
same precedence (`CLI / config.toml > TURNSTONE_DB_* env > defaults`).
Operators with DB credentials in `config.toml` no longer need to
re-export `TURNSTONE_DB_URL` before every admin invocation. Newly
plumbed through to `init_storage`: `pool_size`, `sslmode`,
`sslrootcert`, `sslcert`, `sslkey` — previously the admin CLI
silently dropped these. A new `--config PATH` flag mirrors the
one already on `turnstone-server`.
### Security
- **Permissive `config.toml` now warns** — `turnstone.core.config.load_config`
logs a single warning when the resolved config file is group- or
world-readable (any bit in `0o077`). DB password and TLS key paths
live in `[database]`; operators usually want the file at `0600`.
## [1.5.17]
Backports a clutch of coordinator-tool clarity fixes plus a watch-delivery
correctness fix from `main` to the `stable/1.5` track, plus a previously-
latent intent-verdicts persistence bug exposed by the new heuristic-verdict
INSERT paths. No schema changes.
### Fixed
- **`intent_verdicts` PK collisions on every llm_fallback delivery** —
async LLM-tier "llm_fallback" verdicts (`turnstone/core/judge.py`
`_deliver_fallbacks` and the in-loop fallback path) deliberately
reuse the heuristic verdict's `verdict_id` so the row gets
"upgraded in place" from `tier="heuristic"``tier="llm_fallback"`
when the LLM judge times out, is cancelled, or returns no content.
The consumer `_persist_intent_verdict` was doing a plain INSERT,
hitting the `intent_verdicts_pkey` constraint on every fallback
delivery; Postgres logged the duplicate-key error, the application
try/except swallowed it at `log.debug`, and the row never actually
got upgraded — the LLM judge's annotation
(`"(LLM judge did not return a verdict)"`) was lost. The collision
rate exploded on this release because the new heuristic-INSERT
paths in the auto-approve early-return branches of `approve_tools`
(introduced below) leave no gap for the fallback to land cleanly
into. Fix: new `upsert_intent_verdict` storage method using
`ON CONFLICT (verdict_id) DO UPDATE` that updates only `tier`,
`reasoning`, `judge_model` — the three fields that genuinely
change between heuristic and llm_fallback. Every other column
(identity, carried-verbatim, and `user_decision`) is excluded;
`user_decision` in particular would otherwise be clobbered back
to `"pending"` when a fallback arrives after the operator has
already resolved the approval. The bulk-INSERT path stays as
plain INSERT — fresh UUIDs in `judge.evaluate` make in-turn dups
impossible; the inverse race (fallback wins before bulk lands) is
reachable but unchanged in observable behavior by this fix,
documented at the bulk site for a future hardening pass.
- **Coordinator LLM re-spawn loops on large fan-outs** — the spawn-tool
return JSON used `ws_id` as its key, which primed the model's recency
bias to feed the spawn result straight back into another
`spawn_workstream(ws_id=...)` call instead of progressing to
`wait_for_workstream(ws_ids=[...])`. On 10+ child fan-outs this cascaded
into self-inflicted re-spawn loops. The LLM-facing tool result now emits
`child_ws_id` (the storage column / HTTP API contract is unchanged); the
field name is already an existing project term so the rename aligns
rather than introduces new vocabulary. Also handles the silent
upstream-omits-ws_id success-shape edge that previously emitted
`{"child_ws_id": null}` to the LLM — now surfaces a tool error so the
model retries rather than chasing a null id.
- **`inspect_workstream` blowing the coordinator context budget** — a
coord doing a fan-out wave against tool-heavy children could land
>100 KB of raw output per inspect call, and the previous safety net
(`_truncate_output`'s head+tail strategy) silently dropped *middle*
messages — exactly the wrong shape for understanding a child's
trajectory (the FIRST sets the brief, the LAST shows the conclusion,
the middle is the connective tissue). Output now goes through a
three-tier degradation ladder mirroring the search tool's
`_format_search_results`: `_tier="full"` (every message verbatim) →
`_tier="compact"` (per-message head/tail-snipped content + snipped
`tool_calls.arguments`, falling through a `(20,30)` / `(10,20)` /
`(5,10)` message-list trim ladder) → `_tier="skeleton"` (counts, role
distribution, last-assistant preview). Budget 32 KiB matches the
search tool's; the chosen tier is annotated on the response so the
model can recall with a tighter `message_limit` if signal was lost.
- **Auto-approved verdicts indistinguishable from pending review** —
`intent_verdict` rows for auto-approved tool calls landed with
`user_decision=""`, which read identically to "still waiting for the
operator" in the audit trail and led to a real misdiagnosis incident.
The column now carries an explicit vocabulary at insert: `pending` /
`approved` / `denied` / `timeout` / `policy` / `blanket` / `skill` /
`always` / `auto_approve_tools`. The auto-approve early-return
branches in `approve_tools` now persist heuristic verdicts stamped
with their reason (previously dropped on the floor), and late LLM-tier
verdicts that arrive for an already-auto-approved call_id are stamped
via a TTL-pruned lookup map — so the audit row carries the
auto-approve reason even when the LLM judge daemon completes after
the synchronous approval cycle finished. `resolve_approval` gains a
`timeout` kwarg writing `"timeout"` (the previous shape collapsed
passive timeouts and active denials into the same column).
- **`list_skills` empty `allowed_tools` misread as "no tool access"** —
the response previously emitted `"allowed_tools": []` for every skill
that hadn't declared an auto-approve allowlist, which a coordinator
model read as "this skill can't use any tools" (real misdiagnosis: a
code-review child appeared to have been spawned with zero tool
access). The field is now omitted entirely when empty — absence
carries the unambiguous meaning "no tool is pre-approved for this
skill", presence (non-empty list) keeps the standard Claude Code
skill-spec shape. The tool description rewrite makes the
auto-approve-allowlist semantics explicit so a future reader doesn't
re-derive the gating misread.
- **Watch terminal-fires silently dropped on backpressure** —
delivery now routes terminal events through the same path as
normal fires instead of being filtered out when the consumer was
saturated.
### Documentation
- **Storage `LIKE_ESCAPE` contract** — clarify that callers passing
`.like(escape=...)` must use the same escape character that the
storage helper assumes; previous wording let a reader pass a
different escape and silently produce no matches.
## [1.5.15]
### Fixed
- **Admin console blank-page on MCP server rows with consented users** — a
Phase 9 (1.5.14) regression in `admin.js` used double-quote string
delimiters on the bulk-revoke button HTML literal, but the literal embeds
a `"` mid-attribute. JS closed the string early, turned `bulk-revoke (`
into bare tokens, and the resulting `SyntaxError` wiped out every global
in `admin.js``showAdmin` and all other admin entry points became
undefined, so the console UI was non-functional whenever the rendered MCP
server list contained at least one row with `consented_users_count > 0`.
Switch the literal to single-quote delimiters to match the surrounding
block.
## [1.5.14]
Backports OAuth-MCP Phase 9 from `main` to the `stable/1.5` track.
+1 -1
View File
@@ -8,7 +8,7 @@ FROM python:3.14-slim
LABEL org.opencontainers.image.title="turnstone" \
org.opencontainers.image.description="Multi-node AI orchestration platform"
COPY --from=ghcr.io/astral-sh/uv:0.11.8 /uv /usr/local/bin/uv
COPY --from=ghcr.io/astral-sh/uv:0.11.14 /uv /usr/local/bin/uv
# Remove the slim image's man page exclusion so man-db has actual content
RUN rm -f /etc/dpkg/dpkg.cfg.d/docker
+1 -1
View File
@@ -91,7 +91,7 @@ turnstone/
discord/ Discord adapter (bot, cog, views, streaming, config)
slack/ Slack adapter (Socket Mode bot, DM routing, approval buttons)
shared_static/ Shared design system (base.css, auth.js, theme.js, toast.js, utils.js, kb.js)
katex-0.16.45/ Vendored KaTeX math rendering library (MIT, woff2 fonts)
katex-0.16.47/ Vendored KaTeX math rendering library (MIT, woff2 fonts)
ui/
colors.py ANSI color constants with NO_COLOR support
markdown.py Streaming terminal markdown renderer (line-buffered)
+9 -2
View File
@@ -110,11 +110,18 @@ owns it; the node is just currently unreachable.
### Example — `spawn_batch`
This is the coordinator-tool result shape (the JSON the LLM receives),
not an HTTP API response — the table above keys it under "model tool"
to distinguish it from the `/v1/api/...` endpoints in the same table.
The underlying HTTP spawn endpoint still returns `ws_id`; the tool
result re-keys it to `child_ws_id` to defuse a coordinator-LLM recency
bias (see `docs/coordinator-skills.md`).
```json
{
"results": {
"0": {"ws_id": "d4e5f6...", "name": "csrf-audit", "node_id": "gpu-3"},
"2": {"ws_id": "f1a2b3...", "name": "xss-audit", "node_id": "gpu-1"}
"0": {"child_ws_id": "d4e5f6...", "name": "csrf-audit", "node_id": "gpu-3"},
"2": {"child_ws_id": "f1a2b3...", "name": "xss-audit", "node_id": "gpu-1"}
},
"denied": [
{"idx": 1, "reason": "skill not found: nonexistent-skill"}
+12 -5
View File
@@ -169,14 +169,21 @@ validates ws_id against `parent_ws_id=coord_ws_id` AND
the wait into reporting "complete".
Pattern: capture each spawn result in the next tool call's input.
The JSON tool-result carries `{"ws_id": "...", "name": "...",
The JSON tool-result carries `{"child_ws_id": "...", "name": "...",
"node_id": "...", "routing_strategy": "..."}`; the model should
extract the ws_id and pass it to `inspect_workstream` /
`wait_for_workstream` / `send_to_workstream` / `close_workstream`
verbatim.
extract the `child_ws_id` and pass it as `ws_id` (or in the `ws_ids`
list) to `inspect_workstream` / `wait_for_workstream` /
`send_to_workstream` / `close_workstream` verbatim. The asymmetry
— spawn returns `child_ws_id` but the other tools accept `ws_id` /
`ws_ids` — is intentional: it defuses a coordinator-LLM recency
bias where seeing `ws_id` in a spawn return primed re-spawn loops
instead of progression to the wait phase.
A UI that wants human-readable identifiers should render the `name`
field and keep the ws_id as the click-through key.
field and keep the workstream id as the click-through key — note
that the id *value* is the same regardless of whether it arrived
under the `child_ws_id` key (spawn return) or the `ws_id` key
(every other tool's input/output); only the field name differs.
---
+3 -3
View File
@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
[project]
name = "turnstone"
version = "1.5.14"
version = "1.5.18"
description = "Multi-node AI orchestration platform with tool use, agent routing, and cluster simulation."
readme = "README.md"
license = "BUSL-1.1"
@@ -82,9 +82,9 @@ include = [
"turnstone/console/static/coordinator/*.js",
"turnstone/shared_static/*.css",
"turnstone/shared_static/*.js",
"turnstone/shared_static/katex-0.16.45/**/*",
"turnstone/shared_static/katex-0.16.47/**/*",
"turnstone/shared_static/hljs-11.11.1/**/*",
"turnstone/shared_static/mermaid-11.14.0/**/*",
"turnstone/shared_static/mermaid-11.15.0/**/*",
"turnstone/shared_static/hls-1.6.16/**/*",
"turnstone/sdk/py.typed",
"turnstone/deploy/*.yaml",
+6 -3
View File
@@ -50,11 +50,14 @@ update_refs() {
local old_pattern="$1" # e.g. katex-0.16.38
local new_pattern="$2" # e.g. katex-0.16.39
# Find all files with version references (excludes vendored JS and worktrees)
# Find all files with version references. Excludes the old versioned vendor
# directory itself (about to be rm -rf'd anyway) so we don't bother rewriting
# self-references inside it — but does NOT exclude all of shared_static/,
# because shared_static/renderer.js loads the vendored libs and needs the bump.
local files
files=$(grep -rl --include='*.toml' --include='*.html' --include='*.js' --include='*.md' \
files=$(grep -rl --include='*.toml' --include='*.html' --include='*.js' --include='*.md' --include='*.py' \
-F "$old_pattern" . \
--exclude-dir='.claude' --exclude-dir='node_modules' --exclude-dir='shared_static' \
--exclude-dir='.claude' --exclude-dir='node_modules' --exclude-dir="$old_pattern" \
2>/dev/null || true)
for f in $files; do
sed -i "s|${old_pattern}|${new_pattern}|g" "$f"
+119 -119
View File
@@ -74,9 +74,9 @@
}
},
"node_modules/@oxc-project/types": {
"version": "0.129.0",
"resolved": "https://registry.npmjs.org/@oxc-project/types/-/types-0.129.0.tgz",
"integrity": "sha512-3oz8m3FGdr2nDXVqmFUw7jolKliC4MoyXYIG2c7gpjBnzUWQpUGIYcXYKxTdTi+N2jusvt610ckTMkxdwHkYEg==",
"version": "0.130.0",
"resolved": "https://registry.npmjs.org/@oxc-project/types/-/types-0.130.0.tgz",
"integrity": "sha512-ibD2usx9JRu7f5pu2tMKMI4cpA4NgXJQoYRP4pQ7Pxmn1l6k/53qWtQWZayhYy3X4QZkt90Ot+mJEaeXouio6Q==",
"dev": true,
"license": "MIT",
"funding": {
@@ -84,9 +84,9 @@
}
},
"node_modules/@rolldown/binding-android-arm64": {
"version": "1.0.0",
"resolved": "https://registry.npmjs.org/@rolldown/binding-android-arm64/-/binding-android-arm64-1.0.0.tgz",
"integrity": "sha512-TWMZnRLMe63C2Lhyicviu7ZHaU4kxa6PS3rofvc9GmcvptzNN11BcfQ4Sl7MwTOsisQoa2keB/EBdNCAnUo8vA==",
"version": "1.0.1",
"resolved": "https://registry.npmjs.org/@rolldown/binding-android-arm64/-/binding-android-arm64-1.0.1.tgz",
"integrity": "sha512-fJI3I0r3C3Oj/zdBCpaCmBRZYf07xpaq4yCfDDoSFm+beWNzbIl26puW8RraUdugoJw/95zerNOn6jasAhzSmg==",
"cpu": [
"arm64"
],
@@ -101,9 +101,9 @@
}
},
"node_modules/@rolldown/binding-darwin-arm64": {
"version": "1.0.0",
"resolved": "https://registry.npmjs.org/@rolldown/binding-darwin-arm64/-/binding-darwin-arm64-1.0.0.tgz",
"integrity": "sha512-6XcD+8k0gPVItNagEw78/qqcBDwKcwDYS8V2hRmVsfUSIrd8cWe/CBvRDI5toqFyPfj+FJr6t8U6Xj2P2prEew==",
"version": "1.0.1",
"resolved": "https://registry.npmjs.org/@rolldown/binding-darwin-arm64/-/binding-darwin-arm64-1.0.1.tgz",
"integrity": "sha512-cKnAhWEsV7TPcA/5EAteDp6KcJZBQ2G+BqE7zayMMi7kMvwRsbv7WT9aOnn0WNl4SKEIf43vjS31iUPu80nzXg==",
"cpu": [
"arm64"
],
@@ -118,9 +118,9 @@
}
},
"node_modules/@rolldown/binding-darwin-x64": {
"version": "1.0.0",
"resolved": "https://registry.npmjs.org/@rolldown/binding-darwin-x64/-/binding-darwin-x64-1.0.0.tgz",
"integrity": "sha512-iN/tWVXRQDWvmZlKdceP1Dwug9GDpEymhb9p4xnEe6zvCg5lFmzVljl+1qR1NVx3yfGpr2Na+CuLmv5IU8uzfQ==",
"version": "1.0.1",
"resolved": "https://registry.npmjs.org/@rolldown/binding-darwin-x64/-/binding-darwin-x64-1.0.1.tgz",
"integrity": "sha512-YKrVwQjIRBPo+5G/u03wGjbdy4q7pyzCe93DK9VJ7zkVmeg8LJ7GbgsiHWdR4xSoe4CAXRD7Bcjgbtr64bkXNg==",
"cpu": [
"x64"
],
@@ -135,9 +135,9 @@
}
},
"node_modules/@rolldown/binding-freebsd-x64": {
"version": "1.0.0",
"resolved": "https://registry.npmjs.org/@rolldown/binding-freebsd-x64/-/binding-freebsd-x64-1.0.0.tgz",
"integrity": "sha512-jjQMDvvwSOuhOwMszD/klSOjyWMM3zI64hWTj9KT5x4MxRbZAf+7vLQ6qouRhtsLVFHr3f0ILaJAfgENPiQdAQ==",
"version": "1.0.1",
"resolved": "https://registry.npmjs.org/@rolldown/binding-freebsd-x64/-/binding-freebsd-x64-1.0.1.tgz",
"integrity": "sha512-z/oBsREo46SsFqBwYtFe0kpJeBijAT48O/WXLI4suiCLBkr03RTtTJMCzSdDd2znlh8VJizL09XVkQgk8IZonw==",
"cpu": [
"x64"
],
@@ -152,9 +152,9 @@
}
},
"node_modules/@rolldown/binding-linux-arm-gnueabihf": {
"version": "1.0.0",
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-arm-gnueabihf/-/binding-linux-arm-gnueabihf-1.0.0.tgz",
"integrity": "sha512-d//Dtg2x6/m3mbV64yUGNnDGNZaDGRpDLLNGerHQUVObuNaIQaaDp25yUiqGXtHEXX+NP2d0wAlmKgpYgIAJ2A==",
"version": "1.0.1",
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-arm-gnueabihf/-/binding-linux-arm-gnueabihf-1.0.1.tgz",
"integrity": "sha512-ik8q7GM11zxvYxFc2PeDcT6TBvhCQMaUxfph/M5l9sKuTs/Sjg3L+Byw0F7w0ZVLBZmx30P+gG0ECzzN+MFcmQ==",
"cpu": [
"arm"
],
@@ -169,9 +169,9 @@
}
},
"node_modules/@rolldown/binding-linux-arm64-gnu": {
"version": "1.0.0",
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-arm64-gnu/-/binding-linux-arm64-gnu-1.0.0.tgz",
"integrity": "sha512-n7Ofp0mx+aB2cC+Sdy5YtMnXtY9lchnHbY+3Yt0uq9JsWQExf4f5Whu0tK0R8Jdc9S6RchTHjIFY7uc92puOVQ==",
"version": "1.0.1",
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-arm64-gnu/-/binding-linux-arm64-gnu-1.0.1.tgz",
"integrity": "sha512-QoSx2EkyrrdZ6kcyE8stqZ62t0Yra8Fs5ia9lOxJrh6TMQJK7gQKmscdTHf7pOXKREKrVwOtJcQG3qVSfc866A==",
"cpu": [
"arm64"
],
@@ -189,9 +189,9 @@
}
},
"node_modules/@rolldown/binding-linux-arm64-musl": {
"version": "1.0.0",
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-arm64-musl/-/binding-linux-arm64-musl-1.0.0.tgz",
"integrity": "sha512-EIVjy2cgd7uuMMo94FVkBp7F6DhcZAUwNURkSG3RwUmvAXR6s0ISxM81U+IydcZByPG0pZIHsf1b6kTxoFDgJA==",
"version": "1.0.1",
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-arm64-musl/-/binding-linux-arm64-musl-1.0.1.tgz",
"integrity": "sha512-uwNwFpwKeNiZawfAWBgg0VIztPTV3ihhh1vV334h9ivnNLorxnQMU6Fz8wG1Zb4Qh9LC1/MkcyT3YlDXG3Rsgg==",
"cpu": [
"arm64"
],
@@ -209,9 +209,9 @@
}
},
"node_modules/@rolldown/binding-linux-ppc64-gnu": {
"version": "1.0.0",
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-ppc64-gnu/-/binding-linux-ppc64-gnu-1.0.0.tgz",
"integrity": "sha512-JEwwOPcwTLAcpDQlqSmjEmfs63xJnSiUNIGvLcDLUHCWK4XowpS/7c7tUsUH6uT/ct6bMUTdXKfI8967FYj6mg==",
"version": "1.0.1",
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-ppc64-gnu/-/binding-linux-ppc64-gnu-1.0.1.tgz",
"integrity": "sha512-zY1bul7OWr7DFBiJ++wofXvnr8B45ce3QsQUhKrIhXsygAh7bTkwyeM1bi1a2g5C/yC/N8TZyGDEoMfm/l9mpg==",
"cpu": [
"ppc64"
],
@@ -229,9 +229,9 @@
}
},
"node_modules/@rolldown/binding-linux-s390x-gnu": {
"version": "1.0.0",
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-s390x-gnu/-/binding-linux-s390x-gnu-1.0.0.tgz",
"integrity": "sha512-0wjCFhLrihtAubnT9iA0N++0pSV0z5Hg7tNGdNJ4RFaINceHadoF+kiFGyY1qSSNVIAZtLotG8Ju1bgDPkjnFA==",
"version": "1.0.1",
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-s390x-gnu/-/binding-linux-s390x-gnu-1.0.1.tgz",
"integrity": "sha512-0frlsT/f4Ft6I7SMESTKnF3cZsdicQn1dCMkF/jT9wDLE+gGoiQfv1nmT9e+s7s/fekvvy6tZM2jHvI2tkbJDQ==",
"cpu": [
"s390x"
],
@@ -249,9 +249,9 @@
}
},
"node_modules/@rolldown/binding-linux-x64-gnu": {
"version": "1.0.0",
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-x64-gnu/-/binding-linux-x64-gnu-1.0.0.tgz",
"integrity": "sha512-Dfn7iak9BcMMePxcoJfpSbWqnEyrp/dRF63/8qW/eHBdOZov6x5aShLLEYGYdIeSJ6vMLK/XCVB+lGIxm41bQA==",
"version": "1.0.1",
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-x64-gnu/-/binding-linux-x64-gnu-1.0.1.tgz",
"integrity": "sha512-XABVmGp9Tg0WspTVvwduTc4fpqy6JnAUrSQe6OuyqD/03nI7r0O9OWUkMIwFrjKAIqolvqoA4ZrJppgwE0Gxmw==",
"cpu": [
"x64"
],
@@ -269,9 +269,9 @@
}
},
"node_modules/@rolldown/binding-linux-x64-musl": {
"version": "1.0.0",
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-x64-musl/-/binding-linux-x64-musl-1.0.0.tgz",
"integrity": "sha512-5/utzzDmD/pD/bmuaUcbTf/sZYy0aztwIVlfpoW1fTjCZ0BaPOMVWGZL1zvgxyi7ZIVYWlxKONHmSbHuiOh8Jw==",
"version": "1.0.1",
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-x64-musl/-/binding-linux-x64-musl-1.0.1.tgz",
"integrity": "sha512-bV4fzswuzVcKD90o/VM6QqKxnxlDq0g2BISDLNVmxrnhpv1DDbyPhCIjYfvzYLV+MvkKKnQt2Q6AO86SEBULUQ==",
"cpu": [
"x64"
],
@@ -289,9 +289,9 @@
}
},
"node_modules/@rolldown/binding-openharmony-arm64": {
"version": "1.0.0",
"resolved": "https://registry.npmjs.org/@rolldown/binding-openharmony-arm64/-/binding-openharmony-arm64-1.0.0.tgz",
"integrity": "sha512-ouJs8VcUomfLfpbUECqFMRqdV4x6aeAK3MA4m6vTrJJjKyWTV5KnxZx7Jd9G+GlDaQQxubcba00x16OyJ1meig==",
"version": "1.0.1",
"resolved": "https://registry.npmjs.org/@rolldown/binding-openharmony-arm64/-/binding-openharmony-arm64-1.0.1.tgz",
"integrity": "sha512-/Mh0Zhq3OP7fVs0kcQHZP6lZEthMGTaSf8UBQYSFEZDWGXXlEC+nJ6EqenaK2t4LBXMe3A+K/G2BVXXdtOr4PQ==",
"cpu": [
"arm64"
],
@@ -306,9 +306,9 @@
}
},
"node_modules/@rolldown/binding-wasm32-wasi": {
"version": "1.0.0",
"resolved": "https://registry.npmjs.org/@rolldown/binding-wasm32-wasi/-/binding-wasm32-wasi-1.0.0.tgz",
"integrity": "sha512-E+oHKGiDA+lsKMmFtffDDw91EryDT7uJocrIuCHqhm6bCTM6xFK+3gaCkYOHfPwQr0cCNarSM2xaELoQDz9jJg==",
"version": "1.0.1",
"resolved": "https://registry.npmjs.org/@rolldown/binding-wasm32-wasi/-/binding-wasm32-wasi-1.0.1.tgz",
"integrity": "sha512-+1xc9X45l8ufsBAm6Gjvx2qDRIY9lTVt0cgWNcJ+1gdhXvkbxePA60yRTwSTuXL09CMhyJmjpV7E3NoyxbqFQQ==",
"cpu": [
"wasm32"
],
@@ -325,9 +325,9 @@
}
},
"node_modules/@rolldown/binding-win32-arm64-msvc": {
"version": "1.0.0",
"resolved": "https://registry.npmjs.org/@rolldown/binding-win32-arm64-msvc/-/binding-win32-arm64-msvc-1.0.0.tgz",
"integrity": "sha512-yYK02n8Rngo+gbm1y6G0+7jk1sJ/2Wt7K0me0Y7k/ErBpyf+LJ2gFpqWVTcRV1rUepBlQRmpgWkTQCiiwrK0Ow==",
"version": "1.0.1",
"resolved": "https://registry.npmjs.org/@rolldown/binding-win32-arm64-msvc/-/binding-win32-arm64-msvc-1.0.1.tgz",
"integrity": "sha512-1D+UqZdfnuR+Jy1GgMJwi85bD40H21uNmOPRWQhw4oRSuolZ/B5rixZ45DK2KXOTCvmVCecauWgEhbw8bI7tOw==",
"cpu": [
"arm64"
],
@@ -342,9 +342,9 @@
}
},
"node_modules/@rolldown/binding-win32-x64-msvc": {
"version": "1.0.0",
"resolved": "https://registry.npmjs.org/@rolldown/binding-win32-x64-msvc/-/binding-win32-x64-msvc-1.0.0.tgz",
"integrity": "sha512-14bpChMahXRRXiTwahSl+zzHPW6qQTXtkMuJBFlbo+pqSAews2d4BdCSHfrJ/MBsCZtpmTafsY+1QhBzitcmdg==",
"version": "1.0.1",
"resolved": "https://registry.npmjs.org/@rolldown/binding-win32-x64-msvc/-/binding-win32-x64-msvc-1.0.1.tgz",
"integrity": "sha512-INAycaWuhlOK3wk4mRHGsdgwYWmd9cChdPdE9bwWmy6rn9VqVNYNFGhOdXrofXUxwHIncSiPNb8tNm8knDVIeQ==",
"cpu": [
"x64"
],
@@ -359,9 +359,9 @@
}
},
"node_modules/@rolldown/pluginutils": {
"version": "1.0.0",
"resolved": "https://registry.npmjs.org/@rolldown/pluginutils/-/pluginutils-1.0.0.tgz",
"integrity": "sha512-aKs/3GSWyV0mrhNmt/96/Z3yczC3yvrzYATCiCXQebBsGyYzjNdUphRVLeJQ67ySKVXRfMxt2lm12pmXvbPFQQ==",
"version": "1.0.1",
"resolved": "https://registry.npmjs.org/@rolldown/pluginutils/-/pluginutils-1.0.1.tgz",
"integrity": "sha512-2j9bGt5Jh8hj+vPtgzPtl72j0yRxHAyumoo6TNfAjsLB04UtpSvPbPcDcBMxz7n+9CYB0c1GxQFxYRg2jimqGw==",
"dev": true,
"license": "MIT"
},
@@ -409,16 +409,16 @@
"license": "MIT"
},
"node_modules/@vitest/expect": {
"version": "4.1.5",
"resolved": "https://registry.npmjs.org/@vitest/expect/-/expect-4.1.5.tgz",
"integrity": "sha512-PWBaRY5JoKuRnHlUHfpV/KohFylaDZTupcXN1H9vYryNLOnitSw60Mw9IAE2r67NbwwzBw/Cc/8q9BK3kIX8Kw==",
"version": "4.1.6",
"resolved": "https://registry.npmjs.org/@vitest/expect/-/expect-4.1.6.tgz",
"integrity": "sha512-7EHDquPthALSV0jhhjgEW8FXaviMx7rSqu8W6oqCoAuOhKov814P99QDV1pxMA3QPv21YudvJngIhjrNI4opLg==",
"dev": true,
"license": "MIT",
"dependencies": {
"@standard-schema/spec": "^1.1.0",
"@types/chai": "^5.2.2",
"@vitest/spy": "4.1.5",
"@vitest/utils": "4.1.5",
"@vitest/spy": "4.1.6",
"@vitest/utils": "4.1.6",
"chai": "^6.2.2",
"tinyrainbow": "^3.1.0"
},
@@ -427,13 +427,13 @@
}
},
"node_modules/@vitest/mocker": {
"version": "4.1.5",
"resolved": "https://registry.npmjs.org/@vitest/mocker/-/mocker-4.1.5.tgz",
"integrity": "sha512-/x2EmFC4mT4NNzqvC3fmesuV97w5FC903KPmey4gsnJiMQ3Be1IlDKVaDaG8iqaLFHqJ2FVEkxZk5VmeLjIItw==",
"version": "4.1.6",
"resolved": "https://registry.npmjs.org/@vitest/mocker/-/mocker-4.1.6.tgz",
"integrity": "sha512-MCFc63czMjEInOlcY2cpQCvCN+KgbAn+60xu9cMgP4sKaLC5JNAKw7JH8QdAnoAC88hW1IiSNZ+GgVXlN1UcMQ==",
"dev": true,
"license": "MIT",
"dependencies": {
"@vitest/spy": "4.1.5",
"@vitest/spy": "4.1.6",
"estree-walker": "^3.0.3",
"magic-string": "^0.30.21"
},
@@ -454,9 +454,9 @@
}
},
"node_modules/@vitest/pretty-format": {
"version": "4.1.5",
"resolved": "https://registry.npmjs.org/@vitest/pretty-format/-/pretty-format-4.1.5.tgz",
"integrity": "sha512-7I3q6l5qr03dVfMX2wCo9FxwSJbPdwKjy2uu/YPpU3wfHvIL4QHwVRp57OfGrDFeUJ8/8QdfBKIV12FTtLn00g==",
"version": "4.1.6",
"resolved": "https://registry.npmjs.org/@vitest/pretty-format/-/pretty-format-4.1.6.tgz",
"integrity": "sha512-h5SxD/IzNhZYnrSZRsUZQIC+vD0GY8cUvq0iwsmkFKixRCKLLWqCXa/FIQ4S1R+sI+PGoojkHsdNrbZiM9Qpgw==",
"dev": true,
"license": "MIT",
"dependencies": {
@@ -467,13 +467,13 @@
}
},
"node_modules/@vitest/runner": {
"version": "4.1.5",
"resolved": "https://registry.npmjs.org/@vitest/runner/-/runner-4.1.5.tgz",
"integrity": "sha512-2D+o7Pr82IEO46YPpoA/YU0neeyr6FTerQb5Ro7BUnBuv6NQtT/kmVnczngiMEBhzgqz2UZYl5gArejsyERDSQ==",
"version": "4.1.6",
"resolved": "https://registry.npmjs.org/@vitest/runner/-/runner-4.1.6.tgz",
"integrity": "sha512-nOPCmn2+yD0ZNmKdsXGv/UxMMWbMuKeD6GyYncNwdkYDxpQvrPSKYj2rWuDjC2Y4b6w6hjip5dBKFzEUuZe3vA==",
"dev": true,
"license": "MIT",
"dependencies": {
"@vitest/utils": "4.1.5",
"@vitest/utils": "4.1.6",
"pathe": "^2.0.3"
},
"funding": {
@@ -481,14 +481,14 @@
}
},
"node_modules/@vitest/snapshot": {
"version": "4.1.5",
"resolved": "https://registry.npmjs.org/@vitest/snapshot/-/snapshot-4.1.5.tgz",
"integrity": "sha512-zypXEt4KH/XgKGPUz4eC2AvErYx0My5hfL8oDb1HzGFpEk1P62bxSohdyOmvz+d9UJwanI68MKwr2EquOaOgMQ==",
"version": "4.1.6",
"resolved": "https://registry.npmjs.org/@vitest/snapshot/-/snapshot-4.1.6.tgz",
"integrity": "sha512-YhsdE6xAVfTDmzjxL2ZDUvjj+ZsgyOKe+TdQzqkD72wIOmHka8NuGQ6NpTNZv9D2Z63fbwWKJPeVpEw4EQgYxw==",
"dev": true,
"license": "MIT",
"dependencies": {
"@vitest/pretty-format": "4.1.5",
"@vitest/utils": "4.1.5",
"@vitest/pretty-format": "4.1.6",
"@vitest/utils": "4.1.6",
"magic-string": "^0.30.21",
"pathe": "^2.0.3"
},
@@ -497,9 +497,9 @@
}
},
"node_modules/@vitest/spy": {
"version": "4.1.5",
"resolved": "https://registry.npmjs.org/@vitest/spy/-/spy-4.1.5.tgz",
"integrity": "sha512-2lNOsh6+R2Idnf1TCZqSwYlKN2E/iDlD8sgU59kYVl+OMDmvldO1VDk39smRfpUNwYpNRVn3w4YfuC7KfbBnkQ==",
"version": "4.1.6",
"resolved": "https://registry.npmjs.org/@vitest/spy/-/spy-4.1.6.tgz",
"integrity": "sha512-JFKxMx6udhwKh/Ldo270e17QX710vgunMkuPAvXjHSvC6oqLWAHhVhjg/I71q0u0CBSErIODV1Kjv0FQNSWjdg==",
"dev": true,
"license": "MIT",
"funding": {
@@ -507,13 +507,13 @@
}
},
"node_modules/@vitest/utils": {
"version": "4.1.5",
"resolved": "https://registry.npmjs.org/@vitest/utils/-/utils-4.1.5.tgz",
"integrity": "sha512-76wdkrmfXfqGjueGgnb45ITPyUi1ycZ4IHgC2bhPDUfWHklY/q3MdLOAB+TF1e6xfl8NxNY0ZYaPCFNWSsw3Ug==",
"version": "4.1.6",
"resolved": "https://registry.npmjs.org/@vitest/utils/-/utils-4.1.6.tgz",
"integrity": "sha512-FxIY+U81R3LGKCxaHHFRQ5+g6/iRgGLmeHWdp2Amj4ljQRrEIWHmZyDfDYBRZlpyqA7qKxtS9DD1dhk8RnRIVQ==",
"dev": true,
"license": "MIT",
"dependencies": {
"@vitest/pretty-format": "4.1.5",
"@vitest/pretty-format": "4.1.6",
"convert-source-map": "^2.0.0",
"tinyrainbow": "^3.1.0"
},
@@ -988,14 +988,14 @@
}
},
"node_modules/rolldown": {
"version": "1.0.0",
"resolved": "https://registry.npmjs.org/rolldown/-/rolldown-1.0.0.tgz",
"integrity": "sha512-yD986aXDESFGS95spT1LAv0jssywP4npMEjmMHyN2/5+eE8qQJUype2AaKkRiLgBgyD0LFlubwAht7VmY8rGoA==",
"version": "1.0.1",
"resolved": "https://registry.npmjs.org/rolldown/-/rolldown-1.0.1.tgz",
"integrity": "sha512-X0KQHljNnEkWNqqiz9zJrGunh1B0HgOxLXvnFpCOcadzcy5qohZ3tqMEUg00vncoRovXuK3ZqCT9KnnKzoInFQ==",
"dev": true,
"license": "MIT",
"dependencies": {
"@oxc-project/types": "=0.129.0",
"@rolldown/pluginutils": "1.0.0"
"@oxc-project/types": "=0.130.0",
"@rolldown/pluginutils": "^1.0.0"
},
"bin": {
"rolldown": "bin/cli.mjs"
@@ -1004,21 +1004,21 @@
"node": "^20.19.0 || >=22.12.0"
},
"optionalDependencies": {
"@rolldown/binding-android-arm64": "1.0.0",
"@rolldown/binding-darwin-arm64": "1.0.0",
"@rolldown/binding-darwin-x64": "1.0.0",
"@rolldown/binding-freebsd-x64": "1.0.0",
"@rolldown/binding-linux-arm-gnueabihf": "1.0.0",
"@rolldown/binding-linux-arm64-gnu": "1.0.0",
"@rolldown/binding-linux-arm64-musl": "1.0.0",
"@rolldown/binding-linux-ppc64-gnu": "1.0.0",
"@rolldown/binding-linux-s390x-gnu": "1.0.0",
"@rolldown/binding-linux-x64-gnu": "1.0.0",
"@rolldown/binding-linux-x64-musl": "1.0.0",
"@rolldown/binding-openharmony-arm64": "1.0.0",
"@rolldown/binding-wasm32-wasi": "1.0.0",
"@rolldown/binding-win32-arm64-msvc": "1.0.0",
"@rolldown/binding-win32-x64-msvc": "1.0.0"
"@rolldown/binding-android-arm64": "1.0.1",
"@rolldown/binding-darwin-arm64": "1.0.1",
"@rolldown/binding-darwin-x64": "1.0.1",
"@rolldown/binding-freebsd-x64": "1.0.1",
"@rolldown/binding-linux-arm-gnueabihf": "1.0.1",
"@rolldown/binding-linux-arm64-gnu": "1.0.1",
"@rolldown/binding-linux-arm64-musl": "1.0.1",
"@rolldown/binding-linux-ppc64-gnu": "1.0.1",
"@rolldown/binding-linux-s390x-gnu": "1.0.1",
"@rolldown/binding-linux-x64-gnu": "1.0.1",
"@rolldown/binding-linux-x64-musl": "1.0.1",
"@rolldown/binding-openharmony-arm64": "1.0.1",
"@rolldown/binding-wasm32-wasi": "1.0.1",
"@rolldown/binding-win32-arm64-msvc": "1.0.1",
"@rolldown/binding-win32-x64-msvc": "1.0.1"
}
},
"node_modules/siginfo": {
@@ -1119,16 +1119,16 @@
}
},
"node_modules/vite": {
"version": "8.0.12",
"resolved": "https://registry.npmjs.org/vite/-/vite-8.0.12.tgz",
"integrity": "sha512-w2dDofOWv2QB09ZITZBsvKTVAlYvPR4IAmrY/v0ir9KvLs0xybR7i48wxhM1/oyBWO34wPns+bPGw5ZrZqDpZg==",
"version": "8.0.13",
"resolved": "https://registry.npmjs.org/vite/-/vite-8.0.13.tgz",
"integrity": "sha512-MFtjBYgzmSxmgA4RAfjIyXWpGe1oALnjgUTzzV7QLx/TKxCzjtMH6Fd9/eVK+5Fg1qNoz5VAwsmMs/NofrmJvw==",
"dev": true,
"license": "MIT",
"dependencies": {
"lightningcss": "^1.32.0",
"picomatch": "^4.0.4",
"postcss": "^8.5.14",
"rolldown": "1.0.0",
"rolldown": "1.0.1",
"tinyglobby": "^0.2.16"
},
"bin": {
@@ -1197,19 +1197,19 @@
}
},
"node_modules/vitest": {
"version": "4.1.5",
"resolved": "https://registry.npmjs.org/vitest/-/vitest-4.1.5.tgz",
"integrity": "sha512-9Xx1v3/ih3m9hN+SbfkUyy0JAs72ap3r7joc87XL6jwF0jGg6mFBvQ1SrwaX+h8BlkX6Hz9shdd1uo6AF+ZGpg==",
"version": "4.1.6",
"resolved": "https://registry.npmjs.org/vitest/-/vitest-4.1.6.tgz",
"integrity": "sha512-6lvjbS3p9b4CrdCmguzbh2/4uoXhGE2q71R4OX5sqF9R1bo9Xd6fGrMAfvp5wnCzlBnFVdCOp6onuTQVbo8iUQ==",
"dev": true,
"license": "MIT",
"dependencies": {
"@vitest/expect": "4.1.5",
"@vitest/mocker": "4.1.5",
"@vitest/pretty-format": "4.1.5",
"@vitest/runner": "4.1.5",
"@vitest/snapshot": "4.1.5",
"@vitest/spy": "4.1.5",
"@vitest/utils": "4.1.5",
"@vitest/expect": "4.1.6",
"@vitest/mocker": "4.1.6",
"@vitest/pretty-format": "4.1.6",
"@vitest/runner": "4.1.6",
"@vitest/snapshot": "4.1.6",
"@vitest/spy": "4.1.6",
"@vitest/utils": "4.1.6",
"es-module-lexer": "^2.0.0",
"expect-type": "^1.3.0",
"magic-string": "^0.30.21",
@@ -1237,12 +1237,12 @@
"@edge-runtime/vm": "*",
"@opentelemetry/api": "^1.9.0",
"@types/node": "^20.0.0 || ^22.0.0 || >=24.0.0",
"@vitest/browser-playwright": "4.1.5",
"@vitest/browser-preview": "4.1.5",
"@vitest/browser-webdriverio": "4.1.5",
"@vitest/coverage-istanbul": "4.1.5",
"@vitest/coverage-v8": "4.1.5",
"@vitest/ui": "4.1.5",
"@vitest/browser-playwright": "4.1.6",
"@vitest/browser-preview": "4.1.6",
"@vitest/browser-webdriverio": "4.1.6",
"@vitest/coverage-istanbul": "4.1.6",
"@vitest/coverage-v8": "4.1.6",
"@vitest/ui": "4.1.6",
"happy-dom": "*",
"jsdom": "*",
"vite": "^6.0.0 || ^7.0.0 || ^8.0.0"
+220
View File
@@ -0,0 +1,220 @@
"""Tests for turnstone-admin DB configuration precedence.
Locks in the alignment with turnstone-server:
CLI / config.toml [database] > TURNSTONE_DB_* env > hardcoded default
The motivation is to keep DB secrets in config.toml (see
feedback_secrets_not_in_env) rather than forcing operators to export
TURNSTONE_DB_URL before every admin invocation.
"""
from __future__ import annotations
import argparse
from typing import TYPE_CHECKING
from unittest.mock import patch
import pytest
if TYPE_CHECKING:
from collections.abc import Iterator
from pathlib import Path
import turnstone.core.config as config_mod
from turnstone.admin import _get_storage
def _reset_cache() -> None:
config_mod._cache = None
config_mod._config_path = None
def _build_args(config_path: str | None) -> argparse.Namespace:
"""Build an args namespace the way admin.main() does.
Skips ``add_config_arg`` (which reads ``sys.argv``) the test
constructs the args programmatically instead.
"""
config_mod.set_config_path(config_path or "/nonexistent/turnstone-admin-test.toml")
parser = argparse.ArgumentParser()
config_mod.apply_config(parser, ["database"])
sub = parser.add_subparsers(dest="command")
sub.add_parser("list-users")
return parser.parse_args(["list-users"])
@pytest.fixture(autouse=True)
def _clear_db_env(monkeypatch: pytest.MonkeyPatch) -> Iterator[None]:
"""Clean slate: no TURNSTONE_DB_* env vars unless a test sets them."""
for var in (
"TURNSTONE_DB_BACKEND",
"TURNSTONE_DB_URL",
"TURNSTONE_DB_PATH",
"TURNSTONE_DB_POOL_SIZE",
"TURNSTONE_DB_SSLMODE",
"TURNSTONE_DB_SSLROOTCERT",
"TURNSTONE_DB_SSLCERT",
"TURNSTONE_DB_SSLKEY",
"TURNSTONE_CONFIG",
):
monkeypatch.delenv(var, raising=False)
_reset_cache()
yield
_reset_cache()
def test_defaults_to_sqlite_when_neither_config_nor_env_set() -> None:
args = _build_args(None)
with patch("turnstone.core.storage.init_storage") as init:
_get_storage(args)
assert init.call_args.args == ("sqlite",)
assert init.call_args.kwargs["url"] == ""
assert init.call_args.kwargs["path"] == ""
assert init.call_args.kwargs["pool_size"] == 2
def test_config_toml_database_section_drives_init_storage(tmp_path: Path) -> None:
cfg = tmp_path / "config.toml"
cfg.write_text(
"[database]\n"
'backend = "postgresql"\n'
'url = "postgresql+psycopg://fromconfig:x@host/db"\n'
"pool_size = 5\n"
'sslmode = "verify-full"\n'
'sslrootcert = "/etc/ssl/ca.pem"\n'
'sslcert = "/etc/ssl/client.pem"\n'
'sslkey = "/etc/ssl/client.key"\n'
)
args = _build_args(str(cfg))
with patch("turnstone.core.storage.init_storage") as init:
_get_storage(args)
assert init.call_args.args == ("postgresql",)
kw = init.call_args.kwargs
assert kw["url"] == "postgresql+psycopg://fromconfig:x@host/db"
assert kw["pool_size"] == 5
assert kw["sslmode"] == "verify-full"
assert kw["sslrootcert"] == "/etc/ssl/ca.pem"
assert kw["sslcert"] == "/etc/ssl/client.pem"
assert kw["sslkey"] == "/etc/ssl/client.key"
def test_env_used_as_fallback_when_config_absent(monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.setenv("TURNSTONE_DB_BACKEND", "postgresql")
monkeypatch.setenv("TURNSTONE_DB_URL", "postgresql+psycopg://fromenv:x@host/db")
monkeypatch.setenv("TURNSTONE_DB_POOL_SIZE", "7")
monkeypatch.setenv("TURNSTONE_DB_SSLMODE", "require")
args = _build_args(None)
with patch("turnstone.core.storage.init_storage") as init:
_get_storage(args)
assert init.call_args.args == ("postgresql",)
kw = init.call_args.kwargs
assert kw["url"] == "postgresql+psycopg://fromenv:x@host/db"
assert kw["pool_size"] == 7
assert kw["sslmode"] == "require"
def test_config_toml_wins_over_env(monkeypatch: pytest.MonkeyPatch, tmp_path: Path) -> None:
"""config.toml beats env — operators should put secrets in TOML."""
monkeypatch.setenv("TURNSTONE_DB_BACKEND", "sqlite")
monkeypatch.setenv("TURNSTONE_DB_URL", "postgresql+psycopg://fromenv:x@host/db")
monkeypatch.setenv("TURNSTONE_DB_SSLMODE", "require")
cfg = tmp_path / "config.toml"
cfg.write_text(
"[database]\n"
'backend = "postgresql"\n'
'url = "postgresql+psycopg://fromconfig:x@host/db"\n'
'sslmode = "verify-full"\n'
)
args = _build_args(str(cfg))
with patch("turnstone.core.storage.init_storage") as init:
_get_storage(args)
assert init.call_args.args == ("postgresql",)
kw = init.call_args.kwargs
assert kw["url"] == "postgresql+psycopg://fromconfig:x@host/db"
assert kw["sslmode"] == "verify-full"
def test_partial_config_falls_through_to_env_per_key(
monkeypatch: pytest.MonkeyPatch, tmp_path: Path
) -> None:
"""A key missing from [database] should fall back to its env var."""
monkeypatch.setenv("TURNSTONE_DB_SSLMODE", "require")
monkeypatch.setenv("TURNSTONE_DB_POOL_SIZE", "9")
cfg = tmp_path / "config.toml"
cfg.write_text(
'[database]\nbackend = "postgresql"\nurl = "postgresql+psycopg://fromconfig:x@host/db"\n'
)
args = _build_args(str(cfg))
with patch("turnstone.core.storage.init_storage") as init:
_get_storage(args)
kw = init.call_args.kwargs
assert kw["url"] == "postgresql+psycopg://fromconfig:x@host/db"
assert kw["sslmode"] == "require"
assert kw["pool_size"] == 9
def test_empty_string_in_config_beats_env(monkeypatch: pytest.MonkeyPatch, tmp_path: Path) -> None:
"""`url = ""` in config.toml beats an env var.
Locks in the `is not None` guard a falsy-but-present TOML value
should NOT silently fall through to the env fallback.
"""
monkeypatch.setenv("TURNSTONE_DB_URL", "postgresql+psycopg://fromenv:x@host/db")
cfg = tmp_path / "config.toml"
cfg.write_text('[database]\nbackend = "sqlite"\nurl = ""\n')
args = _build_args(str(cfg))
with patch("turnstone.core.storage.init_storage") as init:
_get_storage(args)
assert init.call_args.kwargs["url"] == ""
def test_main_threads_config_toml_through_real_argv(
monkeypatch: pytest.MonkeyPatch, tmp_path: Path
) -> None:
"""End-to-end: ``turnstone-admin --config <toml> list-users`` honors TOML.
Covers the ``add_config_arg`` -> ``apply_config`` -> ``_get_storage``
chain that the programmatic ``_build_args`` helper skips.
"""
cfg = tmp_path / "config.toml"
cfg.write_text(
'[database]\nbackend = "postgresql"\nurl = "postgresql+psycopg://fromcli:x@host/db"\n'
)
monkeypatch.setattr("sys.argv", ["turnstone-admin", "--config", str(cfg), "list-users"])
fake_storage = patch("turnstone.core.storage.init_storage").start()
fake_storage.return_value.list_users.return_value = []
try:
from turnstone.admin import main
main()
finally:
patch.stopall()
assert fake_storage.call_args.args == ("postgresql",)
assert fake_storage.call_args.kwargs["url"] == "postgresql+psycopg://fromcli:x@host/db"
def test_get_storage_initializes_real_sqlite_backend(tmp_path: Path) -> None:
"""Drives the real ``init_storage`` boundary on a fresh sqlite file.
Mock-only tests would miss a kwarg-name typo (sslmode -> ssl_mode).
This test trips on any such drift because Alembic + the backend
actually run.
"""
from turnstone.core.storage import reset_storage
db_file = tmp_path / "admin.db"
cfg = tmp_path / "config.toml"
cfg.write_text(f'[database]\nbackend = "sqlite"\npath = "{db_file}"\n')
args = _build_args(str(cfg))
reset_storage()
try:
storage = _get_storage(args)
assert storage.list_users() == []
finally:
reset_storage()
+35
View File
@@ -50,6 +50,41 @@ def test_load_config_invalid_toml(tmp_path):
assert load_config() == {}
def test_load_config_warns_when_world_readable(tmp_path, caplog):
"""Secrets in config.toml — warn if anyone but the owner can read it."""
import logging
import os
_reset_cache()
cfg = tmp_path / "config.toml"
cfg.write_text('[database]\nurl = "postgresql+psycopg://u:secret@h/d"\n')
os.chmod(cfg, 0o644)
set_config_path(str(cfg))
with caplog.at_level(logging.WARNING, logger="turnstone.core.config"):
load_config()
messages = [r.getMessage() for r in caplog.records]
assert any("group/world-readable" in m for m in messages)
def test_load_config_quiet_when_mode_0600(tmp_path, caplog):
import logging
import os
_reset_cache()
cfg = tmp_path / "config.toml"
cfg.write_text('[database]\nurl = "postgresql+psycopg://u:secret@h/d"\n')
os.chmod(cfg, 0o600)
set_config_path(str(cfg))
with caplog.at_level(logging.WARNING, logger="turnstone.core.config"):
load_config()
messages = [r.getMessage() for r in caplog.records]
assert not any("group/world-readable" in m for m in messages)
def test_load_config_caches(tmp_path):
_reset_cache()
cfg = tmp_path / "config.toml"
+402
View File
@@ -1228,6 +1228,49 @@ def test_list_skills_hides_interactive_only_skills(tmp_path):
assert skill["kind"] in {"coordinator", "any"}
def test_list_skills_omits_allowed_tools_when_empty(tmp_path):
"""``allowed_tools`` is the auto-approve allowlist (tools exempt
from the operator approval gate), NOT the set of tools the skill
can use. An empty list reads as "no tool access" to a model
that doesn't know the semantics — real misdiagnosis source: a
code-review skill with no auto-approve allowlist looked like it
had been spawned with zero tools. Dropping the key when empty
removes the ambiguity at the source; absence of the field carries
the unambiguous meaning "no tool is pre-approved for this skill"
while a tool list reads as "these specific tools bypass the prompt".
"""
st = SQLiteBackend(str(tmp_path / "skills_empty.db"))
st.create_prompt_template(
template_id="s-empty",
name="empty-skill",
category="ops",
content="",
variables="[]",
is_default=False,
org_id="",
created_by="test",
tags="[]",
allowed_tools="[]",
)
st.create_prompt_template(
template_id="s-nonempty",
name="nonempty-skill",
category="ops",
content="",
variables="[]",
is_default=False,
org_id="",
created_by="test",
tags="[]",
allowed_tools='["read_file"]',
)
client = _make_read_client(st)
result = client.list_skills()
by_name = {s["name"]: s for s in result["skills"]}
assert "allowed_tools" not in by_name["empty-skill"]
assert by_name["nonempty-skill"]["allowed_tools"] == ["read_file"]
def test_list_skills_projects_allowed_tools_capped_with_sentinel(tmp_path):
"""Each row carries the skill's allowed_tools (capped at the projection
cap with a +N more sentinel) so coordinators can pick a skill without
@@ -2616,3 +2659,362 @@ def test_cleanup_dead_task_child_refs_storage_batch_failure_swallows(populated_s
populated_storage.get_workstreams_batch = _boom # type: ignore[method-assign]
assert client.cleanup_dead_task_child_refs("coord-1") == 0
# ---------------------------------------------------------------------------
# inspect_workstream — three-tier output compression
# ---------------------------------------------------------------------------
#
# A coord doing a fan-out wave against tool-heavy children would
# otherwise blow the context budget on raw output alone. Mirrors the
# search tool's Tier-1/Tier-2/Tier-3 ladder.
def _make_inspect_result(
*, ws_id: str = "ws-test", state: str = "running", n_messages: int = 5
) -> dict[str, Any]:
"""Build an inspect-result dict shaped like ``coordinator_client.inspect()``.
Production output keys (``ws_id``, ``skill_id``) mirror the storage
row that ``inspect()`` spreads from ``get_workstream``. Tests that
synthesize an inspect result must match these keys otherwise a
formatter that looks at the production keys silently emits null
values against a fixture that uses different ones (real bug-1
regression source: skeleton tier read ``skill`` from a fixture
that wrote ``skill`` while production wrote ``skill_id``).
"""
return {
"ws_id": ws_id,
"state": state,
"title": "test workstream",
"skill_id": "researcher",
"messages": [
{"role": "user" if i % 2 == 0 else "assistant", "content": f"msg {i} content"}
for i in range(n_messages)
],
"verdicts": [],
}
def test_format_inspect_tiered_full_fits_returns_full_tier():
"""Small payloads pass through with `_tier='full'` — no compression."""
from turnstone.console.coordinator_client import _format_inspect_tiered
result = _make_inspect_result(n_messages=3)
out = _format_inspect_tiered(result)
parsed = json.loads(out)
assert parsed["_tier"] == "full"
# Every message verbatim.
assert len(parsed["messages"]) == 3
assert parsed["messages"][0]["content"] == "msg 0 content"
def test_format_inspect_tiered_compact_when_full_exceeds_budget():
"""Large messages trigger the compact tier — head/tail-snipped
content with the rest of the row intact."""
from turnstone.console.coordinator_client import (
_INSPECT_MSG_CONTENT_HEAD,
_INSPECT_MSG_CONTENT_TAIL,
_INSPECT_OUTPUT_BUDGET,
_format_inspect_tiered,
)
# Each message ~5KB; with 20 messages, full tier blows the 32KB budget.
fat = "X" * 5000
result = {
"id": "ws-fat",
"state": "running",
"messages": [{"role": "assistant", "content": fat} for _ in range(20)],
"verdicts": [],
}
out = _format_inspect_tiered(result)
parsed = json.loads(out)
assert parsed["_tier"] == "compact"
# Every message preserved (compact keeps the count, just snips content).
assert len(parsed["messages"]) == 20
# Head/tail snip kicked in.
msg_content = parsed["messages"][0]["content"]
assert msg_content.startswith("X" * _INSPECT_MSG_CONTENT_HEAD)
assert msg_content.endswith("X" * _INSPECT_MSG_CONTENT_TAIL)
assert "chars elided" in msg_content
# Budget invariant — the load-bearing contract of the formatter.
# Without this assertion, a future change to ``_tier_note`` or
# ``_compact_message`` could push the output over budget and the
# ``_truncate_output`` head+tail safety net would silently mask
# the regression, re-introducing the middle-message-drop pathology.
assert len(out) <= _INSPECT_OUTPUT_BUDGET
def test_format_inspect_tiered_compact_when_content_below_snip_threshold():
"""When per-message content is below the snip threshold but the
message COUNT alone overflows the budget, compact tier must still
stay within budget by trimming the message list (head + tail of
messages) rather than degrading straight to skeleton. Bug-3
regression cover: with 400 × 100-char messages, the original
formatter fell through to skeleton because adding ``_tier_note``
to an un-snipped tier-2 produced output strictly larger than
tier-1 (both over budget). The fix preserves messages from both
ends of the list and inserts an ``_omitted`` sentinel."""
from turnstone.console.coordinator_client import (
_INSPECT_OUTPUT_BUDGET,
_format_inspect_tiered,
)
# 400 × ~100 chars → Tier-1 ~53 KB (over budget), per-message
# content under the 964-char snip threshold so content-snipping
# saves nothing. Without the list-trim rung the formatter would
# fall to skeleton and drop all 400 messages.
smallish = "S" * 100
result = {
"ws_id": "ws-many-small",
"state": "running",
"messages": [
{"role": "assistant" if i % 2 == 0 else "user", "content": smallish} for i in range(400)
],
"verdicts": [],
}
out = _format_inspect_tiered(result)
parsed = json.loads(out)
# Should NOT fall through to skeleton — message-list trim preserves
# head + tail of the conversation.
assert parsed["_tier"] == "compact"
assert "messages" in parsed
# Some messages must survive; the trim shape is head + tail with an
# ``_omitted`` sentinel between them.
assert len(parsed["messages"]) > 0
assert len(parsed["messages"]) < 400
# Budget invariant.
assert len(out) <= _INSPECT_OUTPUT_BUDGET
def test_format_inspect_tiered_skeleton_when_compact_also_exceeds_budget():
"""Tier 3 fallback: counts + last assistant preview only. Trigger by
flooding with messages whose content is a multi-block list the
snipper correctly leaves non-string content unchanged (mirrors
Anthropic/OpenAI multi-block content shape), so even after the
(5, 10) message-list trim the surviving 15 messages don't fit in
the 32 KB budget."""
from turnstone.console.coordinator_client import (
_INSPECT_OUTPUT_BUDGET,
_format_inspect_tiered,
)
# 50 messages × multi-block content (~30 KB each — list-shape
# content bypasses the head/tail string snipper because lists
# aren't strings). Even (5, 10) trim leaves 15 × 30 KB which
# blows the 32 KB budget — forces skeleton.
fat_block = {"type": "text", "text": "Y" * 3000}
result = {
"ws_id": "ws-flood",
"state": "running",
"title": "flood",
"skill_id": "researcher",
"messages": [
{
"role": "assistant" if i % 2 == 0 else "user",
"content": [fat_block] * 10,
}
for i in range(50)
],
"verdicts": [],
}
out = _format_inspect_tiered(result)
parsed = json.loads(out)
assert parsed["_tier"] == "skeleton"
assert parsed["message_count"] == 50
# Role distribution surfaces — the "what shape of activity" signal.
assert parsed["roles"]["assistant"] == 25
assert parsed["roles"]["user"] == 25
# No `messages` field at skeleton tier — only the aggregate signal.
assert "messages" not in parsed
# Budget invariant.
assert len(out) <= _INSPECT_OUTPUT_BUDGET
def test_format_inspect_tiered_skeleton_keeps_terminal_state_fields():
"""``close_reason`` / ``last_error`` survive the skeleton fall — they're
small, load-bearing, and the operator needs them to understand WHY
a terminal child landed in its state."""
from turnstone.console.coordinator_client import (
_INSPECT_OUTPUT_BUDGET,
_format_inspect_tiered,
)
# Same flood pattern as the bare-skeleton test (multi-block content
# bypasses the string snipper) — paired with terminal-state fields
# that must survive the skeleton fall.
fat_block = {"type": "text", "text": "Z" * 3000}
result = {
"ws_id": "ws-closed",
"state": "closed",
"title": "done",
"skill_id": "researcher",
"messages": [{"role": "user", "content": [fat_block] * 10} for _ in range(50)],
"verdicts": [],
"close_reason": "task complete: report attached",
"live": None, # filtered by truthy check
}
out = _format_inspect_tiered(result)
parsed = json.loads(out)
assert parsed["_tier"] == "skeleton"
assert parsed["close_reason"] == "task complete: report attached"
# Falsy ``live`` doesn't bleed through.
assert "live" not in parsed
assert len(out) <= _INSPECT_OUTPUT_BUDGET
def test_format_inspect_tiered_error_shapes_bypass_tiering():
"""Cross-tenant / not-found responses keep their original shape — they
carry no messages, are already tiny, and changing them would break
callers that key on the ``error`` field."""
from turnstone.console.coordinator_client import _format_inspect_tiered
result = {"error": "workstream not found", "ws_id": "ws-foreign"}
out = _format_inspect_tiered(result)
parsed = json.loads(out)
assert parsed == {"error": "workstream not found", "ws_id": "ws-foreign"}
# No `_tier` annotation — error shapes are self-describing.
assert "_tier" not in parsed
def test_format_inspect_tiered_compact_preserves_tool_call_linkage():
"""Compact tier keeps ``tool_name`` / ``tool_call_id`` / ``name`` so a
model reading the snipped trace can still pair a tool call to its
response the linkage is load-bearing for "what happened" signal."""
from turnstone.console.coordinator_client import _format_inspect_tiered
fat = "Q" * 5000
result = {
"ws_id": "ws-tools",
"state": "running",
"messages": [
{
"role": "assistant",
"content": fat,
"tool_name": "bash",
"tool_call_id": "call-1",
}
for _ in range(20)
],
"verdicts": [],
}
out = _format_inspect_tiered(result)
parsed = json.loads(out)
assert parsed["_tier"] == "compact"
first = parsed["messages"][0]
assert first["tool_name"] == "bash"
assert first["tool_call_id"] == "call-1"
def test_format_inspect_tiered_compact_preserves_assistant_tool_calls():
"""Compact tier must preserve the assistant-side ``tool_calls`` list
(OpenAI shape: ``[{id, type, function: {name, arguments}}]``) so a
model reading the snipped trace can see WHICH tool was called and
pair it with the corresponding result row via ``id`` ``tool_call_id``.
Bug-2 regression cover: the pre-fix compactor stripped ``tool_calls``,
leaving the audit reader with a tool-result orphan against an
invisible call.
``function.arguments`` strings are snipped head/tail (analogous to
content) because they can be multi-KB JSON; ``id`` and
``function.name`` are preserved verbatim they're the linkage."""
from turnstone.console.coordinator_client import (
_INSPECT_TOOL_ARG_HEAD,
_INSPECT_TOOL_ARG_TAIL,
_format_inspect_tiered,
)
fat_content = "C" * 5000 # forces compact tier
fat_args = "A" * 5000 # forces argument snipping
tool_calls = [
{
"id": "call-abc-123",
"type": "function",
"function": {"name": "bash", "arguments": fat_args},
},
{
"id": "call-def-456",
"type": "function",
"function": {"name": "read_file", "arguments": fat_args},
},
]
result = {
"ws_id": "ws-tool-calls",
"state": "running",
"messages": [
{"role": "assistant", "content": fat_content, "tool_calls": tool_calls}
for _ in range(20)
],
"verdicts": [],
}
out = _format_inspect_tiered(result)
parsed = json.loads(out)
assert parsed["_tier"] == "compact"
first = parsed["messages"][0]
# tool_calls survives compaction.
assert "tool_calls" in first
assert len(first["tool_calls"]) == 2
# Linkage fields verbatim.
assert first["tool_calls"][0]["id"] == "call-abc-123"
assert first["tool_calls"][0]["function"]["name"] == "bash"
assert first["tool_calls"][1]["id"] == "call-def-456"
assert first["tool_calls"][1]["function"]["name"] == "read_file"
# arguments snipped head/tail — both prefix and suffix preserved.
snipped_args = first["tool_calls"][0]["function"]["arguments"]
assert snipped_args.startswith("A" * _INSPECT_TOOL_ARG_HEAD)
assert snipped_args.endswith("A" * _INSPECT_TOOL_ARG_TAIL)
assert "chars elided" in snipped_args
def test_format_inspect_tiered_compact_passes_small_messages_through_unsnipped():
"""Messages under the snip threshold pass through verbatim at compact
tier snipping a 100-byte message costs more bytes (the elision
marker) than it saves."""
from turnstone.console.coordinator_client import _format_inspect_tiered
# Mix: a few large messages force compact tier; small messages must
# not be snipped.
big = "B" * 5000
small = "S" * 50
result = {
"id": "ws-mixed",
"state": "running",
"messages": [{"role": "assistant", "content": big} for _ in range(15)]
+ [{"role": "user", "content": small}],
"verdicts": [],
}
out = _format_inspect_tiered(result)
parsed = json.loads(out)
assert parsed["_tier"] == "compact"
# The trailing small message is exact, not snipped.
assert parsed["messages"][-1]["content"] == small
def test_format_inspect_tiered_emits_tier_note_when_compressed():
"""The ``_tier_note`` advisory tells the LLM how to ask for a tighter
or fuller view next time actionable feedback rather than a bare
"we compressed your output" signal."""
from turnstone.console.coordinator_client import _format_inspect_tiered
fat = "F" * 5000
result = {
"id": "ws-noted",
"state": "running",
"messages": [{"role": "assistant", "content": fat} for _ in range(20)],
"verdicts": [],
}
out = _format_inspect_tiered(result)
parsed = json.loads(out)
assert "_tier_note" in parsed
assert "message_limit" in parsed["_tier_note"]
def test_format_inspect_tiered_full_tier_omits_tier_note():
"""When the full tier fits, no note is emitted — the absence of a
note is the signal that nothing was compressed."""
from turnstone.console.coordinator_client import _format_inspect_tiered
out = _format_inspect_tiered(_make_inspect_result(n_messages=2))
parsed = json.loads(out)
assert parsed["_tier"] == "full"
assert "_tier_note" not in parsed
+33 -4
View File
@@ -215,7 +215,12 @@ def test_spawn_exec_does_not_surface_misleading_status_field(coord_session):
summary tempted callers to write ``if result["status"] == "idle"``
which silently never matched. The summary now omits the field
entirely; lifecycle state lives on the workstream row and is read
via inspect_workstream."""
via inspect_workstream.
Also asserts the return key is ``child_ws_id`` (not ``ws_id``) so
the coordinator LLM doesn't recency-bias toward feeding the spawn
output back into another ``spawn_workstream(ws_id=...)`` call.
"""
sess, coord, _ui = coord_session
coord.spawn.return_value = {
"ws_id": "child-7",
@@ -227,8 +232,9 @@ def test_spawn_exec_does_not_surface_misleading_status_field(coord_session):
_call_id, output = sess._exec_spawn_workstream(item)
body = json.loads(output)
assert "status" not in body
assert "ws_id" not in body
# The substantive fields are still here.
assert body["ws_id"] == "child-7"
assert body["child_ws_id"] == "child-7"
assert body["node_id"] == "node-1"
@@ -248,6 +254,10 @@ def test_spawn_batch_exec_does_not_surface_misleading_status_field(coord_session
body = json.loads(output)
assert "0" in body["results"]
assert "status" not in body["results"]["0"]
# Per-result entries surface ``child_ws_id``, not ``ws_id`` — same
# recency-bias rationale as the spawn_workstream test above.
assert body["results"]["0"]["child_ws_id"] == "c-x"
assert "ws_id" not in body["results"]["0"]
def test_spawn_exec_surfaces_client_error(coord_session):
@@ -260,6 +270,21 @@ def test_spawn_exec_surfaces_client_error(coord_session):
assert ui.tool_results[-1][3] is True # is_error
def test_spawn_exec_treats_missing_ws_id_on_success_path_as_error(coord_session):
"""A malformed upstream response (200-success-shape with no
``ws_id``) used to emit ``{"child_ws_id": null}`` to the LLM,
which then chased a null id through follow-up tools. Now matches
the matching guard in ``_exec_spawn_batch``: surface as a tool
error so the model retries instead of acting on garbage."""
sess, coord, ui = coord_session
# No ``error`` field, but ``ws_id`` is missing — the silent-null path.
coord.spawn.return_value = {"name": "c", "node_id": "node-1", "status": 200}
item = sess._prepare_tool(_tc("spawn_workstream", {"initial_message": "hi"}))
_call_id, output = sess._exec_spawn_workstream(item)
assert "no ws_id" in output
assert ui.tool_results[-1][3] is True # is_error
# ---------------------------------------------------------------------------
# inspect_workstream
# ---------------------------------------------------------------------------
@@ -1410,9 +1435,13 @@ def test_spawn_batch_exec_serialises_spawns_and_returns_results(coord_session):
assert body["denied"] == []
# Keyed by input index (stringified).
assert set(body["results"].keys()) == {"0", "1", "2"}
assert body["results"]["0"]["ws_id"] == "child-0"
assert body["results"]["0"]["child_ws_id"] == "child-0"
assert body["results"]["1"]["node_id"] == "n-1"
assert body["results"]["2"]["ws_id"] == "child-2"
assert body["results"]["2"]["child_ws_id"] == "child-2"
# Confirm we don't leak the old ``ws_id`` key alongside the new
# ``child_ws_id`` — see test_spawn_exec_does_not_surface_misleading_status_field
# for the rationale on the rename.
assert "ws_id" not in body["results"]["0"]
def test_spawn_batch_exec_surfaces_per_item_errors_in_denied(coord_session):
+122 -1
View File
@@ -51,7 +51,11 @@ class TestIntentVerdictCRUD:
assert v["tier"] == "heuristic"
assert v["judge_model"] == ""
assert v["latency_ms"] == 2
assert v["user_decision"] == ""
# ``user_decision`` defaults to ``"pending"`` (not the empty
# string) so an audit reader can distinguish in-flight rows
# from pre-convention legacy rows that carry the column's
# server_default of ``""``.
assert v["user_decision"] == "pending"
assert "created" in v
def test_get_nonexistent(self, db):
@@ -114,6 +118,123 @@ class TestIntentVerdictCRUD:
assert ok is False
class TestIntentVerdictUpsert:
"""``upsert_intent_verdict`` — the LLM-tier-aware persistence path.
Backs the heuristic llm_fallback "upgrade in place" pattern.
The async judge's fallback verdicts deliberately reuse the
heuristic ``verdict_id``; a plain INSERT would collide on the
PK and the upgrade would be lost to a silently-swallowed
exception (Postgres logged ``intent_verdicts_pkey`` violations
for every fallback delivery on stable/1.5 smoke tests).
"""
def test_upsert_on_fresh_id_inserts(self, db):
"""No conflict — behaves like a regular INSERT."""
db.upsert_intent_verdict(**_make_verdict_kwargs())
v = db.get_intent_verdict("v_001")
assert v is not None
assert v["tier"] == "heuristic"
assert v["user_decision"] == "pending"
def test_upsert_on_conflict_upgrades_tier_reasoning_judge_model(self, db):
"""On PK conflict: tier, reasoning, judge_model update — every
other field is preserved. Mirrors what the judge emits when
promoting heuristic llm_fallback."""
db.upsert_intent_verdict(
**_make_verdict_kwargs(
tier="heuristic",
reasoning="initial heuristic reasoning",
judge_model="",
)
)
db.upsert_intent_verdict(
**_make_verdict_kwargs(
tier="llm_fallback",
reasoning="initial heuristic reasoning (LLM judge did not return a verdict)",
judge_model="gpt-5-judge",
)
)
v = db.get_intent_verdict("v_001")
assert v is not None
# The three fields that should change.
assert v["tier"] == "llm_fallback"
assert "LLM judge did not return" in v["reasoning"]
assert v["judge_model"] == "gpt-5-judge"
def test_upsert_on_conflict_preserves_user_decision(self, db):
"""LOAD-BEARING: a manually-resolved approval (user_decision=
``"approved"``) or auto-approve-stamped row (user_decision=
``"policy"``/``"blanket"``/etc.) must NOT be clobbered back to
``"pending"`` when the late LLM-fallback verdict lands.
``IntentVerdict.to_dict()`` doesn't project user_decision, so
the upsert's defaulted ``"pending"`` would silently overwrite
the real value if user_decision were in the on-conflict
SET clause."""
db.upsert_intent_verdict(**_make_verdict_kwargs())
ok = db.update_intent_verdict("v_001", user_decision="approved")
assert ok is True
# Simulate the late LLM-fallback delivery — same verdict_id,
# default user_decision (the IntentVerdict.to_dict() shape).
db.upsert_intent_verdict(
**_make_verdict_kwargs(
tier="llm_fallback",
reasoning="extended (LLM judge did not return a verdict)",
judge_model="gpt-5-judge",
)
)
v = db.get_intent_verdict("v_001")
assert v is not None
assert v["user_decision"] == "approved" # NOT clobbered to "pending"
assert v["tier"] == "llm_fallback" # but the upgrade did land
def test_upsert_on_conflict_preserves_identity_and_carried_fields(self, db):
"""Identity columns (ws_id, call_id, func_name, func_args) and
carried-verbatim columns (intent_summary, risk_level,
confidence, recommendation, evidence, latency_ms) are
excluded from the on-conflict SET verify they aren't
changed even when the second upsert passes different values
(defensive against a future judge bug that ships divergent
carried fields)."""
db.upsert_intent_verdict(**_make_verdict_kwargs())
db.upsert_intent_verdict(
**_make_verdict_kwargs(
# Same verdict_id (conflict trigger), divergent everything else.
ws_id="ws-different",
call_id="tc_different",
func_name="bash_v2",
func_args='{"command":"rm -rf /"}',
intent_summary="totally different summary",
risk_level="critical",
confidence=0.0,
recommendation="deny",
evidence='["dangerous"]',
latency_ms=99999,
# The three fields that DO update.
tier="llm_fallback",
reasoning="upgraded reasoning",
judge_model="judge-v2",
)
)
v = db.get_intent_verdict("v_001")
assert v is not None
# All preserved from the first upsert (identity + carried).
assert v["ws_id"] == "ws-abc"
assert v["call_id"] == "tc_001"
assert v["func_name"] == "bash"
assert v["func_args"] == '{"command":"echo hello"}'
assert v["intent_summary"] == "Echo a greeting to stdout"
assert v["risk_level"] == "low"
assert v["confidence"] == 0.85
assert v["recommendation"] == "approve"
assert v["evidence"] == '["The command only prints text."]'
assert v["latency_ms"] == 2
# Only the three updated.
assert v["tier"] == "llm_fallback"
assert v["reasoning"] == "upgraded reasoning"
assert v["judge_model"] == "judge-v2"
# ---------------------------------------------------------------------------
# Bulk insert
# ---------------------------------------------------------------------------
+35 -6
View File
@@ -2,6 +2,7 @@
from __future__ import annotations
import logging
import threading
import pytest
@@ -401,8 +402,10 @@ class TestValidation:
class TestValidUntil:
"""``valid_until`` predicate: drain re-checks freshness; falsy /
raising predicates drop the entry without delivery.
"""``valid_until`` predicate: drain re-checks freshness. Falsy
predicates drop the entry without delivery and log at ``info``
(normal lifecycle outcome); raising predicates drop the entry and
log at ``warning`` with ``exc_info`` (misbehaving predicate).
"""
def test_valid_until_true_delivers(self):
@@ -411,26 +414,52 @@ class TestValidUntil:
out = q.drain({"any"})
assert out == [("a", "1", None)]
def test_valid_until_false_drops_silently(self):
def test_valid_until_false_drops_with_info_log(self, caplog: pytest.LogCaptureFixture):
q = NudgeQueue()
q.enqueue("a", "1", "any", valid_until=lambda: False)
out = q.drain({"any"})
with caplog.at_level(logging.INFO, logger="turnstone.core.nudge_queue"):
out = q.drain({"any"})
assert out == []
# Already removed from queue (drain partition removes BEFORE
# predicate check — falsy doesn't return to queue).
assert len(q) == 0
# The drop emits a structured info record so a wiring
# regression (a predicate that always returns False) is still
# observable, without spamming ``warning`` for the routine
# lifecycle case where ``valid_until`` is doing its job.
# structlog renders the event name + extras into ``msg`` as a
# single rendered string, so substring-match like the
# ``watch_dispatch.queue_full`` assertion in
# tests/test_watch_dispatch.py.
drops = [r for r in caplog.records if "nudge_queue.predicate_dropped" in r.getMessage()]
assert len(drops) == 1
assert drops[0].levelno == logging.INFO
assert "predicate_false" in drops[0].getMessage()
assert "'nudge_type': 'a'" in drops[0].getMessage()
assert "'channel': 'any'" in drops[0].getMessage()
assert "'text_len': 1" in drops[0].getMessage()
def test_valid_until_exception_drops_silently(self):
def test_valid_until_exception_drops_with_warning(self, caplog: pytest.LogCaptureFixture):
q = NudgeQueue()
def boom() -> bool:
raise RuntimeError("predicate crash")
q.enqueue("a", "1", "any", valid_until=boom)
out = q.drain({"any"})
with caplog.at_level(logging.WARNING, logger="turnstone.core.nudge_queue"):
out = q.drain({"any"})
assert out == []
# Crash-on-predicate is treated as "no longer valid" — drop, not propagate.
assert len(q) == 0
# Stays at ``warning`` (with ``exc_info``) because a raising
# predicate is a bug, not a normal lifecycle outcome.
drops = [r for r in caplog.records if "nudge_queue.predicate_dropped" in r.getMessage()]
assert len(drops) == 1
assert drops[0].levelno == logging.WARNING
rendered = drops[0].getMessage()
assert "predicate_raised" in rendered
assert "RuntimeError" in rendered
assert "predicate crash" in rendered
def test_valid_until_evaluated_outside_lock(self):
"""The predicate may do non-trivial work (e.g. storage I/O)
+188 -2
View File
@@ -167,8 +167,8 @@ def test_on_intent_verdict_persists_verdict_row() -> None:
}
with _patch_get_storage(storage):
ui.on_intent_verdict(verdict)
storage.create_intent_verdict.assert_called_once()
kwargs = storage.create_intent_verdict.call_args.kwargs
storage.upsert_intent_verdict.assert_called_once()
kwargs = storage.upsert_intent_verdict.call_args.kwargs
assert kwargs["verdict_id"] == "v1"
assert kwargs["ws_id"] == "ws-1"
assert kwargs["call_id"] == "c1"
@@ -352,6 +352,192 @@ def test_resolve_approval_stamps_all_pending_verdicts() -> None:
assert ui._last_verdict_decision == "denied"
# ---------------------------------------------------------------------------
# user_decision value space — pending / approved / denied / timeout
# / auto-approve reasons (policy / blanket / skill / always / auto_approve_tools).
# Guards the "user_decision is never empty for new rows" invariant.
# ---------------------------------------------------------------------------
def test_resolve_approval_timeout_kwarg_writes_timeout_value() -> None:
"""``resolve_approval(False, ..., timeout=True)`` writes
``user_decision="timeout"`` so the audit trail can distinguish a
passive timeout expiry from an active user denial the feedback
string used to carry this distinction but the column alone could not."""
storage = MagicMock()
ui = _make_ui()
with _patch_get_storage(storage):
ui.on_intent_verdict({"verdict_id": "v1", "call_id": "c1"})
with _patch_get_storage(storage):
ui.resolve_approval(False, "expired", timeout=True)
storage.update_intent_verdict.assert_any_call("v1", user_decision="timeout")
assert ui._last_verdict_decision == "timeout"
def test_resolve_approval_timeout_with_approved_raises() -> None:
"""``timeout=True`` is mutually exclusive with ``approved=True`` —
the combination would land a row whose audit column says
``"timeout"`` while the SSE event reports ``approved=True``. Fail
loud so the inconsistency can't ship silently."""
import pytest
ui = _make_ui()
with pytest.raises(ValueError, match="timeout"):
ui.resolve_approval(True, timeout=True)
def test_record_auto_approves_populates_reason_lookup() -> None:
"""``_record_auto_approves`` must seed
``_auto_approve_reasons[call_id]`` with the per-item reason so a
late-arriving LLM judge verdict can recover the auto-approve
reason via ``on_intent_verdict``."""
storage = MagicMock()
ui = _make_ui()
items = [
{
"call_id": "c-policy",
"func_name": "bash",
"auto_approved": True,
"auto_approve_reason": "policy",
},
{
"call_id": "c-blanket",
"func_name": "list_workstreams",
"auto_approved": True,
"auto_approve_reason": "blanket",
},
]
with _patch_get_storage(storage):
ui._record_auto_approves(items)
assert "c-policy" in ui._auto_approve_reasons
assert "c-blanket" in ui._auto_approve_reasons
assert ui._auto_approve_reasons["c-policy"][0] == "policy"
assert ui._auto_approve_reasons["c-blanket"][0] == "blanket"
def test_on_intent_verdict_consumes_auto_approve_reason() -> None:
"""A late LLM verdict for a previously auto-approved call_id picks
up the reason from ``_auto_approve_reasons``, stamps it on the
verdict before persist, and pops the entry so re-use isn't
possible. Closes the misdiagnosis bug where auto-approved tools
landed verdict rows with ``user_decision=""``."""
storage = MagicMock()
ui = _make_ui()
ui._auto_approve_reasons["c-x"] = ("auto_approve_tools", 0.0)
with _patch_get_storage(storage):
ui.on_intent_verdict({"verdict_id": "v-x", "call_id": "c-x"})
storage.upsert_intent_verdict.assert_called_once()
kwargs = storage.upsert_intent_verdict.call_args.kwargs
assert kwargs["user_decision"] == "auto_approve_tools"
# Consumed on read so the same call_id can't double-stamp later.
assert "c-x" not in ui._auto_approve_reasons
# Auto-stamped verdicts must NOT join _pending_verdicts — the
# row's final decision is already set; appending would let a
# later resolve_approval overwrite the auto-reason with the
# manual decision (real audit-trail clobber bug).
assert ui._pending_verdicts == []
def test_on_intent_verdict_auto_reason_survives_resolve_cycle() -> None:
"""Mixed-batch case: one tool was auto-approved (policy), another
needs manual approval. The LLM judge fires for the auto-approved
sibling DURING the manual-approval wait. The verdict must land
with ``user_decision="policy"`` and stay that way even after
``resolve_approval`` fires for the pending sibling the prior
bug was that the auto-stamped row got overwritten with
``"approved"``/``"denied"`` by the resolve path."""
storage = MagicMock()
ui = _make_ui()
ui._auto_approve_reasons["c-auto"] = ("policy", 0.0)
with _patch_get_storage(storage):
# LLM verdict fires for the auto-approved sibling.
ui.on_intent_verdict({"verdict_id": "v-auto", "call_id": "c-auto"})
# Now the pending sibling gets a verdict + manual resolve.
ui.on_intent_verdict({"verdict_id": "v-pending", "call_id": "c-pending"})
ui.resolve_approval(True, "looks good")
# Only the pending verdict should be UPDATEd to "approved" — the
# auto-stamped one stays "policy" via its INSERT.
update_calls = {
c.args[0]: c.kwargs.get("user_decision")
for c in storage.update_intent_verdict.call_args_list
}
assert update_calls == {"v-pending": "approved"}
# The auto verdict's INSERT carried the policy reason.
insert_calls = {
c.kwargs["verdict_id"]: c.kwargs["user_decision"]
for c in storage.upsert_intent_verdict.call_args_list
}
assert insert_calls["v-auto"] == "policy"
assert insert_calls["v-pending"] == "pending"
def test_persist_auto_approved_heuristic_verdicts_stamps_reason() -> None:
"""The auto-approve early-return branches in ``approve_tools`` used
to drop heuristic verdicts on the floor auditors couldn't tell
whether the judge ran or the call was simply silently auto-approved.
``_persist_auto_approved_heuristic_verdicts`` closes that gap and
stamps each verdict with the item's reason."""
storage = MagicMock()
ui = _make_ui()
items = [
{
"call_id": "c-1",
"auto_approved": True,
"auto_approve_reason": "blanket",
"_heuristic_verdict": {
"verdict_id": "v-1",
"call_id": "c-1",
"risk_level": "low",
"recommendation": "review",
},
},
# No _heuristic_verdict — skipped (judge didn't run for this item).
{"call_id": "c-2", "auto_approved": True, "auto_approve_reason": "blanket"},
# Not auto_approved — skipped (this helper only handles auto-approved).
{
"call_id": "c-3",
"_heuristic_verdict": {"verdict_id": "v-3", "call_id": "c-3"},
},
]
with _patch_get_storage(storage):
ui._persist_auto_approved_heuristic_verdicts(items)
storage.create_intent_verdicts_bulk.assert_called_once()
rows = storage.create_intent_verdicts_bulk.call_args.args[0]
assert len(rows) == 1
assert rows[0]["verdict_id"] == "v-1"
assert rows[0]["user_decision"] == "blanket"
def test_auto_approve_reasons_ttl_prune_drops_stale_entries() -> None:
"""Lazy TTL eviction at write time: entries older than
``_AUTO_APPROVE_REASON_TTL`` are pruned on the next
``_record_auto_approves`` call. Without this, a session with the
LLM judge disabled would accumulate entries that never get
consumed."""
import time as time_module
storage = MagicMock()
ui = _make_ui()
# Seed two stale entries (well past the TTL).
stale_ts = time_module.time() - ui._AUTO_APPROVE_REASON_TTL - 30.0
ui._auto_approve_reasons["c-stale-1"] = ("policy", stale_ts)
ui._auto_approve_reasons["c-stale-2"] = ("blanket", stale_ts)
items = [
{
"call_id": "c-fresh",
"auto_approved": True,
"auto_approve_reason": "skill",
"func_name": "bash",
}
]
with _patch_get_storage(storage):
ui._record_auto_approves(items)
# Stale entries pruned; only the fresh one remains.
assert "c-stale-1" not in ui._auto_approve_reasons
assert "c-stale-2" not in ui._auto_approve_reasons
assert "c-fresh" in ui._auto_approve_reasons
# ---------------------------------------------------------------------------
# Output guard persistence
# ---------------------------------------------------------------------------
+32 -39
View File
@@ -232,59 +232,52 @@ class TestSoftCap:
# ---------------------------------------------------------------------------
# valid_until predicate
# Predicate independence
# ---------------------------------------------------------------------------
class TestValidUntil:
"""The ``valid_until`` predicate captured at dispatch time re-checks
the watch's ``active`` flag at drain time, so a cancelled watch's
last splat doesn't ride out a future wake.
class TestPredicateIndependence:
"""The watch closure does NOT wire a ``valid_until`` predicate.
Earlier the closure wired ``_still_active`` (re-reading
``is_watch_active`` at drain time). That predicate raced
``WatchRunner._poll_watch``'s commit of ``active=False`` and silently
dropped every terminal fire. The closure now enqueues without a
predicate; entries survive drain regardless of the row's ``active``
column state.
"""
def test_valid_until_drops_when_watch_inactive(self, tmp_db, monkeypatch):
def test_drain_delivers_even_when_storage_reports_inactive(self, tmp_db, monkeypatch):
session = _make_session_for_dispatch()
_runner, dispatch = _register_runner(session)
# Storage stub returns False at drain time.
is_active_calls = patch_session_storage(monkeypatch, active=False)
dispatch(_reminder("body"), "watch-1")
# Drain fires the predicate; entry should NOT be delivered.
out = session._nudge_queue.drain({"any"})
assert out == []
# Predicate ran once with the dispatched watch_id.
assert is_active_calls == ["watch-1"]
def test_valid_until_drops_when_storage_raises(self, tmp_db, monkeypatch):
"""The closure's broad-except in the predicate translates a
storage-layer exception to ``False`` so the drain doesn't
propagate; the predicate captured ``watch_id`` correctly
(otherwise storage wouldn't even be touched).
"""
session = _make_session_for_dispatch()
_runner, dispatch = _register_runner(session)
patch_session_storage(monkeypatch, raise_on_is_active=True)
dispatch(_reminder("body"), "watch-bound-id")
out = session._nudge_queue.drain({"any"})
assert out == []
def test_valid_until_delivers_when_watch_active(self, tmp_db, monkeypatch):
"""Happy-path counter-test for the predicate above: the entry
DOES drain when the watch is still active.
"""
session = _make_session_for_dispatch()
_runner, dispatch = _register_runner(session)
patch_session_storage(monkeypatch, active=True)
# Even if storage reports active=False, the entry should still
# drain — no predicate to drop it.
patch_session_storage(monkeypatch, active=False)
dispatch(_reminder("body"), "watch-1")
out = session._nudge_queue.drain({"any"})
assert len(out) == 1
assert out[0][0] == "watch_triggered"
def test_dispatch_never_calls_is_watch_active(self, tmp_db, monkeypatch):
"""Pin the invariant directly: the closure must NOT consult
``storage.is_watch_active`` anywhere along the enqueue + drain
path. Without this assertion, a future change that re-wires
an ``is_watch_active`` predicate would silently bring back the
bug that motivates this whole module.
"""
session = _make_session_for_dispatch()
_runner, dispatch = _register_runner(session)
is_active_calls = patch_session_storage(monkeypatch, active=True)
dispatch(_reminder("body"), "watch-bound-id")
session._nudge_queue.drain({"any"})
assert is_active_calls == [], (
f"watch closure must not call is_watch_active; got {is_active_calls!r}"
)
# ---------------------------------------------------------------------------
# Concurrency
+284
View File
@@ -24,11 +24,15 @@ the structural integration gate for the watch switchover.
from __future__ import annotations
import contextlib
from typing import Any
from unittest.mock import MagicMock, patch
import pytest
from tests._helpers import patch_session_storage
from turnstone.core.session import ChatSession
from turnstone.core.storage import get_storage
from turnstone.core.watch import WatchRunner
@@ -272,3 +276,283 @@ def test_watch_dispatch_through_restore_fn_lands_on_rehydrated_session(tmp_db, m
# Original session's queue stays empty — the dispatch did NOT
# accidentally route back to it.
assert len(original._nudge_queue) == 0
@pytest.mark.parametrize(
("stop_on", "max_polls", "label"),
[
('"HIT" in output', 100, "stop_on_fired"),
(None, 1, "max_polls_reached"),
],
)
def test_poll_watch_terminal_fire_survives_drain(
tmp_db: str,
monkeypatch: pytest.MonkeyPatch,
stop_on: str | None,
max_polls: int,
label: str,
) -> None:
"""Regression for the dispatch-ordering bug.
With the broken ordering (``update_watch(active=False)`` before
``_dispatch_result``) plus the ``_still_active`` ``valid_until``
predicate that re-reads ``is_watch_active`` at drain time, every
terminal watch fire was silently dropped the closure enqueued
the entry but the predicate immediately invalidated it because
the row's ``active`` flag had already been flipped to ``0`` in
the same poll. The model never saw the fire.
This test drives a REAL ``WatchRunner._poll_watch`` against a real
``tmp_db`` watch row (no ``patch_session_storage(active=True)``
stub that stub is exactly what masked the bug in earlier tests).
Covers both terminal paths: ``stop_on`` condition matched and
``poll_count >= max_polls`` reached.
"""
session = _make_session()
storage = get_storage()
runner = WatchRunner(storage=storage, node_id="test-node")
session.set_watch_runner(runner)
storage.create_watch(
watch_id=f"w-regression-{label}",
ws_id=session._ws_id,
node_id="test-node",
name=f"regression-{label}",
command="echo HIT",
interval_secs=10.0,
stop_on=stop_on,
max_polls=max_polls,
created_by="model",
next_poll="1970-01-01T00:00:00",
)
# Spy ``enqueue`` so the assertion can distinguish "dispatch never
# called" (a different bug class) from "dispatch enqueued but the
# predicate dropped it at drain" (this bug).
enqueue_calls: list[tuple[str, str, str]] = []
real_enqueue = session._nudge_queue.enqueue
def _spy_enqueue(*args: Any, **kwargs: Any) -> None:
enqueue_calls.append((args[0], args[1][:40], args[2]))
return real_enqueue(*args, **kwargs)
monkeypatch.setattr(session._nudge_queue, "enqueue", _spy_enqueue)
# For the max_polls=1 case the first poll has prev_output=None and
# would not normally fire on output change; the max_polls branch
# at watch.py:412-414 still marks is_final=True so dispatch runs.
due = storage.list_due_watches("2099-01-01T00:00:00")
matching = [r for r in due if r["watch_id"] == f"w-regression-{label}"]
assert len(matching) == 1, f"watch row not picked up by list_due_watches: {due!r}"
runner._poll_watch(matching[0])
assert len(enqueue_calls) == 1, (
f"_poll_watch did not enqueue exactly one fire (got {enqueue_calls!r}); "
"this is a different bug from the predicate-drop regression"
)
assert enqueue_calls[0][0] == "watch_triggered"
assert storage.is_watch_active(f"w-regression-{label}") is False, (
"terminal fire should have committed active=False on the row"
)
# The key assertion: drain delivers the entry. Pre-fix this
# returned ``[]`` because the ``_still_active`` predicate re-read
# ``active=0``. Post-fix the watch closure no longer wires a
# predicate and the entry survives.
out = session._nudge_queue.drain({"any"})
assert len(out) == 1, (
"Watch fire was enqueued but never reached drain — dispatch-ordering "
"regression. Check that WatchRunner._poll_watch dispatches BEFORE "
"committing active=False, and that the watch closure in "
"ChatSession.set_watch_runner does not wire an is_watch_active "
"predicate."
)
nt, text, _meta = out[0]
assert nt == "watch_triggered"
assert "HIT" in text
def test_cancel_reports_already_completed_for_auto_cancelled_watch(tmp_db: str) -> None:
"""After a watch fires and auto-cancels, the cancel-by-name path
should report 'already completed' rather than 'not found'.
Pre-fix, ``_exec_watch`` cancel looked the watch up via
``list_watches_for_ws`` which filters ``active==1``, so a recently-
auto-cancelled row was invisible and the model got the same
'not found' message it would for a typo'd name. Post-fix the
cancel path uses ``find_watch_by_name`` (no active filter) and
branches on ``row["active"]``.
"""
session = _make_session()
storage = get_storage()
storage.create_watch(
watch_id="w-completed-1",
ws_id=session._ws_id,
node_id="test-node",
name="completed-watch",
command="echo x",
interval_secs=10.0,
stop_on=None,
max_polls=100,
created_by="model",
next_poll="",
)
# Simulate the post-fire state.
storage.update_watch("w-completed-1", active=False, next_poll="")
_call_id, msg = session._exec_watch(
{"call_id": "c1", "action": "cancel", "watch_name": "completed-watch"}
)
assert "not found" not in msg.lower()
assert "completed" in msg.lower()
def test_cancel_reports_not_found_for_unknown_watch(tmp_db: str) -> None:
"""The 'not found' message still applies when the watch genuinely
does not exist make sure the new ``find_watch_by_name`` path
didn't accidentally turn every cancel into 'already completed'.
"""
session = _make_session()
_call_id, msg = session._exec_watch(
{"call_id": "c1", "action": "cancel", "watch_name": "ghost-watch"}
)
assert "not found" in msg.lower()
def test_poll_watch_retry_deactivate_after_update_watch_failure(
tmp_db: str, monkeypatch: pytest.MonkeyPatch
) -> None:
"""``_terminal_dispatched`` lifecycle: if ``update_watch`` raises
AFTER ``_dispatch_result`` shipped the reminder for a terminal
fire, the next ``_poll_watch`` tick MUST retry the row write
(so the row stops appearing in ``list_due_watches``) and MUST NOT
re-dispatch the reminder the model already saw.
This is the keystone path that prevents duplicate-fire under
transient storage failure. Pre-this-test, the entire branch was
unexercised.
"""
session = _make_session()
storage = get_storage()
runner = WatchRunner(storage=storage, node_id="test-node")
session.set_watch_runner(runner)
watch_id = "w-retry-1"
storage.create_watch(
watch_id=watch_id,
ws_id=session._ws_id,
node_id="test-node",
name="retry-watch",
command="echo HIT",
interval_secs=10.0,
stop_on='"HIT" in output',
max_polls=100,
created_by="model",
next_poll="1970-01-01T00:00:00",
)
enqueue_calls: list[tuple[str, str]] = []
real_enqueue = session._nudge_queue.enqueue
def _spy_enqueue(*args: Any, **kwargs: Any) -> None:
enqueue_calls.append((args[0], args[1][:32]))
return real_enqueue(*args, **kwargs)
monkeypatch.setattr(session._nudge_queue, "enqueue", _spy_enqueue)
# Stage 1 — first poll. ``update_watch`` raises AFTER dispatch.
real_update = storage.update_watch
update_raise = {"armed": True}
def _failing_update(wid: str, **fields: Any) -> bool:
if update_raise["armed"]:
raise RuntimeError("simulated transient storage failure")
return real_update(wid, **fields)
monkeypatch.setattr(storage, "update_watch", _failing_update)
due = storage.list_due_watches("2099-01-01T00:00:00")
matching = [r for r in due if r["watch_id"] == watch_id]
assert len(matching) == 1
# ``_poll_watch`` doesn't catch the storage error; the outer
# ``_tick`` would log it. Suppress here so the test owns the
# boundary and continues to its assertions.
with contextlib.suppress(RuntimeError):
runner._poll_watch(matching[0])
# Dispatch ran exactly once and the watch_id sits in the
# terminal-dispatched set awaiting retry.
assert len(enqueue_calls) == 1
assert enqueue_calls[0][0] == "watch_triggered"
assert watch_id in runner._terminal_dispatched
# The row is still active=1 because update_watch raised. It
# would re-appear in list_due_watches on the next tick.
assert storage.is_watch_active(watch_id) is True
# Stage 2 — second poll. Storage now succeeds; retry-deactivate
# branch must commit active=False WITHOUT re-dispatching.
update_raise["armed"] = False
due = storage.list_due_watches("2099-01-01T00:00:00")
matching = [r for r in due if r["watch_id"] == watch_id]
assert len(matching) == 1
runner._poll_watch(matching[0])
# Exactly one dispatch in total — the retry path took the
# short-circuit return at the top of _poll_watch.
assert len(enqueue_calls) == 1, f"retry-deactivate must not re-dispatch; got {enqueue_calls!r}"
# Row is now inactive (the retry path's update_watch landed).
assert storage.is_watch_active(watch_id) is False
# Set is cleared so future watches with the same id (unlikely) /
# process memory doesn't accumulate.
assert watch_id not in runner._terminal_dispatched
def test_cancel_clears_pending_terminal_dispatched_entry(
tmp_db: str, monkeypatch: pytest.MonkeyPatch
) -> None:
"""If ``update_watch`` raised after dispatch, leaving a pending
entry in ``_terminal_dispatched``, and the user then cancels the
watch out-of-band, the retry-deactivate branch never gets to run
(the cancel sets ``next_poll=""`` which removes the row from
``list_due_watches``). The cancel path itself must discard the
pending entry; otherwise the runner leaks ``watch_id``s for the
process lifetime.
"""
session = _make_session()
storage = get_storage()
runner = WatchRunner(storage=storage, node_id="test-node")
session.set_watch_runner(runner)
watch_id = "w-leak-1"
storage.create_watch(
watch_id=watch_id,
ws_id=session._ws_id,
node_id="test-node",
name="leak-watch",
command="echo x",
interval_secs=10.0,
stop_on=None,
max_polls=100,
created_by="model",
next_poll="",
)
# Simulate: dispatch shipped, update_watch raised, watch_id sits
# in the runner's pending set.
with runner._terminal_dispatched_lock:
runner._terminal_dispatched.add(watch_id)
# User cancels. Because the cancel writes active=False, next_poll="",
# the row leaves list_due_watches and the runner's retry-deactivate
# branch never executes for it. The cancel must discard the entry.
storage.update_watch(watch_id, active=False, next_poll="")
session._exec_watch({"call_id": "c1", "action": "cancel", "watch_name": "leak-watch"})
assert watch_id not in runner._terminal_dispatched
+89
View File
@@ -2,6 +2,10 @@
from __future__ import annotations
import sqlalchemy as sa
from turnstone.core.storage._schema import watches as watches_table
def _make_watch_kwargs(**overrides):
"""Build default kwargs for create_watch."""
@@ -101,6 +105,91 @@ class TestWatchListQueries:
db.update_watch("w1", active=False)
assert db.list_watches_for_ws("ws-1") == []
def test_find_by_name_returns_inactive(self, db):
"""``find_watch_by_name`` ignores the active filter — that is
what lets the cancel-by-name UX distinguish 'already completed'
from 'no such watch.'
"""
db.create_watch(**_make_watch_kwargs(watch_id="w1", ws_id="ws-1", name="completed"))
db.update_watch("w1", active=False)
row = db.find_watch_by_name("ws-1", "completed")
assert row is not None
assert row["watch_id"] == "w1"
assert not row["active"]
def test_find_by_name_matches_watch_id_prefix(self, db):
db.create_watch(**_make_watch_kwargs(watch_id="abcdef123", ws_id="ws-1", name="x"))
row = db.find_watch_by_name("ws-1", "abc")
assert row is not None
assert row["watch_id"] == "abcdef123"
def test_find_by_name_scoped_to_ws(self, db):
db.create_watch(**_make_watch_kwargs(watch_id="w1", ws_id="ws-1", name="shared"))
db.create_watch(**_make_watch_kwargs(watch_id="w2", ws_id="ws-2", name="shared"))
row = db.find_watch_by_name("ws-1", "shared")
assert row is not None
assert row["watch_id"] == "w1"
def test_find_by_name_returns_none_when_missing(self, db):
assert db.find_watch_by_name("ws-1", "ghost") is None
def test_find_by_name_empty_input_returns_none(self, db):
db.create_watch(**_make_watch_kwargs(watch_id="w1", ws_id="ws-1", name="x"))
assert db.find_watch_by_name("ws-1", "") is None
def test_find_by_name_treats_percent_as_literal(self, db):
"""A model-supplied '%' must NOT match arbitrary watch_ids.
Pre-escape, ``watch_id.like(f"{name_or_prefix}%")`` would
interpret '%' as 'match anything' and pick up the first row in
the workstream regardless of name.
"""
db.create_watch(**_make_watch_kwargs(watch_id="w1", ws_id="ws-1", name="real-watch"))
assert db.find_watch_by_name("ws-1", "%") is None
def test_find_by_name_treats_underscore_as_literal(self, db):
"""Same as the '%' case for the single-char LIKE wildcard."""
db.create_watch(**_make_watch_kwargs(watch_id="abcd", ws_id="ws-1", name="real-watch"))
# '_' would otherwise match any single char, picking up
# watch_ids beginning with 'a', 'b', etc.
assert db.find_watch_by_name("ws-1", "_") is None
def test_find_by_name_prefers_active_over_newer_inactive(self, db):
"""If a same-name pair exists where the inactive row is NEWER
than the active row, find_watch_by_name must still return the
active row. Pre-fix the query was ``ORDER BY created DESC
LIMIT 1`` which would return the newer inactive row and
cause the cancel UX to report 'already completed' for a name
whose live row is still polling.
Reachable in practice because storage allows out-of-band
writes (e.g. ``delete_watches_for_ws`` cleanup followed by
re-create, an admin manually flipping ``active``, or test
scaffolding) that bypass the create-time duplicate-name
guard.
"""
# Older active watch.
db.create_watch(**_make_watch_kwargs(watch_id="w-active", ws_id="ws-1", name="recurring"))
# Newer inactive watch with the same name. ``create_watch``
# stamps ``created`` to ``now`` at second resolution, so we
# bypass the API to give the inactive row a deterministically
# later timestamp.
db.create_watch(**_make_watch_kwargs(watch_id="w-inactive", ws_id="ws-1", name="recurring"))
with db._conn() as conn:
conn.execute(
sa.update(watches_table)
.where(watches_table.c.watch_id == "w-inactive")
.values(active=0, next_poll="", created="2099-01-01T00:00:00")
)
conn.commit()
row = db.find_watch_by_name("ws-1", "recurring")
assert row is not None
assert row["watch_id"] == "w-active"
assert row["active"]
def test_list_for_node(self, db):
db.create_watch(**_make_watch_kwargs(watch_id="w1", node_id="n1"))
db.create_watch(**_make_watch_kwargs(watch_id="w2", node_id="n1"))
+5 -5
View File
@@ -29,7 +29,7 @@ class TestVersionHtml:
def test_vendored_katex_skipped(self):
from turnstone.core.web_helpers import version_html
html = '<link rel="stylesheet" href="/shared/katex-0.16.44/katex.min.css">'
html = '<link rel="stylesheet" href="/shared/katex-0.16.47/katex.min.css">'
result = version_html(html)
assert result == html # unchanged
@@ -43,14 +43,14 @@ class TestVersionHtml:
def test_vendored_mermaid_skipped(self):
from turnstone.core.web_helpers import version_html
html = '<script src="/shared/mermaid-11.14.0/mermaid.min.js"></script>'
html = '<script src="/shared/mermaid-11.15.0/mermaid.min.js"></script>'
result = version_html(html)
assert result == html # unchanged
def test_vendored_hls_skipped(self):
from turnstone.core.web_helpers import version_html
html = '<script src="/shared/hls-1.6.15/hls.min.js"></script>'
html = '<script src="/shared/hls-1.6.16/hls.min.js"></script>'
result = version_html(html)
assert result == html # unchanged
@@ -76,7 +76,7 @@ class TestVersionHtml:
html = (
'<link rel="stylesheet" href="/shared/base.css">\n'
'<link rel="stylesheet" href="/shared/katex-0.16.44/katex.min.css">\n'
'<link rel="stylesheet" href="/shared/katex-0.16.47/katex.min.css">\n'
'<link rel="stylesheet" href="/static/style.css">\n'
'<script src="/shared/utils.js"></script>\n'
'<script src="/shared/hljs-11.11.1/highlight.min.js"></script>\n'
@@ -88,7 +88,7 @@ class TestVersionHtml:
assert f'/shared/utils.js?v={__version__}"' in result
assert f'/static/app.js?v={__version__}"' in result
# Vendored libs unchanged
assert '/shared/katex-0.16.44/katex.min.css"' in result
assert '/shared/katex-0.16.47/katex.min.css"' in result
assert '/shared/hljs-11.11.1/highlight.min.js"' in result
def test_version_matches_package(self):
+1 -1
View File
@@ -1,3 +1,3 @@
"""turnstone - Multi-node AI orchestration platform with tool use, agent routing, and cluster simulation."""
__version__ = "1.5.14"
__version__ = "1.5.18"
+40 -14
View File
@@ -12,14 +12,36 @@ import uuid
from typing import Any
def _get_storage() -> Any:
"""Initialize and return the storage backend."""
def _get_storage(args: argparse.Namespace) -> Any:
"""Initialize and return the storage backend.
Precedence (matches turnstone-server): CLI / config.toml ``[database]``
> ``TURNSTONE_DB_*`` env vars > hardcoded defaults.
"""
from turnstone.core.storage import init_storage
db_backend = os.environ.get("TURNSTONE_DB_BACKEND", "sqlite")
db_url = os.environ.get("TURNSTONE_DB_URL", "")
db_path = os.environ.get("TURNSTONE_DB_PATH", "")
return init_storage(db_backend, path=db_path, url=db_url)
def _pick(arg_name: str, env_name: str, default: str = "") -> Any:
# `is not None` (not truthy) so a legitimate falsy TOML value
# like `pool_size = 0` or `url = ""` still beats the env fallback.
val = getattr(args, arg_name, None)
if val is not None:
return val
return os.environ.get(env_name, default)
db_backend = str(_pick("db_backend", "TURNSTONE_DB_BACKEND", "sqlite"))
db_url = str(_pick("db_url", "TURNSTONE_DB_URL"))
db_path = str(_pick("db_path", "TURNSTONE_DB_PATH"))
db_pool_size = int(_pick("db_pool_size", "TURNSTONE_DB_POOL_SIZE", "2"))
return init_storage(
db_backend,
path=db_path,
url=db_url,
pool_size=db_pool_size,
sslmode=str(_pick("db_sslmode", "TURNSTONE_DB_SSLMODE")),
sslrootcert=str(_pick("db_sslrootcert", "TURNSTONE_DB_SSLROOTCERT")),
sslcert=str(_pick("db_sslcert", "TURNSTONE_DB_SSLCERT")),
sslkey=str(_pick("db_sslkey", "TURNSTONE_DB_SSLKEY")),
)
def _cmd_create_user(args: argparse.Namespace) -> None:
@@ -37,7 +59,7 @@ def _cmd_create_user(args: argparse.Namespace) -> None:
print("Error: invalid username (1-64 chars: letters, digits, . _ -)", file=sys.stderr)
sys.exit(1)
storage = _get_storage()
storage = _get_storage(args)
user_id = uuid.uuid4().hex
# Prompt for password
@@ -76,7 +98,7 @@ def _cmd_create_user(args: argparse.Namespace) -> None:
def _cmd_create_token(args: argparse.Namespace) -> None:
from turnstone.core.auth import generate_token, hash_token, token_prefix
storage = _get_storage()
storage = _get_storage(args)
if storage.get_user(args.user) is None:
print(f"Error: user {args.user} not found", file=sys.stderr)
@@ -110,7 +132,7 @@ def _cmd_create_token(args: argparse.Namespace) -> None:
def _cmd_list_users(args: argparse.Namespace) -> None:
storage = _get_storage()
storage = _get_storage(args)
users = storage.list_users()
if not users:
print("No users found.")
@@ -120,7 +142,7 @@ def _cmd_list_users(args: argparse.Namespace) -> None:
def _cmd_list_tokens(args: argparse.Namespace) -> None:
storage = _get_storage()
storage = _get_storage(args)
tokens = storage.list_api_tokens(args.user)
if not tokens:
print(f"No tokens found for user {args.user}.")
@@ -134,7 +156,7 @@ def _cmd_list_tokens(args: argparse.Namespace) -> None:
def _cmd_revoke_token(args: argparse.Namespace) -> None:
storage = _get_storage()
storage = _get_storage(args)
if storage.delete_api_token(args.token_id):
print(f"Revoked token {args.token_id}")
else:
@@ -297,7 +319,7 @@ def _cmd_list_node_metadata(args: argparse.Namespace) -> None:
"""List metadata for a node."""
import json
storage = _get_storage()
storage = _get_storage(args)
rows = storage.get_node_metadata(args.node_id)
if not rows:
print(f"No metadata for node: {args.node_id}")
@@ -324,7 +346,7 @@ def _cmd_set_node_metadata(args: argparse.Namespace) -> None:
"""Set a metadata key on a node."""
import json
storage = _get_storage()
storage = _get_storage(args)
# Check for auto-source conflict
existing = storage.get_node_metadata(args.node_id)
@@ -345,7 +367,7 @@ def _cmd_set_node_metadata(args: argparse.Namespace) -> None:
def _cmd_delete_node_metadata(args: argparse.Namespace) -> None:
"""Delete a metadata key from a node."""
storage = _get_storage()
storage = _get_storage(args)
existing = storage.get_node_metadata(args.node_id)
for r in existing:
@@ -395,6 +417,10 @@ def main() -> None:
prog="turnstone-admin",
description="Turnstone user and token administration",
)
from turnstone.core.config import add_config_arg, apply_config
add_config_arg(parser)
apply_config(parser, ["database"])
sub = parser.add_subparsers(dest="command")
p_cu = sub.add_parser("create-user", help="Create a new user")
+310 -15
View File
@@ -1278,21 +1278,29 @@ class CoordinatorClient:
allowed_tools: list[str] = [str(t) for t in allowed_full[:_SKILL_TOOLS_PROJECTION_CAP]]
if len(allowed_full) > _SKILL_TOOLS_PROJECTION_CAP:
allowed_tools.append(f"+{len(allowed_full) - _SKILL_TOOLS_PROJECTION_CAP} more")
skills.append(
{
"name": r.get("name") or "",
"category": r.get("category") or "",
"tags": tags,
"version": r.get("version") or "",
"description": r.get("description") or "",
"model": r.get("model") or "",
"enabled": bool(r.get("enabled")),
"risk_level": r.get("risk_level") or "",
"activation": r.get("activation") or "",
"kind": r["kind"],
"allowed_tools": allowed_tools,
}
)
skill_row: dict[str, Any] = {
"name": r.get("name") or "",
"category": r.get("category") or "",
"tags": tags,
"version": r.get("version") or "",
"description": r.get("description") or "",
"model": r.get("model") or "",
"enabled": bool(r.get("enabled")),
"risk_level": r.get("risk_level") or "",
"activation": r.get("activation") or "",
"kind": r["kind"],
}
# Omit ``allowed_tools`` when empty: an empty list reads as
# "no tools are usable by this skill" to a model that doesn't
# know the semantics, but the actual meaning is "no tools are
# pre-approved (auto-approve exemption list)". Real
# misdiagnosis happened in testing when a code-review skill
# with no auto-approve allowlist looked like it had been
# spawned with zero tool access. Dropping the key altogether
# when empty removes the ambiguity at the source.
if allowed_tools:
skill_row["allowed_tools"] = allowed_tools
skills.append(skill_row)
return {"skills": skills, "truncated": truncated}
# ------------------------------------------------------------------
@@ -1791,6 +1799,293 @@ def _serialize_verdicts(rows: list[Any]) -> list[dict[str, Any]]:
return out
# ---------------------------------------------------------------------------
# inspect_workstream — tiered output compression
# ---------------------------------------------------------------------------
#
# A coord doing a fan-out wave of inspect_workstream calls against
# tool-heavy children can blow the context budget on raw output alone
# (one child with a 100 KB bash result × N children). The previous
# safety net was ``_truncate_output``'s head+tail strategy, which
# silently drops *middle* messages — exactly the wrong shape for a
# coordinator trying to understand a child's trajectory (the LAST
# message tells the model what the child concluded; the FIRST sets
# the brief; the middle is the connective tissue).
#
# The three-tier degradation pattern matches the ``search`` tool's
# Tier-1/Tier-2/Tier-3 ladder at ``session.py:_format_search_results``.
# First tier whose serialized size fits the budget wins; the LLM
# learns which tier it got via the ``_tier`` field in the response
# (no API change to the coordinator tool).
#
# Budget chosen well under ``tool_truncation`` (typically 256 KB+) so
# the head+tail safety net never fires for inspect_workstream — that
# strategy silently drops middle messages, which is exactly the
# pathology this formatter exists to avoid.
_INSPECT_OUTPUT_BUDGET: int = 32_768
# Per-message head/tail snip when Tier 2 needs to compress content.
# Head dominates because the first ~600 chars of an assistant message
# usually contains the conclusion / direction; the tail is the
# follow-through. Tool results compress similarly: head shows what
# the tool was asked / what it found at the top; tail shows the final
# state / error suffix.
_INSPECT_MSG_CONTENT_HEAD: int = 600
_INSPECT_MSG_CONTENT_TAIL: int = 300
# Skeleton-tier preview length on the last assistant message. Single
# value because the skeleton wants ONE meaningful signal ("what did
# the child last say"), not a head/tail snip.
_INSPECT_SKELETON_LAST_PREVIEW: int = 400
# Snip lengths for tool-call ``function.arguments`` strings on
# assistant turns. Tighter than content snipping because tool calls
# often appear in clusters (10+ per turn for a fan-out) and the
# arguments JSON is dense — keep just enough to see what was invoked
# and the head of the args structure.
_INSPECT_TOOL_ARG_HEAD: int = 300
_INSPECT_TOOL_ARG_TAIL: int = 100
# Bytes ``_snip_head_tail`` reserves for the elision marker itself
# (``\n...[N chars elided]...\n``). A text shorter than
# ``head + tail + this margin`` passes through unsnipped — snipping
# would cost more bytes (the marker) than it saves.
_INSPECT_ELISION_MARGIN: int = 64
# Message-list trim ladder for the compact tier when per-message
# content snipping alone doesn't free enough budget. Each rung is
# ``(head_count, tail_count)`` — keep the first N + last M messages,
# elide the middle as ``{"_omitted": K}``. Tail-weighted because the
# last assistant turn carries the load-bearing "what did the child
# conclude" signal (same rationale as ``_inspect_skeleton``'s
# last-assistant preview). Tried in order; first rung whose
# serialized emission fits the budget wins. Mirrors the per-file
# sample ladder in ``_format_search_results`` at session.py:254.
_INSPECT_LIST_TRIM_LADDER: tuple[tuple[int, int], ...] = ((20, 30), (10, 20), (5, 10))
def _snip_head_tail(text: str, head: int, tail: int) -> str:
"""Head/tail snip with elision marker; passthrough when shorter than threshold."""
if not isinstance(text, str) or len(text) <= head + tail + _INSPECT_ELISION_MARGIN:
return text
elided = len(text) - head - tail
return text[:head] + f"\n...[{elided} chars elided]...\n" + text[-tail:]
def _compact_tool_calls(tool_calls: Any) -> Any:
"""Snip ``function.arguments`` on each tool-call entry; keep ``id`` and
``function.name`` verbatim.
OpenAI shape: ``[{"id": ..., "type": "function", "function":
{"name": ..., "arguments": "<json-string>"}}, ...]``. The
arguments string is the dominant size term on a fan-out turn that
issued many tool calls with multi-KB JSON arguments each;
preserving them verbatim re-opens the same size pressure the
compact tier is trying to relieve. Non-list / non-dict entries
pass through so a future shape change doesn't crash the formatter.
"""
if not isinstance(tool_calls, list):
return tool_calls
out: list[Any] = []
for call in tool_calls:
if not isinstance(call, dict):
out.append(call)
continue
compact_call: dict[str, Any] = {}
for k in ("id", "type"):
v = call.get(k)
if v:
compact_call[k] = v
func = call.get("function")
if isinstance(func, dict):
compact_func: dict[str, Any] = {}
name = func.get("name")
if name:
compact_func["name"] = name
args = func.get("arguments", "")
if args:
compact_func["arguments"] = _snip_head_tail(
args, _INSPECT_TOOL_ARG_HEAD, _INSPECT_TOOL_ARG_TAIL
)
compact_call["function"] = compact_func
out.append(compact_call)
return out
def _compact_message(msg: dict[str, Any]) -> dict[str, Any]:
"""Tier-2 per-message projection: keep role + identifier keys, snip content + tool_calls.
Tool-call linkage is the load-bearing "what happened" signal:
``tool_call_id`` on the result side matches an ``id`` in
``tool_calls`` on the issuing assistant turn. Stripping
``tool_calls`` (the pre-fix shape) left tool results dangling
against an invisible call the audit reader could see "bash
returned X" but not "the assistant asked for ``ls /tmp``". The
``arguments`` string is the size offender, so we snip it head/tail
rather than dropping the call entirely.
"""
content = msg.get("content", "")
snipped = _snip_head_tail(content, _INSPECT_MSG_CONTENT_HEAD, _INSPECT_MSG_CONTENT_TAIL)
compact: dict[str, Any] = {"role": msg.get("role"), "content": snipped}
# Tool-result linkage (result-side keys).
for k in ("tool_name", "tool_call_id", "name"):
v = msg.get(k)
if v:
compact[k] = v
# Tool-call request linkage (issuing-side list), snipped per-call.
tool_calls = msg.get("tool_calls")
if tool_calls:
compact["tool_calls"] = _compact_tool_calls(tool_calls)
return compact
def _inspect_skeleton(result: dict[str, Any]) -> dict[str, Any]:
"""Tier-3 fallback: state + counts + last assistant preview + terminal info.
Drops every message, keeping only aggregate signal: state, message
count, role distribution, verdict count + risk distribution, and a
short preview of the most recent assistant turn (the "what did this
child last say" signal). Terminal-state fields (``close_reason``,
``last_error``) and the ``live`` block pass through unchanged
because they're already small and load-bearing.
"""
messages = result.get("messages") or []
verdicts = result.get("verdicts") or []
role_counts: dict[str, int] = {}
for m in messages:
role = m.get("role") if isinstance(m, dict) else None
if role:
role_counts[role] = role_counts.get(role, 0) + 1
verdicts_by_risk: dict[str, int] = {}
for v in verdicts:
if isinstance(v, dict):
risk = v.get("risk_level") or "unknown"
verdicts_by_risk[risk] = verdicts_by_risk.get(risk, 0) + 1
last_preview = ""
for m in reversed(messages):
if not isinstance(m, dict) or m.get("role") != "assistant":
continue
c = m.get("content", "")
if isinstance(c, str) and c:
last_preview = c[:_INSPECT_SKELETON_LAST_PREVIEW]
if len(c) > _INSPECT_SKELETON_LAST_PREVIEW:
last_preview += "..."
break
skeleton: dict[str, Any] = {
# Storage row keys verbatim from ``get_workstreams_batch``
# (the projection backing ``get_workstream`` → ``inspect()``):
# ``ws_id``, ``skill_id``. No fallback to ``id`` / ``skill``
# — fail loud on storage column drift rather than silently
# emitting null.
"ws_id": result["ws_id"],
"state": result.get("state"),
"title": result.get("title"),
"skill": result["skill_id"],
"message_count": len(messages),
"roles": role_counts,
"verdict_count": len(verdicts),
"verdicts_by_risk": verdicts_by_risk,
"last_assistant_preview": last_preview,
"_tier": "skeleton",
"_tier_note": (
"Output exceeded the inspect_workstream budget at both full and compact "
"tiers; skeleton-only. Re-call with a smaller ``message_limit`` to fit "
"the compact tier, or read individual messages via the storage admin path."
),
}
for k in ("close_reason", "last_error", "live"):
v = result.get(k)
if v:
skeleton[k] = v
return skeleton
def _format_inspect_tiered(result: dict[str, Any], *, budget: int = _INSPECT_OUTPUT_BUDGET) -> str:
"""Serialize an ``inspect_workstream`` result with tiered degradation.
Tier 1 (full): every message verbatim used when the size fits.
Tier 2 (compact): per-message ``{role, head/tail-snipped content,
tool linkage, snipped tool_calls.arguments}`` for
every message, then a head+tail message-list trim
ladder when content snipping alone doesn't free
enough budget.
Tier 3 (skeleton): no messages counts + last assistant preview only.
First emission whose JSON serialization fits ``budget`` wins.
``_tier`` appears on every non-error emission so the coordinator
LLM (and any audit reader) can see which compression rung the
output landed on without inferring from length. Error-shape
results (missing or cross-tenant ws_id) bypass tiering entirely
they're already small and the ``error`` key signals the shape.
The intermediate Tier-2 list-trim rungs exist because content
snipping alone fails on workloads where many small messages
overflow the budget by sheer count (``message_limit=200`` × a few
hundred chars each). In that regime, dropping content-snipping
saves zero bytes per message, so without the list-trim ladder
Tier-2 produces output strictly larger than Tier-1 (added
``_tier_note``) and the formatter fell through to skeleton
losing every message when a head+tail message-list trim would
have preserved dozens. Mirrors the per-file sample ladder in
``_format_search_results`` (session.py:_SEARCH_TIER2_SAMPLE_LADDER).
"""
if "error" in result:
# Cross-tenant guard / not-found responses — pass through.
return json.dumps(result, default=str, separators=(",", ":"))
tier1 = {**result, "_tier": "full"}
out1 = json.dumps(tier1, default=str, separators=(",", ":"))
if len(out1) <= budget:
return out1
messages = result.get("messages") or []
compact_msgs = [_compact_message(m) if isinstance(m, dict) else m for m in messages]
tier2_note_full = (
"Output exceeded the inspect_workstream budget at the full tier; messages "
"are head/tail-snipped at "
f"{_INSPECT_MSG_CONTENT_HEAD}/{_INSPECT_MSG_CONTENT_TAIL} chars. Re-call "
"with a smaller ``message_limit`` for a tighter tail, or include_provider_"
"content=False if it was on."
)
tier2 = {
**result,
"messages": compact_msgs,
"_tier": "compact",
"_tier_note": tier2_note_full,
}
out2 = json.dumps(tier2, default=str, separators=(",", ":"))
if len(out2) <= budget:
return out2
# Tier-2 list-trim ladder: keep head N + tail M, elide the middle.
# Tail-weighted because the recent turns carry the load-bearing
# signal ("what did the child conclude") — same reason
# ``_inspect_skeleton`` keeps a last-assistant preview rather than
# a first-user preview.
total = len(compact_msgs)
for head_n, tail_n in _INSPECT_LIST_TRIM_LADDER:
if head_n + tail_n >= total:
# Rung doesn't actually trim — would re-emit Tier-2 verbatim.
continue
omitted = total - head_n - tail_n
trimmed: list[Any] = (
compact_msgs[:head_n] + [{"_omitted": omitted}] + compact_msgs[-tail_n:]
)
tier2_trim_note = (
f"Output exceeded the inspect_workstream budget at the compact tier; "
f"keeping first {head_n} + last {tail_n} of {total} messages, eliding "
f"{omitted} middle messages. Re-call with a smaller ``message_limit`` "
"to fit the full compact tier."
)
tier2_trim = {
**result,
"messages": trimmed,
"_tier": "compact",
"_tier_note": tier2_trim_note,
}
out2_trim = json.dumps(tier2_trim, default=str, separators=(",", ":"))
if len(out2_trim) <= budget:
return out2_trim
skeleton = _inspect_skeleton(result)
return json.dumps(skeleton, default=str, separators=(",", ":"))
# ---------------------------------------------------------------------------
# wait_for_workstream — last-message extraction
# ---------------------------------------------------------------------------
+7 -3
View File
@@ -3370,11 +3370,15 @@ function _renderMcpServers(items) {
// outcome for this server.
var refreshPill = "";
if (newestRefreshAt !== null) {
var ageSeconds = Math.max(0, Math.floor(Date.now() / 1000 - newestRefreshAt));
var ageSeconds = Math.max(
0,
Math.floor(Date.now() / 1000 - newestRefreshAt),
);
var ageShort;
if (ageSeconds < 60) ageShort = ageSeconds + "s";
else if (ageSeconds < 3600) ageShort = Math.floor(ageSeconds / 60) + "m";
else if (ageSeconds < 86400) ageShort = Math.floor(ageSeconds / 3600) + "h";
else if (ageSeconds < 86400)
ageShort = Math.floor(ageSeconds / 3600) + "h";
else ageShort = Math.floor(ageSeconds / 86400) + "d";
var outcomeText = newestRefreshOutcome || "unknown";
var pillCls =
@@ -3443,7 +3447,7 @@ function _renderMcpServers(items) {
consentCount +
'" title="Drop all ' +
consentCount +
" user consents for this server">bulk-revoke (" +
' user consents for this server">bulk-revoke (' +
consentCount +
")</button>";
}
@@ -10,7 +10,7 @@
<link rel="stylesheet" href="/shared/base.css">
<link rel="stylesheet" href="/shared/ui-base.css">
<link rel="stylesheet" href="/shared/chat.css">
<link rel="stylesheet" href="/shared/katex-0.16.45/katex.min.css">
<link rel="stylesheet" href="/shared/katex-0.16.47/katex.min.css">
<link rel="stylesheet" href="/static/style.css">
<link rel="stylesheet" href="/static/coordinator/coordinator.css">
<style>
@@ -634,7 +634,7 @@
<script src="/shared/composer_attachments.js"></script>
<script src="/shared/composer_queue.js"></script>
<script src="/shared/status_bar.js"></script>
<script src="/shared/katex-0.16.45/katex.min.js"></script>
<script src="/shared/katex-0.16.47/katex.min.js"></script>
<script src="/shared/hljs-11.11.1/highlight.min.js"></script>
<script src="/shared/renderer.js"></script>
<script src="/static/coordinator/coordinator.js"></script>
+22
View File
@@ -54,6 +54,27 @@ def set_config_path(path: str) -> None:
_cache = None # invalidate cache so next load_config() re-reads
def _warn_if_world_readable(cfg_path: Path) -> None:
"""Warn once if config.toml is group- or world-readable.
DB passwords, OIDC client secrets, and TLS key paths live in this
file operators usually want it at 0600. POSIX-only; no-ops where
``stat()`` modes are meaningless (Windows).
"""
try:
mode = cfg_path.stat().st_mode & 0o777
except OSError:
return
if mode & 0o077:
log.warning(
"%s is mode %04o (group/world-readable); secrets live here — "
"run `chmod 0600 %s` to restrict access",
cfg_path,
mode,
cfg_path,
)
def load_config(section: str | None = None) -> dict[str, Any]:
"""Load config.toml and return the full dict or a specific section.
@@ -66,6 +87,7 @@ def load_config(section: str | None = None) -> dict[str, Any]:
cfg_path = _resolve_config_path()
if cfg_path.is_file():
try:
_warn_if_world_readable(cfg_path)
_cache = tomllib.loads(cfg_path.read_text(encoding="utf-8"))
except Exception as exc:
log.warning("Failed to parse %s: %s", cfg_path, exc)
+35 -10
View File
@@ -19,11 +19,11 @@ Channels:
Drain preserves FIFO order; non-matching entries stay queued. Each
entry can carry an optional ``valid_until`` predicate that drain
evaluates outside the queue lock; entries whose predicate returns
``False`` (or raises) are silently dropped without delivery used by
producers whose payload becomes stale if the underlying state changes
between enqueue and drain (e.g. ``idle_children`` re-checks the active
child set, dropping the nudge if every child finished while the queue
sat). Operations are atomic under an internal :class:`threading.Lock`.
``False`` are dropped (logged at ``info`` normal lifecycle outcome,
e.g. ``idle_children`` after every child closed) and entries whose
predicate raises are dropped (logged at ``warning`` with ``exc_info``
a misbehaving predicate). Operations are atomic under an internal
:class:`threading.Lock`.
"""
from __future__ import annotations
@@ -32,9 +32,13 @@ import threading
from collections import deque
from typing import TYPE_CHECKING, Any, Literal, NamedTuple
from turnstone.core.log import get_logger
if TYPE_CHECKING:
from collections.abc import Callable
log = get_logger(__name__)
Channel = Literal["user", "tool", "any"]
_VALID_CHANNELS: frozenset[str] = frozenset({"user", "tool", "any"})
@@ -139,7 +143,12 @@ class NudgeQueue:
self._items = kept
# Predicates evaluate outside the lock — they may do storage
# I/O or other work that shouldn't block other producers /
# the drain consumer's other queues.
# the drain consumer's other queues. Drop-level distinction:
# a ``False`` return is a normal lifecycle outcome (the
# producer's snapshot is stale — e.g. ``idle_children`` after
# every child closed) and logs at ``info``; a raised exception
# is a wiring bug (predicate is misbehaving) and stays at
# ``warning`` with ``exc_info`` so the traceback surfaces.
out: list[tuple[str, str, dict[str, Any] | None]] = []
for entry in candidates:
if entry.valid_until is None:
@@ -148,11 +157,27 @@ class NudgeQueue:
try:
if entry.valid_until():
out.append((entry.nudge_type, entry.text, entry.metadata))
continue
log.info(
"nudge_queue.predicate_dropped",
extra={
"nudge_type": entry.nudge_type,
"channel": entry.channel,
"reason": "predicate_false",
"text_len": len(entry.text),
},
)
except Exception:
# Predicate raising is treated as "no longer valid" —
# drop silently rather than letting one bad predicate
# poison the whole drain batch.
pass
log.warning(
"nudge_queue.predicate_dropped",
extra={
"nudge_type": entry.nudge_type,
"channel": entry.channel,
"reason": "predicate_raised",
"text_len": len(entry.text),
},
exc_info=True,
)
return out
def __len__(self) -> int:
+53 -30
View File
@@ -1670,13 +1670,17 @@ class ChatSession:
The closure carries:
- a soft cap on per-session ``"watch_triggered"`` depth via
:data:`_WATCH_QUEUE_SOFT_CAP` + drop-oldest-on-saturation.
- a ``valid_until`` predicate that re-checks
``storage.is_watch_active(watch_id)`` at drain time so a
cancelled watch's last splat doesn't ride out a future wake.
- producer-side :func:`sanitize_payload` over the whole
formatted message so steering-vector / control-char payloads
sourced from arbitrary shell output can't tamper with the
envelope at interpolation time.
No ``valid_until`` predicate is wired: ``WatchRunner._poll_watch``
commits ``active=False`` for terminal fires right after dispatch
returns, and an ``is_watch_active`` predicate would race that
write at drain time and drop the fire the model was meant to see.
A user-cancelled watch's last splat is informative (the reminder
carries ``is_final=True``), not stale-noise to suppress.
"""
self._watch_runner = runner
nudge_queue = self._nudge_queue
@@ -1707,18 +1711,6 @@ class ChatSession:
_WATCH_QUEUE_SOFT_CAP,
)
def _still_active() -> bool:
# Re-checked at drain time outside the queue lock — if
# the watch was cancelled between fire and drain, the
# entry gets dropped silently rather than splicing a
# stale result onto the user's next turn. Single-column
# ``is_watch_active`` avoids the full-row marshal of
# ``get_watch`` on this hot path.
try:
return get_storage().is_watch_active(watch_id)
except Exception:
return False
def _maybe_sanitize(v: Any) -> Any:
return sanitize_payload(v) if isinstance(v, str) else v
@@ -1731,7 +1723,6 @@ class ChatSession:
"watch_triggered",
sanitized,
"any",
valid_until=_still_active,
metadata=metadata or None,
)
@@ -7014,9 +7005,20 @@ class ChatSession:
msg = f"Error: {result['error']}"
self._report_tool_result(call_id, "spawn_workstream", msg, is_error=True)
return call_id, msg
# Successful spawn — surface ws_id + node_id + name + routing
# strategy so the coordinator can follow up with inspect / send
# and explain why a given node was chosen. ``status`` was
# Defensive: absence of ``error`` is the success signal, but
# a malformed upstream response could land here with no
# ``ws_id``. Without this check the LLM gets
# ``{"child_ws_id": null}`` and chases a null id through
# follow-up tools. Mirrors the matching guard in
# ``_exec_spawn_batch`` (denied row on empty ws_id).
child_ws_id = str(result.get("ws_id") or "")
if not child_ws_id:
msg = "Error: spawn returned no ws_id"
self._report_tool_result(call_id, "spawn_workstream", msg, is_error=True)
return call_id, msg
# Successful spawn — surface child_ws_id + node_id + name +
# routing strategy so the coordinator can follow up with inspect
# / send and explain why a given node was chosen. ``status`` was
# historically included but it was the routing-proxy's HTTP
# code (always 200 on this branch); the absence of an
# ``error`` field is the success signal. Dropped here to
@@ -7027,14 +7029,19 @@ class ChatSession:
# ``inspect_workstream``.
summary = json.dumps(
{
"ws_id": result.get("ws_id"),
# Key is ``child_ws_id`` (not ``ws_id``) so the coordinator
# LLM doesn't recency-bias toward feeding the spawn-return
# straight back into another ``spawn_workstream(ws_id=...)``
# call. On large fan-outs this cascaded into self-inflicted
# re-spawn loops instead of progressing to ``wait_for_workstream``.
"child_ws_id": child_ws_id,
"name": result.get("name"),
"node_id": result.get("node_id"),
"routing_strategy": result.get("routing_strategy"),
},
separators=(",", ":"),
)
self._report_tool_result(call_id, "spawn_workstream", f"spawned {result.get('ws_id', '?')}")
self._report_tool_result(call_id, "spawn_workstream", f"spawned {child_ws_id}")
return call_id, summary
# Cap per batch call. Matches the ``wait_for_workstream`` ws_ids
@@ -7165,7 +7172,9 @@ class ChatSession:
denied.append({"idx": idx, "reason": "spawn returned no ws_id"})
continue
results[str(idx)] = {
"ws_id": ws_id,
# ``child_ws_id`` (not ``ws_id``) — see the matching
# comment in ``_exec_spawn_workstream``.
"child_ws_id": ws_id,
"name": result.get("name", ""),
"node_id": result.get("node_id", ""),
# ``status`` deliberately omitted — see the matching
@@ -7243,6 +7252,8 @@ class ChatSession:
}
def _exec_inspect_workstream(self, item: dict[str, Any]) -> tuple[str, str]:
from turnstone.console.coordinator_client import _format_inspect_tiered
call_id = item["call_id"]
ws_id = item["ws_id"]
try:
@@ -7255,8 +7266,13 @@ class ChatSession:
msg = f"Error: inspect_workstream failed: {e}"
self._report_tool_result(call_id, "inspect_workstream", msg, is_error=True)
return call_id, msg
output = json.dumps(result, default=str, separators=(",", ":"))
# Summary for UI: state + message count
# Tiered output: full → compact (head/tail-snipped messages) →
# skeleton (counts + last-assistant preview). First tier that
# fits the budget wins; the LLM sees a ``_tier`` field on every
# non-error response. ``_truncate_output`` remains the safety
# net for the (rare) skeleton-exceeds-budget case — guarding
# against a single-field blowup we didn't anticipate.
output = _format_inspect_tiered(result)
desc = f"{result.get('state', '?')} ({len(result.get('messages', []))} msgs)"
self._report_tool_result(call_id, "inspect_workstream", desc)
return call_id, self._truncate_output(output)
@@ -10473,16 +10489,23 @@ class ChatSession:
msg = "Error: storage unavailable"
self._report_tool_result(call_id, "watch", msg, is_error=True)
return call_id, msg
watches = storage.list_watches_for_ws(self._ws_id)
target = None
for w in watches:
if w["name"] == name or w["watch_id"].startswith(name):
target = w
break
target = storage.find_watch_by_name(self._ws_id, name)
if target is None:
msg = f'Watch "{name}" not found.'
self._report_tool_result(call_id, "watch", msg, is_error=True)
return call_id, msg
# In either branch below the row leaves ``list_due_watches``
# view (already-inactive or just-cancelled with empty
# next_poll), so the runner's retry-deactivate branch will
# never reclaim a pending ``_terminal_dispatched`` entry.
# Clear it here to bound the lifetime of any leftover from
# a previous dispatch-then-failed-row-write.
if self._watch_runner is not None:
self._watch_runner.forget_terminal_dispatched(target["watch_id"])
if not target["active"]:
msg = f'Watch "{target["name"]}" already completed (auto-cancelled).'
self._report_tool_result(call_id, "watch", msg)
return call_id, msg
storage.update_watch(target["watch_id"], active=False, next_poll="")
msg = f'Watch "{target["name"]}" cancelled.'
self._report_tool_result(call_id, "watch", msg)
+205 -18
View File
@@ -179,6 +179,24 @@ class SessionUIBase:
# ``/dashboard`` payload. Capped so a long-running skill
# workstream can't fill the live block with stale rows.
self._recent_auto_approvals: list[dict[str, Any]] = []
# Maps ``call_id`` → ``(auto_approve_reason, inserted_ts)`` for
# verdicts that arrive AFTER ``approve_tools`` already returned.
# The LLM judge tier is asynchronous: ``on_intent_verdict`` can
# fire seconds later for a tool that ``approve_tools``
# short-circuited via one of the auto-approve branches. Without
# this lookup the late-arriving LLM verdict lands with
# ``user_decision="pending"`` and stays that way forever (no
# ``resolve_approval`` cycle on the auto-approve path).
#
# Lifetime is bounded by ``_AUTO_APPROVE_REASON_TTL`` rather
# than by a count-cap or by session lifetime: a fixed cap
# would silently break the fix on the (N+1)th in-flight
# auto-approve; "evict on consume" alone would leak entries
# whenever the LLM judge is disabled (no ``on_intent_verdict``
# ever fires to drain them). TTL means entries clear lazily
# on the next ``_record_auto_approves`` write whether or not
# the LLM judge tier is active. Guarded by ``_ws_lock``.
self._auto_approve_reasons: dict[str, tuple[str, float]] = {}
# Foreground gate — used by the CLI's WorkstreamTerminalUI to
# block output when the workstream is in the background.
# Starts set so non-CLI UIs can skip any explicit management.
@@ -367,6 +385,7 @@ class SessionUIBase:
feedback: str | None = None,
*,
always: bool = False,
timeout: bool = False,
) -> None:
"""Unblock a pending approval with the caller's decision.
@@ -383,8 +402,22 @@ class SessionUIBase:
can label their resolved-status pill correctly). Keyword-only
+ default ``False`` so the four pre-existing callers (cancel,
timeout, channel adapters) compile unchanged.
``timeout`` flips the persisted ``user_decision`` from
``"denied"`` to ``"timeout"`` so the audit trail can
distinguish an active user denial from a passive
approval-timeout expiry the feedback string carries the
same information today but operators querying on the
``user_decision`` column alone could not tell them apart.
Mutually exclusive with ``approved=True`` (a timeout is a
passive denial); the combination raises ``ValueError`` so a
future caller can't accidentally ship a row whose audit
column says ``"timeout"`` while the SSE event reports
``approved=True``.
"""
decision_str = "approved" if approved else "denied"
if timeout and approved:
raise ValueError("resolve_approval: timeout=True is incompatible with approved=True")
decision_str = "timeout" if timeout else ("approved" if approved else "denied")
# Swap-and-clear + set decision under lock to avoid racing
# with the daemon judge thread's ``on_intent_verdict`` appends.
with self._ws_lock:
@@ -544,8 +577,15 @@ class SessionUIBase:
# the early return — the fall-through
# branch never runs on this path, so without
# this the policy bypass is invisible to
# /dashboard + audit.
# /dashboard + audit. ``_record_auto_approves``
# MUST run before ``_persist_auto_approved_*``
# so the call_id → reason lookup map is
# populated before the heuristic INSERTs go
# in: otherwise an LLM judge verdict firing
# in the gap lands with ``user_decision=
# "pending"`` and stays that way.
self._record_auto_approves(items)
self._persist_auto_approved_heuristic_verdicts(items)
self._enqueue(
{
"type": "tool_info",
@@ -608,7 +648,12 @@ class SessionUIBase:
self._ws_current_activity = f"{label}: {preview}" if label else ""
self._ws_activity_state = "tool" if label else ""
self._broadcast_activity()
# ``_record_auto_approves`` runs FIRST so the call_id → reason
# lookup is populated before the heuristic INSERT can race
# against a concurrent LLM judge verdict — see the matching
# comment on the policy-deny branch above.
self._record_auto_approves(items)
self._persist_auto_approved_heuristic_verdicts(items)
self._enqueue({"type": "tool_info", "items": self._serialize_approval_items(items)})
return True, None
@@ -628,19 +673,34 @@ class SessionUIBase:
# commit instead of N (was visible as time-to-render-prompt
# latency for fan-out turns); the per-item Prometheus call stays
# in the loop because it's a lock+increment, not a DB round-trip.
#
# ``user_decision`` is stamped per-verdict here so the row lands
# with a meaningful value at insert: auto-approved items
# (mixed-path case: policy allowed some, others still prompt)
# carry their auto_approve_reason directly; items still pending
# operator decision carry ``"pending"`` and get updated by
# ``resolve_approval`` on close. ``_pending_verdicts`` only
# tracks the latter — auto-approved verdicts are already final.
heuristic_verdicts: list[dict[str, Any]] = []
pending_verdicts: list[dict[str, Any]] = []
for item in items:
hv = item.get("_heuristic_verdict")
if hv:
heuristic_verdicts.append(hv)
# Subclass-overridden Prometheus surface: WebUI feeds
# the per-node /metrics endpoint, ConsoleCoordinatorUI
# feeds the console's /metrics endpoint via ConsoleMetrics.
self._record_judge_metric(hv)
if not hv:
continue
if item.get("auto_approved"):
hv["user_decision"] = item.get("auto_approve_reason", "") or "pending"
else:
hv["user_decision"] = "pending"
pending_verdicts.append(hv)
heuristic_verdicts.append(hv)
# Subclass-overridden Prometheus surface: WebUI feeds
# the per-node /metrics endpoint, ConsoleCoordinatorUI
# feeds the console's /metrics endpoint via ConsoleMetrics.
self._record_judge_metric(hv)
self._persist_intent_verdicts_bulk(heuristic_verdicts, default_tier="heuristic")
with self._ws_lock:
self._pending_verdicts = heuristic_verdicts
self._pending_verdicts = pending_verdicts
# Record any items the policy block already auto-approved
# before falling through to the prompt — without this the
@@ -676,8 +736,14 @@ class SessionUIBase:
if not self._approval_event.wait(timeout=self._APPROVAL_WAIT_TIMEOUT):
# Approval timed out (e.g., user disconnected). Deny via
# resolve_approval so verdicts and state are updated consistently.
# Feedback string derives from ``_APPROVAL_WAIT_TIMEOUT`` so the
# text follows the constant if the timeout knob moves.
log.warning("Approval timed out for ws_id=%s", self.ws_id)
self.resolve_approval(False, "Approval timed out after 1 hour")
self.resolve_approval(
False,
f"Approval timed out after {self._APPROVAL_WAIT_TIMEOUT}s",
timeout=True,
)
self._pending_approval = None
approved, feedback = self._approval_result
@@ -714,8 +780,17 @@ class SessionUIBase:
``user_decision`` immediately (if the approval already
resolved) or parks the verdict in ``_pending_verdicts`` for
``resolve_approval`` to stamp on close.
When the verdict arrives for a call_id that ``approve_tools``
already auto-approved (the LLM judge is async and can fire
seconds after the auto-approve path returned), stamp the
``auto_approve_reason`` onto the verdict before persist so
the row lands with a meaningful ``user_decision`` instead of
the default ``"pending"`` (which would never be updated for
this code path).
"""
call_id = verdict.get("call_id", "")
auto_reason = ""
if call_id:
with self._ws_lock:
if (
@@ -725,6 +800,14 @@ class SessionUIBase:
oldest_key = next(iter(self._llm_verdicts))
del self._llm_verdicts[oldest_key]
self._llm_verdicts[call_id] = verdict
# Pop (not get) — once consumed the entry isn't useful
# again; TTL pruning at the writer side keeps the
# never-consumed case bounded too.
entry = self._auto_approve_reasons.pop(call_id, None)
if entry is not None:
auto_reason = entry[0]
if auto_reason:
verdict["user_decision"] = auto_reason
self._enqueue({"type": "intent_verdict", **verdict})
# Kind-specific cross-stream broadcast — ConsoleCoordinatorUI
# overrides to push onto the cluster bus so a coord parent's
@@ -743,6 +826,17 @@ class SessionUIBase:
# WRONG decision. Storage UPDATE happens outside the lock
# on the already-resolved path — no contention with other
# ws-scoped work.
# If ``auto_reason`` was stamped above, the verdict already
# carries the final ``user_decision`` for this row. Neither
# path below applies: appending to ``_pending_verdicts`` would
# cause ``resolve_approval`` (on the manual-approval sibling
# in a mixed batch) to overwrite the auto-reason with
# ``"approved"``/``"denied"``/``"timeout"``; the
# ``_persist_verdict_decisions`` immediate-stamp path would
# overwrite it the same way from a prior cycle's decision.
# Skip both so the audit trail keeps the auto-approve reason.
if auto_reason:
return
with self._ws_lock:
decision = self._last_verdict_decision
if not decision:
@@ -811,9 +905,29 @@ class SessionUIBase:
"tier": v.get("tier", default_tier),
"judge_model": v.get("judge_model", ""),
"latency_ms": v.get("latency_ms", 0),
"user_decision": v.get("user_decision", "pending"),
}
for v in verdicts
]
# Plain INSERT (not UPSERT) at the bulk site. The race
# where a daemon-judge verdict lands BEFORE this bulk
# write IS reachable today: ``_evaluate_intent``
# (session.py) spawns the daemon thread before
# ``approve_tools`` is called, and the daemon's first
# emission (heuristic-only short batch, fast LLM response,
# or cancel-event ``_deliver_fallbacks`` from judge.py)
# can fire ``_persist_intent_verdict`` before this bulk
# INSERT runs. Outcome of that race is unchanged by the
# per-row UPSERT switch: the bulk INSERT statement aborts
# on PK collision regardless of whether the colliding row
# was planted by INSERT or UPSERT, and the wrapping
# ``try/except`` swallows it. Race A (daemon fires AFTER
# bulk) IS improved by the fix: heuristic→llm_fallback
# upgrade-in-place now lands. Future bulk-side hardening
# (``ON CONFLICT DO NOTHING``) would preserve the OTHER
# rows in the batch when one collides, but would keep the
# daemon's ``tier`` ("llm"/"llm_fallback") for the
# colliding row instead of the bulk's heuristic stamp.
storage.create_intent_verdicts_bulk(rows)
except Exception:
log.debug("Failed to bulk-persist intent verdicts", exc_info=True)
@@ -824,15 +938,21 @@ class SessionUIBase:
*,
default_tier: str = "llm",
) -> None:
"""Persist an intent-judge verdict row.
"""Persist an intent-judge verdict row via UPSERT.
Used by both the async LLM-tier path (``on_intent_verdict``,
default tier ``"llm"``) and the synchronous heuristic-tier
path (``approve_tools``, caller passes ``default_tier="heuristic"``).
Used by the async LLM-tier path (``on_intent_verdict``,
default tier ``"llm"``). Routes through ``upsert_intent_verdict``
because ``tier="llm_fallback"`` verdicts deliberately reuse the
heuristic verdict's ``verdict_id`` (see ``judge.py`` —
``_deliver_fallbacks`` and the in-loop fallback path)
so the row gets "upgraded in place" from heuristic
llm_fallback. A plain INSERT would collide on the PK and the
upgrade would be lost to a silently-swallowed exception.
``default_tier`` only matters when the verdict dict doesn't
already carry a ``tier`` key both real producers always set it,
but the fallback is the right call-site label so a malformed
verdict still lands on the correct row classification.
already carry a ``tier`` key both real producers always set
it, but the fallback is the right call-site label so a
malformed verdict still lands on the correct row
classification.
"""
try:
from turnstone.core.storage._registry import get_storage
@@ -840,7 +960,7 @@ class SessionUIBase:
storage = get_storage()
if storage is None:
return
storage.create_intent_verdict(
storage.upsert_intent_verdict(
verdict_id=verdict.get("verdict_id", ""),
ws_id=self.ws_id,
call_id=verdict.get("call_id", ""),
@@ -855,6 +975,7 @@ class SessionUIBase:
tier=verdict.get("tier", default_tier),
judge_model=verdict.get("judge_model", ""),
latency_ms=verdict.get("latency_ms", 0),
user_decision=verdict.get("user_decision", "pending"),
)
except Exception:
log.debug("Failed to persist intent verdict", exc_info=True)
@@ -955,6 +1076,14 @@ class SessionUIBase:
# workstreams that auto-approve dozens of tool calls per turn.
_RECENT_AUTO_APPROVALS_MAX = 10
# TTL on the call_id → auto_approve_reason map. Sized to comfortably
# cover the LLM judge's worst-case latency (cold start + a slow model
# + queue depth). Pruning happens lazily at write time so the cost
# is paid only on the next auto-approve event; a session that goes
# quiet after auto-approving never pays the prune cost at all but
# the resident-set is also tiny.
_AUTO_APPROVE_REASON_TTL = 60.0
@staticmethod
def _serialize_approval_items(items: list[dict[str, Any]]) -> list[dict[str, Any]]:
"""Project each item to the wire shape the SSE event payload uses.
@@ -1026,6 +1155,38 @@ class SessionUIBase:
else:
it["auto_approve_reason"] = reason
def _persist_auto_approved_heuristic_verdicts(self, items: list[dict[str, Any]]) -> None:
"""Persist heuristic verdicts for items the auto-approve path resolved.
The manual-approval block at the bottom of ``approve_tools``
handles its own verdict persistence (and stamps
``user_decision`` per item auto-approved items in a
mixed-path batch carry their reason, pending items carry
``"pending"``). The auto-approve early-return branches
(policy-allow-with-deny, blanket flag, auto_approve_tools
match) used to drop heuristic verdicts on the floor an
operator querying ``user_decision`` for the auto-approve
reason would find no row at all, conflating "we auto-approved
silently" with "the judge didn't run". This helper closes
that gap: walk ``items``, persist each auto-approved verdict
with its reason stamped, and fan a metric row per verdict.
Safe to call with empty items.
"""
if not items:
return
verdicts: list[dict[str, Any]] = []
for it in items:
if not it.get("auto_approved"):
continue
hv = it.get("_heuristic_verdict")
if not hv:
continue
hv["user_decision"] = it.get("auto_approve_reason", "") or "pending"
verdicts.append(hv)
self._record_judge_metric(hv)
if verdicts:
self._persist_intent_verdicts_bulk(verdicts, default_tier="heuristic")
def _record_auto_approves(self, items: list[dict[str, Any]]) -> None:
"""Append auto-approved items to the per-ws ring buffer + audit log.
@@ -1062,6 +1223,32 @@ class SessionUIBase:
overflow = len(self._recent_auto_approvals) - self._RECENT_AUTO_APPROVALS_MAX
if overflow > 0:
self._recent_auto_approvals = self._recent_auto_approvals[overflow:]
# Mirror call_id → reason into the lookup map so a late
# ``on_intent_verdict`` (LLM judge tier) can stamp the
# right ``user_decision`` instead of leaving the verdict
# stuck as ``"pending"`` forever. Prune expired entries
# first (lazy TTL eviction) so a session with the LLM
# judge disabled doesn't accumulate entries that will
# never be consumed. Skip the rebuild when the map is
# empty or all entries are still fresh — the common case
# on a healthy LLM-judge-enabled session where entries
# drain via ``on_intent_verdict.pop`` within the TTL.
cutoff = ts - self._AUTO_APPROVE_REASON_TTL
if self._auto_approve_reasons and any(
ins_ts < cutoff for _, ins_ts in self._auto_approve_reasons.values()
):
self._auto_approve_reasons = {
cid: (reason, ins_ts)
for cid, (reason, ins_ts) in self._auto_approve_reasons.items()
if ins_ts >= cutoff
}
for entry in appended:
cid = entry["call_id"]
if cid:
self._auto_approve_reasons[cid] = (
entry["auto_approve_reason"],
ts,
)
# Audit emission — one row per ``approve_tools`` call (not one
# per item) keeps the audit table from blowing up on
# tool-heavy turns while still capturing every tool name +
+93 -7
View File
@@ -72,6 +72,9 @@ from turnstone.core.storage._schema import (
from turnstone.core.storage._utils import (
HEURISTIC_RULE_MUTABLE as _HEURISTIC_RULE_MUTABLE,
)
from turnstone.core.storage._utils import (
LIKE_ESCAPE as _LIKE_ESCAPE,
)
from turnstone.core.storage._utils import (
MCP_SERVER_MUTABLE as _MCP_SERVER_MUTABLE,
)
@@ -102,6 +105,9 @@ from turnstone.core.storage._utils import (
from turnstone.core.storage._utils import (
VERDICT_MUTABLE as _VERDICT_MUTABLE,
)
from turnstone.core.storage._utils import (
escape_like as _escape_like,
)
from turnstone.core.storage._utils import (
normalize_search_terms as _normalize_search_terms,
)
@@ -120,11 +126,6 @@ from turnstone.core.workstream import BULK_CLOSE_STATE_VALUES, WorkstreamKind
log = get_logger(__name__)
def _escape_ilike(s: str) -> str:
"""Escape ILIKE metacharacters for use with ESCAPE '\\\\'."""
return s.replace("\\", "\\\\").replace("%", "\\%").replace("_", "\\_")
def _resolve_pg_listen_url(override: str, sqlalchemy_url: str) -> str:
"""Resolve the URL used by the dedicated LISTEN connection.
@@ -1835,6 +1836,33 @@ class PostgreSQLBackend:
).fetchall()
return [dict(r._mapping) for r in rows]
def find_watch_by_name(self, ws_id: str, name_or_prefix: str) -> dict[str, Any] | None:
if not name_or_prefix:
return None
like_pattern = _escape_like(name_or_prefix) + "%"
with self._conn() as conn:
row = conn.execute(
sa.select(watches)
.where(
(watches.c.ws_id == ws_id)
& (
(watches.c.name == name_or_prefix)
| watches.c.watch_id.like(like_pattern, escape=_LIKE_ESCAPE)
)
)
# Active rows win over inactive ones with the same name.
# _prepare_watch's duplicate-name guard filters active=1,
# so a model can recreate a name after the previous one
# auto-cancelled; a cancel-by-name request on the live
# row must not be shadowed by the older completed row.
.order_by(watches.c.active.desc(), watches.c.created.desc())
.limit(1)
).fetchone()
if row is None:
return None
return dict(row._mapping)
def list_watches_for_node(self, node_id: str) -> list[dict[str, Any]]:
with self._conn() as conn:
@@ -3334,6 +3362,7 @@ class PostgreSQLBackend:
tier: str,
judge_model: str,
latency_ms: int,
user_decision: str = "pending",
) -> None:
now = datetime.now(UTC).strftime("%Y-%m-%dT%H:%M:%S")
with self._conn() as conn:
@@ -3354,11 +3383,67 @@ class PostgreSQLBackend:
"tier": tier,
"judge_model": judge_model,
"latency_ms": latency_ms,
"user_decision": user_decision,
"created": now,
},
)
conn.commit()
def upsert_intent_verdict(
self,
verdict_id: str,
ws_id: str,
call_id: str,
func_name: str,
func_args: str,
intent_summary: str,
risk_level: str,
confidence: float,
recommendation: str,
reasoning: str,
evidence: str,
tier: str,
judge_model: str,
latency_ms: int,
user_decision: str = "pending",
) -> None:
from sqlalchemy.dialects.postgresql import insert as pg_insert
now = datetime.now(UTC).strftime("%Y-%m-%dT%H:%M:%S")
stmt = pg_insert(intent_verdicts).values(
verdict_id=verdict_id,
ws_id=ws_id,
call_id=call_id,
func_name=func_name,
func_args=func_args,
intent_summary=intent_summary,
risk_level=risk_level,
confidence=confidence,
recommendation=recommendation,
reasoning=reasoning,
evidence=evidence,
tier=tier,
judge_model=judge_model,
latency_ms=latency_ms,
user_decision=user_decision,
created=now,
)
# On verdict_id conflict, update only the three fields that
# genuinely change between heuristic and llm_fallback. See the
# protocol docstring for the full exclusion rationale —
# ``user_decision`` exclusion in particular is load-bearing.
stmt = stmt.on_conflict_do_update(
index_elements=[intent_verdicts.c.verdict_id],
set_={
"tier": tier,
"reasoning": reasoning,
"judge_model": judge_model,
},
)
with self._conn() as conn:
conn.execute(stmt)
conn.commit()
def create_intent_verdicts_bulk(self, verdicts: list[dict[str, Any]]) -> None:
if not verdicts:
return
@@ -3379,6 +3464,7 @@ class PostgreSQLBackend:
"tier": v.get("tier", "heuristic"),
"judge_model": v.get("judge_model", ""),
"latency_ms": v.get("latency_ms", 0),
"user_decision": v.get("user_decision", "pending"),
"created": now,
}
for v in verdicts
@@ -3671,7 +3757,7 @@ class PostgreSQLBackend:
clauses = []
params: dict[str, str] = {}
for i, t in enumerate(terms):
escaped = _escape_ilike(t)
escaped = _escape_like(t)
clauses.append(
f"(name ILIKE :n{i} ESCAPE '\\' "
f"OR description ILIKE :d{i} ESCAPE '\\' "
@@ -3750,7 +3836,7 @@ class PostgreSQLBackend:
scope_clauses, params = self._build_scope_or_clause(scopes)
term_clauses = []
for i, t in enumerate(terms):
escaped = _escape_ilike(t)
escaped = _escape_like(t)
term_clauses.append(
f"(name ILIKE :n{i} ESCAPE '\\' "
f"OR description ILIKE :d{i} ESCAPE '\\' "
+93 -5
View File
@@ -1003,6 +1003,23 @@ class StorageBackend(Protocol):
"""Return active watches for a workstream, ordered by created DESC."""
...
def find_watch_by_name(self, ws_id: str, name_or_prefix: str) -> dict[str, Any] | None:
"""Return a watch in ``ws_id`` whose ``name`` matches
``name_or_prefix`` exactly, or whose ``watch_id`` starts with it.
Unlike :meth:`list_watches_for_ws` this DOES NOT filter on the
``active`` flag callers can inspect ``row["active"]`` to
distinguish a still-running watch from one that fired and
auto-cancelled. Returns ``None`` if no match.
When multiple rows match, prefers active rows over inactive
ones, then most-recently-created. Without the active
preference, a recreated-after-completion name would let the
older inactive row shadow the new active one in the cancel
path.
"""
...
def list_watches_for_node(self, node_id: str) -> list[dict[str, Any]]:
"""Return all active watches on a node, ordered by created DESC."""
...
@@ -1590,8 +1607,77 @@ class StorageBackend(Protocol):
tier: str,
judge_model: str,
latency_ms: int,
user_decision: str = "pending",
) -> None:
"""Record an intent validation verdict."""
"""Record an intent validation verdict.
``user_decision`` defaults to ``"pending"`` rather than ``""``
so an audit reader can distinguish "in-flight" rows from
legacy pre-fix rows (which carry ``""`` from the column's
server_default and indicate "convention not yet established
when this row was written"). Resolution writers
(:meth:`update_intent_verdict`) later overwrite the field with
``"approved"`` / ``"denied"`` / ``"timeout"`` (user-driven) or
``"policy"`` / ``"blanket"`` / ``"auto_approve_tools"``
(auto-approve reason, mirroring :class:`AutoApproveReason`).
"""
...
def upsert_intent_verdict(
self,
verdict_id: str,
ws_id: str,
call_id: str,
func_name: str,
func_args: str,
intent_summary: str,
risk_level: str,
confidence: float,
recommendation: str,
reasoning: str,
evidence: str,
tier: str,
judge_model: str,
latency_ms: int,
user_decision: str = "pending",
) -> None:
"""INSERT a verdict row, or UPDATE the judge-output fields on conflict.
Async LLM judge verdicts with ``tier="llm_fallback"`` deliberately
reuse the heuristic verdict's ``verdict_id`` so the row gets
"upgraded in place" from heuristic fallback when the LLM tier
doesn't return a real verdict (timeout / cancelled / no-content).
A plain INSERT collides on ``intent_verdicts_pkey``; this method
``ON CONFLICT (verdict_id) DO UPDATE`` updates only the columns
that genuinely change between the two tiers:
- ``tier`` (the upgrade itself)
- ``reasoning`` (gets " (LLM judge did not return a verdict)" appended)
- ``judge_model`` (heuristic carries "", fallback carries the model)
Every other column is EXCLUDED from the on-conflict SET clause:
- Identity columns (``verdict_id``, ``ws_id``, ``call_id``,
``func_name``, ``func_args``) already the same row.
- Carried-verbatim columns (``intent_summary``, ``risk_level``,
``confidence``, ``recommendation``, ``evidence``, ``latency_ms``)
the fallback copies them from the heuristic verdict; updating
would be a no-op.
- ``user_decision`` LOAD-BEARING exclusion. ``IntentVerdict.to_dict()``
doesn't project it, so a fallback verdict reaching this layer
defaults the kwarg to ``"pending"``. If the operator already
resolved the approval between heuristic INSERT and fallback
fire, the row's ``user_decision`` was already updated to
``"approved"``/``"denied"``/``"timeout"`` (or stamped to an
auto-approve reason at heuristic-INSERT time). Clobbering it
back to ``"pending"`` would undo that.
- ``created`` preserve the original timestamp.
Used by :meth:`SessionUIBase._persist_intent_verdict` for every
async LLM-tier delivery; the synchronous heuristic-bulk path
(:meth:`create_intent_verdicts_bulk`) stays as plain INSERT
since each heuristic UUID is freshly generated per turn.
"""
...
def create_intent_verdicts_bulk(self, verdicts: list[dict[str, Any]]) -> None:
@@ -1601,10 +1687,12 @@ class StorageBackend(Protocol):
(``verdict_id`` / ``ws_id`` / ``call_id`` / ``func_name`` /
``func_args`` / ``intent_summary`` / ``risk_level`` /
``confidence`` / ``recommendation`` / ``reasoning`` / ``evidence`` /
``tier`` / ``judge_model`` / ``latency_ms``). Used by the
synchronous heuristic-verdict persistence loop in
``approve_tools`` so a tool-heavy turn doesn't pay N×commit
latency before the approval prompt renders.
``tier`` / ``judge_model`` / ``latency_ms`` /
``user_decision``). ``user_decision`` defaults to ``"pending"``
when absent see :meth:`create_intent_verdict` for the
vocabulary. Used by the synchronous heuristic-verdict
persistence loop in ``approve_tools`` so a tool-heavy turn
doesn't pay N×commit latency before the approval prompt renders.
"""
...
+91 -5
View File
@@ -72,6 +72,9 @@ from turnstone.core.storage._schema import (
from turnstone.core.storage._utils import (
HEURISTIC_RULE_MUTABLE as _HEURISTIC_RULE_MUTABLE,
)
from turnstone.core.storage._utils import (
LIKE_ESCAPE as _LIKE_ESCAPE,
)
from turnstone.core.storage._utils import (
MCP_SERVER_MUTABLE as _MCP_SERVER_MUTABLE,
)
@@ -102,6 +105,9 @@ from turnstone.core.storage._utils import (
from turnstone.core.storage._utils import (
VERDICT_MUTABLE as _VERDICT_MUTABLE,
)
from turnstone.core.storage._utils import (
escape_like as _escape_like,
)
from turnstone.core.storage._utils import (
normalize_search_terms as _normalize_search_terms,
)
@@ -120,11 +126,6 @@ from turnstone.core.workstream import BULK_CLOSE_STATE_VALUES, WorkstreamKind
log = get_logger(__name__)
def _escape_like(s: str) -> str:
"""Escape LIKE metacharacters for use with ESCAPE '\\\\'."""
return s.replace("\\", "\\\\").replace("%", "\\%").replace("_", "\\_")
def _fts5_query(query: str) -> str:
"""Convert a plain search string into a safe FTS5 query."""
terms = query.split()
@@ -1976,6 +1977,33 @@ class SQLiteBackend:
).fetchall()
return [dict(r._mapping) for r in rows]
def find_watch_by_name(self, ws_id: str, name_or_prefix: str) -> dict[str, Any] | None:
if not name_or_prefix:
return None
like_pattern = _escape_like(name_or_prefix) + "%"
with self._conn() as conn:
row = conn.execute(
sa.select(watches)
.where(
(watches.c.ws_id == ws_id)
& (
(watches.c.name == name_or_prefix)
| watches.c.watch_id.like(like_pattern, escape=_LIKE_ESCAPE)
)
)
# Active rows win over inactive ones with the same name.
# _prepare_watch's duplicate-name guard filters active=1,
# so a model can recreate a name after the previous one
# auto-cancelled; a cancel-by-name request on the live
# row must not be shadowed by the older completed row.
.order_by(watches.c.active.desc(), watches.c.created.desc())
.limit(1)
).fetchone()
if row is None:
return None
return dict(row._mapping)
def list_watches_for_node(self, node_id: str) -> list[dict[str, Any]]:
with self._conn() as conn:
@@ -3496,6 +3524,7 @@ class SQLiteBackend:
tier: str,
judge_model: str,
latency_ms: int,
user_decision: str = "pending",
) -> None:
now = datetime.now(UTC).strftime("%Y-%m-%dT%H:%M:%S")
with self._conn() as conn:
@@ -3516,11 +3545,67 @@ class SQLiteBackend:
"tier": tier,
"judge_model": judge_model,
"latency_ms": latency_ms,
"user_decision": user_decision,
"created": now,
},
)
conn.commit()
def upsert_intent_verdict(
self,
verdict_id: str,
ws_id: str,
call_id: str,
func_name: str,
func_args: str,
intent_summary: str,
risk_level: str,
confidence: float,
recommendation: str,
reasoning: str,
evidence: str,
tier: str,
judge_model: str,
latency_ms: int,
user_decision: str = "pending",
) -> None:
from sqlalchemy.dialects.sqlite import insert as sqlite_insert
now = datetime.now(UTC).strftime("%Y-%m-%dT%H:%M:%S")
stmt = sqlite_insert(intent_verdicts).values(
verdict_id=verdict_id,
ws_id=ws_id,
call_id=call_id,
func_name=func_name,
func_args=func_args,
intent_summary=intent_summary,
risk_level=risk_level,
confidence=confidence,
recommendation=recommendation,
reasoning=reasoning,
evidence=evidence,
tier=tier,
judge_model=judge_model,
latency_ms=latency_ms,
user_decision=user_decision,
created=now,
)
# On verdict_id conflict, update only the three fields that
# genuinely change between heuristic and llm_fallback. See the
# protocol docstring for the full exclusion rationale —
# ``user_decision`` exclusion in particular is load-bearing.
stmt = stmt.on_conflict_do_update(
index_elements=["verdict_id"],
set_={
"tier": tier,
"reasoning": reasoning,
"judge_model": judge_model,
},
)
with self._conn() as conn:
conn.execute(stmt)
conn.commit()
def create_intent_verdicts_bulk(self, verdicts: list[dict[str, Any]]) -> None:
if not verdicts:
return
@@ -3541,6 +3626,7 @@ class SQLiteBackend:
"tier": v.get("tier", "heuristic"),
"judge_model": v.get("judge_model", ""),
"latency_ms": v.get("latency_ms", 0),
"user_decision": v.get("user_decision", "pending"),
"created": now,
}
for v in verdicts
+31
View File
@@ -98,6 +98,37 @@ def sanitize_text(value: str | None) -> str | None:
return value
# ---------------------------------------------------------------------------
# SQL LIKE escaping
# ---------------------------------------------------------------------------
# The escape character paired with :func:`escape_like`. Callers MUST
# pass ``escape=LIKE_ESCAPE`` to SQLAlchemy's ``.like()`` — without
# that kwarg, ``.like()`` uses no escape character at all and the
# ``\%`` / ``\_`` sequences produced by :func:`escape_like` would be
# interpreted as a literal backslash followed by a wildcard. ``\``
# is the SQL standard escape character and works identically on SQLite
# and PostgreSQL when passed explicitly.
LIKE_ESCAPE = "\\"
def escape_like(value: str) -> str:
"""Escape ``%`` and ``_`` (and the escape character itself) so the
string can be safely embedded in a SQL ``LIKE`` pattern.
Pair with ``column.like(escape_like(prefix) + "%", escape=LIKE_ESCAPE)``
to do a true prefix match against caller-supplied input. Without
this, untrusted text containing ``%`` or ``_`` is interpreted as a
wildcard e.g. a model-supplied watch name of ``"%"`` would match
every row in the queried partition.
"""
return (
value.replace(LIKE_ESCAPE, LIKE_ESCAPE * 2)
.replace("%", LIKE_ESCAPE + "%")
.replace("_", LIKE_ESCAPE + "_")
)
# ---------------------------------------------------------------------------
# Row helper
# ---------------------------------------------------------------------------
+92 -24
View File
@@ -289,6 +289,17 @@ class WatchRunner:
self._dispatch_fns: dict[str, Callable[[dict[str, Any], str], None]] = {}
self._dispatch_lock = threading.Lock()
# Watch ids whose terminal reminder has already been dispatched
# but whose row write has not yet been confirmed. Populated
# between ``_dispatch_result`` and ``update_watch`` in
# :meth:`_poll_watch`; on a subsequent tick the same row will
# still appear in ``list_due_watches`` (active=1, next_poll
# unchanged) — the guard at the top of ``_poll_watch`` retries
# the row write WITHOUT re-dispatching. Bounded by transient
# storage failure depth (~MAX_WATCHES_PER_WS × num_ws).
self._terminal_dispatched: set[str] = set()
self._terminal_dispatched_lock = threading.Lock()
self._stop_event = threading.Event()
self._thread: threading.Thread | None = None
@@ -314,14 +325,18 @@ class WatchRunner:
def set_dispatch_fn(self, ws_id: str, fn: Callable[[dict[str, Any], str], None]) -> None:
"""Register a per-workstream dispatch fn.
The fn signature is ``(reminder, watch_id)`` the runner passes
the originating ``watch_id`` so dispatch closures can capture
per-watch metadata (e.g. a ``valid_until`` predicate that
re-checks ``storage.is_watch_active(watch_id)`` before
delivering a stale entry). ``reminder`` is the structured dict
returned by :func:`build_watch_reminder` ``text`` carries the
formatted body, the remaining fields ride as queue-entry
metadata so the frontend can render a ``.msg.watch-result`` card.
The fn signature is ``(reminder, watch_id)``. ``reminder`` is
the structured dict returned by :func:`build_watch_reminder`
``text`` carries the formatted body, the remaining fields ride
as queue-entry metadata so the frontend can render a
``.msg.watch-result`` card. ``watch_id`` is passed for
closures that need per-watch metadata in their queue plumbing
(e.g. correlating a fire back to the originating row in logs);
do NOT use it to gate delivery against
``storage.is_watch_active(watch_id)`` see
:meth:`ChatSession.set_watch_runner` for why that pattern
races :meth:`_poll_watch`'s commit of ``active=False`` and
drops fires the model was meant to see.
"""
with self._dispatch_lock:
self._dispatch_fns[ws_id] = fn
@@ -339,6 +354,22 @@ class WatchRunner:
with self._dispatch_lock:
return self._dispatch_fns.get(ws_id)
def forget_terminal_dispatched(self, watch_id: str) -> None:
"""Discard ``watch_id`` from the pending-terminal-dispatched
set if present. Called by paths that take a watch out of
:meth:`StorageBackend.list_due_watches` view independent of
the runner's own poll (most importantly the user-cancel path
in :meth:`ChatSession._exec_watch`). Without this, a
``_poll_watch`` whose row write failed AFTER dispatch would
leak ``watch_id`` in ``_terminal_dispatched`` indefinitely
the user-cancel writes ``next_poll=''`` which excludes the
row from ``list_due_watches``, so the retry-deactivate branch
at the top of :meth:`_poll_watch` never fires to clear the
entry.
"""
with self._terminal_dispatched_lock:
self._terminal_dispatched.discard(watch_id)
# -- Main loop -----------------------------------------------------------
def _run(self) -> None:
@@ -383,6 +414,21 @@ class WatchRunner:
prev_output = watch_row.get("last_output")
created = watch_row.get("created", "")
# Re-poll of a row whose terminal reminder already shipped but
# whose ``active=False`` write didn't land — retry just the row
# write so the row stops appearing in ``list_due_watches``; do
# NOT re-dispatch the reminder, which the model already saw.
with self._terminal_dispatched_lock:
already_dispatched = watch_id in self._terminal_dispatched
if already_dispatched:
try:
self._storage.update_watch(watch_id, active=False, next_poll="")
with self._terminal_dispatched_lock:
self._terminal_dispatched.discard(watch_id)
except Exception:
log.exception("watch_runner.retry_deactivate_failed", extra={"watch_id": watch_id})
return
# Safety check
blocked = is_command_blocked(command)
if blocked:
@@ -416,22 +462,17 @@ class WatchRunner:
now = datetime.now(UTC)
now_str = now.strftime("%Y-%m-%dT%H:%M:%S")
# Update DB
update_fields: dict[str, Any] = {
"poll_count": poll_count,
"last_output": output,
"last_exit_code": exit_code,
"last_poll": now_str,
}
if is_final:
update_fields["active"] = False
update_fields["next_poll"] = ""
else:
next_poll = now + timedelta(seconds=watch_row["interval_secs"])
update_fields["next_poll"] = next_poll.strftime("%Y-%m-%dT%H:%M:%S")
self._storage.update_watch(watch_id, **update_fields)
# Dispatch result if condition fired or final
# Dispatch before committing the row update. Belt-and-braces
# given the rest of the fix (closure no longer wires a
# ``valid_until`` predicate, cancel-by-name uses
# :meth:`find_watch_by_name` which ignores the ``active``
# filter): either order would deliver the reminder today, but
# this ordering preserves the invariant against re-wiring an
# ``is_watch_active`` predicate or adding a new
# ``active``-filtered read on this hot path. Combined with the
# ``_terminal_dispatched`` guard above it also bounds the
# duplicate-fire blast radius if the row write fails after the
# reminder shipped.
if fired or is_final:
# Compute elapsed from created time
elapsed_secs = 0.0
@@ -454,6 +495,33 @@ class WatchRunner:
reason=reason,
)
self._dispatch_result(ws_id, reminder, watch_id)
if is_final:
# Mark BEFORE the row write so a raise below routes the
# next tick into the retry-deactivate branch instead of
# re-firing the reminder.
with self._terminal_dispatched_lock:
self._terminal_dispatched.add(watch_id)
# Update DB
update_fields: dict[str, Any] = {
"poll_count": poll_count,
"last_output": output,
"last_exit_code": exit_code,
"last_poll": now_str,
}
if is_final:
update_fields["active"] = False
update_fields["next_poll"] = ""
else:
next_poll = now + timedelta(seconds=watch_row["interval_secs"])
update_fields["next_poll"] = next_poll.strftime("%Y-%m-%dT%H:%M:%S")
self._storage.update_watch(watch_id, **update_fields)
if is_final:
# Row write committed; the retry-deactivate branch will
# never be reached for this watch_id.
with self._terminal_dispatched_lock:
self._terminal_dispatched.discard(watch_id)
log.debug(
"watch_runner.polled",
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long

Some files were not shown because too many files have changed in this diff Show More