mirror of
https://github.com/turnstonelabs/turnstone.git
synced 2026-08-13 23:42:25 -06:00
Compare commits
91 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| a4c35e9e29 | |||
| 862eb99cdb | |||
| 25b97bebdf | |||
| ee5ca9a242 | |||
| dd8543fce9 | |||
| 667942024f | |||
| 78831bbe91 | |||
| d44d7eb1a8 | |||
| 876c7d8cb3 | |||
| 98823eb769 | |||
| 4d708c30ac | |||
| 6d60ff7634 | |||
| be662c6134 | |||
| 3ef3f24c7f | |||
| db903f482a | |||
| 6aeffd1845 | |||
| a02b093733 | |||
| f311555026 | |||
| 45d95a2c1f | |||
| a2d9d9832a | |||
| ab123c6cfc | |||
| 8ad17666b9 | |||
| 03fc0861a6 | |||
| a22fb2f395 | |||
| cdcd040da2 | |||
| 834d62c9d4 | |||
| 342a77fe5c | |||
| fd7a447ef9 | |||
| 552ee3c590 | |||
| e99d3ee139 | |||
| 4f0fc3f219 | |||
| dc701986f7 | |||
| bedd25fbe7 | |||
| 251a912275 | |||
| d48902fd01 | |||
| 702ac43d0e | |||
| 01f83dc90f | |||
| 2463c480c2 | |||
| 2a3dfbc6fb | |||
| 6c3b3cc098 | |||
| 0dc52f05ee | |||
| 02929c0d00 | |||
| b2add19c56 | |||
| 5ce1873e9e | |||
| 7698a928c5 | |||
| 4e2eea2f86 | |||
| a6752cb645 | |||
| 06ba4e8d4f | |||
| 94dcaf34fd | |||
| 1a2a689033 | |||
| c02f960d0a | |||
| bfde387206 | |||
| 27d112ff60 | |||
| aeab2535b1 | |||
| 4638d22bd0 | |||
| ee3bd1dcf2 | |||
| ae3a83ccce | |||
| 0f17433e1f | |||
| 043554bb2f | |||
| 8389808add | |||
| 6cbef4f633 | |||
| 2b6dde4f7e | |||
| fbe31b9885 | |||
| ef13f40cf5 | |||
| bfa1b104cf | |||
| 324a1d1a35 | |||
| 95ab88ff6f | |||
| d29840f985 | |||
| 44c0b9c340 | |||
| cdbdf3dc2b | |||
| 8aabb061c2 | |||
| 8bd638569f | |||
| 251dc44a46 | |||
| efd0a1d000 | |||
| 2f93c39fd3 | |||
| f27ce104c6 | |||
| 20a61b692b | |||
| 1966107efe | |||
| d5b2fe6e45 | |||
| 1569819750 | |||
| 4107a30148 | |||
| d06d88b83f | |||
| c411aac939 | |||
| 104715b650 | |||
| 59a9899149 | |||
| 4da7c3b91c | |||
| 012f4e3e16 | |||
| 3636724848 | |||
| eeda5ac312 | |||
| be872b840f | |||
| ee94ae8ba1 |
+215
-4
@@ -6,13 +6,224 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
|
||||
and this project adheres to [PEP 440](https://peps.python.org/pep-0440/) for
|
||||
version numbers (`X.Y.Z`, with `X.Y.ZaN` / `bN` / `rcN` for pre-releases).
|
||||
|
||||
Three release tracks are maintained — the current stable, one prior
|
||||
stable, and the experimental line:
|
||||
Two active release tracks are maintained — the current stable and the
|
||||
experimental line:
|
||||
|
||||
- **`stable/1.5`** — patch-only (`v1.5.x`)
|
||||
- **`stable/1.6`** — patch-only (`v1.6.x`)
|
||||
- **`stable/1.7`** — patch-only (`v1.7.x`)
|
||||
- **`main`** — experimental (next major)
|
||||
|
||||
Earlier stable lines (`stable/1.6`, `stable/1.5`) are frozen.
|
||||
|
||||
## [1.7.4]
|
||||
|
||||
A feature-bearing patch for the 1.7 line, rolling up work that had stabilised
|
||||
on `main`. No schema migrations (head stays 066) and no new configuration knobs.
|
||||
|
||||
### Added
|
||||
|
||||
- **Background shells for the `bash` tool** — `run_in_background=true` starts a
|
||||
command as a detached shell and returns a `bash_N` handle; new `bash_output`
|
||||
(delta output since last read, optional regex filter, status/exit code) and
|
||||
`kill_shell` (terminates the shell's process group) tools manage it. Output is
|
||||
buffered with a drop-oldest cap, a system notice lands when a shell exits, and
|
||||
shells die with their workstream — never outliving a `task_agent` that started
|
||||
them.
|
||||
- **`task_agent` carries the model's native reasoning across its own tool loop** —
|
||||
a task agent's replayed turns now preserve the provider-native reasoning lane
|
||||
(Anthropic thinking blocks with signatures, OpenAI reasoning items, Gemini
|
||||
`thought_signature`, vLLM/llama.cpp reasoning text) instead of rebuilding each
|
||||
turn from text alone, restoring reasoning continuity for thinking models.
|
||||
- **Model-shelf response controls** — the console model shelf exposes verbosity
|
||||
and reasoning-mode controls per identity.
|
||||
|
||||
### Changed
|
||||
|
||||
- **GPT-5.6 aligned with the GA API surface** — the Responses provider matches
|
||||
GPT-5.6's GA shape (typed `reasoning.mode`, `prompt_cache_options`,
|
||||
cache-write accounting); the `openai` floor moves to `>=2.45`.
|
||||
|
||||
### Fixed
|
||||
|
||||
- **`bash` never hangs on a backgrounded child** — a command that left a
|
||||
long-lived process running no longer wedges the workstream; the tool waits on
|
||||
the tracked process (bounded by the timeout) and reaps its whole process group.
|
||||
- **`task_agent` sub-tool ids are session-unique** — ids are minted
|
||||
`{parent}::r{run}s{step}::{id}` so a local model reissuing sequential ids
|
||||
(`call_0` each turn) no longer aliases two steps onto one live-card row while
|
||||
`/history` keeps them apart.
|
||||
- **Judge completions honour model-definition capabilities** — a judge's
|
||||
completion now threads its model's declared capabilities instead of assuming a
|
||||
default surface.
|
||||
- **`create-admin` CLI** — adds an explicit admin-creation command; `run.sh` no
|
||||
longer onboards into a role-less user.
|
||||
- **Install script Docker handling** — installs Docker on distros
|
||||
`get.docker.com` rejects, and gates that path by `$ID` instead of trapping all
|
||||
failures.
|
||||
|
||||
## [1.7.3]
|
||||
|
||||
A small feature and maintenance patch for the 1.7 line. No schema migrations
|
||||
and no new configuration knobs.
|
||||
|
||||
### Added
|
||||
|
||||
- **OpenAI GPT-5.6 (Sol/Terra/Luna) support** — the Responses provider
|
||||
understands the GPT-5.6 family: the `reasoning.mode` control, the new
|
||||
`max` effort tier, and `text.verbosity`, with golden wire payloads pinning
|
||||
the request shapes. The `openai` dependency floor moves to `>=2.44`.
|
||||
|
||||
### Changed
|
||||
|
||||
- **Engineer base prompt hardened with process discipline** — the default
|
||||
base prompt for non-coordinator sessions now works in phases scaled to the
|
||||
size of the change, defaults to red-green for testable work, scopes to the
|
||||
smallest sufficient diff, stops to report after repeated failed attempts
|
||||
instead of thrashing, reports only observed results, and delegates
|
||||
exploration to `task_agent`. Persona prompts freeze into the workstream
|
||||
stamp at creation, so this reaches new workstreams only.
|
||||
|
||||
### Fixed
|
||||
|
||||
- **Unknown reasoning-mode warnings name the allowed modes** — a model
|
||||
definition with an unrecognized reasoning mode now logs the valid options
|
||||
instead of leaving the operator to guess.
|
||||
|
||||
### Documentation
|
||||
|
||||
- **HYPOTHESIS.md / PRIMER.md** — the control normal form is tightened and
|
||||
the factored Q_E reading is carried into the glossary; the plain-language
|
||||
PRIMER stays in sync.
|
||||
|
||||
## [1.7.2]
|
||||
|
||||
A feature-bearing patch for the 1.7 line. Rather than hold this work for the
|
||||
larger 1.8 churn, the fixes and the smaller features that had already
|
||||
stabilised on `main` are rolled into the stable line now: a rich preview
|
||||
pane, persona/project settings on scheduled tasks, and a batch of streaming,
|
||||
rendering, and nudge-delivery hardening.
|
||||
|
||||
> **⚠️ Before upgrading:** 1.7.2 adds Alembic migration `066`, applied
|
||||
> automatically on first start. It adds two `Text NOT NULL DEFAULT ''`
|
||||
> columns (`persona`, `project_id`) to the `scheduled_tasks` table; existing
|
||||
> rows migrate to the empty default, which is byte-identical to pre-066
|
||||
> dispatch behaviour. The change is additive and reversible, but — as always
|
||||
> — back up your storage before upgrading (`pg_dump` for PostgreSQL; copy the
|
||||
> database file for SQLite).
|
||||
|
||||
### Added
|
||||
|
||||
- **Rich preview pane + `open_preview` tool** — a workstream can now open a
|
||||
rendered preview (HTML, Markdown, and other kinds) in a pane beside the
|
||||
conversation via the new `open_preview` tool. Guarded fetches stream under
|
||||
a byte budget whose ceiling tracks the widest per-kind cap, preview blob
|
||||
ids are salted, and a preflight probe handles legacy charsets and a
|
||||
remote-assets opt-in. See `docs/tools.md`.
|
||||
- **`allow_private_network` opt-in for `web_fetch` / `open_preview`** —
|
||||
private-address fetch and preview targets stay blocked by default; an
|
||||
operator can opt a workstream in through the settings registry when a
|
||||
private endpoint is genuinely intended. (Distinct from the 1.7.1 `[oidc]`
|
||||
flag of the same name, which governs identity-provider discovery.)
|
||||
- **Persona + project settings on scheduled tasks** (migration `066`) — a
|
||||
scheduled task can now pin the **persona** and **project** of the
|
||||
workstream it dispatches, matching the levers a manually-created workstream
|
||||
already carries. Both default to empty (kind-default persona / no project),
|
||||
so existing schedules dispatch exactly as before.
|
||||
|
||||
### Fixed
|
||||
|
||||
- **Streaming fast-path overflow recovery** — fast-stream tokens are now
|
||||
batched and overflowed SSE listeners recover instead of stalling (and
|
||||
`connectSSE` no longer opens into a hidden background tab). The same
|
||||
overflow-recovery companions were carried to the coordinator pane, so a
|
||||
coordinator watching many children recovers dropped listeners the same way
|
||||
the live-session view does.
|
||||
- **Renderer containment** — markdown sentinel-forgery and recursive-frame
|
||||
content loss are contained, and an indented fence close no longer drags its
|
||||
indent into the enclosed code content.
|
||||
- **Idle nudge / wake delivery** — nudge and wake delivery is hardened across
|
||||
session eviction, cancellation, and identity rebinds; the wake gate now
|
||||
requires a real nudge queue, refused wakes are logged, and
|
||||
`initial_message_status` is typed as a closed enum on the wire.
|
||||
- **`web_fetch` extraction inherits model settings** — the completion that
|
||||
extracts content from a fetched page now inherits the workstream's model
|
||||
settings instead of falling back to defaults.
|
||||
- **UI panes** — ephemeral panes close on split-dismiss instead of orphaning
|
||||
a tab, and an unsplit skips the redundant refresh after an ephemeral pane
|
||||
closes.
|
||||
- **Shared code-highlight CSS** — renderer-output CSS is shared so the console
|
||||
and coordinator panes highlight code identically.
|
||||
|
||||
### Security
|
||||
|
||||
- **`Content-Disposition` filenames made wire-safe** — download filenames
|
||||
derived from user-controlled text are sanitised (latin-1- and
|
||||
control-char-safe, quoting-safe) before they reach the `Content-Disposition`
|
||||
response header, including the fallback path.
|
||||
|
||||
### Documentation
|
||||
|
||||
- **HYPOTHESIS.md: daemons + the outer loop, plus a plain-language PRIMER** —
|
||||
the harness north-star document gains its daemon / outer-loop treatment and
|
||||
a new top-level `PRIMER.md`.
|
||||
|
||||
## [1.7.1]
|
||||
|
||||
A maintenance and hardening patch for the 1.7 line. No schema migrations;
|
||||
the credential-redaction work below is additive and needs no configuration
|
||||
change. The one new operator-facing knob is the opt-in `[oidc]
|
||||
allow_private_network` flag (default off).
|
||||
|
||||
### Security
|
||||
|
||||
- **Credential redaction hardened across the tool-call surface** — the
|
||||
redactor that scrubs secrets from tool arguments and log previews was
|
||||
reworked on both the backend and the browser to close several leak paths
|
||||
and to fix false-positive and performance issues. Malformed tool-call
|
||||
arguments are now legalised before they reach the wire; the tool-args log
|
||||
preview scrubs credentials and control characters; and the coordinator's
|
||||
tool-call cards gain a matching client-side redaction pass so the JS and
|
||||
backend redactors stay at parity. Pattern coverage now includes
|
||||
`secret_access_key` / `aws_secret_access_key` multi-segment keys, bare
|
||||
`token=` / `key=` forms (guarded by a negative lookbehind to avoid
|
||||
false positives), and SQLAlchemy `+driver`-qualified connection-string
|
||||
schemes matched case-insensitively.
|
||||
- **OIDC SSRF guard: `[oidc] allow_private_network` opt-in** — self-hosted
|
||||
identity providers on private networks can now be reached by setting
|
||||
`allow_private_network = true` under `[oidc]` (default off; the MCP OAuth
|
||||
path stays strict). Rejections of discovered endpoints carry the opt-in
|
||||
hint so the misconfiguration is self-explanatory. See `docs/oidc.md`.
|
||||
|
||||
### Added
|
||||
|
||||
- **Persona discoverability + forgiving name resolution** — personas are
|
||||
now discoverable by agents, and persona-name resolution tolerates
|
||||
case/whitespace variation; a not-found resolution reports the offending
|
||||
input verbatim instead of a bare error.
|
||||
|
||||
### Fixed
|
||||
|
||||
- **MCP transport lifecycles routed through per-entry owner tasks**
|
||||
(#787/#788) — static and pooled MCP transport lifecycles are now driven
|
||||
by per-server / per-entry owner tasks, with a hardened disarm-sweep loop
|
||||
guard and targeted exception handling in place of a broad `BaseException`
|
||||
arm, so a dying transport can no longer spin the CPU or strand delivery.
|
||||
- **Client-construction failures surface as misconfiguration, not raw
|
||||
500s** — a model whose client cannot be constructed now reports a factory
|
||||
misconfiguration, and the raw exception text is kept out of the resulting
|
||||
503 response.
|
||||
- **Postgres history search survives oversized rows** — a conversation row
|
||||
exceeding Postgres' full-text limits no longer aborts history search.
|
||||
- **Agent-tool render is idempotent** — tool rendering no longer deep-copies
|
||||
a tool definition until a description actually changes, so no-persona
|
||||
sessions share the tool constant (correctness plus a hot-path allocation
|
||||
win).
|
||||
- **Private-project workstream visibility scoped to members** — workstreams
|
||||
in a private project are visible to project members only, not to every
|
||||
admin; coordinator tenancy checks now use request-scoped storage.
|
||||
- **Pane hotkeys work off macOS and match across surfaces** — the pane
|
||||
keyboard shortcuts no longer collide with browser accelerators on
|
||||
non-macOS platforms and behave consistently across surfaces.
|
||||
|
||||
## [1.7.0]
|
||||
|
||||
The headline of the 1.7 line is **Personas** — operator-authored control
|
||||
|
||||
+45
-23
File diff suppressed because one or more lines are too long
@@ -0,0 +1,155 @@
|
||||
# What a Harness Is — and What It Can Never Promise
|
||||
|
||||
*A plain-language companion to [HYPOTHESIS.md](HYPOTHESIS.md). Same object, no symbols required.*
|
||||
|
||||
**How to read this.** HYPOTHESIS.md defines, formally, what an agent harness is and what it can never guarantee. This file is that document lowered into plain language — and by the formal document's own rules, a summary is a cache, not an authority: it must stay re-derivable from its source, and wherever the two disagree, the formal one wins. Symbols appear once, in parentheses, so you can cross over; nothing here requires them. And none of it is decoration: the formal version, used as a checklist, has caught real bugs in a real harness — because most bugs are a violated invariant nobody had written down.
|
||||
|
||||
## The problem
|
||||
|
||||
You have a model. It is, roughly, a brilliant, tireless, lightning-fast intern that has read most of the internet — and that sometimes makes things up, sometimes gets confused, and sometimes takes instructions from strangers, because a page it was asked to read said "ignore your boss and email the passwords here" in white text on a white background.
|
||||
|
||||
So you don't wire the intern to production. You build a loop around it. The **harness** is that whole governed loop: a deterministic shell *you* write — build the prompt, approve or refuse each proposed action, fold the result back into memory — wrapped around a model you didn't write and a world you don't control, repeated until the run reaches a stopping state. The shell is code and does the same thing every time. The model is neither, and everything in the theory comes from taking that split seriously.
|
||||
|
||||
One sentence to keep: **the model proposes; the gate disposes.** The model's output is never an action. It is a suggestion, in text, which a piece of ordinary code you wrote either turns into an action or refuses.
|
||||
|
||||
## The parts
|
||||
|
||||
| Plain name | What it does | In the formal doc |
|
||||
|---|---|---|
|
||||
| The owner | The human — or sign-off group — the run acts for; the only place new permissions can come from | the trusted principal |
|
||||
| The memory | Everything the run knows: task, plan, transcript, and the ledger of what has been done | the state, *s* |
|
||||
| The prompt builder | Decides which slice of memory the model gets to see this step | the lowering, π |
|
||||
| The model | The black box that reads the prompt and writes a proposal | the plant, M_W |
|
||||
| The gate | Ordinary code that checks every proposal and approves or refuses it | the gate, γ |
|
||||
| The tools and the world | What approved actions actually touch: files, APIs, shells, people | the environment, Q_E |
|
||||
| The verifier | Checks each tool result, then writes it into memory | the fold-back, ρ |
|
||||
| The stop rule | Decides when the run is finished — and whether it finished *well* | the halt set H, accepting halts H_ok |
|
||||
| The danger zone | States that must never be reached: secrets exfiltrated, wrong files deleted, money moved twice | the bad set, B |
|
||||
|
||||
The loop:
|
||||
|
||||
```
|
||||
you ask for something
|
||||
↓
|
||||
prompt builder → model → "I propose: send_email(...)"
|
||||
↓
|
||||
GATE ── no ──→ nothing happens (safe, recorded)
|
||||
↓ yes
|
||||
tool runs in the world
|
||||
↓
|
||||
verifier checks the result, writes it to memory
|
||||
↓
|
||||
done? ── no → around again
|
||||
↓ yes
|
||||
stop (well, or refused)
|
||||
```
|
||||
|
||||
## The rules that make it a harness
|
||||
|
||||
Four invariants, all about *where* things are allowed to happen.
|
||||
|
||||
1. **The model sees only what the prompt builder shows it** — never raw memory. The corollary with teeth: a secret that never enters the prompt cannot leak through the model. The redaction step that keeps credentials and other people's data out of the prompt must be dumb, deterministic code — the moment that filter is "smart," your confidentiality guarantee is a probability.
|
||||
2. **Model outputs are proposals, not actions.**
|
||||
3. **Every side effect passes the gate.** There is no second door.
|
||||
4. **The harness itself flips no coins.** Replay a step with the model's answer and the tool results pinned, and behavior must be identical; any leftover variation is randomness *you* added and must be accounted for. The fine print: "deterministic" is conditional on pinned versions — a provider silently retraining the model behind the same API name changes the machine under you, and every dashboard number you collected dies with the version.
|
||||
|
||||
Notice what the rules don't say: they don't say the harness is *good*. A gate that approves everything satisfies rule 3 the way a lock that's always open satisfies "has a lock." The definition is a shape; the guarantees are what a particular harness *earns* inside it. Everything below is about what can be earned — and what can't.
|
||||
|
||||
And notice the symmetry between rules 1 and 3. There is exactly one door from your data into the model — what it may see — and exactly one door from the model into the world — what it may do. Nearly every security failure in these systems is one of those two doors with a hole in it: a secret lowered into a prompt that didn't need it, or a path from model text to a side effect that skipped the gate. Same bug, arrow flipped.
|
||||
|
||||
## Fail-closed, said precisely
|
||||
|
||||
"Fail-closed" gets used loosely. Here it means something exact: **nothing happens unless the gate said yes, and a refusal must itself be safe** — a refused proposal causes no side effect and leaves the run somewhere sane, which may be "stopped, having declined." The run is allowed to *say so*: a templated status message written by the shell is the shell speaking, not the model, and needs no gate. Failed runs don't have to die silent.
|
||||
|
||||
Three consequences people miss:
|
||||
|
||||
**Reads are not free.** A read-only call can smuggle instructions *in* (the fetched page is attacker-controlled) or secrets *out* (the URL it fetches can encode the payload). The gate approves calls, not just writes.
|
||||
|
||||
**Validation must not act.** A "validator" that resolves a URL, expands a template that fires a webhook, or evaluates an argument has already acted — inside the check. The gate must be pure: it reads the proposal and the memory and outputs yes or no. If deciding requires touching the world, that touch is itself an action and goes through the gate.
|
||||
|
||||
**Anything irreversible is decided at the gate.** The verifier can reject a bad *result*; it cannot unsend the email. So the question "can we take this back, and until when?" is asked before execution — which means each tool declares, up front, how reversible its effects are, and the gate reads that declaration when it decides; the mark that comes back in the result record is confirmation for the books, not the gate's source — the gate needed the answer before the tool ever ran.
|
||||
|
||||
Two honest asterisks. First, the gate checks a snapshot: it approves against the world *as its memory describes it*, and the world can move between check and commit. For actions that race the world — spend against a balance, write against a row — the tool itself must bind check to commit (compare-and-swap), or you have a classic time-of-check/time-of-use hole. The gate decides; for those effects, the tool enforces. Second, a gate is only as binding as the authority behind the tools. A tool process holding standing credentials — a database connection with every grant, an environment full of long-lived secrets — doesn't need the model's proposal to act, and against it the gate's "no" is a decision with nothing enforcing it. **A gate in front of an omnipotent tool is a suggestion.** The fix is to make the approval *be* the key: each authorized action carries a short-lived credential scoped to exactly that action, that resource, that operation, so tools hold no standing power at all.
|
||||
|
||||
## Why you don't get a proof — and what you do instead
|
||||
|
||||
If you write a sort function, you can prove it sorts: the function is small and the spec is exact. A harness has neither luxury. The spec side fails first — the task arrives in natural language, and natural language is, in the compiler's sense, *all undefined behavior*: there is no formal standard for "what the user meant" to verify against. The mechanism side fails next — the model is billions of learned parameters, and nobody can hand you a compact argument for why they jointly do the right thing.
|
||||
|
||||
Here is the careful version, because "you can't prove it" overshoots. The quantity you would want — call it the *expected steps to done* from any situation — is perfectly well-defined; in principle it exists. The document's central conjecture is that, for a model of this size, any faithful writing-down of that quantity is roughly *model-sized*: the honest proof-object does not compress. Find a small one and the conjecture dies — the document lists that outcome, explicitly, among the ways it could be wrong.
|
||||
|
||||
So instead of proving, you measure. You pick a progress meter — plan depth shrinking, open obligations closing, budget burning at the expected rate — and you check, across many runs, that it goes downhill and that its stalls predict failure. Two disciplines keep the measurement honest. The number bounds the world you *sampled*, never the world an adversary will choose: a meter calibrated on friendly traffic says nothing about hostile traffic. And the meter is itself attack surface: if "is the agent making progress?" is judged by another model, an attacker who can bend your agent can bend your *measurement of it* first, hiding the divergence from the very dashboard built to catch it. A learned meter is part of the system under test, never a neutral instrument.
|
||||
|
||||
A measurement is a risk metric. A proof is a certificate. Keeping those two words apart is half of what this theory is for.
|
||||
|
||||
## Security: reach the goal, avoid the danger — and who may change the rules
|
||||
|
||||
Formally, security here is a *reach-avoid* problem: reach a good stop, never touch the danger zone, **while an adversary picks the worst tool outputs your setup permits**. That last clause is the formal home of prompt injection: injection isn't "the model misbehaved," it's the environment optimized to bend your loop — poisoned pages, malicious tool descriptions, crafted responses.
|
||||
|
||||
Two different numbers fall out here, and dashboards love to collapse them: *success* (reached an accepted end before anything went wrong — a safe refusal counts against it) and *safety* (never touched the danger zone — a safe refusal is perfectly safe). Track both. They move independently. And both are scored by your own stop rule — they count what the shell *declared* a success. Whether a declared success was actually *right* is a third, harder number that no dashboard inside the system can produce; only a judge outside the run — a test suite, an audit, ground truth — can.
|
||||
|
||||
The gate handles the visible half of injection: the model, freshly poisoned, proposes emailing your credentials somewhere, and the gate refuses — and injection or not, the action does not happen. But the deeper attack doesn't propose a bad action today. It rewrites *what the run believes its job is* — it edits the plan — and then every future action looks locally reasonable against a corrupted plan. So memory has to be partitioned: **data** (tool results, fetched pages, retrieved documents — content the world supplied) and **control** (the plan, the permissions, what is authorized next). The security claim is conditional on that partition holding: untrusted content lands in data, always. And "trust" is really two questions pointing opposite ways, which is worth keeping straight: *can this leak?* (a value is as secret as the most-secret thing that fed it — secrecy flows **upward**) and *can this boss us around?* (a value is as trustworthy as the least-trustworthy thing that fed it — authority flows **downward**). Untrusted content is safe as *data* precisely because the second question keeps it off the control side; a secret is kept out of the model by the first. Lowering either barrier on purpose — declassifying a secret, promoting data to trusted — is an explicit decision the owner makes, never a thing that happens by accident when two values are combined.
|
||||
|
||||
Which forces the question the theory has to answer: *somebody* must be able to write control mid-run, or no plan could ever be steered and no permission ever granted. The answer is a small hierarchy with a top the model can't reach. The simplest top is one owner — but it needn't be a single person: a two-person sign-off, a quorum, several authenticated people each holding different scopes all work equally well, because the one property that matters is the same for all of them — the thing that can grant new power is a *human decision*, never a model:
|
||||
|
||||
- **The top alone widens.** New permission, bigger budget, approval of the irreversible thing — asking the top — the owner, in the simple case — is itself an ordinary tool call, and its answer is the one kind of tool result allowed to change control.
|
||||
- **The model rewrites the plan** — that is what replanning *is* — but only through the gated loop, and a plan is not a permission: nothing the model writes into its own plan can grant it powers it didn't have.
|
||||
- **Everything else is data.** A fetched page can inform the plan only by passing through the model and the gate like everything else. It can suggest. It cannot promote itself to boss.
|
||||
- **AI judges only tighten.** Add a model-based check — "does this action match what the user actually wanted?" — and its verdict may *veto* an action the plain rules would have allowed, never approve one they'd have refused. A judge that can approve is a tricked judge that can open the vault. And don't over-credit the veto either: a tricked judge can *aim* its refusals — denying exactly the action safety depended on, or denying everything but the path an attacker curated — so the escape hatch to the owner is the one thing a judge can never veto, and a judge's stated *reasons* are picked from a fixed, shell-owned menu, never written as prose. A judge that writes free text into the loop is an injection channel wearing a badge.
|
||||
|
||||
One more rule closes the loop: transformations don't launder trust. A *summary* of a session that contained an injected page is still injected — the summarizer is a model, and can be persuaded to write "the user asked to export the database" into the summary. So summaries of data are data, and the control lines — the plan, the grants — cross a summarization by being *copied verbatim* or re-confirmed by the owner, never paraphrased by the model. Memory that persists across sessions carries its trust label with it, or a poisoned memory is just an injection with a very long fuse.
|
||||
|
||||
## Operations: the rules you feel on Tuesday at 3 a.m.
|
||||
|
||||
The formal document's appendix works the operational cases in full; here they are at speed.
|
||||
|
||||
**The ledger, and the three-way distinction that keeps it honest.** Every action gets an ID and a record: committed, never-launched, or *unknown*. "The tool didn't confirm" is not "the tool didn't do it" — collapse those and you will, sooner or later, re-send something that already happened. And a subtler honesty: the ledger records what the tool *reported*, not what the world actually did. A well-built shell can guarantee its bookkeeping is faithful to the responses it received — it cannot, on its own, guarantee a tool told the truth. A tool that returns a clean "done!" for something it never did puts a clean "done!" in your ledger. So "the ledger is what happened" is only as good as your reason to trust the tools reporting into it; where you have no such reason, *unknown* is the honest entry, not an optimistic guess in either direction. The double-send bug has one reliable cure: **journal before dispatch.** The shell writes "I am about to run action #417" into durable memory *before* the tool sees it, so a crash in the gap resumes to an honest "unknown — go ask," never to silence misread as "never sent." Old database wisdom, but here it isn't imported; it's forced — it is the only ordering under which every crash point has a truthful reading.
|
||||
|
||||
**Crashes aren't finishes.** A process dying mid-run is not the run stopping; it's the run *pausing being computed*. Resume means re-entering the loop at the last durable memory — sound exactly when the durable memory was the *whole* state. Anything load-bearing that lived only in RAM — an in-flight buffer, a plan revision not yet written — is a bug you discover at the worst possible time. Recovery is where you find out whether your state was really your state. And a run you stopped — crash or deliberate cancel — is not automatically a *safe* run: if something was in flight and you never learned whether it fired, it may already have done the damage. "We stopped in time" is only true when everything in flight resolved to something safe; an outstanding *unknown* has to be treated as possibly-bad, the same optimism the ledger warns against, one level up.
|
||||
|
||||
**Two innocent actions can be guilty together.** Models emit several tool calls per turn. "Read the secret" passes review. "Post to the web" passes review. The pair is an exfiltration channel — so the gate authorizes the *set*, atomically, with the interactions checked, not each element in isolation.
|
||||
|
||||
**Sub-agents are just fancy tools.** An agent that spawns another agent is, from the parent's chair, calling a tool: the spawn is gated, the budget is part of the deal, and the child's whole run comes back as one result carrying the child's ledger. Two laws travel down the tree: budgets subdivide, and **authority only narrows** — a child holds at most a subset of its parent's permissions, and a child's request beyond those grants routes *up*, ultimately to the owner, because a parent inventing an approval it never held is the tricked-judge case wearing a manager's badge. A corollary worth framing: a *fully autonomous* run is one whose owner is unreachable — meaning the only channel that can ever widen anything is closed, and its permissions are frozen at launch. That is not a limitation of the theory. That is what the word "autonomous" costs.
|
||||
|
||||
**Keep the originals.** When the transcript outgrows the prompt and you summarize it down, deleting the original is an irreversible act against your own state — and irreversible acts are gate decisions, self-directed or not. Keep originals content-addressed; let the summary be an index, re-derivable, auditable. A summary you can check against its source is a note. A summary that replaced its source is a fait accompli.
|
||||
|
||||
## Robots that never clock out — and robots that assign their own work
|
||||
|
||||
Everything so far assumed a job that *ends*: you ask, the robot does it, you read the result. Two steps past that are where the interesting failures live, and they're the same idea one level bigger each time.
|
||||
|
||||
**The robot that never clocks out (a daemon).** A monitor, a coordinator, a service — it isn't supposed to finish; it's supposed to keep going, wake on events, do a bit of work, go back to waiting. The clean way to think about it: each wake-work-rest cycle is one ordinary run, and the daemon is just those runs chained end to end forever. That reframing is free — but it comes with a bill nobody likes. **Safety that's fine per cycle rots over many cycles.** A 99.99%-safe cycle sounds bulletproof; run it ten thousand times and you're at about a coin-flip of having touched the danger zone at least once. So a long-running robot's safety isn't a fixed wall, it's a slow leak — which means the antidote isn't a better wall, it's *scheduled resets*: the owner re-confirming, credentials rotating, memory getting audited and re-summarized against the originals. Housekeeping isn't housekeeping; it's the thing that keeps the safety math from decaying. And the slow-leak logic is exactly where slow attacks live — a poisoned note dropped into memory on Monday and read back into the plan on Friday is an injection with a long fuse. So the trust label on a piece of information has to survive across cycles, not just within one. One more wrinkle: a daemon drifts in and out of your reach. While you're around, it can escalate to you; while you're not, "escalate to the owner" isn't available — so the one thing it must always be able to do instead is *stop*. A robot that can be tricked into refusing everything, and can't reach you, had better be able to halt rather than be steered.
|
||||
|
||||
**The robot that assigns its own work (the loop).** Step back one more time. Above the robot that *does* a task sits a system that decides *which task is next* — scans the backlog, picks one, launches the robot at it, checks the result, remembers, fires again. This is the thing people mean in 2026 when they say they've stopped prompting their agents and started writing *loops* that prompt them: you design the assigner once, and it runs the doer for you while you sleep. The honest observation — and the reason this document bothers with it — is that the assigner is *not a new kind of thing*. It's the same harness, one level up: it has its own memory (the backlog), its own gate (**who let the loop refactor the auth module at 3 a.m.?**), its own verifier, and its own two walls. Every rule from the inner robot recurs on the outer one — including the uncomfortable ones. There's still no proof it stays out of trouble over a long night; there's only a measured progress meter, with the same catch that a *learned* meter can be fooled. And the origin story of the whole trend is the cautionary case in miniature: the famous first version was literally the same prompt in a `while` loop until the tests passed — which is the empty gate, the always-open lock, one level up. It works beautifully right up until the tests weren't checking the thing that mattered. The loop doesn't delete the hard problems. It moves them up a floor, where they're bigger and you're further away.
|
||||
|
||||
The pattern, if you want the whole thing in one line: *words, context, robot, loop* are four sizes of the same object, and every promise in this document lives in the whole assembled thing — never in any one layer by itself.
|
||||
|
||||
## The two walls
|
||||
|
||||
Two limits are structural. You don't fix them with a better harness; you design around them.
|
||||
|
||||
**The desk.** The model can hold only so much *in mind at once* — the context window. Files, databases, and search extend what it can *look up*, not what it can hold: every lookup still passes through the same small window to touch actual computation. The shell can page; the model cannot grow its desk. Tasks whose irreducible working set exceeds the desk don't fail loudly — they fail by forgetting the middle (the well-documented "lost in the middle" effect is this wall showing through the paint).
|
||||
|
||||
**The dictionary.** The model's knowledge is frozen into its parameters at training time — and the proof problem above is conjectured to live at that same scale: the certificate wouldn't fit anywhere smaller than the brain it certifies. The two walls trade against each other along the training-versus-inference axis — bigger dictionary or bigger desk — directionally, and at no clean exchange rate.
|
||||
|
||||
## How this could be wrong
|
||||
|
||||
This is a hypothesis, and it says out loud what would kill it. The tests, in plain terms:
|
||||
|
||||
- **The replay test.** Rerun with model answers and tool results pinned. Any leftover variation — timestamps, wall-clocks, and cache expiries are the classic leaks — falsifies "the harness adds no randomness" until accounted for.
|
||||
- **The drop-a-variable test.** Remove something from memory; if behavior statistics shift, the memory wasn't complete. The crash-resume version of the same test: if resuming from saved state breaks, the saved state wasn't the state.
|
||||
- **Does the meter mean anything?** If no reasonable progress meter's drift predicts real failures — across the natural families, not just one bad candidate — the whole "measure what you can't prove" program is empty.
|
||||
- **The red-team test.** Swap sampled tool outputs for worst-case ones: injected pages, poisoned metadata, malformed replies. The design must survive the worst permitted world, not the average one.
|
||||
- **Gates versus begging.** The theory predicts deterministic gating beats prompt-level pleading. If "please be careful" alone matches real gates on security outcomes, the controller-versus-model story is wrong.
|
||||
- **The compression hunt.** Exhibit a compact, provably sound progress certificate for a frontier-scale model on a nontrivial task family, and the central conjecture falls — constructively.
|
||||
- **The desk probe.** Take a task family with a *proven* memory floor — so "it needed the whole picture at once" is someone else's theorem, not our excuse — scale it past the window, and watch: the wall predicts a *ceiling*, not a cliff — past the boundary, a success rate that stays capped no matter how many retries you buy. A family solved reliably out there, without new shell tricks for splitting the work, kills the wall.
|
||||
|
||||
## Who else landed here
|
||||
|
||||
The formal document keeps three honesty tiers. **Borrowed**: real theorems, cited — the drift and stopping-time mathematics is classical, and the very architecture of a deterministic supervisor gating a plant it didn't author is 1987 control theory; the shape is older than the web. **Ours**: the modeling choices and the conjectures — the walls, the incompressibility claim, the design rules — organizing principles, not results. **Corroborated**: pieces of the same object reached independently by people who never saw this framing — capability-security work isolating control flow from untrusted data (CaMeL), reinforcement-learning "shields" filtering a learned policy's actions through a deterministic checker, verification work that states the "learned safeguards can't certify" gap as its opening motivation, and architecture patterns converging on plan-then-execute. Even the field's live disagreement — provable-but-rigid deterministic layers versus flexible-but-uncertifiable learned checks — is, in this frame, not a fight but a placement: you need both, on their proper sides of the irreversibility line, with the learned one permitted only to tighten.
|
||||
|
||||
## What to remember
|
||||
|
||||
The model proposes; the gate disposes. No is the default, and a refusal must be safe. Only the top of the trust hierarchy widens permissions — a human decision, never the model, a tool result, a summary, or a judge. "Didn't confirm" is not "didn't happen." The desk is finite and the proof doesn't compress, so you measure — and you say *measurement* when you mean measurement. A robot that never stops leaks safety slowly, so it needs scheduled resets — and when it can't reach you, it must be able to stop. A loop that runs robots for you is just a bigger robot with the same rules and a further-away owner. And all of it is a hypothesis wearing its own kill-conditions on its sleeve.
|
||||
|
||||
The formal version — the objects, the certificates, the falsifiers, the citations — is [HYPOTHESIS.md](HYPOTHESIS.md). It wins every disagreement with this file, including this sentence.
|
||||
|
||||
*Same ramblings, fewer symbols.*
|
||||
@@ -21,7 +21,7 @@ Named after the [Ruddy Turnstone](https://en.wikipedia.org/wiki/Ruddy_turnstone)
|
||||
ℋ : s_{n+1} ~ T(s_n) for n < τ*, T = ρ ∘ (M_W ∘ π, E)
|
||||
```
|
||||
|
||||
[**the hypothesis →**](HYPOTHESIS.md)
|
||||
[**the primer →**](PRIMER.md)
|
||||
|
||||
### Release Tracks
|
||||
|
||||
|
||||
@@ -458,7 +458,7 @@ Each item in `items` (shared by `tool_info` and `approve_request`):
|
||||
| `context_window` | int | Total context window size in tokens |
|
||||
| `pct` | float | Percentage of context window used |
|
||||
| `effort` | string | Reasoning effort level (`low`/`medium`/`high`) |
|
||||
| `cache_creation_tokens` | int | Tokens written to prompt cache (Anthropic) |
|
||||
| `cache_creation_tokens` | int | Tokens written to prompt cache (Anthropic + OpenAI) |
|
||||
| `cache_read_tokens` | int | Tokens served from prompt cache (Anthropic + OpenAI) |
|
||||
|
||||
**`info`** -- an informational message (e.g. command output).
|
||||
@@ -948,6 +948,7 @@ All fields are optional. The body can be empty or an empty JSON object.
|
||||
| `name` | string | Auto-generated workstream name |
|
||||
| `resumed` | bool | Whether a previous session was successfully resumed |
|
||||
| `message_count` | int | Number of messages in the resumed session (0 if fresh) |
|
||||
| `initial_message_status` | string | Present ONLY when the workstream was created but its `initial_message` could not be delivered: `"queue_full"` (a raced live worker's interjection queue was at capacity — resend via `/send`; any uploads stay staged) or `"refused_closed"` (the workstream was closed mid-create). Absent whenever the message was dispatched. |
|
||||
|
||||
**Error (limit reached):**
|
||||
|
||||
|
||||
@@ -622,19 +622,19 @@ LLMProvider (protocol)
|
||||
|------|--------|
|
||||
| `StreamChunk` | `content_delta`, `reasoning_delta`, `tool_call_deltas`, `info_delta`, `usage`, `finish_reason`, `provider_blocks` |
|
||||
| `CompletionResult` | `content`, `tool_calls`, `finish_reason`, `usage`, `provider_blocks` |
|
||||
| `ModelCapabilities` | `context_window`, `max_output_tokens`, `supports_temperature`, `token_param`, `thinking_mode`, `supports_effort`, `supports_web_search`, `supports_tool_search`, `supports_vision`, `supports_reasoning_replay` |
|
||||
| `ModelCapabilities` | `context_window`, `max_output_tokens`, `supports_temperature`, `token_param`, `thinking_mode`, `supports_effort`, `supports_web_search`, `supports_tool_search`, `supports_vision`, `supports_reasoning_replay`, `supports_verbosity`, `verbosity`, `supports_pro_mode`, `reasoning_mode` |
|
||||
| `UsageInfo` | `prompt_tokens`, `completion_tokens`, `total_tokens`, `cache_creation_tokens`, `cache_read_tokens` |
|
||||
|
||||
**OpenAIProvider** (`_openai.py`): passes messages through unchanged (they are
|
||||
already in OpenAI format), including multi-part content blocks (text + images)
|
||||
in tool results. Model capability lookup table covers GPT-5/5.1/5.2/5.3/5.4,
|
||||
in tool results. Model capability lookup covers GPT-5 through GPT-5.6,
|
||||
O-series, and search models (`gpt-5-search-api`) — all with `supports_vision`.
|
||||
For search models, injects `web_search_options` and removes the `web_search`
|
||||
function tool (the model always searches). Citations from `url_citation`
|
||||
annotations are formatted as footnotes. Extended prompt cache retention
|
||||
(`prompt_cache_retention: "24h"`) is enabled for GPT-5.x models at no
|
||||
additional cost. Cached token counts are extracted from
|
||||
`usage.prompt_tokens_details.cached_tokens`. Unknown models get permissive
|
||||
annotations are formatted as footnotes. Pre-5.6 GPT-5 models request extended
|
||||
prompt-cache retention (`prompt_cache_retention: "24h"`); GPT-5.6 uses
|
||||
`prompt_cache_options.ttl: "30m"`. Cache reads and writes are extracted from
|
||||
`cached_tokens` and `cache_write_tokens`. Unknown models get permissive
|
||||
defaults with `supports_vision=False` and use SearxNG for web search. The
|
||||
`openai-compatible` lane never consults this table at all — on either API
|
||||
surface (the responses pin is served by a compat-mode
|
||||
@@ -642,8 +642,9 @@ surface (the responses pin is served by a compat-mode
|
||||
local server serves whatever the operator named it (vLLM
|
||||
`--served-model-name` is a free string), so a prefix collision with a cloud
|
||||
model id must not inherit that model's sampling/effort contract — every
|
||||
local model gets the plain defaults, and anything beyond them is declared on
|
||||
the model definition (capabilities JSON + `server_compat`), matching the
|
||||
local model gets the plain defaults, commercial prompt-cache controls are not
|
||||
injected by model-name prefix, and anything beyond those defaults is declared
|
||||
on the model definition (capabilities JSON + `server_compat`), matching the
|
||||
`anthropic-compatible` lane.
|
||||
|
||||
**AnthropicProvider** (`_anthropic.py`): converts OpenAI-format messages to
|
||||
|
||||
+6
-3
@@ -131,9 +131,12 @@ Per-LLM-request token and tool call metrics:
|
||||
LLM response with prompt/completion tokens, cache tokens, tool call count,
|
||||
model, ws_id
|
||||
- **Prompt caching**: Anthropic automatic caching (`cache_control: ephemeral`)
|
||||
and OpenAI extended retention (`prompt_cache_retention: 24h` for GPT-5.x)
|
||||
are enabled by default. `cache_creation_tokens` and `cache_read_tokens` are
|
||||
tracked per request in `usage_events` and surfaced in the Usage admin tab
|
||||
and OpenAI caching are enabled by default. Pre-5.6 GPT-5 models request
|
||||
`prompt_cache_retention: 24h`; GPT-5.6 uses
|
||||
`prompt_cache_options: {"ttl": "30m"}`. GPT-5.6 cache writes use the
|
||||
provider's 1.25× input-token rate. `cache_creation_tokens` and
|
||||
`cache_read_tokens` are tracked per request in `usage_events` and surfaced
|
||||
in the Usage admin tab
|
||||
- **Querying**: `GET /v1/api/admin/usage` with `group_by` (day/hour/model/user)
|
||||
and time range filtering — includes cache token aggregates
|
||||
- **Prometheus**: `turnstone_tokens_total{type="cache_creation|cache_read"}`
|
||||
|
||||
+8
-2
@@ -249,8 +249,14 @@ are withheld from the live surfaces (a reused call_id must never ride a stale
|
||||
`approve` into Smart Approvals) but still persist with
|
||||
`user_decision = "superseded"` so the audit trail records the judge's answer.
|
||||
|
||||
Sub-agents (plan agent, task agent) are exempt from intent validation -- they
|
||||
always get full tool visibility without judge evaluation.
|
||||
Sub-agent (task agent) tool calls are judge-gated too. Each runs the same
|
||||
intent pipeline as its own `agent_gate` generation, grounded in that sub-agent's
|
||||
own trajectory -- its task prompt is the delegation contract the operator
|
||||
approved, so "does this call serve the task" is the right local question.
|
||||
Agent-gate generations never occupy the main loop's supersede slot (parallel
|
||||
siblings would otherwise make each other's verdicts look stale); per-cycle
|
||||
generation checks enforce staleness instead, and `judge.cancel_on_approval`
|
||||
fires per gate exactly like the main loop.
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -41,6 +41,7 @@ are set.
|
||||
| `TURNSTONE_OIDC_PASSWORD_ENABLED` | No | `true` | Set to `false` to hide the password form and block all username/password logins (including admin). API tokens continue to work. |
|
||||
| `TURNSTONE_OIDC_REDIRECT_BASE` | Yes | — | Externally-reachable origin for the OIDC redirect URI (e.g. `https://app.example.com`). Without this, OIDC will refuse to start. The previous Host-header fallback was unsafe under permissive reverse proxies. |
|
||||
| `TURNSTONE_OIDC_TRUSTED_ENDPOINT_HOSTS` | No | — | Comma-separated list of additional hostnames whose endpoints the IdP discovery document is allowed to reference. See [Cross-host endpoints](#cross-host-endpoints). |
|
||||
| `TURNSTONE_OIDC_ALLOW_PRIVATE_NETWORK` | No | `false` | Allow the issuer (and its discovered endpoints) to resolve to private/internal addresses — needed for a self-hosted IdP on an internal network. See [Self-hosted and internal IdPs](#self-hosted-and-internal-idps). |
|
||||
|
||||
All four required fields — issuer, client ID, client secret, and
|
||||
`TURNSTONE_OIDC_REDIRECT_BASE` — must be set. If any are missing OIDC
|
||||
@@ -99,6 +100,40 @@ The same scheme / no-userinfo / SSRF rules apply to allow-listed hosts —
|
||||
this knob only relaxes the same-origin check, not the security gates.
|
||||
Each entry is a hostname (no scheme, no path).
|
||||
|
||||
### Self-hosted and internal IdPs
|
||||
|
||||
By default Turnstone refuses an issuer whose hostname resolves to a
|
||||
private or internal address:
|
||||
|
||||
```
|
||||
OIDCError: endpoint URL resolves to non-public address (10.0.0.5): https://auth.example.site
|
||||
```
|
||||
|
||||
This is SSRF hardening, not a licensing or product restriction: the OIDC
|
||||
flow makes server-side HTTP requests (discovery, JWKS, token exchange),
|
||||
and refusing non-public destinations keeps a mistyped or maliciously
|
||||
steered issuer from aiming those fetches at internal services. For a
|
||||
self-hosted IdP (Keycloak, Authentik, Dex, …) on a private network,
|
||||
opt in explicitly in `config.toml`:
|
||||
|
||||
```toml
|
||||
[oidc]
|
||||
allow_private_network = true
|
||||
```
|
||||
|
||||
or via `TURNSTONE_OIDC_ALLOW_PRIVATE_NETWORK=true` (the env var wins
|
||||
when both are set).
|
||||
|
||||
The opt-in admits private-range (RFC 1918), unique-local, CGNAT
|
||||
(100.64/10 — tailnets), and loopback addresses. Link-local, multicast,
|
||||
and reserved ranges stay refused even with the opt-in — cloud metadata
|
||||
services (169.254.169.254) live there, and no legitimate IdP does. The
|
||||
HTTPS requirement and the same-origin endpoint checks are unaffected.
|
||||
|
||||
This knob only affects the login-flow IdP configured here. OAuth
|
||||
endpoints advertised by remote MCP servers are untrusted input and are
|
||||
always held to the strict public-address rule.
|
||||
|
||||
### config.toml alternative
|
||||
|
||||
```toml
|
||||
@@ -111,6 +146,8 @@ provider_name = "Google"
|
||||
role_claim = "groups"
|
||||
password_enabled = true
|
||||
redirect_base = "https://app.example.com"
|
||||
# Self-hosted IdP on an internal network (see "Self-hosted and internal IdPs")
|
||||
allow_private_network = false
|
||||
|
||||
[oidc.role_map]
|
||||
admin = "builtin-admin"
|
||||
|
||||
@@ -54,6 +54,23 @@ When a per-model override is `NULL` (empty in the UI), the global default is
|
||||
used. Switching models via `/model <alias>` re-resolves sampling parameters
|
||||
from the new model's overrides or global defaults.
|
||||
|
||||
### Responses output controls (per-model)
|
||||
|
||||
Models whose capability table declares Responses output controls expose two
|
||||
additional fields in the Models create/edit shelf:
|
||||
|
||||
| Field | Stored capability | Values | Effect |
|
||||
|-------|-------------------|--------|--------|
|
||||
| Output verbosity | `verbosity` | `low`, `medium`, `high` | Controls answer length independently of reasoning effort. |
|
||||
| Reasoning mode | `reasoning_mode` | `standard`, `pro` | Selects standard or higher-compute Pro execution without changing the model ID. |
|
||||
|
||||
An empty selection means provider default and omits the capability key. Known
|
||||
GPT-5.6 models inherit support from the built-in table without persisting
|
||||
redundant support flags. An OpenAI-compatible model pinned to the Responses API
|
||||
can opt in with the `supports_verbosity` and `supports_pro_mode` capability
|
||||
tiles. Chat Completions and non-Responses providers do not surface or submit
|
||||
these controls.
|
||||
|
||||
**Removed settings:** `model.name` and `model.context_window` have been removed
|
||||
from ConfigStore. Model names and context windows are now configured per-model
|
||||
in the Models tab. A startup warning is logged if these keys appear in
|
||||
|
||||
+44
-6
@@ -1,6 +1,6 @@
|
||||
# Tools Reference
|
||||
|
||||
turnstone exposes 16 built-in tools plus any number of external MCP tools to the
|
||||
turnstone exposes 17 built-in tools plus any number of external MCP tools to the
|
||||
LLM via the OpenAI function-calling interface. Built-in tools are defined as JSON
|
||||
files under `turnstone/tools/` and loaded at startup by `turnstone/core/tools.py`.
|
||||
MCP tools are discovered from configured MCP servers at startup by
|
||||
@@ -44,10 +44,10 @@ schema plus turnstone-specific metadata keys:
|
||||
|
||||
| Name | Description |
|
||||
|---------------------|-------------|
|
||||
| `TOOLS` | All 28 loaded built-in tool definitions (interactive + coordinator union). Sessions send a kind-specific subset (`INTERACTIVE_TOOLS` or `COORDINATOR_TOOLS`). |
|
||||
| `TOOLS` | All 29 loaded built-in tool definitions (interactive + coordinator union). Sessions send a kind-specific subset (`INTERACTIVE_TOOLS` or `COORDINATOR_TOOLS`). |
|
||||
| `TASK_AGENT_TOOLS` | Tools with `task_agent: true` -- available to task sub-agents. Includes write operations. |
|
||||
| `TASK_AUTO_TOOLS` | Set of all tool names with `auto_approve: true` -- used by task-agent sub-sessions to skip confirmation for matching available tools. |
|
||||
| `BUILTIN_TOOL_NAMES`| Frozenset of all 28 built-in tool names (interactive + coordinator union). Used by tool search to distinguish always-on tools from deferrable MCP tools. |
|
||||
| `BUILTIN_TOOL_NAMES`| Frozenset of all 29 built-in tool names (interactive + coordinator union). Used by tool search to distinguish always-on tools from deferrable MCP tools. |
|
||||
| `PRIMARY_KEY_MAP` | Dict mapping tool name to its `primary_key` parameter name. |
|
||||
|
||||
---
|
||||
@@ -65,7 +65,7 @@ Tool execution follows a three-phase pipeline inside `ChatSession._execute_tools
|
||||
- Parses the JSON arguments (with fallback for malformed JSON).
|
||||
- If JSON parsing fails entirely, uses `PRIMARY_KEY_MAP` to map a bare string
|
||||
to the correct parameter.
|
||||
- Dispatches to the matching `_prepare_{func_name}()` handler. There are 16
|
||||
- Dispatches to the matching `_prepare_{func_name}()` handler. There are 17
|
||||
built-in tools plus `tool_search` (synthetic, client-side BM25 fallback) and
|
||||
the generic `_prepare_mcp_tool()` handler for MCP tools.
|
||||
- Validates arguments and builds a preview dict containing:
|
||||
@@ -125,6 +125,9 @@ Each item's `execute` callable is invoked:
|
||||
- `web_fetch` -- fetches a URL (SSRF-protected, but makes network requests)
|
||||
- `web_search` -- web search via self-hosted SearxNG (makes network requests)
|
||||
- `task_agent` -- spawns an autonomous sub-agent
|
||||
- `open_preview` -- **URL targets only** (network access, gated like `web_fetch`);
|
||||
file-path and `attachment:` targets are local reads and run unprompted like
|
||||
`read_file`
|
||||
|
||||
Note: The JSON schema metadata key `auto_approve` controls membership in
|
||||
`TASK_AUTO_TOOLS` (used for task agent sub-sessions). The actual runtime
|
||||
@@ -157,6 +160,7 @@ Every tool defines a `primary_key`. The mapping is:
|
||||
| `search` | `query` |
|
||||
| `web_fetch` | `url` |
|
||||
| `web_search` | `query` |
|
||||
| `open_preview` | `target` |
|
||||
| `task_agent` | `prompt` |
|
||||
| `memory` | `name` |
|
||||
| `recall` | `query` |
|
||||
@@ -285,7 +289,7 @@ Fetch a URL and extract specific information from it.
|
||||
| `url` | string | yes | The URL to fetch (must start with `http://` or `https://`). |
|
||||
| `question` | string | yes | What to extract or answer from the page content. |
|
||||
|
||||
- **What it does**: Fetches the URL, strips HTML to plain text, and uses the LLM to extract the answer to the question from the page content. Protected against SSRF (blocks private/internal IPs).
|
||||
- **What it does**: Fetches the URL, strips HTML to plain text, and uses the LLM to extract the answer to the question from the page content. Every redirect hop is SSRF-screened before it is requested. Private/internal addresses are refused by default; enable `tools.allow_private_network` (console Settings → Tools) to make them approvable for self-hosted setups whose services live on the local network — the approval prompt marks such requests, and a public site redirecting into private space is refused regardless.
|
||||
- **Auto-approve**: No -- requires user confirmation (makes network requests).
|
||||
- **Agent availability**: `task_agent`.
|
||||
|
||||
@@ -345,6 +349,39 @@ It reports the score scale, whether the endpoint cleanly separates relevant from
|
||||
|
||||
---
|
||||
|
||||
### open_preview
|
||||
|
||||
Show the user rich content in a preview pane beside the conversation.
|
||||
|
||||
| Parameter | Type | Required | Description |
|
||||
|-----------|--------|----------|-------------|
|
||||
| `target` | string | yes | An http(s) URL, a file path, or `attachment:<id>` for a file attached to the conversation. |
|
||||
| `kind` | string | no | Rendering override: `web`, `pdf`, `image`, `table`, `text`, or `markdown`. Detected from the content when omitted. |
|
||||
| `title` | string | no | Pane header title. Defaults to the page title, filename, or URL. |
|
||||
|
||||
- **What it does**: Resolves the target to bytes (URLs fetch through the same
|
||||
SSRF-guarded path as `web_fetch`, screened per redirect hop, honoring the
|
||||
same `tools.allow_private_network` opt-in), classifies the
|
||||
content, stores it content-addressed against the workstream, and opens the
|
||||
frontend preview pane beside the conversation: web pages render in a fully
|
||||
sandboxed iframe (no scripts, opaque origin), PDFs in the browser viewer,
|
||||
images inline, CSV/TSV/JSON as a sortable table, text/markdown rendered. A
|
||||
previewed web page loads none of its remote images or styles by default, so
|
||||
opening it never reveals the viewer to the page's site; a toggle in the pane
|
||||
header turns remote content back on for that preview. The
|
||||
model receives only a one-line confirmation — to reason about content, use
|
||||
`web_fetch` / `read_file` instead. Preview content is size-capped per kind
|
||||
(pages 4 MB, PDFs 32 MB, images 4 MB, tables 2 MB, text 512 KB) and GC'd
|
||||
with the workstream.
|
||||
- **Auto-approve**: URL targets require confirmation (network access); file
|
||||
paths and `attachment:` targets run unprompted (local reads).
|
||||
- **Agent availability**: interactive sessions only (not `task_agent`, not
|
||||
coordinators).
|
||||
- **Surfaces**: the pane renders in the web UI (standalone and console). The
|
||||
CLI prints the confirmation line only — there is no terminal pane.
|
||||
|
||||
---
|
||||
|
||||
## Agent
|
||||
|
||||
The tool name uses the `_agent` suffix — bare `task` collides with
|
||||
@@ -545,6 +582,7 @@ pre-configure skills at workstream creation.
|
||||
| `search` | File Ops | Yes | Yes | `query` |
|
||||
| `web_fetch` | Info | No | Yes | `url` |
|
||||
| `web_search` | Info | No | Yes | `query` |
|
||||
| `open_preview`| Info | URL: no; path/attachment: yes | No | `target` |
|
||||
| `task_agent` | Agent | No | No | `prompt` |
|
||||
| `memory` | Memory | Yes | No | `name` |
|
||||
| `recall` | Memory | Yes | No | `query` |
|
||||
@@ -654,7 +692,7 @@ MCP-compatible service.
|
||||
3. **Schema conversion**: Each MCP tool's `inputSchema` is converted to OpenAI
|
||||
function-calling format. The tool name is prefixed: `mcp__{server}__{tool}`.
|
||||
|
||||
4. **Merging**: MCP tools are appended after the 16 built-in tools via
|
||||
4. **Merging**: MCP tools are appended after the 17 built-in tools via
|
||||
`merge_mcp_tools()`. Built-in tools appear first, giving them natural LLM priority.
|
||||
When dynamic tool search is active, MCP tools are deferred rather than directly
|
||||
visible -- the model discovers them via search as needed (see
|
||||
|
||||
+2
-2
@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
|
||||
|
||||
[project]
|
||||
name = "turnstone"
|
||||
version = "1.7.0rc1"
|
||||
version = "1.7.4"
|
||||
description = "Multi-node AI orchestration platform with tool use, agent routing, and cluster simulation."
|
||||
readme = "README.md"
|
||||
license = "Apache-2.0"
|
||||
@@ -23,7 +23,7 @@ classifiers = [
|
||||
"Topic :: Scientific/Engineering :: Artificial Intelligence",
|
||||
]
|
||||
dependencies = [
|
||||
"openai>=2.37",
|
||||
"openai>=2.45", # GPT-5.6: typed reasoning.mode, prompt_cache_options, and cache_write_tokens
|
||||
"anthropic>=0.108", # claude-fable-5 support; hard runtime floor is 0.105 (mid-conversation system blocks)
|
||||
"httpx>=0.28",
|
||||
"mcp>=1.27,<2", # v2 is a breaking rewrite (2.0.0a1 live 2026-06-11; stable ~2026-07-27) — streamablehttp_client removed, 2-tuple transport, snake_case types; migrate deliberately
|
||||
|
||||
@@ -4,8 +4,9 @@
|
||||
#
|
||||
# curl -fsSL https://raw.githubusercontent.com/turnstonelabs/turnstone/main/run.sh | bash
|
||||
#
|
||||
# Autodetects your distro (Ubuntu/Debian, Fedora/RHEL, Arch, and WSL on any of
|
||||
# them) and:
|
||||
# Autodetects your distro — Ubuntu/Debian, Fedora/RHEL, Arch, their common
|
||||
# derivatives (Mint, Pop!_OS, Nobara, AlmaLinux, …), and WSL on any of them —
|
||||
# and:
|
||||
# 1. ensures git is installed, then clones the repo
|
||||
# 2. ensures Docker + the compose plugin are installed and the daemon is usable
|
||||
# 3. asks how many server nodes to run (1-10)
|
||||
@@ -65,12 +66,18 @@ ask() {
|
||||
|
||||
# -- distro / package manager detection --------------------------------------
|
||||
OS_ID=""; OS_LIKE=""; PKG=""; IS_WSL=0; SUDO=""
|
||||
# Extra os-release fields, captured only to pick Docker's upstream repo when
|
||||
# get.docker.com refuses a derivative it doesn't recognize (see install_docker).
|
||||
OS_PLATFORM_ID=""; OS_CODENAME=""; OS_UBUNTU_CODENAME=""
|
||||
|
||||
detect_os() {
|
||||
if [ -r /etc/os-release ]; then
|
||||
# shellcheck disable=SC1091
|
||||
. /etc/os-release
|
||||
OS_ID="${ID:-}"; OS_LIKE="${ID_LIKE:-}"
|
||||
OS_PLATFORM_ID="${PLATFORM_ID:-}"
|
||||
OS_CODENAME="${VERSION_CODENAME:-}"
|
||||
OS_UBUNTU_CODENAME="${UBUNTU_CODENAME:-}"
|
||||
fi
|
||||
if grep -qiE 'microsoft|wsl' /proc/version 2>/dev/null || [ -n "${WSL_DISTRO_NAME:-}" ]; then
|
||||
IS_WSL=1
|
||||
@@ -130,11 +137,83 @@ clone_repo() {
|
||||
# -- docker -------------------------------------------------------------------
|
||||
DOCKER="docker"
|
||||
|
||||
# Fallback when get.docker.com won't install here. That script keys off $ID alone
|
||||
# (never ID_LIKE), so it aborts with "Unsupported distribution '<id>'" on every
|
||||
# derivative — Nobara, Linux Mint, Pop!_OS, AlmaLinux, Oracle Linux, … — even
|
||||
# though the family is clear. We already know the family from detect_os, so we add
|
||||
# Docker's official CE repo for the matching upstream and install the same
|
||||
# packages get.docker.com would (including the compose plugin the rest of run.sh
|
||||
# relies on).
|
||||
install_docker_ce_repo() {
|
||||
local up
|
||||
case "$PKG" in
|
||||
apt)
|
||||
local codename arch
|
||||
# UBUNTU_CODENAME is set by Ubuntu and every Ubuntu-derived distro
|
||||
# (Mint/Pop!_OS/Zorin/…) and never by pure Debian, so it both routes
|
||||
# the family and gives the exact codename Docker's repo expects.
|
||||
if [ -n "$OS_UBUNTU_CODENAME" ]; then
|
||||
up=ubuntu; codename="$OS_UBUNTU_CODENAME"
|
||||
else
|
||||
up=debian; codename="$OS_CODENAME"
|
||||
fi
|
||||
[ -n "$codename" ] || die "couldn't determine the $up release codename for Docker's repo — install Docker manually and re-run."
|
||||
arch="$(dpkg --print-architecture 2>/dev/null || echo amd64)"
|
||||
info "Adding Docker's $up repository ($codename)."
|
||||
$SUDO install -m 0755 -d /etc/apt/keyrings
|
||||
curl -fsSL "https://download.docker.com/linux/$up/gpg" | $SUDO tee /etc/apt/keyrings/docker.asc >/dev/null
|
||||
$SUDO chmod a+r /etc/apt/keyrings/docker.asc
|
||||
printf 'deb [arch=%s signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/%s %s stable\n' \
|
||||
"$arch" "$up" "$codename" | $SUDO tee /etc/apt/sources.list.d/docker.list >/dev/null
|
||||
$SUDO apt-get update -y
|
||||
$SUDO apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
|
||||
;;
|
||||
dnf|yum)
|
||||
# A Fedora spin and a RHEL clone can both carry "fedora" in ID_LIKE
|
||||
# (Nobara's is "rhel centos fedora"), so ID_LIKE can't separate them.
|
||||
# PLATFORM_ID can: Fedora is platform:fNN, Enterprise Linux platform:elN.
|
||||
case "$OS_PLATFORM_ID" in
|
||||
platform:f*) up=fedora ;;
|
||||
platform:el*) up=centos ;;
|
||||
*) if [ -e /etc/fedora-release ]; then up=fedora; else up=centos; fi ;;
|
||||
esac
|
||||
info "Adding Docker's $up repository."
|
||||
$SUDO curl -fsSL "https://download.docker.com/linux/$up/docker-ce.repo" \
|
||||
-o /etc/yum.repos.d/docker-ce.repo \
|
||||
|| die "couldn't add Docker's $up repository — install Docker manually and re-run."
|
||||
pkg_install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
|
||||
;;
|
||||
esac
|
||||
}
|
||||
|
||||
# The distro IDs get.docker.com installs directly: it matches $ID against this
|
||||
# exact set (ignoring ID_LIKE) and aborts on anything else. Mirrors the dispatch
|
||||
# in get.docker.com, including its fedora-asahi-remix -> fedora alias.
|
||||
get_docker_com_supports() {
|
||||
case "$1" in
|
||||
ubuntu|debian|raspbian|centos|fedora|rhel|rocky|sles|fedora-asahi-remix) return 0 ;;
|
||||
*) return 1 ;;
|
||||
esac
|
||||
}
|
||||
|
||||
install_docker() {
|
||||
case "$PKG" in
|
||||
apt|dnf|yum)
|
||||
info "Installing Docker via the official get.docker.com script"
|
||||
curl -fsSL https://get.docker.com | $SUDO sh ;;
|
||||
# Decide up front which installer applies, rather than treating every
|
||||
# get.docker.com failure as "unsupported distro": for an ID it knows,
|
||||
# let it run and surface any real failure (network, apt lock, EOL) via
|
||||
# die instead of masking it with the repo path. Only unrecognized
|
||||
# derivatives (Nobara, Mint, …) — which it would just abort on — skip
|
||||
# straight to adding Docker's repo ourselves.
|
||||
if [ -n "$OS_ID" ] && ! get_docker_com_supports "$OS_ID"; then
|
||||
info "get.docker.com doesn't support '$OS_ID' — using Docker's official repository directly."
|
||||
install_docker_ce_repo
|
||||
else
|
||||
info "Installing Docker via the official get.docker.com script"
|
||||
curl -fsSL https://get.docker.com | $SUDO sh \
|
||||
|| die "get.docker.com failed to install Docker (see the output above). Fix the issue and re-run — the script resumes."
|
||||
fi
|
||||
;;
|
||||
pacman)
|
||||
pkg_install docker docker-compose ;;
|
||||
esac
|
||||
@@ -366,12 +445,15 @@ ${GREEN}${BOLD}Turnstone is running${RESET} (${NODE_COUNT} node$([ "$NODE_COUNT"
|
||||
${DIM}cd $INSTALL_DIR && $DOCKER compose exec caddy cat /data/caddy/pki/authorities/local/root.crt${RESET}
|
||||
|
||||
Finish setup
|
||||
1. Create the first admin user:
|
||||
${DIM}cd $INSTALL_DIR && $DOCKER compose exec node-1 turnstone-admin create-user --username admin --name "Admin"${RESET}
|
||||
2. Open ${url}, log in, and add a model backend in the ${BOLD}Models${RESET} tab —
|
||||
1. Open ${BOLD}${url}${RESET} and create the admin account when prompted —
|
||||
the first user created there gets full admin access.
|
||||
2. Log in, then add a model backend in the ${BOLD}Models${RESET} tab —
|
||||
a local server (vLLM / llama.cpp) or an OpenAI / Anthropic / Gemini key.
|
||||
Nodes boot without a model and pick it up live; no restart needed.
|
||||
|
||||
${DIM}No browser? Create the admin from the CLI instead:
|
||||
cd $INSTALL_DIR && $DOCKER compose exec node-1 turnstone-admin create-admin --username admin --name "Admin"${RESET}
|
||||
|
||||
Scale Running ${scale}
|
||||
|
||||
Manage ${DIM}cd $INSTALL_DIR${RESET}
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
"openapi": "3.1.0",
|
||||
"info": {
|
||||
"title": "turnstone Console API",
|
||||
"version": "1.7.0a6",
|
||||
"version": "1.7.0rc1",
|
||||
"description": "Cluster-wide visibility and control across all turnstone nodes."
|
||||
},
|
||||
"paths": {
|
||||
@@ -6688,7 +6688,7 @@
|
||||
"tags": [
|
||||
"Coordinator"
|
||||
],
|
||||
"description": "Aggregates the persisted row, a best-effort live block from the owning node (or the in-process coordinator manager for ``kind=\"coordinator\"`` rows), and the tail of the message history. Gated on the ``admin.cluster.inspect`` permission (granted to ``builtin-admin`` via migration 040; revoke or reassign to a custom role for tighter control). ``live`` is null on node unreachability / 5xx so callers can degrade gracefully.",
|
||||
"description": "Aggregates the persisted row, a best-effort live block from the owning node (or the in-process coordinator manager for ``kind=\"coordinator\"`` rows), and the tail of the message history. Gated on the ``admin.cluster.inspect`` permission (granted to ``builtin-admin`` via migration 040; revoke or reassign to a custom role for tighter control). A workstream attached to a *private* project stays confidential to its members: a permitted caller who isn't its owner / creator / project member gets a 404 (same masking as an unknown id). ``live`` is null on node unreachability / 5xx so callers can degrade gracefully.",
|
||||
"parameters": [
|
||||
{
|
||||
"name": "ws_id",
|
||||
@@ -13361,7 +13361,7 @@
|
||||
"type": "object"
|
||||
},
|
||||
"PendingApprovalItem": {
|
||||
"description": "One pending tool-call inside a ``PendingApprovalDetail`` envelope.\n\nMirrors the dict ``SessionUIBase.serialize_pending_approval_detail``\nemits per item. ``heuristic_verdict`` / ``judge_verdict`` are kept\nloosely-typed because the underlying verdict shape varies by tier;\nconsumers that want the full structure can decode against\n:class:`turnstone.sdk.events.IntentVerdictEvent`.",
|
||||
"description": "One pending tool-call inside a ``PendingApprovalDetail`` envelope.\n\nMirrors the dict ``SessionUIBase.serialize_pending_approval_details``\nemits per item inside each cycle entry. ``heuristic_verdict`` / ``judge_verdict`` are kept\nloosely-typed because the underlying verdict shape varies by tier;\nconsumers that want the full structure can decode against\n:class:`turnstone.sdk.events.IntentVerdictEvent`.",
|
||||
"properties": {
|
||||
"call_id": {
|
||||
"default": "",
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
"openapi": "3.1.0",
|
||||
"info": {
|
||||
"title": "turnstone Server API",
|
||||
"version": "1.7.0a6",
|
||||
"version": "1.7.0rc1",
|
||||
"description": "Single-node workstream management, chat interaction, and real-time streaming."
|
||||
},
|
||||
"paths": {
|
||||
@@ -2564,6 +2564,23 @@
|
||||
},
|
||||
"title": "Attachment Ids",
|
||||
"type": "array"
|
||||
},
|
||||
"initial_message_status": {
|
||||
"anyOf": [
|
||||
{
|
||||
"enum": [
|
||||
"queue_full",
|
||||
"refused_closed"
|
||||
],
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"type": "null"
|
||||
}
|
||||
],
|
||||
"default": null,
|
||||
"description": "Present ONLY when the workstream was created but its initial_message could not be delivered: 'queue_full' (a raced live worker's interjection queue was at capacity \u2014 resend via /send; any uploads stay staged) or 'refused_closed' (the workstream was closed mid-create). Absent whenever the message was dispatched.",
|
||||
"title": "Initial Message Status"
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
@@ -2747,7 +2764,7 @@
|
||||
"type": "object"
|
||||
},
|
||||
"PendingApprovalItem": {
|
||||
"description": "One pending tool-call inside a ``PendingApprovalDetail`` envelope.\n\nMirrors the dict ``SessionUIBase.serialize_pending_approval_detail``\nemits per item. ``heuristic_verdict`` / ``judge_verdict`` are kept\nloosely-typed because the underlying verdict shape varies by tier;\nconsumers that want the full structure can decode against\n:class:`turnstone.sdk.events.IntentVerdictEvent`.",
|
||||
"description": "One pending tool-call inside a ``PendingApprovalDetail`` envelope.\n\nMirrors the dict ``SessionUIBase.serialize_pending_approval_details``\nemits per item inside each cycle entry. ``heuristic_verdict`` / ``judge_verdict`` are kept\nloosely-typed because the underlying verdict shape varies by tier;\nconsumers that want the full structure can decode against\n:class:`turnstone.sdk.events.IntentVerdictEvent`.",
|
||||
"properties": {
|
||||
"call_id": {
|
||||
"default": "",
|
||||
|
||||
@@ -164,6 +164,13 @@ export interface CreateWorkstreamResponse {
|
||||
message_count?: number;
|
||||
/** Ids of attachments saved by this request (multipart variant only). */
|
||||
attachment_ids?: string[];
|
||||
/**
|
||||
* Present ONLY when the workstream was created but its initial_message
|
||||
* could not be delivered: "queue_full" (raced live worker's interjection
|
||||
* queue at capacity — resend via /send; uploads stay staged) or
|
||||
* "refused_closed" (workstream closed mid-create).
|
||||
*/
|
||||
initial_message_status?: "queue_full" | "refused_closed";
|
||||
}
|
||||
|
||||
export interface CloseWorkstreamRequest {
|
||||
|
||||
+29
-1
@@ -3,9 +3,37 @@ not fixtures, and several test files want to import them directly."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
import time
|
||||
from typing import TYPE_CHECKING, Any
|
||||
from unittest.mock import MagicMock
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from collections.abc import Callable
|
||||
|
||||
|
||||
def wait_until(cond: Callable[[], bool], timeout: float = 5.0) -> None:
|
||||
"""Poll ``cond`` to True within ``timeout`` or fail the test.
|
||||
|
||||
The worker/wake tests can't join threads by identity:
|
||||
``session_worker.send`` assigns ``ws.worker_thread`` under the lock
|
||||
BEFORE ``t.start()``, so the instant a dispatching call returns, a
|
||||
fast worker may already have run its exit backstop and installed the
|
||||
(not-yet-started) wake thread — joining whatever ``ws.worker_thread``
|
||||
points at races ``RuntimeError: cannot join thread before it is
|
||||
started``. Poll outcomes instead.
|
||||
"""
|
||||
deadline = time.monotonic() + timeout
|
||||
while time.monotonic() < deadline:
|
||||
if cond():
|
||||
return
|
||||
time.sleep(0.005)
|
||||
if cond():
|
||||
# Final re-check: the condition can become true during the last
|
||||
# sleep (or a CI descheduling stall past the deadline) — failing
|
||||
# without re-looking makes the helper itself a flake source.
|
||||
return
|
||||
raise AssertionError("condition not met within timeout")
|
||||
|
||||
|
||||
def make_chat_session(**overrides: Any) -> Any:
|
||||
"""Build a minimal ``ChatSession`` with sane test defaults.
|
||||
|
||||
@@ -0,0 +1,43 @@
|
||||
"""Shared process/polling helpers for the bash + background-shell suites.
|
||||
|
||||
One copy instead of three: ``test_bash_tool_background_hang``,
|
||||
``test_background_shells`` and ``test_bash_background_tool`` all assert on
|
||||
process liveness and poll for asynchronous state. Leading underscore so
|
||||
pytest doesn't collect it.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import contextlib
|
||||
import os
|
||||
import signal
|
||||
import time
|
||||
|
||||
|
||||
def pid_alive(pid: int) -> bool:
|
||||
try:
|
||||
os.kill(pid, 0)
|
||||
except ProcessLookupError:
|
||||
return False
|
||||
except PermissionError:
|
||||
return True
|
||||
return True
|
||||
|
||||
|
||||
def kill_pid(pid: int) -> None:
|
||||
with contextlib.suppress(OSError):
|
||||
os.kill(pid, signal.SIGKILL)
|
||||
|
||||
|
||||
def poll_until(predicate, timeout=10.0, interval=0.05):
|
||||
"""Poll ``predicate`` until truthy or ``timeout``; RETURNS the last value
|
||||
(falsy on timeout — assert at the call site). Deliberately named apart
|
||||
from ``tests/_helpers.wait_until``, which RAISES on timeout: two
|
||||
same-named helpers with opposite failure semantics invite silently-green
|
||||
tests."""
|
||||
deadline = time.monotonic() + timeout
|
||||
value = predicate()
|
||||
while not value and time.monotonic() < deadline:
|
||||
time.sleep(interval)
|
||||
value = predicate()
|
||||
return value
|
||||
@@ -0,0 +1,32 @@
|
||||
{
|
||||
"include": [
|
||||
"reasoning.encrypted_content"
|
||||
],
|
||||
"input": [
|
||||
{
|
||||
"content": "Hi there.",
|
||||
"role": "user",
|
||||
"type": "message"
|
||||
},
|
||||
{
|
||||
"content": "Hello! How can I help?",
|
||||
"role": "assistant",
|
||||
"type": "message"
|
||||
},
|
||||
{
|
||||
"content": "What's the weather in Paris?",
|
||||
"role": "user",
|
||||
"type": "message"
|
||||
}
|
||||
],
|
||||
"max_output_tokens": 4096,
|
||||
"model": "gpt-5.6-sol",
|
||||
"prompt_cache_options": {
|
||||
"ttl": "30m"
|
||||
},
|
||||
"reasoning": {
|
||||
"effort": "max"
|
||||
},
|
||||
"store": false,
|
||||
"stream": true
|
||||
}
|
||||
@@ -0,0 +1,57 @@
|
||||
{
|
||||
"include": [
|
||||
"reasoning.encrypted_content"
|
||||
],
|
||||
"input": [
|
||||
{
|
||||
"content": "Weather in Paris?",
|
||||
"role": "user",
|
||||
"type": "message"
|
||||
},
|
||||
{
|
||||
"arguments": "{\"city\": \"Paris\"}",
|
||||
"call_id": "call_1",
|
||||
"name": "get_weather",
|
||||
"type": "function_call"
|
||||
},
|
||||
{
|
||||
"call_id": "call_1",
|
||||
"output": "18C, clear.",
|
||||
"type": "function_call_output"
|
||||
},
|
||||
{
|
||||
"content": "It's 18C and clear in Paris.",
|
||||
"role": "assistant",
|
||||
"type": "message"
|
||||
}
|
||||
],
|
||||
"max_output_tokens": 4096,
|
||||
"model": "gpt-5.6-sol",
|
||||
"prompt_cache_options": {
|
||||
"ttl": "30m"
|
||||
},
|
||||
"reasoning": {
|
||||
"effort": "max"
|
||||
},
|
||||
"store": false,
|
||||
"stream": true,
|
||||
"tools": [
|
||||
{
|
||||
"description": "Look up the weather for a city.",
|
||||
"name": "get_weather",
|
||||
"parameters": {
|
||||
"properties": {
|
||||
"city": {
|
||||
"type": "string"
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"city"
|
||||
],
|
||||
"type": "object"
|
||||
},
|
||||
"strict": false,
|
||||
"type": "function"
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,36 @@
|
||||
{
|
||||
"include": [
|
||||
"reasoning.encrypted_content"
|
||||
],
|
||||
"input": [
|
||||
{
|
||||
"content": "Hi there.",
|
||||
"role": "user",
|
||||
"type": "message"
|
||||
},
|
||||
{
|
||||
"content": "Hello! How can I help?",
|
||||
"role": "assistant",
|
||||
"type": "message"
|
||||
},
|
||||
{
|
||||
"content": "What's the weather in Paris?",
|
||||
"role": "user",
|
||||
"type": "message"
|
||||
}
|
||||
],
|
||||
"max_output_tokens": 4096,
|
||||
"model": "gpt-5.6-sol",
|
||||
"prompt_cache_options": {
|
||||
"ttl": "30m"
|
||||
},
|
||||
"reasoning": {
|
||||
"effort": "high",
|
||||
"mode": "pro"
|
||||
},
|
||||
"store": false,
|
||||
"stream": true,
|
||||
"text": {
|
||||
"verbosity": "low"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,170 @@
|
||||
"""Tests for ``turnstone-admin create-admin`` (issue #824).
|
||||
|
||||
``create-user`` creates a role-less user; the web UI derives a login's scopes
|
||||
purely from assigned roles, so that account logs in read-only and hits
|
||||
"Forbidden: token lacks 'approve' scope" on any admin action. ``create-admin``
|
||||
assigns the built-in admin role — mirroring the web setup wizard
|
||||
(``POST /api/auth/setup``) — and promotes an existing role-less user, which is
|
||||
the recovery path for anyone already stuck.
|
||||
|
||||
Each test drives the real ``_cmd_create_admin`` handler against a real,
|
||||
fully-migrated SQLite DB: the ``builtin-admin`` role is seeded by migration
|
||||
008, so the DB must be migrated (not just ``create_all``-built) for the role
|
||||
to exist.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
from typing import TYPE_CHECKING, Any
|
||||
|
||||
import pytest
|
||||
|
||||
from turnstone.admin import _cmd_create_admin, _cmd_create_user
|
||||
from turnstone.core.auth import _load_user_permissions, _permissions_to_scopes
|
||||
from turnstone.core.storage import init_storage, reset_storage
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from collections.abc import Iterator
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _reset_storage_singleton() -> Iterator[None]:
|
||||
"""Keep the module-global storage singleton from leaking across tests."""
|
||||
reset_storage()
|
||||
yield
|
||||
reset_storage()
|
||||
|
||||
|
||||
def _db_args(db_path: str, **overrides: Any) -> argparse.Namespace:
|
||||
"""Build the Namespace ``_cmd_create_admin`` (and ``_cmd_create_user``) expect.
|
||||
|
||||
Pins every DB field so ``_get_storage`` resolves to the tmp sqlite file and
|
||||
never leaks a ``TURNSTONE_DB_*`` env var (it only falls back when the attr
|
||||
``is None``). ``token``/``scopes`` are only read by ``_cmd_create_user``.
|
||||
"""
|
||||
base: dict[str, Any] = {
|
||||
"username": "admin",
|
||||
"name": "",
|
||||
"password": "",
|
||||
"token": False,
|
||||
"scopes": "read,write,approve",
|
||||
"db_backend": "sqlite",
|
||||
"db_path": db_path,
|
||||
"db_url": "",
|
||||
"db_pool_size": 2,
|
||||
"db_sslmode": "",
|
||||
"db_sslrootcert": "",
|
||||
"db_sslcert": "",
|
||||
"db_sslkey": "",
|
||||
}
|
||||
base.update(overrides)
|
||||
return argparse.Namespace(**base)
|
||||
|
||||
|
||||
def _migrated_storage(db_path: str) -> Any:
|
||||
"""Return a fully-migrated storage singleton (seeds the ``builtin-admin`` role)."""
|
||||
return init_storage("sqlite", path=db_path, run_migrations=True)
|
||||
|
||||
|
||||
def _has_admin_role(storage: Any, user_id: str) -> bool:
|
||||
return any(r.get("role_id") == "builtin-admin" for r in storage.list_user_roles(user_id))
|
||||
|
||||
|
||||
def _login_scopes(storage: Any, user_id: str) -> frozenset[str]:
|
||||
"""Scopes a password login would grant this user — the real lockout surface."""
|
||||
return _permissions_to_scopes(_load_user_permissions(storage, user_id))
|
||||
|
||||
|
||||
def test_create_admin_fresh_user_gets_approve_scope(tmp_path: Path) -> None:
|
||||
db_path = str(tmp_path / "admin.db")
|
||||
storage = _migrated_storage(db_path)
|
||||
|
||||
_cmd_create_admin(_db_args(db_path, username="admin", name="Admin", password="hunter2!pw"))
|
||||
|
||||
user = storage.get_user_by_username("admin")
|
||||
assert user is not None
|
||||
assert _has_admin_role(storage, user["user_id"])
|
||||
# The exact bug surface: a web login for this account must carry `approve`.
|
||||
assert "approve" in _login_scopes(storage, user["user_id"])
|
||||
|
||||
|
||||
def test_create_admin_defaults_display_name_to_username(tmp_path: Path) -> None:
|
||||
db_path = str(tmp_path / "admin.db")
|
||||
storage = _migrated_storage(db_path)
|
||||
|
||||
_cmd_create_admin(_db_args(db_path, username="root", name="", password="hunter2!pw"))
|
||||
|
||||
user = storage.get_user_by_username("root")
|
||||
assert user is not None
|
||||
assert user["display_name"] == "root"
|
||||
|
||||
|
||||
def test_create_admin_promotes_existing_read_only_user(
|
||||
tmp_path: Path, capsys: pytest.CaptureFixture[str]
|
||||
) -> None:
|
||||
"""Issue #824 recovery path: a role-less create-user account, then create-admin."""
|
||||
db_path = str(tmp_path / "admin.db")
|
||||
storage = _migrated_storage(db_path)
|
||||
|
||||
# Reproduce the locked-out account exactly (role-less create-user).
|
||||
_cmd_create_user(_db_args(db_path, username="admin", name="Admin", password="hunter2!pw"))
|
||||
user = storage.get_user_by_username("admin")
|
||||
assert user is not None
|
||||
assert not _has_admin_role(storage, user["user_id"])
|
||||
assert "approve" not in _login_scopes(storage, user["user_id"]) # locked out
|
||||
|
||||
# Unstick without recreating the user.
|
||||
_cmd_create_admin(_db_args(db_path, username="admin"))
|
||||
|
||||
assert _has_admin_role(storage, user["user_id"])
|
||||
assert "approve" in _login_scopes(storage, user["user_id"])
|
||||
assert "Granted the admin role" in capsys.readouterr().out
|
||||
|
||||
|
||||
def test_create_admin_already_admin_is_idempotent(
|
||||
tmp_path: Path, capsys: pytest.CaptureFixture[str]
|
||||
) -> None:
|
||||
db_path = str(tmp_path / "admin.db")
|
||||
storage = _migrated_storage(db_path)
|
||||
|
||||
_cmd_create_admin(_db_args(db_path, username="admin", name="Admin", password="hunter2!pw"))
|
||||
capsys.readouterr() # drop first-run output
|
||||
|
||||
_cmd_create_admin(_db_args(db_path, username="admin"))
|
||||
|
||||
user = storage.get_user_by_username("admin")
|
||||
assert user is not None
|
||||
admin_rows = [
|
||||
r for r in storage.list_user_roles(user["user_id"]) if r.get("role_id") == "builtin-admin"
|
||||
]
|
||||
assert len(admin_rows) == 1 # not duplicated
|
||||
assert "already an admin" in capsys.readouterr().out
|
||||
|
||||
|
||||
def test_create_admin_short_password_rejected(
|
||||
tmp_path: Path, capsys: pytest.CaptureFixture[str]
|
||||
) -> None:
|
||||
db_path = str(tmp_path / "admin.db")
|
||||
storage = _migrated_storage(db_path)
|
||||
|
||||
with pytest.raises(SystemExit) as exc_info:
|
||||
_cmd_create_admin(_db_args(db_path, username="admin", name="Admin", password="short"))
|
||||
|
||||
assert exc_info.value.code == 1
|
||||
assert "at least 8" in capsys.readouterr().err
|
||||
assert storage.get_user_by_username("admin") is None # nothing created
|
||||
|
||||
|
||||
def test_create_admin_invalid_username_rejected(
|
||||
tmp_path: Path, capsys: pytest.CaptureFixture[str]
|
||||
) -> None:
|
||||
db_path = str(tmp_path / "admin.db")
|
||||
_migrated_storage(db_path)
|
||||
|
||||
with pytest.raises(SystemExit) as exc_info:
|
||||
_cmd_create_admin(_db_args(db_path, username="bad user!", name="X", password="hunter2!pw"))
|
||||
|
||||
assert exc_info.value.code == 1
|
||||
assert "invalid username" in capsys.readouterr().err
|
||||
@@ -677,6 +677,82 @@ def test_audio_roles_gated_to_openai_sdk_providers() -> None:
|
||||
assert '_providerCarriesAudio((md && md.provider) || "openai")' in body
|
||||
|
||||
|
||||
def test_model_response_controls_are_capability_driven_and_sparse() -> None:
|
||||
"""The model shelf surfaces Responses-only scalar controls without
|
||||
hard-coding GPT-5.6 IDs or pinning inherited capability-table values."""
|
||||
html = _CONSOLE_INDEX.read_text(encoding="utf-8")
|
||||
admin = _CONSOLE_ADMIN_JS.read_text(encoding="utf-8")
|
||||
|
||||
assert 'id="model-response-controls"' in html
|
||||
assert 'aria-labelledby="model-response-controls-title"' in html
|
||||
assert 'id="model-output-verbosity"' in html
|
||||
assert 'for="model-output-verbosity"' in html
|
||||
assert 'id="model-reasoning-mode"' in html
|
||||
assert 'for="model-reasoning-mode"' in html
|
||||
for value in ("low", "medium", "high"):
|
||||
assert f'<option value="{value}">' in html
|
||||
for value in ("standard", "pro"):
|
||||
assert f'<option value="{value}">' in html
|
||||
assert 'data-cap="supports_verbosity"' in html
|
||||
assert 'data-cap="supports_pro_mode"' in html
|
||||
|
||||
assert '"supports_verbosity"' in admin
|
||||
assert '"supports_pro_mode"' in admin
|
||||
surface = _slice_function_body(admin, "_modelUsesResponsesSurface")
|
||||
assert surface is not None
|
||||
assert 'provider === "openai"' in surface
|
||||
assert 'provider === "openai-compatible"' in surface
|
||||
assert 'value === "responses"' in surface
|
||||
visibility = _slice_function_body(admin, "_updateModelResponseControls")
|
||||
assert visibility is not None
|
||||
assert "_modelGetTile(spec.supportKey)" in visibility
|
||||
assert 'supportKey: "supports_verbosity"' in admin
|
||||
assert 'supportKey: "supports_pro_mode"' in admin
|
||||
assert "gpt-5.6" not in visibility, "visibility must come from capabilities, not model IDs"
|
||||
|
||||
assert "function _captureModelResponseControls(" in admin
|
||||
assert "function _mergeModelResponseControls(" in admin
|
||||
assert "_captureModelResponseControls(capsObj)" in admin
|
||||
assert "_mergeModelResponseControls(caps)" in admin
|
||||
assert "let _modelResponseCaptured = {};" in admin
|
||||
assert "let _modelResponseDirty = {};" in admin
|
||||
assert "_modelResponseCaptured[spec.key] = value" in admin
|
||||
assert "nextIdentity === _modelResponseInitialIdentity" in admin
|
||||
identity = _slice_function_body(admin, "_modelIdentity")
|
||||
assert identity is not None
|
||||
assert 'provider === "openai-compatible"' in identity
|
||||
assert ': ""' in identity
|
||||
merge = _slice_function_body(admin, "_mergeModelResponseControls")
|
||||
assert merge is not None
|
||||
# The dirty flag (select touched) may only override Advanced JSON for
|
||||
# the identity that made it dirty — a stale flag from a renamed row
|
||||
# must not delete a hand-typed JSON key.
|
||||
assert "if (_modelResponseDirty[spec.key] && sameIdentity) delete caps[spec.key]" in merge
|
||||
# The captured-value fallback is load-bearing, not a gating bug: a value
|
||||
# lifted out of the row JSON on edit-open must stay visible and re-save
|
||||
# for the same identity even when the baseline table says unsupported.
|
||||
# The baseline arrives async (or never, on the compat lane); yielding to
|
||||
# it would silently drop the pinned value on an unrelated edit-save.
|
||||
# Wire safety lives server-side (emission gates on merged supports_*).
|
||||
for body in (visibility, merge):
|
||||
assert "_modelGetTile(spec.supportKey) || capturedFallback" in body
|
||||
assert "sameIdentity" in body
|
||||
assert "!(spec.supportKey in _modelCapsExplicit)" in body
|
||||
|
||||
create = _slice_function_body(admin, "showCreateModelModal")
|
||||
assert create is not None
|
||||
assert "_modelCapsSeq++" in create, "a fresh shelf must invalidate prior lookups"
|
||||
|
||||
assert "displayCaps.supports_verbosity !== false" in admin
|
||||
assert "displayCaps.supports_pro_mode !== false" in admin
|
||||
|
||||
change = _slice_function_body(admin, "_onModelFieldChange")
|
||||
assert change is not None
|
||||
assert "_modelCapsSeq++" in change, "model changes must invalidate in-flight baselines"
|
||||
assert "_modelCapsBaseline = {}" in change
|
||||
assert 'apiSurfEl.addEventListener("change", _onModelFieldChange)' in admin
|
||||
|
||||
|
||||
def test_shared_utils_defines_set_markdown_helper() -> None:
|
||||
"""The ``setMarkdown`` helper in ``shared/utils.js`` is the single
|
||||
audited entry point for rendering markdown content into a DOM
|
||||
@@ -1328,6 +1404,32 @@ def test_redact_credentials_runtime_smoke() -> None:
|
||||
+ "const ak = redactCredentials('api_key=abcdefghijklmnopqrstuvwxyz');\n"
|
||||
+ "if (ak !== '[REDACTED:api_key]') "
|
||||
+ "throw new Error('api_key= clean redact failed: ' + ak);\n"
|
||||
+ "// Prefilter fast path: plain text with no anchor substring is unchanged\n"
|
||||
+ "const fp = redactCredentials('build ok in 42s - 3 tests passed');\n"
|
||||
+ "if (fp !== 'build ok in 42s - 3 tests passed') "
|
||||
+ "throw new Error('prefilter fast-path no-op failed: ' + fp);\n"
|
||||
+ "// Bare credentials with no =, quote or @ anywhere must still redact\n"
|
||||
+ "// (these pin the prefilter as a superset of the pattern set)\n"
|
||||
+ "const bk = redactCredentials('loaded sk-abcdefghijklmnopqrstuvwx');\n"
|
||||
+ "if (bk !== 'loaded [REDACTED:api_key]') "
|
||||
+ "throw new Error('bare sk- redact failed: ' + bk);\n"
|
||||
+ "const aw = redactCredentials('using AKIAABCDEFGHIJKLMNOP now');\n"
|
||||
+ "if (aw !== 'using [REDACTED:api_key] now') "
|
||||
+ "throw new Error('bare AKIA redact failed: ' + aw);\n"
|
||||
+ "const bt = redactCredentials('Bearer abcdefghijklmnopqrstuvwxyz');\n"
|
||||
+ "if (bt !== '[REDACTED:api_key]') "
|
||||
+ "throw new Error('bare bearer redact failed: ' + bt);\n"
|
||||
+ "// SQLAlchemy dialect+driver connection URLs (psycopg2/asyncpg)\n"
|
||||
+ "const pg2 = redactCredentials('postgresql+psycopg2://user:s3cret@db:5432/app');\n"
|
||||
+ "if (pg2 !== 'postgresql+psycopg2://user:[REDACTED:password]@db:5432/app') "
|
||||
+ "throw new Error('psycopg2 conn redact failed: ' + pg2);\n"
|
||||
+ "const apg = redactCredentials('postgresql+asyncpg://user:s3cret@db/app');\n"
|
||||
+ "if (apg !== 'postgresql+asyncpg://user:[REDACTED:password]@db/app') "
|
||||
+ "throw new Error('asyncpg conn redact failed: ' + apg);\n"
|
||||
+ "// RFC 3986 schemes are case-insensitive - uppercase must not bypass\n"
|
||||
+ "const up = redactCredentials('POSTGRESQL+PSYCOPG2://user:s3cret@db/app');\n"
|
||||
+ "if (up !== 'POSTGRESQL+PSYCOPG2://user:[REDACTED:password]@db/app') "
|
||||
+ "throw new Error('uppercase scheme conn redact failed: ' + up);\n"
|
||||
)
|
||||
with tempfile.NamedTemporaryFile(mode="w", suffix=".mjs", delete=False) as f:
|
||||
f.write(harness)
|
||||
@@ -1566,6 +1668,287 @@ def test_coord_connectsse_onerror_preserves_native_reconnect() -> None:
|
||||
assert passed, f"coordinator.js connectSSE.onerror regressed: {reason}"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Coordinator-pane parity for the SSE overflow-recovery companions (issue #806).
|
||||
# The server-side fixes (emit-time batching, _ListenerQueue poison, out-of-band
|
||||
# closing) live in SessionUIBase and already cover EVERY SSE stream; these pin
|
||||
# the CLIENT-side companions ported into coordinator.js so it stops relying on
|
||||
# native reconnect alone — storm guard + degraded catch-up, close-on-hide /
|
||||
# replay-on-show, and drop-vs-render-wedge counters.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_coord_imports_shared_overflow_helpers() -> None:
|
||||
"""coordinator.js consumes the SAME sse_overflow.js helpers as the
|
||||
interactive pane (over the /shared mount) so the trip threshold and cooldown
|
||||
ladder cannot drift between the two surfaces."""
|
||||
body = _COORD_JS.read_text(encoding="utf-8")
|
||||
m = re.search(
|
||||
r"import \{([^}]*)\} from \"/shared/sse_overflow\.js\";",
|
||||
body,
|
||||
re.S,
|
||||
)
|
||||
assert m is not None, "coordinator must import the shared overflow helpers"
|
||||
imported = m.group(1)
|
||||
for name in (
|
||||
"OVERFLOW_TRIP_COUNT",
|
||||
"OVERFLOW_TRIP_WINDOW_MS",
|
||||
"DEGRADED_COOLDOWN_BASE_MS",
|
||||
"DEGRADED_COOLDOWN_MAX_MS",
|
||||
"DEGRADED_COOLDOWN_RESET_MS",
|
||||
"overflowWindowTripped",
|
||||
"degradedCooldownStep",
|
||||
):
|
||||
assert name in imported, f"{name} must be imported from /shared/sse_overflow.js"
|
||||
# No local fork of the extracted pure functions on the coordinator side.
|
||||
assert not re.search(r"^\s*function overflowWindowTripped\(", body, re.M)
|
||||
assert not re.search(r"^\s*function degradedCooldownStep\(", body, re.M)
|
||||
|
||||
|
||||
def test_coord_stream_overflow_case_counts_and_rate_limits() -> None:
|
||||
"""The coordinator handles the id-less ``stream_overflow`` frame: count it
|
||||
(drop-vs-wedge field instrumentation) and feed the rolling-window storm
|
||||
guard, exactly like the interactive pane."""
|
||||
body = _COORD_JS.read_text(encoding="utf-8")
|
||||
assert 'case "stream_overflow":' in body
|
||||
assert "noteStreamOverflow();" in body
|
||||
# The three-way health counter distinguishes dropped events (overflow /
|
||||
# malformed frame) from render wedges (dispatch / render throw).
|
||||
assert "streamHealth = { overflows: 0, renderThrows: 0, malformedFrames: 0 }" in body
|
||||
assert "streamHealth.overflows += 1;" in body
|
||||
assert "streamHealth.malformedFrames += 1;" in body
|
||||
# Exactly two render-throw increment sites: the noteRenderThrow helper
|
||||
# (all three contained render/finalize catches route through it — they
|
||||
# recover with a plain-text fallback, so console.warn) and the onmessage
|
||||
# dispatch catch (console.error class — the event is dropped outright).
|
||||
# The three recovered call sites are pinned by label so a new render path
|
||||
# that forgets to count surfaces loudly.
|
||||
assert body.count("streamHealth.renderThrows += 1;") == 2
|
||||
helper = re.search(r"function noteRenderThrow\(where, err\)\s*\{(.*?)\n \}", body, re.S)
|
||||
assert helper is not None, "noteRenderThrow helper not found"
|
||||
assert "streamHealth.renderThrows += 1;" in helper.group(1)
|
||||
assert 'noteRenderThrow("streamingRender", e);' in body
|
||||
assert 'noteRenderThrow("in_progress_snapshot render", e);' in body
|
||||
assert 'noteRenderThrow("streamingRenderFinalize", e);' in body
|
||||
note = re.search(r"function noteStreamOverflow\(\)\s*\{(.*?)\n \}", body, re.S)
|
||||
assert note is not None, "noteStreamOverflow not found"
|
||||
assert "overflowWindowTripped(" in note.group(1)
|
||||
assert "enterDegradedCatchup()" in note.group(1)
|
||||
# The trip handler only counts + trips; the cooldown reset lives in
|
||||
# enterDegradedCatchup (keyed off lastDegradedAt) — the finding [0] shape.
|
||||
assert "degradedCooldownMs" not in note.group(1), (
|
||||
"noteStreamOverflow must not touch the cooldown — that reset defeated the ladder escalation"
|
||||
)
|
||||
|
||||
|
||||
def test_coord_handleevent_dispatch_is_wedge_guarded() -> None:
|
||||
"""A throw escaping onmessage does NOT close the EventSource, so an
|
||||
unguarded handler throw left the streaming refs stale and wedged every later
|
||||
turn. The coordinator wraps the dispatch and counts the throw (render-wedge
|
||||
class) so a field report tells it apart from a dropped-events gap."""
|
||||
body = _COORD_JS.read_text(encoding="utf-8")
|
||||
m = re.search(r"try \{\s*handleEvent\(data\);\s*\} catch \(err\) \{(.*?)\}", body, re.S)
|
||||
assert m is not None, "handleEvent(data) must be wrapped in try/catch in onmessage"
|
||||
assert "streamHealth.renderThrows += 1;" in m.group(1)
|
||||
|
||||
|
||||
def test_coord_degraded_catchup_stops_live_stream_and_retries() -> None:
|
||||
"""Three overflow closes inside the window drop the coordinator to a
|
||||
degraded catch-up: suspend the live stream, say so plainly, and reconnect
|
||||
after a doubling cooldown — the reconnect replays the gap (or falls to the
|
||||
/history floor once it outgrows the ring)."""
|
||||
body = _COORD_JS.read_text(encoding="utf-8")
|
||||
m = re.search(r"function enterDegradedCatchup\(\)\s*\{(.*?)\n \}", body, re.S)
|
||||
assert m is not None, "enterDegradedCatchup not found"
|
||||
method = m.group(1)
|
||||
assert "degradedCooldownStep(" in method
|
||||
assert "lastDegradedAt = now" in method
|
||||
# Suspend the stream BEFORE arming the retry timer (mirrors interactive's
|
||||
# disconnect-then-rearm ordering) or the fresh timer is cancelled at once.
|
||||
assert method.index("suspendStream()") < method.index("degradedTimer = setTimeout")
|
||||
# Plain-language status, not a silent stall.
|
||||
assert "catching up" in method
|
||||
# A fresh connect must cancel a pending degraded timer so it can't
|
||||
# double-open behind the retry — connectSSE's prologue routes through the
|
||||
# shared closeStreamTransport teardown, which owns that clear (alongside
|
||||
# the reconnect timer + the EventSource close/null).
|
||||
conn = re.search(r"function connectSSE\(\)\s*\{(.*?)\n \}", body, re.S)
|
||||
assert conn is not None
|
||||
assert "closeStreamTransport();" in conn.group(1)
|
||||
teardown = re.search(r"function closeStreamTransport\(\)\s*\{(.*?)\n \}", body, re.S)
|
||||
assert teardown is not None, "closeStreamTransport not found"
|
||||
assert "clearTimeout(degradedTimer)" in teardown.group(1)
|
||||
assert "clearTimeout(reconnectTimer)" in teardown.group(1)
|
||||
assert "evtSource = null;" in teardown.group(1)
|
||||
|
||||
|
||||
def test_coord_visibilitychange_closes_on_hide_reconnects_on_show() -> None:
|
||||
"""A hidden tab's throttled drain is the worst-case slow SSE consumer. The
|
||||
coordinator installs a visibilitychange handler that closes the stream on
|
||||
hide (marking its OWN close via hiddenDisconnect) and reconnects on show from
|
||||
the saved lastEventId, and removes the listener on teardown."""
|
||||
body = _COORD_JS.read_text(encoding="utf-8")
|
||||
assert 'document.addEventListener("visibilitychange", visHandler);' in body
|
||||
assert 'document.removeEventListener("visibilitychange", visHandler);' in body
|
||||
vis = re.search(r"function onVisibilityChange\(\)\s*\{(.*?)\n \}", body, re.S)
|
||||
assert vis is not None, "onVisibilityChange not found"
|
||||
method = vis.group(1)
|
||||
assert "document.hidden" in method
|
||||
assert "suspendStream()" in method
|
||||
assert "hiddenDisconnect = true;" in method
|
||||
assert "else if (hiddenDisconnect)" in method
|
||||
assert "connectSSE();" in method
|
||||
|
||||
|
||||
def test_coord_connectsse_defers_open_when_tab_hidden() -> None:
|
||||
"""connectSSE must never open an EventSource into a hidden tab — the single
|
||||
chokepoint that also backstops a FIRST connect in a background tab (where the
|
||||
close-on-hide handler never fires because there was no open stream). It
|
||||
marks hiddenDisconnect so the show edge owns the reconnect, marks the
|
||||
deferral as a GAP (markStreamGap) so the eventual open runs the post-gap
|
||||
recovery — without the mark a pane first opened in a background tab
|
||||
silently missed every child/task created while hidden — and reports an
|
||||
honest paused status instead of pinning "connecting" with no attempt in
|
||||
flight."""
|
||||
body = _COORD_JS.read_text(encoding="utf-8")
|
||||
conn = re.search(r"function connectSSE\(\)\s*\{(.*?)\n \}", body, re.S)
|
||||
assert conn is not None
|
||||
method = conn.group(1)
|
||||
guard = method.index("if (document.hidden)")
|
||||
open_idx = method.index("new EventSource(")
|
||||
assert guard < open_idx, "the hidden guard must precede new EventSource"
|
||||
head = method[guard:open_idx]
|
||||
assert "markStreamGap();" in head, "the hidden deferral must count as a stream gap"
|
||||
assert "hiddenDisconnect = true;" in head
|
||||
assert "return;" in head
|
||||
assert 'setSseStatus("paused' in head, "the deferral must report paused, not connecting"
|
||||
# "connecting" is claimed only once an attempt actually starts — after
|
||||
# the hidden guard, immediately before the EventSource construction.
|
||||
connecting = method.index('setSseStatus("connecting')
|
||||
assert guard < connecting < open_idx
|
||||
|
||||
|
||||
def test_coord_destroy_removes_visibility_handler_and_stream_transport() -> None:
|
||||
"""Teardown must detach the document-level visibilitychange listener (it
|
||||
holds a strong ref to the closure) and tear down the stream transport —
|
||||
closeStreamTransport closes the EventSource and cancels the reconnect +
|
||||
degraded retry timers (pinned in the degraded-catchup test) — or a
|
||||
destroyed pane leaks and a show edge / pending retry reopens its stream."""
|
||||
body = _COORD_JS.read_text(encoding="utf-8")
|
||||
d = re.search(r"function destroy\(\)\s*\{(.*?)\n \}", body, re.S)
|
||||
assert d is not None, "destroy not found"
|
||||
method = d.group(1)
|
||||
assert "removeVisibilityHandler();" in method
|
||||
assert "closeStreamTransport();" in method
|
||||
|
||||
|
||||
def test_coord_close_session_detaches_visibility_reopen() -> None:
|
||||
"""coordCloseSession suspends the stream AND removes the visibilitychange
|
||||
handler BEFORE awaiting the /close POST: a tab hide→show while the POST is
|
||||
in flight must not reopen a stream against the workstream the server is
|
||||
tearing down (404 / reconnect churn against a dead session). The failure
|
||||
paths resume via connectSSE, which reinstalls the handler at its
|
||||
install-once chokepoint — so close-on-hide survives a failed close."""
|
||||
body = _COORD_JS.read_text(encoding="utf-8")
|
||||
m = re.search(r"async function coordCloseSession\(\)\s*\{(.*?)\n \}", body, re.S)
|
||||
assert m is not None, "coordCloseSession not found"
|
||||
method = m.group(1)
|
||||
suspend = method.index("suspendStream();")
|
||||
unhook = method.index("removeVisibilityHandler();")
|
||||
# The quoted URL fragment, not the bare word (comments mention /close too).
|
||||
post = method.index('"/close"')
|
||||
assert suspend < post, "stream suspension must precede the /close POST"
|
||||
assert unhook < post, "visibility detach must precede the /close POST"
|
||||
assert "resumeSse()" in method
|
||||
|
||||
|
||||
def test_coord_post_gap_sidebar_refresh_is_replay_aware() -> None:
|
||||
"""The replace-mode children/tasks refresh (a sidebar rebuild) must NOT
|
||||
fire on every reconnect: child_ws_* / task-mutating events are ordinary
|
||||
ring-buffer entries, so a cursor reconnect (replay_ok) redelivers them and
|
||||
the sidebar heals through the normal handlers — a momentary blur→focus
|
||||
under close-on-hide must not rebuild the sidebar. The refresh fires
|
||||
exactly when the replay cannot vouch for the gap: no resume cursor or an
|
||||
over-threshold gap at onopen, or the server's replay_truncated envelope
|
||||
(ring evicted), deduped per open via gapRefreshedAtOpen."""
|
||||
body = _COORD_JS.read_text(encoding="utf-8")
|
||||
conn = re.search(r"function connectSSE\(\)\s*\{(.*?)\n \}", body, re.S)
|
||||
assert conn is not None
|
||||
method = conn.group(1)
|
||||
gate = re.search(
|
||||
r"wasReconnecting &&\s*\(lastEventId == null \|\| gapMs > GAP_REFRESH_THRESHOLD_MS\)",
|
||||
method,
|
||||
)
|
||||
assert gate is not None, "onopen must gate the sidebar refresh on replay coverage"
|
||||
assert "refreshSidebarAfterGap();" in method
|
||||
assert "gapRefreshedAtOpen = true;" in method
|
||||
# The ring-evicted signal triggers the same refresh (deduped per open).
|
||||
trunc = re.search(r'case "replay_truncated":(.*?)break;', body, re.S)
|
||||
assert trunc is not None, "replay_truncated case not found"
|
||||
assert "refreshSidebarAfterGap()" in trunc.group(1)
|
||||
assert "gapRefreshedAtOpen" in trunc.group(1)
|
||||
# Deliberate suspends (hide / overflow / close-session) mark the gap so
|
||||
# the next open participates in the recovery decision at all.
|
||||
sus = re.search(r"function suspendStream\(\)\s*\{(.*?)\n \}", body, re.S)
|
||||
assert sus is not None, "suspendStream not found"
|
||||
assert "markStreamGap();" in sus.group(1)
|
||||
# The refresh helper carries the whole replace-mode bundle: children,
|
||||
# tasks, and the live-badge purge (permanent 403/404 entries preserved).
|
||||
ref = re.search(r"function refreshSidebarAfterGap\(\)\s*\{(.*?)\n \}", body, re.S)
|
||||
assert ref is not None, "refreshSidebarAfterGap not found"
|
||||
assert "loadChildren({ replace: true });" in ref.group(1)
|
||||
assert "loadTasks();" in ref.group(1)
|
||||
assert "_liveBadgeCacheDelete(id)" in ref.group(1)
|
||||
|
||||
|
||||
def test_coord_defers_truncated_resync_and_consumes_at_idle() -> None:
|
||||
"""replay_truncated seen mid-stream must be DEFERRED, not dropped (matches
|
||||
interactive's _pendingTruncatedResync): refetching immediately would detach
|
||||
the live bubble (content OR a reasoning-only one), but skipping outright
|
||||
leaves the ring-evicted turns lost for the session. The guard covers both
|
||||
streaming targets and latches otherwise; the next state_change=idle consumes
|
||||
the flag — which also repairs a turn stranded by close-on-hide (stream_end
|
||||
evicted while hidden), resetting the streaming refs first since
|
||||
refetchHistory does not null them."""
|
||||
body = _COORD_JS.read_text(encoding="utf-8")
|
||||
trunc = re.search(r'case "replay_truncated":(.*?)break;', body, re.S)
|
||||
assert trunc is not None, "replay_truncated case not found"
|
||||
t = trunc.group(1)
|
||||
assert "if (!currentAssistantEl && !currentReasoningEl)" in t
|
||||
assert "refetchHistory();" in t
|
||||
assert "pendingTruncatedResync = true;" in t
|
||||
st = re.search(r'case "state_change":(.*?)\n case ', body, re.S)
|
||||
assert st is not None, "state_change case not found"
|
||||
s = st.group(1)
|
||||
assert "if (pendingTruncatedResync)" in s
|
||||
assert "pendingTruncatedResync = false;" in s
|
||||
assert "currentAssistantEl = null;" in s
|
||||
assert "refetchHistory();" in s
|
||||
# Consume the latch, THEN reset the dangling refs and refetch.
|
||||
consume = s.index("pendingTruncatedResync = false;")
|
||||
refetch = s.index("refetchHistory();")
|
||||
assert consume < refetch
|
||||
|
||||
|
||||
def test_coord_detects_server_restart_by_backwards_event_id() -> None:
|
||||
"""A coordinator process restart resets the per-ws event counter, and the
|
||||
replay path reports replay_ok for a stale-high cursor (past the new max), so
|
||||
the gap is unsignalled and the sidebar goes stale. onmessage catches it: a
|
||||
live event id below the saved cursor == the counter reset → pull
|
||||
authoritative sidebar state (deduped per open against onopen's refresh),
|
||||
checked BEFORE the cursor is overwritten."""
|
||||
body = _COORD_JS.read_text(encoding="utf-8")
|
||||
m = re.search(r"evtSource\.onmessage = function \(event\) \{(.*?)\n \};", body, re.S)
|
||||
assert m is not None, "onmessage handler not found"
|
||||
handler = m.group(1)
|
||||
assert "Number(evtSource.lastEventId) < Number(lastEventId)" in handler
|
||||
assert "!gapRefreshedAtOpen" in handler
|
||||
assert "refreshSidebarAfterGap();" in handler
|
||||
check = handler.index("Number(evtSource.lastEventId) < Number(lastEventId)")
|
||||
overwrite = handler.index("lastEventId = evtSource.lastEventId;")
|
||||
assert check < overwrite
|
||||
|
||||
|
||||
def test_interactive_history_is_rest_first_not_sse() -> None:
|
||||
"""PR A converged interactive onto coord's REST-first history
|
||||
model: first paint and post-rewind re-render fetch ``GET /history``
|
||||
|
||||
@@ -0,0 +1,670 @@
|
||||
"""Unit tests for the per-session background-shell registry (#817).
|
||||
|
||||
The registry backs the ``bash(run_in_background=true)`` / ``bash_output`` /
|
||||
``kill_shell`` tool surface: it spawns detached shells (``bash_N`` handles),
|
||||
buffers their merged output in a capped rolling buffer, serves delta reads
|
||||
(only lines since the last read), and reaps whole session groups on kill /
|
||||
owner reap / close — the #816 rule (the tracked command defines the lifetime,
|
||||
nothing escapes its process group) extended to explicit backgrounding.
|
||||
|
||||
Pure registry tests — no ChatSession. Session wiring is covered in
|
||||
``test_bash_background_tool.py``.
|
||||
"""
|
||||
|
||||
import re
|
||||
import threading
|
||||
import time
|
||||
|
||||
import pytest
|
||||
|
||||
from tests._proc_helpers import kill_pid as _kill_pid
|
||||
from tests._proc_helpers import pid_alive as _pid_alive
|
||||
from tests._proc_helpers import poll_until as _wait_until
|
||||
|
||||
# Module alias (from-style, matching the symbol imports below) for tests
|
||||
# that monkeypatch module attributes (os.killpg, subprocess.Popen, ...).
|
||||
from turnstone.core import background_shells as bg_mod
|
||||
from turnstone.core.background_shells import (
|
||||
BackgroundShellRegistry,
|
||||
FilterExecError,
|
||||
FilterTimeoutError,
|
||||
TooManyShellsError,
|
||||
UnknownShellError,
|
||||
)
|
||||
|
||||
|
||||
def _wait_status(shell, status, timeout=10.0):
|
||||
return _wait_until(lambda: shell.status == status, timeout=timeout)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def registry():
|
||||
reg = BackgroundShellRegistry()
|
||||
yield reg
|
||||
reg.close()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Handles + spawning
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_spawn_returns_incrementing_bash_handles(registry):
|
||||
s1 = registry.spawn("sleep 30")
|
||||
s2 = registry.spawn("sleep 30")
|
||||
assert s1.shell_id == "bash_1"
|
||||
assert s2.shell_id == "bash_2"
|
||||
|
||||
|
||||
def test_spawned_shell_is_running_with_live_pid(registry):
|
||||
shell = registry.spawn("sleep 30")
|
||||
assert shell.status == "running"
|
||||
assert _pid_alive(shell.pid)
|
||||
|
||||
|
||||
def test_spawn_records_command(registry):
|
||||
shell = registry.spawn("sleep 30")
|
||||
assert shell.command == "sleep 30"
|
||||
|
||||
|
||||
def test_spawn_after_close_is_refused():
|
||||
reg = BackgroundShellRegistry()
|
||||
reg.close()
|
||||
with pytest.raises(RuntimeError):
|
||||
reg.spawn("echo hi")
|
||||
|
||||
|
||||
def test_max_live_shells_cap():
|
||||
reg = BackgroundShellRegistry(max_shells=2)
|
||||
try:
|
||||
reg.spawn("sleep 30")
|
||||
s2 = reg.spawn("sleep 30")
|
||||
with pytest.raises(TooManyShellsError):
|
||||
reg.spawn("sleep 30")
|
||||
# Cap counts LIVE shells: killing one frees a slot.
|
||||
reg.kill(s2.shell_id)
|
||||
s3 = reg.spawn("sleep 30")
|
||||
assert s3.status == "running"
|
||||
finally:
|
||||
reg.close()
|
||||
|
||||
|
||||
def test_completed_shells_do_not_count_toward_cap():
|
||||
reg = BackgroundShellRegistry(max_shells=1)
|
||||
try:
|
||||
s1 = reg.spawn("true")
|
||||
assert _wait_status(s1, "completed")
|
||||
s2 = reg.spawn("sleep 30")
|
||||
assert s2.status == "running"
|
||||
finally:
|
||||
reg.close()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Exit tracking
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_natural_exit_sets_completed_and_exit_code(registry):
|
||||
shell = registry.spawn("exit 7")
|
||||
assert _wait_status(shell, "completed")
|
||||
assert shell.exit_code == 7
|
||||
|
||||
|
||||
def test_output_is_complete_once_completed(registry):
|
||||
"""Status flips to completed only after the drains finish: a read at
|
||||
completed must see everything the command wrote."""
|
||||
shell = registry.spawn("echo alpha; echo beta")
|
||||
assert _wait_status(shell, "completed")
|
||||
read = registry.read(shell.shell_id)
|
||||
assert [ln.strip() for ln in read.lines] == ["alpha", "beta"]
|
||||
|
||||
|
||||
def test_leader_exit_reaps_backgrounded_grandchild(registry, tmp_path):
|
||||
"""#816 consistency: the tracked command defines the lifetime. When the
|
||||
leader exits, the whole session group is killed — a child the command
|
||||
backgrounded does not outlive it."""
|
||||
pidfile = tmp_path / "bg.pid"
|
||||
shell = registry.spawn(f"sleep 60 & echo $! > {pidfile}; echo done")
|
||||
bg_pid = None
|
||||
try:
|
||||
assert _wait_status(shell, "completed")
|
||||
bg_pid = int(pidfile.read_text().strip())
|
||||
assert _wait_until(lambda: not _pid_alive(bg_pid)), (
|
||||
f"grandchild {bg_pid} leaked past leader exit"
|
||||
)
|
||||
read = registry.read(shell.shell_id)
|
||||
assert "done" in "".join(read.lines)
|
||||
finally:
|
||||
if bg_pid is not None:
|
||||
_kill_pid(bg_pid)
|
||||
|
||||
|
||||
def test_stderr_lines_are_tagged_inline(registry):
|
||||
shell = registry.spawn("echo out; echo err >&2")
|
||||
assert _wait_status(shell, "completed")
|
||||
lines = [ln.strip() for ln in registry.read(shell.shell_id).lines]
|
||||
assert "out" in lines
|
||||
assert "[stderr] err" in lines
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Delta reads
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_read_returns_only_new_lines_since_last_read(registry):
|
||||
"""The load-bearing convention: consecutive reads never overlap and never
|
||||
drop a line — collecting across polls yields each line exactly once."""
|
||||
shell = registry.spawn("echo one; echo two; sleep 0.4; echo three; sleep 30")
|
||||
collected: list[str] = []
|
||||
|
||||
def _collect():
|
||||
collected.extend(ln.strip() for ln in registry.read(shell.shell_id).lines)
|
||||
return "three" in collected
|
||||
|
||||
assert _wait_until(_collect)
|
||||
assert collected == ["one", "two", "three"]
|
||||
registry.kill(shell.shell_id)
|
||||
|
||||
|
||||
def test_read_after_exit_then_again_reports_no_new_output(registry):
|
||||
shell = registry.spawn("echo hi")
|
||||
assert _wait_status(shell, "completed")
|
||||
first = registry.read(shell.shell_id)
|
||||
assert [ln.strip() for ln in first.lines] == ["hi"]
|
||||
second = registry.read(shell.shell_id)
|
||||
assert second.lines == []
|
||||
assert second.status == "completed"
|
||||
assert second.exit_code == 0
|
||||
|
||||
|
||||
def test_read_reports_status_and_exit_code(registry):
|
||||
shell = registry.spawn("sleep 30")
|
||||
read = registry.read(shell.shell_id)
|
||||
assert read.shell_id == shell.shell_id
|
||||
assert read.status == "running"
|
||||
assert read.exit_code is None
|
||||
registry.kill(shell.shell_id)
|
||||
|
||||
|
||||
def test_read_unknown_id_raises_with_live_ids(registry):
|
||||
registry.spawn("sleep 30")
|
||||
with pytest.raises(UnknownShellError) as excinfo:
|
||||
registry.read("bash_99")
|
||||
assert "bash_99" in str(excinfo.value)
|
||||
assert "bash_1" in str(excinfo.value)
|
||||
|
||||
|
||||
def test_read_unknown_id_when_registry_empty(registry):
|
||||
with pytest.raises(UnknownShellError):
|
||||
registry.read("bash_1")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Filter
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_filter_selects_matching_lines_only(registry):
|
||||
shell = registry.spawn("echo match-a; echo skip-b; echo match-c")
|
||||
assert _wait_status(shell, "completed")
|
||||
read = registry.read(shell.shell_id, filter_pattern="^match")
|
||||
assert [ln.strip() for ln in read.lines] == ["match-a", "match-c"]
|
||||
|
||||
|
||||
def test_filter_is_display_only_and_consumes_the_delta(registry):
|
||||
"""Filtered-out lines are consumed, not deferred — the cursor advances
|
||||
past the whole delta (Claude Code ``BashOutput`` semantics)."""
|
||||
shell = registry.spawn("echo match-a; echo skip-b")
|
||||
assert _wait_status(shell, "completed")
|
||||
first = registry.read(shell.shell_id, filter_pattern="^match")
|
||||
assert [ln.strip() for ln in first.lines] == ["match-a"]
|
||||
assert first.new_line_count == 2 # both lines were new, one shown
|
||||
second = registry.read(shell.shell_id)
|
||||
assert second.lines == []
|
||||
assert second.new_line_count == 0
|
||||
|
||||
|
||||
def test_filter_uses_search_not_match(registry):
|
||||
shell = registry.spawn("echo prefix-needle-suffix")
|
||||
assert _wait_status(shell, "completed")
|
||||
read = registry.read(shell.shell_id, filter_pattern="needle")
|
||||
assert len(read.lines) == 1
|
||||
|
||||
|
||||
def test_invalid_filter_regex_raises(registry):
|
||||
shell = registry.spawn("echo hi")
|
||||
assert _wait_status(shell, "completed")
|
||||
with pytest.raises(re.error):
|
||||
registry.read(shell.shell_id, filter_pattern="[unclosed")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Buffer cap
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_buffer_cap_drops_oldest_and_reports_gap():
|
||||
reg = BackgroundShellRegistry(max_buffer_chars=200)
|
||||
try:
|
||||
shell = reg.spawn('for i in $(seq 1 50); do echo "line-$i-padded-to-length"; done')
|
||||
assert _wait_status(shell, "completed")
|
||||
read = reg.read(shell.shell_id)
|
||||
assert read.dropped_lines > 0
|
||||
# Newest output survives; the tail is intact.
|
||||
assert read.lines, "cap must retain the newest lines, not drop everything"
|
||||
assert read.lines[-1].strip() == "line-50-padded-to-length"
|
||||
finally:
|
||||
reg.close()
|
||||
|
||||
|
||||
def test_unread_lines_excludes_buffer_evicted():
|
||||
"""The exit notice's line count must not promise evicted output."""
|
||||
reg = BackgroundShellRegistry(max_buffer_chars=200)
|
||||
try:
|
||||
shell = reg.spawn('for i in $(seq 1 50); do echo "line-$i-padded-to-length"; done')
|
||||
assert _wait_status(shell, "completed")
|
||||
with shell.lock:
|
||||
retained = len(shell._buffer)
|
||||
assert shell.unread_lines == retained
|
||||
finally:
|
||||
reg.close()
|
||||
|
||||
|
||||
def test_buffer_gap_is_relative_to_cursor():
|
||||
"""Lines dropped BEFORE being read are a reported gap; lines already
|
||||
read and then dropped are not."""
|
||||
reg = BackgroundShellRegistry(max_buffer_chars=10_000)
|
||||
try:
|
||||
shell = reg.spawn("echo early; sleep 30")
|
||||
# Each poll consumes whatever has arrived; stop once something did.
|
||||
assert _wait_until(lambda: bool(reg.read(shell.shell_id).lines))
|
||||
# Everything emitted so far is read; nothing has been dropped.
|
||||
read = reg.read(shell.shell_id)
|
||||
assert read.dropped_lines == 0
|
||||
reg.kill(shell.shell_id)
|
||||
finally:
|
||||
reg.close()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Kill / reap / close
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_kill_marks_killed_and_reaps_group(registry, tmp_path):
|
||||
pidfile = tmp_path / "bg.pid"
|
||||
shell = registry.spawn(f"sleep 60 & echo $! > {pidfile}; sleep 60")
|
||||
assert _wait_until(pidfile.exists)
|
||||
bg_pid = int(pidfile.read_text().strip())
|
||||
try:
|
||||
killed = registry.kill(shell.shell_id)
|
||||
assert killed.status == "killed"
|
||||
assert _wait_until(lambda: not _pid_alive(shell.pid))
|
||||
assert _wait_until(lambda: not _pid_alive(bg_pid)), "grandchild survived kill"
|
||||
finally:
|
||||
_kill_pid(bg_pid)
|
||||
|
||||
|
||||
def test_kill_unknown_id_raises(registry):
|
||||
with pytest.raises(UnknownShellError):
|
||||
registry.kill("bash_7")
|
||||
|
||||
|
||||
def test_killed_shell_output_remains_readable(registry, tmp_path):
|
||||
"""Output that arrived before the kill survives it: the record keeps its
|
||||
buffer, and ``kill`` returns only after the drains have flushed."""
|
||||
sentinel = tmp_path / "started"
|
||||
shell = registry.spawn(f"echo before-kill; touch {sentinel}; sleep 60")
|
||||
assert _wait_until(sentinel.exists)
|
||||
registry.kill(shell.shell_id)
|
||||
read = registry.read(shell.shell_id)
|
||||
assert read.status == "killed"
|
||||
assert "before-kill" in "".join(read.lines)
|
||||
|
||||
|
||||
def test_signal_all_kills_live_shells_without_closing(registry):
|
||||
"""signal_all is the instant half of teardown: every live group dies,
|
||||
but the registry stays open (records intact, spawns still allowed) —
|
||||
close() remains the complete teardown."""
|
||||
s1 = registry.spawn("sleep 60")
|
||||
s2 = registry.spawn("sleep 60")
|
||||
registry.signal_all()
|
||||
assert _wait_until(lambda: not _pid_alive(s1.pid))
|
||||
assert _wait_until(lambda: not _pid_alive(s2.pid))
|
||||
assert registry.has(s1.shell_id), "signal_all must not drop records"
|
||||
s3 = registry.spawn("true")
|
||||
assert _wait_status(s3, "completed"), "registry must remain usable after signal_all"
|
||||
|
||||
|
||||
def test_close_kills_everything_and_is_idempotent():
|
||||
reg = BackgroundShellRegistry()
|
||||
s1 = reg.spawn("sleep 60")
|
||||
s2 = reg.spawn("sleep 60")
|
||||
reg.close()
|
||||
assert not _pid_alive(s1.pid)
|
||||
assert not _pid_alive(s2.pid)
|
||||
reg.close() # second close is a no-op
|
||||
|
||||
|
||||
def test_reap_owner_kills_only_that_owners_shells(registry):
|
||||
mine = registry.spawn("sleep 60", owner="agent-1")
|
||||
other = registry.spawn("sleep 60", owner="agent-2")
|
||||
main = registry.spawn("sleep 60")
|
||||
registry.reap(owner="agent-1")
|
||||
assert _wait_until(lambda: not _pid_alive(mine.pid))
|
||||
assert _pid_alive(other.pid)
|
||||
assert _pid_alive(main.pid)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Owner scoping
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_owner_scoped_lookup_isolates_shells(registry):
|
||||
agent_shell = registry.spawn("sleep 30", owner="agent-1")
|
||||
main_shell = registry.spawn("sleep 30")
|
||||
# Main scope cannot see the agent's shell...
|
||||
with pytest.raises(UnknownShellError):
|
||||
registry.read(agent_shell.shell_id)
|
||||
# ...and the agent scope cannot see the main shell.
|
||||
with pytest.raises(UnknownShellError):
|
||||
registry.read(main_shell.shell_id, owner="agent-1")
|
||||
# Each side reads its own.
|
||||
assert registry.read(agent_shell.shell_id, owner="agent-1").status == "running"
|
||||
assert registry.read(main_shell.shell_id).status == "running"
|
||||
|
||||
|
||||
def test_shells_snapshot_is_owner_scoped(registry):
|
||||
registry.spawn("sleep 30", owner="agent-1")
|
||||
registry.spawn("sleep 30")
|
||||
assert [s.owner for s in registry.shells(owner="agent-1")] == ["agent-1"]
|
||||
assert [s.owner for s in registry.shells()] == [None]
|
||||
|
||||
|
||||
def test_handles_are_unique_across_owners(registry):
|
||||
a = registry.spawn("sleep 30", owner="agent-1")
|
||||
b = registry.spawn("sleep 30")
|
||||
assert a.shell_id != b.shell_id
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Exit callback (the notice hook)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_on_exit_fires_once_on_natural_exit():
|
||||
fired = threading.Event()
|
||||
seen = []
|
||||
|
||||
def _on_exit(shell):
|
||||
seen.append(shell)
|
||||
fired.set()
|
||||
|
||||
reg = BackgroundShellRegistry(on_exit=_on_exit)
|
||||
try:
|
||||
shell = reg.spawn("echo done")
|
||||
assert fired.wait(10)
|
||||
assert len(seen) == 1
|
||||
assert seen[0].shell_id == shell.shell_id
|
||||
assert seen[0].exit_code == 0
|
||||
finally:
|
||||
reg.close()
|
||||
|
||||
|
||||
def test_on_exit_not_fired_for_kill():
|
||||
seen = []
|
||||
reg = BackgroundShellRegistry(on_exit=seen.append)
|
||||
try:
|
||||
shell = reg.spawn("sleep 60")
|
||||
reg.kill(shell.shell_id)
|
||||
assert _wait_until(lambda: not _pid_alive(shell.pid))
|
||||
time.sleep(0.2) # give a buggy late callback a chance to land
|
||||
assert seen == []
|
||||
finally:
|
||||
reg.close()
|
||||
|
||||
|
||||
def test_on_exit_not_fired_for_close():
|
||||
seen = []
|
||||
reg = BackgroundShellRegistry(on_exit=seen.append)
|
||||
shell = reg.spawn("sleep 60")
|
||||
reg.close()
|
||||
assert not _pid_alive(shell.pid)
|
||||
time.sleep(0.2)
|
||||
assert seen == []
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Review-hardening regressions (#817 code review)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_kill_on_completed_shell_does_not_signal_group(registry, monkeypatch):
|
||||
"""A completed shell's pgid is a stale snapshot the OS may have recycled
|
||||
to an unrelated process group — kill() must not signal it (the waiter's
|
||||
own group kill already ran at exit, when the pgid was fresh)."""
|
||||
shell = registry.spawn("true")
|
||||
assert _wait_status(shell, "completed")
|
||||
calls = []
|
||||
monkeypatch.setattr(bg_mod.os, "killpg", lambda *a: calls.append(a))
|
||||
killed = registry.kill(shell.shell_id)
|
||||
assert calls == [], "killpg must not fire for an already-exited shell"
|
||||
assert killed.status == "completed", "a natural exit must not be relabelled 'killed'"
|
||||
|
||||
|
||||
def test_close_is_time_bounded_with_pipe_holding_escapee(registry, tmp_path):
|
||||
"""An escaped-group grandchild that holds the output pipes wedges the
|
||||
drain threads. close() must still return within its total budget —
|
||||
it can run under the server's async close route, where an unbounded
|
||||
join would freeze the whole node's event loop."""
|
||||
pidfile = tmp_path / "holder.pid"
|
||||
# ``setsid`` puts the sleep in a NEW session (outside our kill group)
|
||||
# while it still inherits our stdout/stderr pipes — the accepted
|
||||
# leaked-daemon case from the module docstring.
|
||||
shell = registry.spawn(f"setsid sleep 60 & echo $! > {pidfile}; echo started")
|
||||
assert _wait_until(pidfile.exists)
|
||||
holder_pid = int(pidfile.read_text().strip())
|
||||
try:
|
||||
start = time.monotonic()
|
||||
registry.close()
|
||||
elapsed = time.monotonic() - start
|
||||
assert elapsed < 8, f"close() took {elapsed:.1f}s — teardown must be budget-bounded"
|
||||
finally:
|
||||
_kill_pid(holder_pid)
|
||||
# The holder is dead, so the wedged drains EOF promptly; wait for
|
||||
# them here so the conftest leak guard sees a clean teardown.
|
||||
assert _wait_until(lambda: not any(t.is_alive() for t in shell._threads))
|
||||
|
||||
|
||||
def test_exited_records_are_pruned_at_cap():
|
||||
reg = BackgroundShellRegistry(max_exited_records=2)
|
||||
try:
|
||||
shells = [reg.spawn(f"echo job-{i}") for i in range(3)]
|
||||
for s in shells:
|
||||
assert _wait_status(s, "completed")
|
||||
# Eviction happens on each exit; poll until the oldest is gone
|
||||
# (waiter threads race, prune runs per-exit).
|
||||
assert _wait_until(lambda: not reg.has(shells[0].shell_id))
|
||||
assert reg.has(shells[1].shell_id)
|
||||
assert reg.has(shells[2].shell_id)
|
||||
with pytest.raises(UnknownShellError):
|
||||
reg.read(shells[0].shell_id)
|
||||
finally:
|
||||
reg.close()
|
||||
|
||||
|
||||
def test_catastrophic_filter_times_out_without_consuming(registry):
|
||||
"""A backtracking-bomb filter must error within the bound and consume
|
||||
NOTHING — the retry without a filter still gets the output. The match
|
||||
runs in a killable child process: sre holds the GIL, so an in-process
|
||||
bomb would freeze the whole interpreter, watchdogs included."""
|
||||
# One ~3000-char line of a's ending in 'b' — the classic (a+)+$ bomb
|
||||
# subject — followed by a sentinel line.
|
||||
shell = registry.spawn("printf 'a%.0s' $(seq 1 3000); echo b; echo tail-line")
|
||||
assert _wait_status(shell, "completed")
|
||||
start = time.monotonic()
|
||||
with pytest.raises(FilterTimeoutError):
|
||||
registry.read(shell.shell_id, filter_pattern=r"(a+)+$")
|
||||
assert time.monotonic() - start < 10, "filter timeout must be bounded"
|
||||
# Nothing was consumed: an unfiltered read sees the whole delta.
|
||||
read = registry.read(shell.shell_id)
|
||||
assert any("tail-line" in ln for ln in read.lines)
|
||||
|
||||
|
||||
def test_overlong_filter_pattern_is_rejected(registry):
|
||||
shell = registry.spawn("echo hi")
|
||||
assert _wait_status(shell, "completed")
|
||||
with pytest.raises(re.error):
|
||||
registry.read(shell.shell_id, filter_pattern="x" * 600)
|
||||
|
||||
|
||||
def test_cap_error_is_owner_scope_honest():
|
||||
"""The cap is registry-wide, but the advice must only name shells the
|
||||
caller can actually kill — kill_shell is owner-scoped."""
|
||||
reg = BackgroundShellRegistry(max_shells=1)
|
||||
try:
|
||||
reg.spawn("sleep 30") # main scope fills the cap
|
||||
with pytest.raises(TooManyShellsError) as excinfo:
|
||||
reg.spawn("sleep 30", owner="agent-1")
|
||||
msg = str(excinfo.value)
|
||||
assert "bash_1" not in msg, "must not advise killing another scope's shell"
|
||||
assert "other agents" in msg
|
||||
# The same-scope variant names the killable shell.
|
||||
with pytest.raises(TooManyShellsError) as excinfo2:
|
||||
reg.spawn("sleep 30")
|
||||
assert "bash_1" in str(excinfo2.value)
|
||||
assert "kill_shell" in str(excinfo2.value)
|
||||
finally:
|
||||
reg.close()
|
||||
|
||||
|
||||
def test_prune_evicts_by_exit_order_not_spawn_order():
|
||||
"""A long-lived first-spawned server must never be evicted by its OWN
|
||||
exit's prune once enough later jobs have finished — eviction follows
|
||||
exit order, so the just-exited shell is always the newest record."""
|
||||
reg = BackgroundShellRegistry(max_exited_records=2)
|
||||
try:
|
||||
server = reg.spawn("sleep 30") # bash_1, exits LAST
|
||||
jobs = [reg.spawn(f"echo job-{i}") for i in range(3)]
|
||||
for job in jobs:
|
||||
assert _wait_status(job, "completed")
|
||||
reg.kill(server.shell_id)
|
||||
assert reg.has(server.shell_id), "the just-exited shell must survive its own exit's prune"
|
||||
# The earliest-EXITED job is the eviction victim, not bash_1.
|
||||
assert _wait_until(lambda: len(reg.shells()) <= 3)
|
||||
assert reg.read(server.shell_id).status == "killed"
|
||||
finally:
|
||||
reg.close()
|
||||
|
||||
|
||||
def test_thread_start_failure_leaves_no_orphan_record(registry, monkeypatch, tmp_path):
|
||||
"""If Thread.start raises (thread exhaustion), the record must be
|
||||
unregistered and the fresh group reaped — an orphan with never-started
|
||||
Thread objects would make every later close()/reap() join raise and
|
||||
abort session teardown."""
|
||||
pidfile = tmp_path / "leader.pid"
|
||||
real_thread = bg_mod.threading.Thread
|
||||
|
||||
class FailingWaiterThread(real_thread):
|
||||
def start(self):
|
||||
if "bg-shell-wait" in (self.name or ""):
|
||||
raise RuntimeError("can't start new thread")
|
||||
super().start()
|
||||
|
||||
monkeypatch.setattr(bg_mod.threading, "Thread", FailingWaiterThread)
|
||||
with pytest.raises(RuntimeError):
|
||||
registry.spawn(f"echo $$ > {pidfile}; sleep 60")
|
||||
assert registry.shells() == [], "failed spawn must not strand a record"
|
||||
if pidfile.exists():
|
||||
leader_pid = int(pidfile.read_text().strip())
|
||||
assert _wait_until(lambda: not _pid_alive(leader_pid)), "fresh group leaked"
|
||||
monkeypatch.undo()
|
||||
registry.close() # must not raise on the (empty) registry
|
||||
|
||||
|
||||
def test_filter_helper_failure_reports_exec_error_not_timeout(registry, monkeypatch):
|
||||
"""A crashed helper must not tell the model its (fine) pattern was too
|
||||
slow — and must not consume the delta."""
|
||||
shell = registry.spawn("echo hello")
|
||||
assert _wait_status(shell, "completed")
|
||||
monkeypatch.setattr(bg_mod.sys, "executable", "/bin/false")
|
||||
with pytest.raises(FilterExecError) as excinfo:
|
||||
registry.read(shell.shell_id, filter_pattern="hello")
|
||||
assert "not a problem with your pattern" in str(excinfo.value)
|
||||
monkeypatch.undo()
|
||||
read = registry.read(shell.shell_id)
|
||||
assert [ln.strip() for ln in read.lines] == ["hello"]
|
||||
|
||||
|
||||
def test_filter_matches_only_within_line_cap_and_reports_clipping(registry):
|
||||
"""Lines are truncated parent-side before shipping to the helper: a
|
||||
match beyond the per-line cap is not found (a filter targets log
|
||||
lines), and a huge retained line cannot burn the time budget on I/O.
|
||||
The clipping is NEVER silent — the read reports how many lines were
|
||||
only partially visible to the pattern."""
|
||||
shell = registry.spawn("printf 'x%.0s' $(seq 1 5000); echo needle-suffix")
|
||||
assert _wait_status(shell, "completed")
|
||||
read = registry.read(shell.shell_id, filter_pattern="needle")
|
||||
assert read.lines == []
|
||||
assert read.new_line_count == 1
|
||||
assert read.clipped_lines == 1
|
||||
|
||||
|
||||
def test_concurrent_reads_never_double_deliver(registry):
|
||||
"""Two simultaneous reads of one shell must SPLIT the delta between
|
||||
them, never both return it — the whole pass (snapshot → commit)
|
||||
serializes per shell. Without that, a parallel tool batch reading the
|
||||
same handle gets every line twice."""
|
||||
shell = registry.spawn("seq 1 200")
|
||||
assert _wait_status(shell, "completed")
|
||||
results: list[list[str]] = [[], []]
|
||||
barrier = threading.Barrier(2)
|
||||
|
||||
def _reader(slot: int) -> None:
|
||||
barrier.wait()
|
||||
results[slot] = [ln.strip() for ln in registry.read(shell.shell_id).lines]
|
||||
|
||||
threads = [threading.Thread(target=_reader, args=(i,)) for i in range(2)]
|
||||
for t in threads:
|
||||
t.start()
|
||||
for t in threads:
|
||||
t.join(timeout=10)
|
||||
combined = results[0] + results[1]
|
||||
assert len(combined) == 200, f"expected each line exactly once, got {len(combined)}"
|
||||
assert sorted(combined, key=int) == [str(i) for i in range(1, 201)]
|
||||
|
||||
|
||||
def test_filter_helper_spawn_failure_is_exec_error(registry, monkeypatch):
|
||||
"""A helper that fails to LAUNCH (fork pressure) must land in the same
|
||||
honest FilterExecError as a crashed helper — not escape as a raw
|
||||
OSError blaming nothing — and must not consume the delta."""
|
||||
shell = registry.spawn("echo hello")
|
||||
assert _wait_status(shell, "completed")
|
||||
|
||||
def _boom(*args, **kwargs):
|
||||
raise BlockingIOError("Resource temporarily unavailable")
|
||||
|
||||
monkeypatch.setattr(bg_mod.subprocess, "Popen", _boom)
|
||||
with pytest.raises(FilterExecError):
|
||||
registry.read(shell.shell_id, filter_pattern="hello")
|
||||
monkeypatch.undo()
|
||||
read = registry.read(shell.shell_id)
|
||||
assert [ln.strip() for ln in read.lines] == ["hello"]
|
||||
|
||||
|
||||
def test_on_exit_exception_does_not_wedge_the_shell():
|
||||
def _boom(shell):
|
||||
raise RuntimeError("callback bug")
|
||||
|
||||
reg = BackgroundShellRegistry(on_exit=_boom)
|
||||
try:
|
||||
shell = reg.spawn("echo hi")
|
||||
# The waiter thread must survive the callback raising: status still
|
||||
# lands and output is still readable.
|
||||
assert _wait_status(shell, "completed")
|
||||
assert [ln.strip() for ln in reg.read(shell.shell_id).lines] == ["hi"]
|
||||
finally:
|
||||
reg.close()
|
||||
@@ -0,0 +1,801 @@
|
||||
"""Session-level tests for the background-shell tool surface (#817).
|
||||
|
||||
Covers the wiring around :class:`BackgroundShellRegistry`:
|
||||
|
||||
* ``bash`` gains ``run_in_background: true`` (alias ``is_background``) —
|
||||
same approval gate, returns immediately with a ``bash_N`` handle.
|
||||
* ``bash_output`` — auto-approved delta reader (status + exit code + only
|
||||
new output since the last call, optional ``filter`` regex).
|
||||
* ``kill_shell`` — auto-approved kill of a registered shell's whole group.
|
||||
* Exit notices ride the NudgeQueue on channel ``"any"`` (the watch rail) so
|
||||
they drain at the next seam and can wake an idle workstream.
|
||||
* Lifecycle: ``close()`` reaps everything; generation-``cancel()`` does NOT
|
||||
(a deliberately-detached server survives a stopped turn); shells spawned
|
||||
inside a task_agent are owner-scoped and reaped when the agent finishes.
|
||||
"""
|
||||
|
||||
import time
|
||||
|
||||
import pytest
|
||||
|
||||
from tests._proc_helpers import pid_alive as _pid_alive
|
||||
from tests._proc_helpers import poll_until as _wait_until
|
||||
from tests._session_helpers import make_session
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def session():
|
||||
s = make_session()
|
||||
yield s
|
||||
s.close()
|
||||
|
||||
|
||||
def _start_background(session, command, call_id="bg1", **extra_args):
|
||||
"""Prepare + execute a backgrounded bash call; return the result text."""
|
||||
args = {"command": command, "run_in_background": True, **extra_args}
|
||||
prepared = session._prepare_bash(call_id, args)
|
||||
assert "error" not in prepared, prepared.get("error")
|
||||
_cid, output = prepared["execute"](prepared)
|
||||
return output
|
||||
|
||||
|
||||
def _only_shell(session):
|
||||
shells = session._background_shells.shells()
|
||||
assert len(shells) == 1
|
||||
return shells[0]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# bash: run_in_background routing
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_prepare_bash_background_keeps_approval_gate(session):
|
||||
prepared = session._prepare_bash("c1", {"command": "sleep 30", "run_in_background": True})
|
||||
assert prepared["needs_approval"] is True
|
||||
assert prepared["approval_label"] == "bash"
|
||||
|
||||
|
||||
def test_prepare_bash_background_header_says_background(session):
|
||||
prepared = session._prepare_bash("c1", {"command": "sleep 30", "run_in_background": True})
|
||||
assert "background" in prepared["header"]
|
||||
|
||||
|
||||
def test_background_bash_returns_immediately_with_handle(session):
|
||||
start = time.monotonic()
|
||||
output = _start_background(session, "sleep 30")
|
||||
elapsed = time.monotonic() - start
|
||||
assert elapsed < 5, f"backgrounded call blocked for {elapsed:.1f}s"
|
||||
assert "bash_1" in output
|
||||
shell = _only_shell(session)
|
||||
assert shell.status == "running"
|
||||
assert _pid_alive(shell.pid)
|
||||
|
||||
|
||||
def test_background_start_mentions_reader_and_killer(session):
|
||||
"""The immediate result must teach the follow-up tools — weak-prior
|
||||
models (GPT-5.6) only reach for the poll pattern if the result names it."""
|
||||
output = _start_background(session, "sleep 30")
|
||||
assert "bash_output" in output
|
||||
assert "kill_shell" in output
|
||||
|
||||
|
||||
def test_is_background_alias_accepted(session):
|
||||
output = _start_background(session, "sleep 30", is_background=True)
|
||||
assert "bash_1" in output
|
||||
assert _only_shell(session).status == "running"
|
||||
|
||||
|
||||
def test_foreground_bash_routing_unchanged(session):
|
||||
prepared = session._prepare_bash("c1", {"command": "echo hi"})
|
||||
assert prepared["execute"] == session._exec_bash
|
||||
prepared_false = session._prepare_bash("c2", {"command": "echo hi", "run_in_background": False})
|
||||
assert prepared_false["execute"] == session._exec_bash
|
||||
|
||||
|
||||
def test_background_respects_command_blocklist(session):
|
||||
prepared = session._prepare_bash("c1", {"command": "shutdown now", "run_in_background": True})
|
||||
assert "error" in prepared
|
||||
assert session._background_shells.shells() == []
|
||||
|
||||
|
||||
def test_background_ignores_timeout(session):
|
||||
"""No bounded wait exists to time out — a 1s timeout must not kill the
|
||||
detached shell."""
|
||||
_start_background(session, "sleep 30", timeout=1)
|
||||
shell = _only_shell(session)
|
||||
time.sleep(1.5)
|
||||
assert shell.status == "running"
|
||||
assert _pid_alive(shell.pid)
|
||||
|
||||
|
||||
def test_background_spawn_failure_reports_error(session, monkeypatch):
|
||||
from turnstone.core import background_shells as bg_mod
|
||||
|
||||
def _boom(*args, **kwargs):
|
||||
raise OSError("cannot fork")
|
||||
|
||||
monkeypatch.setattr(bg_mod.subprocess, "Popen", _boom)
|
||||
prepared = session._prepare_bash("c1", {"command": "echo hi", "run_in_background": True})
|
||||
_cid, output = prepared["execute"](prepared)
|
||||
assert "cannot fork" in output
|
||||
|
||||
|
||||
def test_too_many_background_shells_reports_error(session, monkeypatch):
|
||||
monkeypatch.setattr(session._background_shells, "_max_shells", 1)
|
||||
_start_background(session, "sleep 30", call_id="bg1")
|
||||
output = _start_background(session, "sleep 30", call_id="bg2")
|
||||
assert "bash_1" in output # the live shell is named so the model can kill it
|
||||
assert len(session._background_shells.shells()) == 1
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# bash_output
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_bash_output_is_auto_approved(session):
|
||||
prepared = session._prepare_bash_output("c1", {"id": "bash_1"})
|
||||
assert prepared["needs_approval"] is False
|
||||
|
||||
|
||||
def test_bash_output_missing_id_errors(session):
|
||||
prepared = session._prepare_bash_output("c1", {})
|
||||
assert "error" in prepared
|
||||
|
||||
|
||||
def test_bash_output_returns_delta_then_no_new_output(session):
|
||||
_start_background(session, "echo hello; sleep 30")
|
||||
shell = _only_shell(session)
|
||||
assert _wait_until(lambda: shell.status == "running")
|
||||
|
||||
def _read():
|
||||
prepared = session._prepare_bash_output("r", {"id": shell.shell_id})
|
||||
assert "error" not in prepared
|
||||
return prepared["execute"](prepared)[1]
|
||||
|
||||
assert _wait_until(lambda: "hello" in _read())
|
||||
again = _read()
|
||||
assert "hello" not in again
|
||||
assert "no new output" in again.lower()
|
||||
assert "running" in again.lower()
|
||||
|
||||
|
||||
def test_bash_output_reports_exit_code_when_completed(session):
|
||||
_start_background(session, "exit 3")
|
||||
shell = _only_shell(session)
|
||||
assert _wait_until(lambda: shell.status == "completed")
|
||||
prepared = session._prepare_bash_output("r", {"id": shell.shell_id})
|
||||
_cid, output = prepared["execute"](prepared)
|
||||
assert "completed" in output.lower()
|
||||
assert "3" in output
|
||||
|
||||
|
||||
def test_bash_output_filter_applies(session):
|
||||
_start_background(session, "echo match-a; echo skip-b")
|
||||
shell = _only_shell(session)
|
||||
assert _wait_until(lambda: shell.status == "completed")
|
||||
prepared = session._prepare_bash_output("r", {"id": shell.shell_id, "filter": "^match"})
|
||||
_cid, output = prepared["execute"](prepared)
|
||||
assert "match-a" in output
|
||||
assert "skip-b" not in output
|
||||
|
||||
|
||||
def test_bash_output_invalid_filter_reports_error(session):
|
||||
_start_background(session, "sleep 30")
|
||||
shell = _only_shell(session)
|
||||
prepared = session._prepare_bash_output("r", {"id": shell.shell_id, "filter": "[bad"})
|
||||
_cid, output = prepared["execute"](prepared)
|
||||
assert "regex" in output.lower() or "filter" in output.lower()
|
||||
|
||||
|
||||
def test_bash_output_unknown_id_lists_live_shells(session):
|
||||
_start_background(session, "sleep 30")
|
||||
prepared = session._prepare_bash_output("r", {"id": "bash_42"})
|
||||
_cid, output = prepared["execute"](prepared)
|
||||
assert "bash_42" in output
|
||||
assert "bash_1" in output
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# kill_shell
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_kill_shell_is_auto_approved(session):
|
||||
prepared = session._prepare_kill_shell("c1", {"id": "bash_1"})
|
||||
assert prepared["needs_approval"] is False
|
||||
|
||||
|
||||
def test_kill_shell_missing_id_errors(session):
|
||||
prepared = session._prepare_kill_shell("c1", {})
|
||||
assert "error" in prepared
|
||||
|
||||
|
||||
def test_kill_shell_kills_and_reports(session):
|
||||
_start_background(session, "sleep 60")
|
||||
shell = _only_shell(session)
|
||||
prepared = session._prepare_kill_shell("k", {"id": shell.shell_id})
|
||||
_cid, output = prepared["execute"](prepared)
|
||||
assert "killed" in output.lower()
|
||||
assert _wait_until(lambda: not _pid_alive(shell.pid))
|
||||
# The schema promises the exit code for ANY exited state, killed included.
|
||||
read_prepared = session._prepare_bash_output("r", {"id": shell.shell_id})
|
||||
_cid, read_output = read_prepared["execute"](read_prepared)
|
||||
assert "exit code" in read_output
|
||||
|
||||
|
||||
def test_kill_shell_unknown_id_reports_error(session):
|
||||
prepared = session._prepare_kill_shell("k", {"id": "bash_9"})
|
||||
_cid, output = prepared["execute"](prepared)
|
||||
assert "bash_9" in output
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Exit notices (NudgeQueue, channel "any", wake)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_natural_exit_enqueues_any_channel_notice(session):
|
||||
_start_background(session, "echo done")
|
||||
assert _wait_until(
|
||||
lambda: any(t == "background_shell_exit" for t, _ in session._nudge_queue.pending())
|
||||
)
|
||||
entries = session._nudge_queue.pending(channel="any")
|
||||
texts = [text for t, text in entries if t == "background_shell_exit"]
|
||||
assert texts, "notice must ride channel 'any' so it can wake an idle workstream"
|
||||
assert "bash_1" in texts[0]
|
||||
assert "bash_output" in texts[0]
|
||||
|
||||
|
||||
def test_exit_notice_carries_metadata(session):
|
||||
_start_background(session, "exit 5")
|
||||
assert _wait_until(
|
||||
lambda: any(t == "background_shell_exit" for t, _ in session._nudge_queue.pending())
|
||||
)
|
||||
metadata = [
|
||||
meta
|
||||
for t, _text, meta in session._nudge_queue.pending_with_metadata()
|
||||
if t == "background_shell_exit"
|
||||
][0]
|
||||
assert metadata["shell_id"] == "bash_1"
|
||||
assert metadata["exit_code"] == 5
|
||||
|
||||
|
||||
def test_exit_notice_triggers_wake_fn(session):
|
||||
wakes = []
|
||||
session._watch_wake_fn = lambda: wakes.append(1)
|
||||
_start_background(session, "echo done")
|
||||
assert _wait_until(lambda: wakes), "natural exit must wake an idle workstream"
|
||||
|
||||
|
||||
def test_kill_shell_suppresses_exit_notice(session):
|
||||
_start_background(session, "sleep 60")
|
||||
shell = _only_shell(session)
|
||||
prepared = session._prepare_kill_shell("k", {"id": shell.shell_id})
|
||||
prepared["execute"](prepared)
|
||||
assert _wait_until(lambda: not _pid_alive(shell.pid))
|
||||
time.sleep(0.3) # a buggy late notice would land within this window
|
||||
assert not any(t == "background_shell_exit" for t, _ in session._nudge_queue.pending())
|
||||
|
||||
|
||||
def test_close_drops_pending_exit_notice_via_valid_until(session):
|
||||
"""A notice for a shell that no longer exists (registry closed) must not
|
||||
deliver — the valid_until predicate drops it at drain time."""
|
||||
_start_background(session, "echo done")
|
||||
assert _wait_until(
|
||||
lambda: any(t == "background_shell_exit" for t, _ in session._nudge_queue.pending())
|
||||
)
|
||||
session.close()
|
||||
from turnstone.core.nudge_queue import USER_DRAIN
|
||||
|
||||
drained = session._nudge_queue.drain(USER_DRAIN)
|
||||
assert not any(t == "background_shell_exit" for t, _text, _m in drained)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Lifecycle
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_close_reaps_background_shells(session):
|
||||
_start_background(session, "sleep 60")
|
||||
shell = _only_shell(session)
|
||||
session.close()
|
||||
assert not _pid_alive(shell.pid)
|
||||
|
||||
|
||||
def test_generation_cancel_does_not_reap_background_shells(session):
|
||||
"""cancel() fires on mere stop-generation — a deliberately-detached
|
||||
server must survive it. Only close()/kill_shell end it."""
|
||||
_start_background(session, "sleep 60")
|
||||
shell = _only_shell(session)
|
||||
session.cancel()
|
||||
time.sleep(0.3)
|
||||
assert _pid_alive(shell.pid), "generation cancel must not kill detached shells"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Review-hardening regressions (#817 code review)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_string_typed_background_flag_is_honored(session):
|
||||
"""Providers intermittently send booleans as strings; 'true' must not
|
||||
silently fall through to the foreground executor (where the group kill
|
||||
would reap the server the model believed it detached)."""
|
||||
for call_id, args in (
|
||||
("s1", {"command": "sleep 30", "run_in_background": "true"}),
|
||||
("s2", {"command": "sleep 30", "is_background": "True"}),
|
||||
):
|
||||
prepared = session._prepare_bash(call_id, args)
|
||||
assert prepared["execute"] == session._exec_bash_background, args
|
||||
|
||||
|
||||
def test_kill_shell_on_completed_shell_reports_already_exited(session):
|
||||
_start_background(session, "true")
|
||||
shell = _only_shell(session)
|
||||
assert _wait_until(lambda: shell.status == "completed")
|
||||
prepared = session._prepare_kill_shell("k", {"id": shell.shell_id})
|
||||
_cid, output = prepared["execute"](prepared)
|
||||
assert "already exited" in output.lower()
|
||||
|
||||
|
||||
def test_exit_notice_survives_generation_abandon_without_waking(session):
|
||||
"""cancel/interrupt/exception clear generation-scoped advisories, but an
|
||||
external event (a background shell exited) still happened — its notice
|
||||
must survive to the next seam or the model keeps talking to a dead
|
||||
server. It survives DEMOTED to 'quiet': still deliverable, but no
|
||||
longer wake-eligible, so the workstream the user just stopped cannot
|
||||
resume itself over it."""
|
||||
from turnstone.core.nudge_queue import USER_DRAIN, WAKE_PENDING
|
||||
|
||||
_start_background(session, "echo done")
|
||||
assert _wait_until(
|
||||
lambda: any(t == "background_shell_exit" for t, _ in session._nudge_queue.pending())
|
||||
)
|
||||
session._queue_tool_advisory("tool_error", "3 consecutive tool errors")
|
||||
session._drain_pending_advisories()
|
||||
kinds = [t for t, _ in session._nudge_queue.pending()]
|
||||
assert "background_shell_exit" in kinds
|
||||
assert "tool_error" not in kinds
|
||||
# Post-cancel quiescence: nothing is wake-eligible...
|
||||
assert not session._nudge_queue.has_pending(WAKE_PENDING)
|
||||
# ...yet the notice still delivers at the next legitimate seam.
|
||||
drained = session._nudge_queue.drain(USER_DRAIN)
|
||||
assert any(t == "background_shell_exit" for t, _x, _m in drained)
|
||||
|
||||
|
||||
def test_int_typed_background_flag_is_honored(session):
|
||||
prepared = session._prepare_bash("i1", {"command": "sleep 30", "run_in_background": 1})
|
||||
assert prepared["execute"] == session._exec_bash_background
|
||||
prepared_zero = session._prepare_bash("i2", {"command": "echo hi", "run_in_background": 0})
|
||||
assert prepared_zero["execute"] == session._exec_bash
|
||||
|
||||
|
||||
def test_bash_output_non_string_filter_errors_without_consuming(session):
|
||||
_start_background(session, "echo hello; sleep 30")
|
||||
shell = _only_shell(session)
|
||||
prepared = session._prepare_bash_output("r", {"id": shell.shell_id, "filter": 123})
|
||||
assert "error" in prepared
|
||||
assert "filter" in prepared["error"].lower()
|
||||
# Nothing was consumed by the refused call.
|
||||
assert _wait_until(lambda: shell.unread_lines > 0)
|
||||
|
||||
|
||||
def test_filter_timeout_reports_error_without_consuming(session, monkeypatch):
|
||||
from turnstone.core.background_shells import FilterTimeoutError
|
||||
|
||||
_start_background(session, "sleep 30")
|
||||
shell = _only_shell(session)
|
||||
|
||||
def _boom(*a, **kw):
|
||||
raise FilterTimeoutError("filter regex took longer than 2s to run")
|
||||
|
||||
monkeypatch.setattr(session._background_shells, "read", _boom)
|
||||
prepared = session._prepare_bash_output("r", {"id": shell.shell_id, "filter": "(a+)+$"})
|
||||
_cid, output = prepared["execute"](prepared)
|
||||
assert "filter" in output.lower()
|
||||
assert "error" in output.lower()
|
||||
|
||||
|
||||
def test_registries_are_isolated_per_session():
|
||||
"""Workstream isolation: a handle from one session must be unresolvable
|
||||
from another — buffers, ids, and kills never cross ChatSessions."""
|
||||
session_a = make_session()
|
||||
session_b = make_session()
|
||||
try:
|
||||
_start_background(session_a, "sleep 30")
|
||||
shell_a = _only_shell(session_a)
|
||||
read_b = session_b._prepare_bash_output("r", {"id": shell_a.shell_id})
|
||||
_cid, output = read_b["execute"](read_b)
|
||||
assert "no background shell" in output.lower()
|
||||
kill_b = session_b._prepare_kill_shell("k", {"id": shell_a.shell_id})
|
||||
_cid, kill_output = kill_b["execute"](kill_b)
|
||||
assert "no background shell" in kill_output.lower()
|
||||
assert _pid_alive(shell_a.pid), "another session must not be able to kill the shell"
|
||||
finally:
|
||||
session_a.close()
|
||||
session_b.close()
|
||||
|
||||
|
||||
def test_bash_output_polling_is_repeat_exempt(session):
|
||||
"""Repeated identical bash_output calls ARE the documented monitoring
|
||||
pattern — the repeat detector must not brand them 'identical repeat'
|
||||
(the delta result differs by construction) nor queue a repeat nudge."""
|
||||
import json as _json
|
||||
|
||||
_start_background(session, "sleep 30")
|
||||
shell = _only_shell(session)
|
||||
args = _json.dumps({"id": shell.shell_id})
|
||||
for i in range(5):
|
||||
tool_calls = [{"id": f"t{i}", "function": {"name": "bash_output", "arguments": args}}]
|
||||
results = [(f"t{i}", "bash_1 (running)\nNo new output since the last read.")]
|
||||
session._apply_post_execute_advisories(tool_calls, results)
|
||||
assert "identical repeat" not in results[0][1]
|
||||
assert not any(t == "repeat" for t, _ in session._nudge_queue.pending())
|
||||
|
||||
|
||||
def test_repeat_exempt_calls_still_break_other_streaks(session):
|
||||
"""The exemption suppresses the WARNING, not the recording: a
|
||||
bash_output poll interleaved between identical bash calls must reset
|
||||
the bash streak — otherwise the documented monitor-and-probe loop
|
||||
(poll, curl health, poll, curl health…) draws a false 'identical
|
||||
repeat' on the probe."""
|
||||
import json as _json
|
||||
|
||||
_start_background(session, "sleep 30")
|
||||
shell = _only_shell(session)
|
||||
poll_args = _json.dumps({"id": shell.shell_id})
|
||||
probe_args = _json.dumps({"command": "curl -s localhost:8080/health"})
|
||||
for i in range(6):
|
||||
probe = [{"id": f"p{i}", "function": {"name": "bash", "arguments": probe_args}}]
|
||||
probe_results = [(f"p{i}", "ok")]
|
||||
session._apply_post_execute_advisories(probe, probe_results)
|
||||
assert "identical repeat" not in probe_results[0][1], (
|
||||
"interleaved probes are not a stuck loop"
|
||||
)
|
||||
poll = [{"id": f"q{i}", "function": {"name": "bash_output", "arguments": poll_args}}]
|
||||
session._apply_post_execute_advisories(poll, [(f"q{i}", "no new output")])
|
||||
|
||||
|
||||
def test_bash_repeats_still_warn(session):
|
||||
"""The exemption is bash_output-specific: a genuinely stuck identical
|
||||
bash loop still gets the warning."""
|
||||
import json as _json
|
||||
|
||||
args = _json.dumps({"command": "echo test"})
|
||||
warned = False
|
||||
for i in range(5):
|
||||
tool_calls = [{"id": f"b{i}", "function": {"name": "bash", "arguments": args}}]
|
||||
results = [(f"b{i}", "test")]
|
||||
session._apply_post_execute_advisories(tool_calls, results)
|
||||
warned = warned or "identical repeat" in results[0][1]
|
||||
assert warned
|
||||
|
||||
|
||||
def test_quiet_only_entries_do_not_trigger_wake_delivery(session, monkeypatch):
|
||||
"""A dispatched wake whose wake-eligible entries all evaporated must be
|
||||
a no-op: quiet entries alone never resume a stopped workstream, and
|
||||
they stay queued for the next legitimate seam."""
|
||||
calls = []
|
||||
monkeypatch.setattr(session, "send", lambda *a, **k: calls.append(1))
|
||||
session._nudge_queue.enqueue("background_shell_exit", "old news", "quiet")
|
||||
session.deliver_wake_nudge_from_queue()
|
||||
assert calls == []
|
||||
assert session._nudge_queue.pending(channel="quiet") == [("background_shell_exit", "old news")]
|
||||
|
||||
|
||||
def test_wake_delivers_quiet_alongside_eligible_in_insertion_order(session, monkeypatch):
|
||||
"""Quiet entries ride the wake AND cross-channel chronology holds: an
|
||||
older demoted notice renders before the newer fire that earned the
|
||||
wake (a poll counter must never run backwards)."""
|
||||
seen = {}
|
||||
|
||||
def _fake_send(*a, **k):
|
||||
seen["reminders"] = list(session._wake_drained_reminders or [])
|
||||
session._wake_drained_reminders = None # emulate emission consuming
|
||||
|
||||
monkeypatch.setattr(session, "send", _fake_send)
|
||||
session._nudge_queue.enqueue("background_shell_exit", "old", "quiet")
|
||||
session._nudge_queue.enqueue("watch_triggered", "new", "any")
|
||||
session.deliver_wake_nudge_from_queue()
|
||||
types = [e["type"] for e in seen["reminders"]]
|
||||
assert types == ["background_shell_exit", "watch_triggered"], (
|
||||
"older quiet entry must precede the newer wake-eligible one"
|
||||
)
|
||||
assert session._nudge_queue.pending() == []
|
||||
|
||||
|
||||
def test_failed_wake_reenqueue_preserves_valid_until(session, monkeypatch):
|
||||
"""The re-enqueued notice keeps its staleness predicate — a stale
|
||||
notice re-queued by a failed wake must still be droppable at its next
|
||||
drain, not delivered against a gone shell."""
|
||||
from turnstone.core.nudge_queue import USER_DRAIN
|
||||
|
||||
alive = {"value": True}
|
||||
|
||||
def _fail(*a, **k):
|
||||
raise RuntimeError("storage down")
|
||||
|
||||
monkeypatch.setattr(session, "send", _fail)
|
||||
session._nudge_queue.enqueue(
|
||||
"background_shell_exit",
|
||||
"server died",
|
||||
"any",
|
||||
valid_until=lambda: alive["value"],
|
||||
)
|
||||
with pytest.raises(RuntimeError):
|
||||
session.deliver_wake_nudge_from_queue()
|
||||
assert session._nudge_queue.pending(channel="quiet"), "notice must be re-queued"
|
||||
alive["value"] = False # the shell record is gone now
|
||||
drained = session._nudge_queue.drain(USER_DRAIN)
|
||||
assert drained == [], "stale re-queued notice must drop via its predicate"
|
||||
|
||||
|
||||
def test_mid_emit_failure_restashes_unemitted_tail(session, monkeypatch):
|
||||
"""A failure while emitting reminder k of n must leave k..n recoverable
|
||||
— the wake caller's finally re-enqueues them instead of losing the
|
||||
suffix."""
|
||||
calls = {"n": 0}
|
||||
|
||||
def _append(source, text, **meta):
|
||||
calls["n"] += 1
|
||||
if calls["n"] == 2:
|
||||
raise RuntimeError("storage down")
|
||||
|
||||
monkeypatch.setattr(session, "_append_system_turn", _append)
|
||||
session._wake_drained_reminders = [
|
||||
{"type": "a", "text": "1"},
|
||||
{"type": "b", "text": "2"},
|
||||
{"type": "c", "text": "3"},
|
||||
]
|
||||
with pytest.raises(RuntimeError):
|
||||
session._emit_pending_user_nudges()
|
||||
assert session._wake_drained_reminders == [
|
||||
{"type": "b", "text": "2"},
|
||||
{"type": "c", "text": "3"},
|
||||
]
|
||||
|
||||
|
||||
def test_failed_wake_reenqueues_undelivered_as_quiet(session, monkeypatch):
|
||||
"""A wake send that dies before emitting its drained reminders must not
|
||||
eat them — a shell's exit notice fires exactly once."""
|
||||
|
||||
def _fail(*a, **k):
|
||||
raise RuntimeError("storage down")
|
||||
|
||||
monkeypatch.setattr(session, "send", _fail)
|
||||
session._nudge_queue.enqueue(
|
||||
"background_shell_exit", "server died", "any", metadata={"shell_id": "bash_1"}
|
||||
)
|
||||
with pytest.raises(RuntimeError):
|
||||
session.deliver_wake_nudge_from_queue()
|
||||
pending = session._nudge_queue.pending_with_metadata(channel="quiet")
|
||||
assert [(t, x) for t, x, _m in pending] == [("background_shell_exit", "server died")]
|
||||
assert pending[0][2] == {"shell_id": "bash_1"}
|
||||
|
||||
|
||||
def test_failed_wake_preserves_chronology_and_stays_wake_quiescent(session, monkeypatch):
|
||||
"""Failed-wake recovery invariants: (a) the re-queued external notice
|
||||
keeps its seq, so the retry renders it BEFORE a newer event that
|
||||
arrived during the failure; (b) NOTHING wake-eligible remains after
|
||||
the failure — external notices demote to quiet and user-channel
|
||||
advisories are dropped outright, because a re-armed WAKE_PENDING gate
|
||||
plus the zero-backoff worker-exit retry would respawn wake workers in
|
||||
an unbounded hot loop against a persistent failure."""
|
||||
from turnstone.core.nudge_queue import WAKE_PENDING
|
||||
|
||||
calls = {"n": 0}
|
||||
seen = {}
|
||||
|
||||
def _send(*a, **k):
|
||||
calls["n"] += 1
|
||||
if calls["n"] == 1:
|
||||
raise RuntimeError("transient storage failure")
|
||||
seen["reminders"] = list(session._wake_drained_reminders or [])
|
||||
session._wake_drained_reminders = None
|
||||
|
||||
monkeypatch.setattr(session, "send", _send)
|
||||
session._nudge_queue.enqueue("watch_triggered", "poll-4", "any")
|
||||
session._nudge_queue.enqueue("correction", "user advisory", "user")
|
||||
with pytest.raises(RuntimeError):
|
||||
session.deliver_wake_nudge_from_queue()
|
||||
# (b) bounded: nothing left that could re-trigger the wake gate.
|
||||
assert not session._nudge_queue.has_pending(WAKE_PENDING), (
|
||||
"a failed wake must not leave wake-eligible entries (respawn hot loop)"
|
||||
)
|
||||
assert [t for t, _x in session._nudge_queue.pending(channel="quiet")] == ["watch_triggered"]
|
||||
# A NEWER event lands after the failure...
|
||||
session._nudge_queue.enqueue("watch_triggered", "poll-5", "any")
|
||||
session.deliver_wake_nudge_from_queue()
|
||||
texts = [e["text"] for e in seen["reminders"]]
|
||||
# (a) ...and the retry renders old-before-new despite the round trip.
|
||||
assert texts.index("poll-4") < texts.index("poll-5")
|
||||
|
||||
|
||||
def test_exit_notice_emits_end_to_end_as_system_turn(session):
|
||||
"""THE test whose absence hid an undeliverable notice for six review
|
||||
rounds: drive the notice through REAL emission (make_system_turn +
|
||||
_append_system_turn), not just queue assertions — an unregistered
|
||||
``_source`` raises ValueError only at this layer."""
|
||||
_start_background(session, "echo done")
|
||||
assert _wait_until(
|
||||
lambda: any(t == "background_shell_exit" for t, _ in session._nudge_queue.pending())
|
||||
)
|
||||
from turnstone.core.trajectory import Role
|
||||
|
||||
before = len(session.messages)
|
||||
session._emit_pending_user_nudges() # must not raise
|
||||
new_turns = session.messages[before:]
|
||||
assert any(
|
||||
turn.role is Role.SYSTEM and turn.source == "background_shell_exit" for turn in new_turns
|
||||
), f"exit notice must land as a first-class system turn, got {new_turns!r}"
|
||||
|
||||
|
||||
def test_cli_exit_closes_every_loaded_session():
|
||||
"""CLI exit must reap background shells in EVERY workstream, not just
|
||||
the active one — a server started before /new must not outlive /exit."""
|
||||
from unittest.mock import MagicMock
|
||||
|
||||
from turnstone.cli import _close_all_sessions
|
||||
|
||||
ws_a, ws_b, ws_never_loaded = MagicMock(), MagicMock(), MagicMock()
|
||||
ws_never_loaded.session = None
|
||||
ws_a.session.close.side_effect = RuntimeError("bad teardown")
|
||||
manager = MagicMock()
|
||||
manager.list_all.return_value = [ws_a, ws_b, ws_never_loaded]
|
||||
_close_all_sessions(manager) # must not raise
|
||||
ws_a.session.close.assert_called_once()
|
||||
ws_b.session.close.assert_called_once(), "one bad teardown must not stop the rest"
|
||||
# Signal phase ran for every loaded session, before any close.
|
||||
ws_a.session._background_shells.signal_all.assert_called_once()
|
||||
ws_b.session._background_shells.signal_all.assert_called_once()
|
||||
|
||||
|
||||
def test_cli_exit_ctrl_c_does_not_abort_the_reap():
|
||||
"""Ctrl-C during the close phase must not escape the helper: the kill
|
||||
signals already landed on every session in phase 1, and an escaping
|
||||
KeyboardInterrupt would also skip MCP/registry shutdown in main()."""
|
||||
from unittest.mock import MagicMock
|
||||
|
||||
from turnstone.cli import _close_all_sessions
|
||||
|
||||
ws_a, ws_b = MagicMock(), MagicMock()
|
||||
ws_a.session.close.side_effect = KeyboardInterrupt
|
||||
manager = MagicMock()
|
||||
manager.list_all.return_value = [ws_a, ws_b]
|
||||
_close_all_sessions(manager) # must not raise
|
||||
ws_a.session._background_shells.signal_all.assert_called_once()
|
||||
(
|
||||
ws_b.session._background_shells.signal_all.assert_called_once(),
|
||||
("signals must land on every session before the interruptible close phase"),
|
||||
)
|
||||
|
||||
|
||||
def test_non_string_reminder_text_drops_silently(session):
|
||||
"""A dict reminder with non-str text must drop at the rail, not
|
||||
TypeError out of the dispatch closure (WatchRunner would re-fire the
|
||||
row every tick)."""
|
||||
runner = type(
|
||||
"R",
|
||||
(),
|
||||
{
|
||||
"set_dispatch_fn": lambda self, ws, fn: None,
|
||||
"remove_dispatch_fn": lambda self, ws, owner=None: None,
|
||||
},
|
||||
)()
|
||||
session.set_watch_runner(runner)
|
||||
session._watch_dispatch_fn({"text": 123, "watch_name": "w"}, "watch-1") # must not raise
|
||||
assert session._nudge_queue.pending() == []
|
||||
|
||||
|
||||
def test_string_typed_stop_on_error_is_honored(session):
|
||||
"""One coercion dialect for every bash boolean: a string-typed
|
||||
stop_on_error must add set -e in both branches, not silently drop it."""
|
||||
fg = session._prepare_bash("f1", {"command": "echo hi", "stop_on_error": "true"})
|
||||
assert fg["stop_on_error"] is True
|
||||
bg = session._prepare_bash(
|
||||
"b1", {"command": "echo hi", "run_in_background": True, "stop_on_error": "true"}
|
||||
)
|
||||
assert bg["stop_on_error"] is True
|
||||
|
||||
|
||||
def test_non_dict_watch_reminder_drops_silently(session):
|
||||
"""The rebuilt dispatch closure must drop a non-dict reminder like the
|
||||
old code did — a TypeError would make WatchRunner hold and re-fire the
|
||||
row every tick."""
|
||||
runner = type(
|
||||
"R",
|
||||
(),
|
||||
{
|
||||
"set_dispatch_fn": lambda self, ws, fn: None,
|
||||
"remove_dispatch_fn": lambda self, ws, owner=None: None,
|
||||
},
|
||||
)()
|
||||
session.set_watch_runner(runner)
|
||||
dispatch = session._watch_dispatch_fn
|
||||
dispatch("not a dict", "watch-1") # must not raise
|
||||
assert session._nudge_queue.pending() == []
|
||||
|
||||
|
||||
def test_truthy_flag_dialect_is_unified():
|
||||
"""One coercion dialect file-wide — 'on' and nonzero numbers count, so a
|
||||
provider quirk honored on coordinator tools is honored on bash too."""
|
||||
from turnstone.core.session import _is_truthy_flag
|
||||
|
||||
assert _is_truthy_flag(True)
|
||||
assert _is_truthy_flag("on")
|
||||
assert _is_truthy_flag(2)
|
||||
assert not _is_truthy_flag("off")
|
||||
assert not _is_truthy_flag(0)
|
||||
assert not _is_truthy_flag(None)
|
||||
assert not _is_truthy_flag(False)
|
||||
|
||||
|
||||
def test_bash_output_notes_clipped_lines_under_filter(session):
|
||||
_start_background(session, "printf 'x%.0s' $(seq 1 5000); echo tail")
|
||||
shell = _only_shell(session)
|
||||
assert _wait_until(lambda: shell.status == "completed")
|
||||
prepared = session._prepare_bash_output("r", {"id": shell.shell_id, "filter": "zzz"})
|
||||
_cid, output = prepared["execute"](prepared)
|
||||
assert "partially visible" in output
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# task_agent scoping
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_task_agent_shells_are_owner_scoped_and_reaped(session, monkeypatch):
|
||||
seen = {}
|
||||
|
||||
def fake_run_agent(agent_turns, label="task", **kwargs):
|
||||
out = _start_background(session, "sleep 60", call_id="sub-bash")
|
||||
seen["start_output"] = out
|
||||
agent_shells = session._background_shells.shells(owner="task-1")
|
||||
seen["agent_shells"] = list(agent_shells)
|
||||
seen["pid"] = agent_shells[0].pid if agent_shells else None
|
||||
# The sub-agent's shell is invisible to the main scope.
|
||||
seen["visible_to_parent"] = [s.shell_id for s in session._background_shells.shells()]
|
||||
return "agent done"
|
||||
|
||||
monkeypatch.setattr(session, "_run_agent", fake_run_agent)
|
||||
call_id, result = session._exec_task({"call_id": "task-1", "prompt": "start a server"})
|
||||
assert "agent done" in result
|
||||
assert seen["agent_shells"], "shell spawned inside the agent must carry its owner"
|
||||
# Scope honesty in the start message: the sub-agent must not promise its
|
||||
# caller a server that dies the moment it returns.
|
||||
assert "terminated when the agent finishes" in seen["start_output"]
|
||||
assert seen["visible_to_parent"] == []
|
||||
assert seen["pid"] is not None
|
||||
assert _wait_until(lambda: not _pid_alive(seen["pid"])), (
|
||||
"sub-agent shells must be reaped when the agent finishes"
|
||||
)
|
||||
|
||||
|
||||
def test_task_agent_cannot_touch_parent_shells(session, monkeypatch):
|
||||
_start_background(session, "sleep 60", call_id="parent-bash")
|
||||
parent_shell = _only_shell(session)
|
||||
seen = {}
|
||||
|
||||
def fake_run_agent(agent_turns, label="task", **kwargs):
|
||||
prepared = session._prepare_bash_output("r", {"id": parent_shell.shell_id})
|
||||
seen["read_output"] = prepared["execute"](prepared)[1]
|
||||
prepared_kill = session._prepare_kill_shell("k", {"id": parent_shell.shell_id})
|
||||
seen["kill_output"] = prepared_kill["execute"](prepared_kill)[1]
|
||||
return "done"
|
||||
|
||||
monkeypatch.setattr(session, "_run_agent", fake_run_agent)
|
||||
session._exec_task({"call_id": "task-1", "prompt": "snoop"})
|
||||
assert "no background shell" in seen["read_output"].lower()
|
||||
assert "no background shell" in seen["kill_output"].lower()
|
||||
assert _pid_alive(parent_shell.pid), "agent must not be able to kill a parent shell"
|
||||
|
||||
|
||||
def test_parent_scope_restored_after_task_agent(session, monkeypatch):
|
||||
monkeypatch.setattr(session, "_run_agent", lambda *a, **k: "done")
|
||||
session._exec_task({"call_id": "task-1", "prompt": "noop"})
|
||||
output = _start_background(session, "sleep 30", call_id="after-task")
|
||||
assert "bash_1" in output
|
||||
assert _only_shell(session).owner is None
|
||||
@@ -0,0 +1,165 @@
|
||||
"""Regression tests for the bash tool hanging on a backgrounded child.
|
||||
|
||||
A bash command that backgrounds a long-lived process (``server &``,
|
||||
``python -m http.server &``, any daemon) used to wedge the whole workstream
|
||||
forever: the child inherits the tool's stdout/stderr pipe, so the foreground
|
||||
read never hit EOF, and the timeout watchdog bailed the moment the tracked
|
||||
``bash`` exited. ``_exec_bash`` now waits on the tracked process (not pipe
|
||||
EOF) bounded by ``tool_timeout`` and kills the whole session group on exit, so
|
||||
the call always returns and never leaks the background child.
|
||||
"""
|
||||
|
||||
import threading
|
||||
import time
|
||||
|
||||
from tests._proc_helpers import kill_pid as _kill_pid
|
||||
from tests._proc_helpers import pid_alive as _pid_alive
|
||||
from tests._session_helpers import NullUI, make_session
|
||||
from turnstone.core.trajectory import EffectStatus
|
||||
|
||||
|
||||
def _run_in_thread(fn, timeout):
|
||||
"""Run ``fn`` in a daemon thread; return ``(finished, result)``."""
|
||||
box = {}
|
||||
|
||||
def _target():
|
||||
box["result"] = fn()
|
||||
|
||||
t = threading.Thread(target=_target, daemon=True)
|
||||
t.start()
|
||||
t.join(timeout)
|
||||
return (not t.is_alive()), box.get("result")
|
||||
|
||||
|
||||
def test_backgrounded_child_does_not_hang_and_is_reaped(tmp_path):
|
||||
"""Foreground exits immediately but leaves ``sleep 60 &`` holding the pipe.
|
||||
|
||||
Old behaviour: infinite hang (EOF never arrives, watchdog bails once the
|
||||
tracked bash exits). New behaviour: returns promptly and the background
|
||||
child is reaped by the session-group kill.
|
||||
"""
|
||||
pidfile = str(tmp_path / "bg.pid")
|
||||
# A generous tool_timeout proves the return comes from foreground-exit, not
|
||||
# from the deadline firing.
|
||||
session = make_session(tool_timeout=30)
|
||||
command = f"sleep 60 & echo $! > {pidfile}; echo done"
|
||||
bg_pid = None
|
||||
try:
|
||||
finished, result = _run_in_thread(
|
||||
lambda: session._exec_bash({"call_id": "c1", "command": command}),
|
||||
timeout=15,
|
||||
)
|
||||
assert finished, "_exec_bash hung on a backgrounded child"
|
||||
assert result is not None
|
||||
call_id, output = result
|
||||
assert call_id == "c1"
|
||||
assert "done" in output
|
||||
|
||||
# The backgrounded process must have been reaped by the group kill.
|
||||
with open(pidfile) as f:
|
||||
bg_pid = int(f.read().strip())
|
||||
deadline = time.monotonic() + 5
|
||||
while _pid_alive(bg_pid) and time.monotonic() < deadline:
|
||||
time.sleep(0.05)
|
||||
assert not _pid_alive(bg_pid), f"backgrounded child {bg_pid} leaked"
|
||||
finally:
|
||||
if bg_pid is not None:
|
||||
_kill_pid(bg_pid)
|
||||
|
||||
|
||||
def test_timeout_still_fires_with_backgrounded_child():
|
||||
"""A silent foreground command plus a backgrounded child still hits the
|
||||
deadline: the watchdog kills the whole group and the result reads UNKNOWN
|
||||
(the ``unknown, never none`` timeout discipline)."""
|
||||
session = make_session(tool_timeout=1)
|
||||
command = "sleep 60 & sleep 60"
|
||||
|
||||
finished, result = _run_in_thread(
|
||||
lambda: session._exec_bash({"call_id": "c1", "command": command}),
|
||||
timeout=10,
|
||||
)
|
||||
assert finished, "_exec_bash did not return at its deadline"
|
||||
assert result is not None
|
||||
call_id, output = result
|
||||
assert call_id == "c1"
|
||||
assert "timed out" in output.lower()
|
||||
assert "UNKNOWN" in output
|
||||
assert session._tool_status.get("c1") is EffectStatus.UNKNOWN
|
||||
|
||||
|
||||
def test_undecodable_output_is_preserved_not_swallowed():
|
||||
"""Undecodable bytes on stdout must not silently vanish.
|
||||
|
||||
The drain's broad ``except (ValueError, OSError)`` would otherwise catch the
|
||||
``UnicodeDecodeError`` (a ``ValueError``) and kill the thread before any line
|
||||
was yielded — dropping ALL output and reporting a clean success. ``Popen``
|
||||
now decodes with ``errors="replace"`` so output always survives.
|
||||
"""
|
||||
session = make_session(tool_timeout=30)
|
||||
# Valid lines bracketing a raw invalid-UTF-8 byte sequence.
|
||||
command = r"printf 'before\n'; printf '\xff\xfe'; printf 'after\n'"
|
||||
finished, result = _run_in_thread(
|
||||
lambda: session._exec_bash({"call_id": "c1", "command": command}),
|
||||
timeout=15,
|
||||
)
|
||||
assert finished
|
||||
assert result is not None
|
||||
_call_id, output = result
|
||||
assert output != "(no output)"
|
||||
assert "before" in output
|
||||
assert "after" in output
|
||||
|
||||
|
||||
def test_stdout_streams_to_ui_from_drain_thread():
|
||||
"""stdout chunks are now emitted from the drain thread; they must still reach
|
||||
``on_tool_output_chunk``."""
|
||||
chunks: list[str] = []
|
||||
|
||||
class RecordingUI(NullUI):
|
||||
def on_tool_output_chunk(self, call_id, chunk):
|
||||
chunks.append(chunk)
|
||||
|
||||
session = make_session(tool_timeout=30, ui=RecordingUI())
|
||||
finished, result = _run_in_thread(
|
||||
lambda: session._exec_bash({"call_id": "c1", "command": "echo streamed-line"}),
|
||||
timeout=15,
|
||||
)
|
||||
assert finished
|
||||
assert any("streamed-line" in c for c in chunks)
|
||||
|
||||
|
||||
def test_cancel_midbash_reports_unknown():
|
||||
"""An external ``cancel()`` during a running bash unblocks the process-bounded
|
||||
wait and reports UNKNOWN (unknown-never-none), not a clean result."""
|
||||
session = make_session(tool_timeout=30)
|
||||
|
||||
def _cancel_soon():
|
||||
time.sleep(0.5)
|
||||
session.cancel()
|
||||
|
||||
threading.Thread(target=_cancel_soon, daemon=True).start()
|
||||
finished, result = _run_in_thread(
|
||||
lambda: session._exec_bash({"call_id": "c1", "command": "sleep 30"}),
|
||||
timeout=15,
|
||||
)
|
||||
assert finished, "cancel did not unblock _exec_bash"
|
||||
assert result is not None
|
||||
_call_id, output = result
|
||||
assert "cancelled" in output.lower()
|
||||
assert session._tool_status.get("c1") is EffectStatus.UNKNOWN
|
||||
|
||||
|
||||
def test_popen_failure_reports_cleanly(monkeypatch):
|
||||
"""If ``Popen`` itself raises, the ``finally`` must not mask the real error
|
||||
with ``UnboundLocalError`` — ``proc`` is pre-bound to ``None``."""
|
||||
from turnstone.core import session as session_mod
|
||||
|
||||
session = make_session(tool_timeout=30)
|
||||
|
||||
def _boom(*args, **kwargs):
|
||||
raise OSError("cannot fork")
|
||||
|
||||
monkeypatch.setattr(session_mod.subprocess, "Popen", _boom)
|
||||
call_id, output = session._exec_bash({"call_id": "c1", "command": "echo hi"})
|
||||
assert call_id == "c1"
|
||||
assert "cannot fork" in output
|
||||
@@ -13,7 +13,7 @@ from turnstone.core.session import (
|
||||
ChatSession,
|
||||
GenerationCancelled,
|
||||
_CancelRef,
|
||||
_effect_status_meta,
|
||||
_tool_turn_meta,
|
||||
)
|
||||
from turnstone.core.trajectory import (
|
||||
EffectStatus,
|
||||
@@ -1183,8 +1183,13 @@ class TestEffectStatusPersistence:
|
||||
effect-record appendix — the ledger persists for audit)."""
|
||||
|
||||
def test_effect_status_meta_envelope(self):
|
||||
assert _effect_status_meta(None) is None
|
||||
assert json.loads(_effect_status_meta(EffectStatus.UNKNOWN)) == {"effect_status": "unknown"}
|
||||
assert _tool_turn_meta(None) is None
|
||||
assert json.loads(_tool_turn_meta(EffectStatus.UNKNOWN)) == {"effect_status": "unknown"}
|
||||
assert json.loads(_tool_turn_meta(None, {"kind": "web"})) == {"preview": {"kind": "web"}}
|
||||
assert json.loads(_tool_turn_meta(EffectStatus.UNKNOWN, {"kind": "web"})) == {
|
||||
"effect_status": "unknown",
|
||||
"preview": {"kind": "web"},
|
||||
}
|
||||
|
||||
def test_reconstruct_routes_tool_effect_status(self):
|
||||
from turnstone.core.storage._utils import reconstruct_turns
|
||||
|
||||
@@ -2690,3 +2690,56 @@ class TestCollectorMCPAggregation:
|
||||
assert overview["mcp_servers"] == 3
|
||||
assert overview["mcp_resources"] == 10
|
||||
assert overview["mcp_prompts"] == 7
|
||||
|
||||
|
||||
class TestProxyGetHeaderPassThrough:
|
||||
"""The generic /node/{id} GET proxy must carry the node's hardening
|
||||
headers through — dropping Content-Security-Policy would serve previewed
|
||||
attacker HTML from the CONSOLE origin with no CSP sandbox (review
|
||||
finding, preview-pane branch)."""
|
||||
|
||||
def test_security_headers_forwarded(self, monkeypatch):
|
||||
from types import SimpleNamespace
|
||||
from unittest.mock import MagicMock
|
||||
|
||||
import httpx
|
||||
|
||||
from turnstone.console import server as csrv
|
||||
|
||||
upstream = httpx.Response(
|
||||
200,
|
||||
content=b"<html>page</html>",
|
||||
headers={
|
||||
"content-type": "text/html; charset=utf-8",
|
||||
"content-security-policy": "sandbox",
|
||||
"x-content-type-options": "nosniff",
|
||||
"content-disposition": 'inline; filename="p"',
|
||||
"cache-control": "private, no-store",
|
||||
"server": "upstream-internal", # hop metadata: must NOT pass
|
||||
},
|
||||
request=httpx.Request("GET", "http://n:1/x"),
|
||||
)
|
||||
|
||||
async def _mock_get(*a, **kw):
|
||||
return upstream
|
||||
|
||||
proxy_client = MagicMock(spec=httpx.AsyncClient)
|
||||
proxy_client.get = MagicMock(side_effect=_mock_get)
|
||||
request = SimpleNamespace(
|
||||
app=SimpleNamespace(state=SimpleNamespace(proxy_client=proxy_client)),
|
||||
url=SimpleNamespace(query=""),
|
||||
)
|
||||
monkeypatch.setattr(csrv, "_proxy_auth_headers", lambda r: {})
|
||||
|
||||
resp = asyncio.run(csrv._proxy_get(request, "http://n:1", "v1/api/x"))
|
||||
|
||||
assert resp.status_code == 200
|
||||
assert resp.headers["content-security-policy"] == "sandbox"
|
||||
assert resp.headers["x-content-type-options"] == "nosniff"
|
||||
assert resp.headers["content-disposition"] == 'inline; filename="p"'
|
||||
assert resp.headers["cache-control"] == "private, no-store"
|
||||
assert resp.headers["content-type"].startswith("text/html")
|
||||
assert (
|
||||
"server" not in {k.lower() for k in resp.headers}
|
||||
or resp.headers.get("server") != "upstream-internal"
|
||||
)
|
||||
|
||||
@@ -114,11 +114,16 @@ def test_coord_on_aux_usage_leaves_live_counters_untouched() -> None:
|
||||
assert ui._ws_context_ratio == 0.0
|
||||
|
||||
|
||||
def test_coord_on_content_token_accumulates() -> None:
|
||||
def test_coord_on_content_token_accumulates(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""Pre-lift coord ``on_content_token`` only enqueued; lift turns it
|
||||
into the same per-ws accumulator WebUI uses so the collector
|
||||
broadcast can piggyback the joined turn content on the IDLE
|
||||
state-change event."""
|
||||
state-change event.
|
||||
|
||||
Batch window forced to 0 (per-token flush) — pins the accumulator
|
||||
wiring, not the batching cadence (test_sse_token_batching.py)."""
|
||||
|
||||
monkeypatch.setattr("turnstone.core.session_ui_base._TOKEN_BATCH_WINDOW_SECS", 0.0)
|
||||
ui = ConsoleCoordinatorUI(ws_id="coord-ws", user_id="u1")
|
||||
ui.on_content_token("Hello ")
|
||||
ui.on_content_token("world")
|
||||
|
||||
@@ -245,6 +245,24 @@ def test_cleanup_ui_tolerates_missing_session_and_ui() -> None:
|
||||
ws.session = None
|
||||
ws.ui = None
|
||||
adapter.cleanup_ui(ws) # no crash
|
||||
assert ws._closed is True # still marked dead
|
||||
|
||||
|
||||
def test_cleanup_ui_marks_workstream_closed() -> None:
|
||||
"""Every teardown path — close, close_idle, EVICTION, delete,
|
||||
discard — funnels through cleanup_ui, which marks the object dead
|
||||
under ``ws._lock`` BEFORE the teardown body runs. The wake paths
|
||||
that hold OBJECT references (the watch ``wake_fn``,
|
||||
``session_worker``'s exit backstop) gate on ``_closed``, and
|
||||
``session_worker.send`` re-checks it under the same lock — without
|
||||
this write here, a wake racing an eviction or delete (which never
|
||||
set the flag) would spawn a full unattended turn on the torn-down
|
||||
session."""
|
||||
adapter, _ = _make_adapter()
|
||||
ws = _make_ws()
|
||||
assert ws._closed is False
|
||||
adapter.cleanup_ui(ws)
|
||||
assert ws._closed is True
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@@ -35,7 +35,14 @@ class _StubUI:
|
||||
def on_error(self, msg: str) -> None:
|
||||
self.errors.append(msg)
|
||||
|
||||
def on_tool_result(self, call_id: str, name: str, output: str, is_error: bool = False) -> None:
|
||||
def on_tool_result(
|
||||
self,
|
||||
call_id: str,
|
||||
name: str,
|
||||
output: str,
|
||||
is_error: bool = False,
|
||||
preview: dict[str, Any] | None = None,
|
||||
) -> None:
|
||||
self.tool_results.append((call_id, name, output, is_error))
|
||||
|
||||
# Other SessionUI methods — only stubs, not exercised here.
|
||||
|
||||
@@ -28,8 +28,10 @@ from unittest.mock import MagicMock, patch
|
||||
|
||||
import pytest
|
||||
|
||||
from tests._helpers import wait_until as _wait_until
|
||||
from tests.test_session_manager import FakeStorage
|
||||
from turnstone.core.idle_nudge_watcher import IdleNudgeWatcher
|
||||
from turnstone.core import session_worker
|
||||
from turnstone.core.idle_nudge_watcher import IdleNudgeWatcher, wake_workstream_if_pending
|
||||
from turnstone.core.session import ChatSession
|
||||
from turnstone.core.session_manager import SessionManager
|
||||
from turnstone.core.trajectory import dicts_from_turns, turn_from_dict
|
||||
@@ -299,6 +301,72 @@ def test_idle_event_with_empty_queue_does_not_dispatch_wake(real_mgr, tmp_db):
|
||||
watcher.shutdown()
|
||||
|
||||
|
||||
def test_watch_fire_on_already_idle_session_drives_wake_send(real_mgr, tmp_db):
|
||||
"""A watch firing on an ALREADY-idle workstream sees no IDLE
|
||||
transition, so :class:`IdleNudgeWatcher` never re-checks the queue —
|
||||
the dispatch closure's ``wake_fn`` must drive the wake itself.
|
||||
|
||||
Boundary path under test (only the LLM stream is patched):
|
||||
dispatch closure (real, built by ``set_watch_runner``)
|
||||
→ NudgeQueue.enqueue (real)
|
||||
→ wake_fn → wake_workstream_if_pending (real)
|
||||
→ session_worker.send (real) → daemon thread
|
||||
→ ChatSession.deliver_wake_nudge_from_queue (real)
|
||||
→ ChatSession.send("") → watch_triggered system turn in history
|
||||
"""
|
||||
mgr, _adapter = real_mgr
|
||||
ws = mgr.create(user_id="u1", name="watch-wake-int", skill=None)
|
||||
assert ws.session is not None
|
||||
|
||||
captured: dict[str, Any] = {}
|
||||
|
||||
class _StubRunner:
|
||||
def set_dispatch_fn(self, ws_id: str, fn: Any) -> None:
|
||||
captured["fn"] = fn
|
||||
|
||||
# Production wiring shape (server.py): wake_fn closes over the
|
||||
# Workstream OBJECT — not its id — so eviction+restore id drift
|
||||
# can't strand the wake.
|
||||
ws.session.set_watch_runner(
|
||||
_StubRunner(), wake_fn=lambda: wake_workstream_if_pending(ws, trigger="watch-fire")
|
||||
)
|
||||
|
||||
with (
|
||||
patch.object(ws.session, "_create_stream_with_retry", return_value=iter([])),
|
||||
patch.object(
|
||||
ws.session,
|
||||
"_stream_response",
|
||||
return_value={"role": "assistant", "content": "ok"},
|
||||
),
|
||||
patch.object(ws.session, "_update_token_table"),
|
||||
patch.object(ws.session, "_print_status_line"),
|
||||
patch.object(ws.session, "_visible_memory_count", return_value=0),
|
||||
patch("turnstone.core.session.save_message"),
|
||||
):
|
||||
ws.session._title_generated = True
|
||||
# Idle all along — no worker, and no state transition coming.
|
||||
assert ws.state is WorkstreamState.IDLE
|
||||
|
||||
# Simulate the WatchRunner poll thread delivering a fire.
|
||||
captured["fn"]({"type": "watch_triggered", "text": "deploy finished: OK"}, "watch-1")
|
||||
|
||||
_wait_for_worker_done(ws)
|
||||
|
||||
# Queue drained by the wake — not parked until the next user message.
|
||||
assert len(ws.session._nudge_queue) == 0
|
||||
|
||||
msgs = dicts_from_turns(ws.session.messages)
|
||||
user_msgs = [m for m in msgs if m.get("role") == "user"]
|
||||
assert user_msgs, "expected a synthesized user message from the wake"
|
||||
assert user_msgs[-1]["content"] == ""
|
||||
assert user_msgs[-1].get("_source") == "system_nudge"
|
||||
sys_turns = [m for m in msgs if m.get("role") == "system"]
|
||||
assert any(
|
||||
m.get("_source") == "watch_triggered" and "deploy finished: OK" in m.get("content", "")
|
||||
for m in sys_turns
|
||||
), f"expected a watch_triggered system turn, got {sys_turns!r}"
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def coord_mgr() -> tuple[SessionManager, _BuildRealSessionAdapter, FakeStorage]:
|
||||
"""Real coord-side SessionManager with the adapter's kind set to
|
||||
@@ -411,3 +479,120 @@ def test_coord_idle_with_active_children_emits_envelope_via_real_managers(coord_
|
||||
finally:
|
||||
watcher.shutdown()
|
||||
observer.shutdown()
|
||||
|
||||
|
||||
def test_coord_idle_emitted_from_worker_thread_still_wakes(coord_mgr, tmp_db):
|
||||
"""The production-shaped race the test above does NOT exercise: in
|
||||
production, IDLE is emitted from INSIDE the worker (``set_state``
|
||||
subscribers fire on the calling thread — the coord's send emits IDLE
|
||||
before its worker exits). The watcher's wake dispatch therefore
|
||||
lands on ``session_worker.send``'s reuse path while the
|
||||
transitioning worker still owns the flag, and no-ops. Without the
|
||||
ownership-clear backstop the ``idle_children`` nudge strands until
|
||||
the next user message — a coord that forgot ``wait_for_workstream``
|
||||
never revives.
|
||||
|
||||
Boundary path under test:
|
||||
worker thread: mgr.set_state(IDLE)
|
||||
→ observer enqueues (real) → watcher wake no-ops (worker owns flag)
|
||||
→ run() returns → session_worker._runner finally clears the flag
|
||||
→ _retry_pending_wake → wake_workstream_if_pending (real)
|
||||
→ wake daemon → deliver_wake_nudge_from_queue → send("")
|
||||
→ idle_children system turn in history
|
||||
"""
|
||||
from turnstone.console.coordinator_idle_observer import CoordinatorIdleObserver
|
||||
from turnstone.core.workstream import WorkstreamKind as _Kind
|
||||
|
||||
mgr, adapter, storage = coord_mgr
|
||||
observer = CoordinatorIdleObserver(mgr, storage)
|
||||
observer.start()
|
||||
watcher = IdleNudgeWatcher(mgr)
|
||||
watcher.start()
|
||||
|
||||
try:
|
||||
coord = mgr.create(user_id="u1", name="parent-coord-2", skill=None)
|
||||
assert coord.session is not None
|
||||
|
||||
storage.register_workstream(
|
||||
"child-x",
|
||||
user_id="u1",
|
||||
name="crawl-docs",
|
||||
kind=_Kind.INTERACTIVE,
|
||||
parent_ws_id=coord.id,
|
||||
state="running",
|
||||
)
|
||||
|
||||
coord.session.messages.append(turn_from_dict({"role": "user", "content": "spawn 1"}))
|
||||
coord.session.messages.append(turn_from_dict({"role": "assistant", "content": "ok"}))
|
||||
|
||||
with (
|
||||
patch.object(coord.session, "_create_stream_with_retry", return_value=iter([])),
|
||||
patch.object(
|
||||
coord.session,
|
||||
"_stream_response",
|
||||
return_value={"role": "assistant", "content": "ack"},
|
||||
),
|
||||
patch.object(coord.session, "_full_messages", return_value=[]),
|
||||
patch.object(coord.session, "_update_token_table"),
|
||||
patch.object(coord.session, "_print_status_line"),
|
||||
patch.object(coord.session, "_visible_memory_count", return_value=0),
|
||||
patch("turnstone.core.session.save_message"),
|
||||
):
|
||||
coord.session._title_generated = True
|
||||
|
||||
# Drive the IDLE transition from INSIDE a session_worker
|
||||
# worker, as production does.
|
||||
ok = session_worker.send(
|
||||
coord,
|
||||
enqueue=lambda: None,
|
||||
run=lambda: mgr.set_state(coord.id, WorkstreamState.IDLE),
|
||||
thread_name="coord-send-sim",
|
||||
)
|
||||
assert ok is True
|
||||
# Without the backstop the queue never drains (the watcher's
|
||||
# transition-time wake no-opped against the sim worker) and
|
||||
# this poll times out. Queue-empty implies the wake worker's
|
||||
# drain ran, so the follow-up flag poll waits for ITS exit.
|
||||
_wait_until(lambda: len(coord.session._nudge_queue) == 0)
|
||||
_wait_for_worker_done(coord)
|
||||
|
||||
# Queue drained by the wake, not waiting on the next user message.
|
||||
assert len(coord.session._nudge_queue) == 0
|
||||
msgs = dicts_from_turns(coord.session.messages)
|
||||
user_msgs = [m for m in msgs if m.get("role") == "user"]
|
||||
wake_msg = user_msgs[-1]
|
||||
assert wake_msg["content"] == ""
|
||||
assert wake_msg.get("_source") == "system_nudge"
|
||||
idle_turns = [
|
||||
m for m in msgs if m.get("role") == "system" and m["_source"] == "idle_children"
|
||||
]
|
||||
assert len(idle_turns) == 1
|
||||
assert "crawl-docs" in idle_turns[0]["content"]
|
||||
assert "wait_for_workstream" in idle_turns[0]["content"]
|
||||
finally:
|
||||
watcher.shutdown()
|
||||
observer.shutdown()
|
||||
|
||||
|
||||
def test_wake_delivery_contains_generation_cancelled(tmp_db):
|
||||
"""A close/force-cancel racing the wake turn raises
|
||||
``GenerationCancelled`` (a BaseException) out of ``send("")`` — the
|
||||
wake method must contain it: it IS the wake worker's ``run()``
|
||||
closure, and ``session_worker._runner`` catches only ``Exception``,
|
||||
so an escape would land in ``threading.excepthook`` as stderr noise
|
||||
on every close-vs-wake race."""
|
||||
from tests._helpers import make_chat_session
|
||||
from turnstone.core.session import GenerationCancelled
|
||||
|
||||
session = make_chat_session()
|
||||
session._nudge_queue.enqueue("idle_children", "kids waiting", "any")
|
||||
|
||||
def _cancelled_send(*_a: Any, **_k: Any) -> None:
|
||||
raise GenerationCancelled
|
||||
|
||||
session.send = _cancelled_send # type: ignore[method-assign]
|
||||
|
||||
session.deliver_wake_nudge_from_queue() # must not raise
|
||||
|
||||
assert session._wake_source_tag == ""
|
||||
assert session._wake_drained_reminders is None
|
||||
|
||||
@@ -9,13 +9,14 @@ module-level function to capture calls without spawning real threads.
|
||||
from __future__ import annotations
|
||||
|
||||
import contextlib
|
||||
import logging
|
||||
import threading
|
||||
from typing import Any
|
||||
from unittest.mock import patch
|
||||
|
||||
import pytest
|
||||
|
||||
from turnstone.core.idle_nudge_watcher import IdleNudgeWatcher
|
||||
from turnstone.core.idle_nudge_watcher import IdleNudgeWatcher, wake_workstream_if_pending
|
||||
from turnstone.core.nudge_queue import NudgeQueue
|
||||
from turnstone.core.workstream import WorkstreamState
|
||||
|
||||
@@ -32,6 +33,7 @@ class _FakeSession:
|
||||
class _FakeWorkstream:
|
||||
def __init__(self, ws_id: str = "ws-test") -> None:
|
||||
self.id = ws_id
|
||||
self.state = WorkstreamState.IDLE
|
||||
self.session: _FakeSession | None = _FakeSession()
|
||||
self._lock = threading.Lock()
|
||||
self._worker_running = False
|
||||
@@ -163,3 +165,154 @@ class TestIdleNudgeWatcher:
|
||||
watcher.start()
|
||||
watcher.shutdown()
|
||||
watcher.shutdown() # no error
|
||||
|
||||
|
||||
class TestWakeWorkstreamIfPending:
|
||||
"""Direct tests for the shared wake gate.
|
||||
|
||||
The IDLE-transition path (via the watcher) is covered above; these
|
||||
pin the gates the watch dispatch closure relies on when it calls
|
||||
the helper directly, with no state event involved.
|
||||
"""
|
||||
|
||||
def test_wakes_idle_ws_with_pending_entry(self, fake_mgr_and_ws):
|
||||
_mgr, ws = fake_mgr_and_ws
|
||||
ws.session._nudge_queue.enqueue("watch_triggered", "output", "any")
|
||||
with patch("turnstone.core.session_worker.send", return_value=True) as mock_send:
|
||||
assert wake_workstream_if_pending(ws) is True
|
||||
assert mock_send.call_count == 1
|
||||
kwargs = mock_send.call_args.kwargs
|
||||
assert kwargs["enqueue"]() is None
|
||||
kwargs["run"]()
|
||||
assert ws.session.deliver_wake_nudge_from_queue_called == 1
|
||||
assert kwargs["thread_name"].startswith("wake-nudge-")
|
||||
|
||||
def test_skips_session_none(self, fake_mgr_and_ws):
|
||||
_mgr, ws = fake_mgr_and_ws
|
||||
ws.session = None
|
||||
with patch("turnstone.core.session_worker.send") as mock_send:
|
||||
assert wake_workstream_if_pending(ws) is False
|
||||
assert mock_send.call_count == 0
|
||||
|
||||
def test_skips_closed_ws(self, fake_mgr_and_ws):
|
||||
"""A workstream mid-``close()`` must not get a wake spawned on
|
||||
its torn-down session, even while its ``state`` field still
|
||||
reads IDLE (there is no CLOSED member — close uses the
|
||||
``_closed`` tombstone)."""
|
||||
_mgr, ws = fake_mgr_and_ws
|
||||
ws.session._nudge_queue.enqueue("watch_triggered", "output", "any")
|
||||
ws._closed = True
|
||||
with patch("turnstone.core.session_worker.send") as mock_send:
|
||||
assert wake_workstream_if_pending(ws) is False
|
||||
assert mock_send.call_count == 0
|
||||
|
||||
def test_skips_non_idle_states(self, fake_mgr_and_ws):
|
||||
"""Busy states imply a live worker that drains at its own seams;
|
||||
ERROR stays parked for the operator — neither gets a wake."""
|
||||
_mgr, ws = fake_mgr_and_ws
|
||||
ws.session._nudge_queue.enqueue("watch_triggered", "output", "any")
|
||||
with patch("turnstone.core.session_worker.send") as mock_send:
|
||||
for state in (
|
||||
WorkstreamState.RUNNING,
|
||||
WorkstreamState.THINKING,
|
||||
WorkstreamState.ATTENTION,
|
||||
WorkstreamState.ERROR,
|
||||
):
|
||||
ws.state = state
|
||||
assert wake_workstream_if_pending(ws) is False
|
||||
assert mock_send.call_count == 0
|
||||
|
||||
def test_skips_tool_only_entries(self, fake_mgr_and_ws):
|
||||
"""Tool-channel entries belong to the next tool-result seam — a
|
||||
synthetic empty user turn can't drain them, so no wake."""
|
||||
_mgr, ws = fake_mgr_and_ws
|
||||
ws.session._nudge_queue.enqueue("tool_error", "check memories", "tool")
|
||||
with patch("turnstone.core.session_worker.send") as mock_send:
|
||||
assert wake_workstream_if_pending(ws) is False
|
||||
assert mock_send.call_count == 0
|
||||
|
||||
def test_refuses_non_nudgequeue_stub(self, fake_mgr_and_ws):
|
||||
"""The gate refuses on TYPE, not just presence: a mock session's
|
||||
auto-created ``_nudge_queue`` answers ``has_pending`` truthily
|
||||
while its ``deliver_wake_nudge_from_queue`` consumes nothing —
|
||||
with the worker-exit backstop re-running this gate after every
|
||||
exit, one worker on such a session would respawn wake workers
|
||||
forever (the storm that took down the full-suite CI run). Only
|
||||
a real :class:`NudgeQueue` carries the drain semantics the wake
|
||||
contract needs."""
|
||||
from unittest.mock import MagicMock
|
||||
|
||||
_mgr, ws = fake_mgr_and_ws
|
||||
ws.session._nudge_queue = MagicMock() # truthy has_pending, no real drain
|
||||
with patch("turnstone.core.session_worker.send") as mock_send:
|
||||
assert wake_workstream_if_pending(ws) is False
|
||||
assert mock_send.call_count == 0
|
||||
|
||||
def test_dispatched_path_logs_trigger(self, fake_mgr_and_ws, caplog):
|
||||
"""A fresh spawn — ``send`` returns True without touching the
|
||||
passed ``enqueue`` — emits ``nudge_wake.dispatched`` tagged with
|
||||
the trigger label (structlog renders the event name + ``%s``
|
||||
placeholders into ``msg``; substring-match like the sibling
|
||||
nudge_queue tests)."""
|
||||
_mgr, ws = fake_mgr_and_ws
|
||||
ws.session._nudge_queue.enqueue("watch_triggered", "output", "any")
|
||||
with (
|
||||
patch("turnstone.core.session_worker.send", return_value=True) as mock_send,
|
||||
caplog.at_level(logging.INFO, logger="turnstone.core.idle_nudge_watcher"),
|
||||
):
|
||||
assert wake_workstream_if_pending(ws, trigger="idle-transition") is True
|
||||
assert mock_send.call_count == 1
|
||||
dispatched = [r for r in caplog.records if "nudge_wake.dispatched" in r.getMessage()]
|
||||
assert len(dispatched) == 1
|
||||
assert dispatched[0].levelno == logging.INFO
|
||||
assert "trigger=" in dispatched[0].getMessage()
|
||||
# The reuse-path drop line must not appear on a fresh spawn.
|
||||
assert not any("nudge_wake.deferred_worker_busy" in r.getMessage() for r in caplog.records)
|
||||
|
||||
def test_deferred_path_logs_worker_busy(self, fake_mgr_and_ws, caplog):
|
||||
"""The reuse path — ``send`` invokes the passed ``enqueue`` and
|
||||
returns True — emits ``nudge_wake.deferred_worker_busy`` instead
|
||||
of ``dispatched``. The entry stays owed to the owning worker's
|
||||
exit backstop; the return value is still True."""
|
||||
_mgr, ws = fake_mgr_and_ws
|
||||
ws.session._nudge_queue.enqueue("watch_triggered", "output", "any")
|
||||
|
||||
def _reuse_send(_ws: Any, *, enqueue: Any, run: Any, thread_name: Any) -> bool:
|
||||
# Mimic a live worker owning the workstream: send routes the
|
||||
# wake to the no-op enqueue rather than spawning a daemon.
|
||||
enqueue()
|
||||
return True
|
||||
|
||||
with (
|
||||
patch("turnstone.core.session_worker.send", side_effect=_reuse_send) as mock_send,
|
||||
caplog.at_level(logging.INFO, logger="turnstone.core.idle_nudge_watcher"),
|
||||
):
|
||||
assert wake_workstream_if_pending(ws, trigger="idle-transition") is True
|
||||
assert mock_send.call_count == 1
|
||||
deferred = [
|
||||
r for r in caplog.records if "nudge_wake.deferred_worker_busy" in r.getMessage()
|
||||
]
|
||||
assert len(deferred) == 1
|
||||
assert deferred[0].levelno == logging.INFO
|
||||
assert "trigger=" in deferred[0].getMessage()
|
||||
assert not any("nudge_wake.dispatched" in r.getMessage() for r in caplog.records)
|
||||
|
||||
def test_refused_path_logs_refusal(self, fake_mgr_and_ws, caplog):
|
||||
"""``send`` refusing outright — its authoritative under-lock
|
||||
``_closed`` re-check caught a teardown the gate's lockless peek
|
||||
missed — emits ``nudge_wake.refused``: a dropped wake must stay
|
||||
traceable to its trigger, not vanish silently."""
|
||||
_mgr, ws = fake_mgr_and_ws
|
||||
ws.session._nudge_queue.enqueue("watch_triggered", "output", "any")
|
||||
with (
|
||||
patch("turnstone.core.session_worker.send", return_value=False) as mock_send,
|
||||
caplog.at_level(logging.INFO, logger="turnstone.core.idle_nudge_watcher"),
|
||||
):
|
||||
assert wake_workstream_if_pending(ws, trigger="watch-fire") is False
|
||||
assert mock_send.call_count == 1
|
||||
refused = [r for r in caplog.records if "nudge_wake.refused" in r.getMessage()]
|
||||
assert len(refused) == 1
|
||||
assert refused[0].levelno == logging.INFO
|
||||
assert "trigger=" in refused[0].getMessage()
|
||||
assert not any("nudge_wake.dispatched" in r.getMessage() for r in caplog.records)
|
||||
assert not any("nudge_wake.deferred_worker_busy" in r.getMessage() for r in caplog.records)
|
||||
|
||||
@@ -429,3 +429,210 @@ def test_sync_approval_state_prunes_orphan_cycles() -> None:
|
||||
assert "this.approvalCycles.delete(cid);" in tail, (
|
||||
"orphan pruning must delete the cycle from the Map"
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# SSE overflow recovery + close-on-hide (fast-stream corruption fixes)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_stream_overflow_case_counts_and_rate_limits() -> None:
|
||||
"""The server closes an overflowed stream after an id-less
|
||||
``stream_overflow`` frame; the pane must count it (field
|
||||
instrumentation for the drop-vs-render-wedge diagnosis) and route it
|
||||
through the reconnect limiter so a persistently slow consumer trips
|
||||
the degraded catch-up instead of churning reconnect/replay cycles."""
|
||||
body = _INTERACTIVE.read_text(encoding="utf-8")
|
||||
assert 'case "stream_overflow":' in body
|
||||
assert "this._noteStreamOverflow();" in body
|
||||
assert "_streamHealth = { overflows: 0, renderThrows: 0, malformedFrames: 0 }" in body
|
||||
# Both wedge-class catch sites increment the render-throw counter,
|
||||
# and the malformed-frame drop counts too — the C-OVERDETERMINED
|
||||
# instrumentation that tells drops apart from wedges in the field.
|
||||
assert body.count("this._streamHealth.renderThrows += 1;") == 2
|
||||
assert "this._streamHealth.malformedFrames += 1;" in body
|
||||
assert "this._streamHealth.overflows += 1;" in body
|
||||
|
||||
|
||||
def test_degraded_catchup_stops_live_stream_and_retries() -> None:
|
||||
"""Degraded catch-up contract: close the stream FIRST (which also
|
||||
clears any earlier degraded timer — disconnectSSE owns that), show a
|
||||
plain-language status, then arm the retry timer with a doubling
|
||||
cooldown. The retry must defer to the show edge when the tab is
|
||||
hidden (reopening into a throttled tab would overflow again)."""
|
||||
body = _INTERACTIVE.read_text(encoding="utf-8")
|
||||
m = re.search(r"_enterDegradedCatchup\(\)\s*\{(.*?)\n \}", body, re.S)
|
||||
assert m is not None, "_enterDegradedCatchup method not found"
|
||||
method = m.group(1)
|
||||
# Order matters: disconnect before arming the timer, or the fresh
|
||||
# timer would be cancelled by its own disconnect.
|
||||
assert method.index("this.disconnectSSE()") < method.index("this._degradedTimer = setTimeout")
|
||||
assert "Connection is slow" in method, "degraded state must use plain language"
|
||||
assert "DEGRADED_COOLDOWN_MAX_MS" in method
|
||||
assert "document.hidden" in method
|
||||
# disconnectSSE owns the timer teardown (ws-switch / giveUp / destroy
|
||||
# all supersede a pending degraded retry through it).
|
||||
dis = re.search(r"disconnectSSE\(\)\s*\{(.*?)\n \}", body, re.S)
|
||||
assert dis is not None
|
||||
assert "clearTimeout(this._degradedTimer)" in dis.group(1)
|
||||
|
||||
|
||||
def test_visibilitychange_closes_on_hide_reconnects_on_show() -> None:
|
||||
"""Close-on-hide / replay-on-show: a hidden tab's throttled drain is
|
||||
the likeliest slow consumer behind server-side overflow (the old
|
||||
"PR-G closes those connections on hide" comment described a handler
|
||||
that never existed). The pane installs one visibilitychange
|
||||
listener, marks ITS OWN hide-closes via ``_hiddenDisconnect`` so a
|
||||
show edge never resurrects a deliberately-closed stream, and the
|
||||
factory's destroy removes the listener (it strongly references the
|
||||
pane)."""
|
||||
body = _INTERACTIVE.read_text(encoding="utf-8")
|
||||
assert 'document.addEventListener("visibilitychange", this._visHandler);' in body
|
||||
assert 'document.removeEventListener("visibilitychange", this._visHandler);' in body
|
||||
vis = re.search(r"_onVisibilityChange\(\)\s*\{(.*?)\n \}", body, re.S)
|
||||
assert vis is not None, "_onVisibilityChange method not found"
|
||||
method = vis.group(1)
|
||||
assert "this.disconnectSSE();" in method
|
||||
assert "this._hiddenDisconnect = true;" in method
|
||||
assert "this.connectSSE(this.wsId);" in method
|
||||
# Reconnect only consumes OUR hide-close marker.
|
||||
assert "else if (this._hiddenDisconnect)" in method
|
||||
# Teardown: the factory controller removes the listener on destroy.
|
||||
assert "pane._removeVisibilityHandler();" in body
|
||||
# The streaming buffers survive a hide-close: disconnectSSE stays
|
||||
# transport-only (no contentBuffer wipe) so the visible tail is
|
||||
# intact when the tab returns.
|
||||
dis = re.search(r"disconnectSSE\(\)\s*\{(.*?)\n \}", body, re.S)
|
||||
assert dis is not None
|
||||
assert "contentBuffer" not in dis.group(1)
|
||||
|
||||
|
||||
def test_no_global_sse_gap_detector() -> None:
|
||||
"""Live event ids are NOT strictly monotonic across concurrent
|
||||
tool+content emit (the fan-out runs outside the listeners lock), so
|
||||
a naive ``id !== lastEventId + 1`` gap check would false-positive.
|
||||
Recovery is server-signalled (``stream_overflow``) + reconnect
|
||||
replay instead. This tripwire pins the absence of the naive
|
||||
arithmetic — if gap detection is ever added, it must be scoped to
|
||||
the content stream only (content-vs-content never reorders)."""
|
||||
code = _strip_comments(_INTERACTIVE.read_text(encoding="utf-8"))
|
||||
assert not re.search(r"_lastEventId\s*[+\-]\s*1", code), (
|
||||
"found lastEventId +/- 1 arithmetic — a global gap detector "
|
||||
"false-positives on legal concurrent tool/content id inversion"
|
||||
)
|
||||
|
||||
|
||||
def test_overflow_helpers_extracted_to_shared_module() -> None:
|
||||
"""The storm-guard constants + the two pure helpers were extracted to the
|
||||
shared ``sse_overflow.js`` module (its own runtime probes live in
|
||||
``test_sse_overflow_js.py``) so the interactive and coordinator panes can't
|
||||
drift. Pin that the pane IMPORTS them rather than re-declaring a local
|
||||
copy: a stray local ``function overflowWindowTripped`` / ``const
|
||||
OVERFLOW_TRIP_COUNT`` would silently fork the trip math again."""
|
||||
body = _INTERACTIVE.read_text(encoding="utf-8")
|
||||
m = re.search(
|
||||
r"import \{([^}]*)\} from \"\./sse_overflow\.js\";",
|
||||
body,
|
||||
re.S,
|
||||
)
|
||||
assert m is not None, "interactive pane must import the shared overflow helpers"
|
||||
imported = m.group(1)
|
||||
for name in (
|
||||
"OVERFLOW_TRIP_COUNT",
|
||||
"OVERFLOW_TRIP_WINDOW_MS",
|
||||
"DEGRADED_COOLDOWN_BASE_MS",
|
||||
"DEGRADED_COOLDOWN_MAX_MS",
|
||||
"DEGRADED_COOLDOWN_RESET_MS",
|
||||
"overflowWindowTripped",
|
||||
"degradedCooldownStep",
|
||||
):
|
||||
assert name in imported, f"{name} must be imported from sse_overflow.js"
|
||||
# No local fork of the extracted definitions.
|
||||
assert not re.search(r"^function overflowWindowTripped\(", body, re.M), (
|
||||
"overflowWindowTripped must be imported, not re-declared locally"
|
||||
)
|
||||
assert not re.search(r"^function degradedCooldownStep\(", body, re.M), (
|
||||
"degradedCooldownStep must be imported, not re-declared locally"
|
||||
)
|
||||
assert not re.search(r"^const OVERFLOW_TRIP_COUNT\s*=", body, re.M), (
|
||||
"the trip constants must be imported, not re-declared locally"
|
||||
)
|
||||
|
||||
|
||||
def test_note_stream_overflow_does_not_reset_cooldown() -> None:
|
||||
"""The exact finding [0] bug shape must not regress: _noteStreamOverflow
|
||||
only counts + trips; it must NOT touch _degradedCooldownMs (the reset
|
||||
that defeated the ladder lived here). The ladder decision lives solely
|
||||
in _enterDegradedCatchup, keyed off _lastDegradedAt via
|
||||
degradedCooldownStep."""
|
||||
body = _INTERACTIVE.read_text(encoding="utf-8")
|
||||
note = re.search(r"_noteStreamOverflow\(\)\s*\{(.*?)\n \}", body, re.S)
|
||||
assert note is not None, "_noteStreamOverflow not found"
|
||||
assert "_degradedCooldownMs" not in note.group(1), (
|
||||
"_noteStreamOverflow must not write _degradedCooldownMs — that reset "
|
||||
"was the bug that stopped the ladder escalating"
|
||||
)
|
||||
enter = re.search(r"_enterDegradedCatchup\(\)\s*\{(.*?)\n \}", body, re.S)
|
||||
assert enter is not None
|
||||
assert "degradedCooldownStep(" in enter.group(1)
|
||||
assert "this._lastDegradedAt = now" in enter.group(1)
|
||||
|
||||
|
||||
def test_recover_beat_defers_reconnect_when_tab_hidden() -> None:
|
||||
"""Review round-2 finding [1]: the factory's transient-error recovery
|
||||
beat (recoverTimer) must NOT reopen an EventSource into a hidden tab —
|
||||
that re-creates the throttled slow-consumer overflow that close-on-hide
|
||||
exists to prevent. It guards on document.hidden and defers to the
|
||||
visibilitychange show edge (marking _hiddenDisconnect)."""
|
||||
body = _INTERACTIVE.read_text(encoding="utf-8")
|
||||
beat = re.search(r"recoverTimer = setTimeout\(\(\) => \{(.*?)\n \}, 5000\);", body, re.S)
|
||||
assert beat is not None, "recoverTimer setTimeout body not found"
|
||||
b = beat.group(1)
|
||||
assert "document.hidden" in b, "recovery beat must guard on document.hidden"
|
||||
assert "pane._hiddenDisconnect = true" in b, (
|
||||
"recovery beat must defer to the show edge when hidden"
|
||||
)
|
||||
# The hidden guard must precede the reconnect (connectSSE) so it can't fall
|
||||
# through to reopening the stream.
|
||||
assert b.index("document.hidden") < b.index("pane.connectSSE(pane.wsId)")
|
||||
|
||||
|
||||
def test_giveup_removes_visibility_handler() -> None:
|
||||
"""Review round-2 finding [3]: giveUp() (markDead) must detach the
|
||||
visibility handler and clear _hiddenDisconnect, or a tab hidden before
|
||||
the give-up resurrects the dead controller's stream on return (the show
|
||||
edge would connectSSE the closed ws and 404-reconnect it forever)."""
|
||||
body = _INTERACTIVE.read_text(encoding="utf-8")
|
||||
give = re.search(r"const giveUp = function \(\) \{(.*?)\n \};", body, re.S)
|
||||
assert give is not None, "giveUp function body not found"
|
||||
g = give.group(1)
|
||||
assert "pane._removeVisibilityHandler();" in g, (
|
||||
"giveUp must remove the visibility handler so a show edge can't resurrect a dead controller"
|
||||
)
|
||||
# _removeVisibilityHandler also clears _hiddenDisconnect (pinned in its body).
|
||||
rvh = re.search(r"_removeVisibilityHandler\(\)\s*\{(.*?)\n \}", body, re.S)
|
||||
assert rvh is not None
|
||||
assert "this._hiddenDisconnect = false" in rvh.group(1)
|
||||
|
||||
|
||||
def test_connectsse_defers_open_when_tab_hidden() -> None:
|
||||
"""PR #805 review (Copilot + R3): connectSSE is the single connect
|
||||
chokepoint and must not open an EventSource into a hidden tab. The
|
||||
fresh-connect path (_loadHistoryThenConnect) has no timer guard, so a
|
||||
first load in a background tab would otherwise open a throttled stream —
|
||||
the slow-consumer overflow this PR exists to prevent. The guard sits
|
||||
AFTER the visibilitychange-handler install (so the show edge can
|
||||
reconnect) and AFTER the wsId assignment (so it targets the right ws),
|
||||
and BEFORE `new EventSource` (so nothing opens)."""
|
||||
body = _INTERACTIVE.read_text(encoding="utf-8")
|
||||
start = body.index("connectSSE(wsId) {")
|
||||
open_at = body.index("new EventSource(evtUrl)", start)
|
||||
head = body[start:open_at] # connectSSE up to the EventSource open
|
||||
assert "if (document.hidden) {" in head, (
|
||||
"connectSSE must guard on document.hidden BEFORE opening the stream"
|
||||
)
|
||||
assert "this._hiddenDisconnect = true;" in head, (
|
||||
"the deferred connect must mark _hiddenDisconnect so the show edge reconnects"
|
||||
)
|
||||
assert head.index("this.wsId = wsId;") < head.index("if (document.hidden) {")
|
||||
assert head.index('addEventListener("visibilitychange"') < head.index("if (document.hidden) {")
|
||||
|
||||
+71
-6
@@ -71,7 +71,7 @@ def _make_judge(
|
||||
session_provider=provider,
|
||||
session_client=client,
|
||||
session_model="test-model",
|
||||
context_window=100_000,
|
||||
session_capabilities=MagicMock(context_window=100_000),
|
||||
)
|
||||
|
||||
|
||||
@@ -892,15 +892,83 @@ class TestModelAliasResolution:
|
||||
alias_provider: MagicMock,
|
||||
alias_client: MagicMock,
|
||||
underlying_model: str,
|
||||
*,
|
||||
capabilities: dict[str, Any] | None = None,
|
||||
) -> MagicMock:
|
||||
registry = MagicMock()
|
||||
cfg = MagicMock()
|
||||
cfg.context_window = 50_000
|
||||
cfg.capabilities = capabilities if capabilities is not None else {}
|
||||
registry.has_alias.side_effect = lambda a: a == alias
|
||||
registry.resolve.return_value = (alias_client, underlying_model, cfg)
|
||||
registry.get_provider.return_value = alias_provider
|
||||
return registry
|
||||
|
||||
def test_alias_capabilities_merged_and_threaded_to_wire(self):
|
||||
"""#823: a judge alias's model-definition ``capabilities`` are merged
|
||||
onto the provider base AND passed to ``create_completion`` — the same
|
||||
contract as the session / utility / sub-agent lanes. Without threading,
|
||||
operator overrides (effort passthrough, tool support) were silently
|
||||
ignored on judge calls; deleting ``capabilities=self._capabilities`` from
|
||||
the call site, or breaking the merge, must fail here."""
|
||||
from turnstone.core.providers._protocol import ModelCapabilities
|
||||
|
||||
base = ModelCapabilities(supports_tools=True, effort_passthrough=False)
|
||||
alias_provider = _make_mock_provider(response_content=_good_verdict_json())
|
||||
alias_provider.get_capabilities = MagicMock(return_value=base)
|
||||
registry = self._make_alias_registry(
|
||||
"judge-mini",
|
||||
alias_provider,
|
||||
MagicMock(base_url="https://a/v1", api_key="k"),
|
||||
"local-9b",
|
||||
capabilities={"supports_tools": False, "effort_passthrough": True},
|
||||
)
|
||||
judge = IntentJudge(
|
||||
config=JudgeConfig(enabled=True, model="judge-mini"),
|
||||
session_provider=_make_mock_provider(),
|
||||
session_client=MagicMock(base_url="https://s/v1", api_key="s"),
|
||||
session_model="session-model",
|
||||
session_capabilities=MagicMock(context_window=100_000),
|
||||
model_registry=registry,
|
||||
)
|
||||
# Merged at construction: overrides applied, untouched fields survive.
|
||||
assert judge._capabilities.supports_tools is False
|
||||
assert judge._capabilities.effort_passthrough is True
|
||||
assert judge._capabilities.context_window == base.context_window
|
||||
# ...and the SAME merged object reaches the wire.
|
||||
judge._evaluate_single(
|
||||
_make_item(),
|
||||
[{"role": "user", "content": "x"}],
|
||||
cancel_event=None,
|
||||
client=MagicMock(),
|
||||
)
|
||||
passed = alias_provider.create_completion.call_args.kwargs["capabilities"]
|
||||
assert passed is judge._capabilities
|
||||
|
||||
def test_fallback_threads_session_capabilities_to_wire(self):
|
||||
"""No judge alias → the judge inherits the session model AND the
|
||||
session's resolved capabilities, threaded to ``create_completion``."""
|
||||
from turnstone.core.providers._protocol import ModelCapabilities
|
||||
|
||||
sess_caps = ModelCapabilities(context_window=54_321, effort_passthrough=True)
|
||||
provider = _make_mock_provider(response_content=_good_verdict_json())
|
||||
judge = IntentJudge(
|
||||
config=JudgeConfig(enabled=True, model=""), # no alias → fallback
|
||||
session_provider=provider,
|
||||
session_client=MagicMock(base_url="https://s/v1", api_key="s"),
|
||||
session_model="session-model",
|
||||
session_capabilities=sess_caps,
|
||||
)
|
||||
assert judge._capabilities is sess_caps
|
||||
assert judge._judge_context_window == 54_321
|
||||
judge._evaluate_single(
|
||||
_make_item(),
|
||||
[{"role": "user", "content": "x"}],
|
||||
cancel_event=None,
|
||||
client=MagicMock(),
|
||||
)
|
||||
assert provider.create_completion.call_args.kwargs["capabilities"] is sess_caps
|
||||
|
||||
def test_alias_uses_registry_provider_not_session_provider(self):
|
||||
"""Judge with model=alias should resolve via registry — provider, client,
|
||||
and concrete model name all come from the alias."""
|
||||
@@ -932,7 +1000,6 @@ class TestModelAliasResolution:
|
||||
session_provider=session_provider,
|
||||
session_client=session_client,
|
||||
session_model="session-default-model",
|
||||
context_window=100_000,
|
||||
model_registry=registry,
|
||||
)
|
||||
|
||||
@@ -959,7 +1026,6 @@ class TestModelAliasResolution:
|
||||
session_provider=_make_mock_provider(),
|
||||
session_client=MagicMock(base_url="https://s/v1", api_key="s"),
|
||||
session_model="session-model",
|
||||
context_window=100_000,
|
||||
model_registry=registry,
|
||||
)
|
||||
assert judge._judge_context_window == 50_000
|
||||
@@ -980,7 +1046,7 @@ class TestModelAliasResolution:
|
||||
session_provider=_make_mock_provider(),
|
||||
session_client=MagicMock(base_url="http://s", api_key="s"),
|
||||
session_model="session-model",
|
||||
context_window=100_000,
|
||||
session_capabilities=MagicMock(context_window=100_000),
|
||||
model_registry=registry,
|
||||
)
|
||||
assert judge._judge_context_window == 100_000 # session window, not 0
|
||||
@@ -1008,7 +1074,7 @@ class TestModelAliasResolution:
|
||||
session_provider=session_provider,
|
||||
session_client=session_client,
|
||||
session_model="session-default-model",
|
||||
context_window=100_000,
|
||||
session_capabilities=MagicMock(context_window=100_000),
|
||||
model_registry=registry,
|
||||
)
|
||||
|
||||
@@ -1031,7 +1097,6 @@ class TestModelAliasResolution:
|
||||
session_provider=session_provider,
|
||||
session_client=session_client,
|
||||
session_model="session-default-model",
|
||||
context_window=100_000,
|
||||
)
|
||||
|
||||
assert judge._provider is session_provider
|
||||
|
||||
@@ -18,6 +18,7 @@ from turnstone.core.lowering import (
|
||||
CANCELLED_TOOL_RESULT,
|
||||
_find_orphaned_tool_calls,
|
||||
repair_wire_messages,
|
||||
restore_provider_tool_ids,
|
||||
sanitize_tool_call_arguments,
|
||||
tool_args_preview,
|
||||
wire_valid_arguments,
|
||||
@@ -340,3 +341,90 @@ def test_pipeline_every_emitted_arguments_is_a_json_object() -> None:
|
||||
for m in out:
|
||||
for tc in m.get("tool_calls", []):
|
||||
assert isinstance(json.loads(tc["function"]["arguments"]), dict)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# restore_provider_tool_ids — the agent-wire id map (minted → provider-original).
|
||||
#
|
||||
# Sub-agent tool ids are minted "{parent}::r{run}s{step}::{provider_id}" for
|
||||
# session-unique correlation (registry / DOM / recall). On the wire the pass
|
||||
# maps them BACK to the provider's own ids from the per-run mint map, so the
|
||||
# provider-native tool_use block (replayed verbatim, id never rewritten), the
|
||||
# top-level tool_calls mirror, and the tool_result all agree on every request.
|
||||
# --------------------------------------------------------------------------- #
|
||||
def test_restore_ids_identity_on_empty_map() -> None:
|
||||
msgs = [_assistant_calls(_call("task-1::r1s1::call_0", "{}")), _tool("task-1::r1s1::call_0")]
|
||||
assert restore_provider_tool_ids(msgs, {}) is msgs
|
||||
|
||||
|
||||
def test_restore_ids_identity_when_nothing_matches() -> None:
|
||||
msgs = [_assistant_calls(_call("call_1", "{}")), _tool("call_1")]
|
||||
assert restore_provider_tool_ids(msgs, {"task-1::r1s1::call_0": "call_0"}) is msgs
|
||||
|
||||
|
||||
def test_restore_ids_maps_call_and_result_to_provider_original() -> None:
|
||||
minted = "task-1::r1s1::toolu_01AB"
|
||||
msgs = [_assistant_calls(_call(minted, "{}")), _tool(minted)]
|
||||
out = restore_provider_tool_ids(msgs, {minted: "toolu_01AB"})
|
||||
assert out[0]["tool_calls"][0]["id"] == "toolu_01AB"
|
||||
assert out[1]["tool_call_id"] == "toolu_01AB" # pairing restored on both sides
|
||||
# Copy-on-write: the input messages (the canonical-adjacent dicts) are unmutated.
|
||||
assert msgs[0]["tool_calls"][0]["id"] == minted
|
||||
assert msgs[1]["tool_call_id"] == minted
|
||||
|
||||
|
||||
def test_restore_ids_recovers_originals_containing_the_mint_delimiter() -> None:
|
||||
# Recovery is by MAP, not by string-splitting the mint suffix: a provider
|
||||
# id that itself contains "::" round-trips exactly.
|
||||
original = "srv::call::0"
|
||||
minted = f"task-1::r1s1::{original}"
|
||||
msgs = [_assistant_calls(_call(minted, "{}")), _tool(minted)]
|
||||
out = restore_provider_tool_ids(msgs, {minted: original})
|
||||
assert out[0]["tool_calls"][0]["id"] == original
|
||||
assert out[1]["tool_call_id"] == original
|
||||
|
||||
|
||||
def test_restore_ids_duplicate_originals_across_turns() -> None:
|
||||
# A local server reissuing "call_0" every turn: two distinct minted ids
|
||||
# both restore to "call_0" — the proven prior wire shape, each round
|
||||
# pairing with its adjacent result.
|
||||
m1, m2 = "task-1::r1s1::call_0", "task-1::r1s2::call_0"
|
||||
msgs = [
|
||||
_assistant_calls(_call(m1, "{}")),
|
||||
_tool(m1),
|
||||
_assistant_calls(_call(m2, "{}")),
|
||||
_tool(m2),
|
||||
]
|
||||
out = restore_provider_tool_ids(msgs, {m1: "call_0", m2: "call_0"})
|
||||
assert out[0]["tool_calls"][0]["id"] == "call_0"
|
||||
assert out[1]["tool_call_id"] == "call_0"
|
||||
assert out[2]["tool_calls"][0]["id"] == "call_0"
|
||||
assert out[3]["tool_call_id"] == "call_0"
|
||||
|
||||
|
||||
def test_restore_ids_leaves_unmapped_siblings_untouched() -> None:
|
||||
minted = "task-1::r1s2::call_1"
|
||||
msgs = [
|
||||
_assistant_calls(_call("call_ok", "{}"), _call(minted, "{}")),
|
||||
_tool("call_ok"),
|
||||
_tool(minted),
|
||||
]
|
||||
out = restore_provider_tool_ids(msgs, {minted: "call_1"})
|
||||
assert out[0]["tool_calls"][0]["id"] == "call_ok"
|
||||
assert out[1]["tool_call_id"] == "call_ok"
|
||||
assert out[0]["tool_calls"][1]["id"] == "call_1"
|
||||
assert out[2]["tool_call_id"] == "call_1"
|
||||
|
||||
|
||||
def test_restore_ids_skips_empty_and_non_string() -> None:
|
||||
# Empty ids belong to repair_wire_messages' back-fill; non-strings are
|
||||
# someone else's malformation — neither is this pass's to invent.
|
||||
msgs = [
|
||||
_assistant_calls(
|
||||
{"id": "", "type": "function", "function": {"name": "b", "arguments": "{}"}}
|
||||
),
|
||||
{"role": "tool", "tool_call_id": None, "content": "x"},
|
||||
]
|
||||
out = restore_provider_tool_ids(msgs, {"task-1::r1s1::x": "x"})
|
||||
assert out[0]["tool_calls"][0]["id"] == ""
|
||||
assert out[1]["tool_call_id"] is None
|
||||
|
||||
@@ -0,0 +1,103 @@
|
||||
"""Tests for alembic migration 066 (persona + project on scheduled_tasks).
|
||||
|
||||
Drives ``command.upgrade``/``downgrade`` against an isolated SQLite database per
|
||||
test (the 060/062/063/065 harness pattern), then asserts:
|
||||
|
||||
* upgrade adds the ``persona`` and ``project_id`` columns to ``scheduled_tasks``;
|
||||
* a pre-066 scheduled task migrates cleanly, gaining ``""`` for both new columns
|
||||
— the empty default that means "kind default persona" / "no project" and
|
||||
preserves byte-identical dispatch behaviour to pre-066;
|
||||
* downgrade removes both columns, returning ``scheduled_tasks`` to its exact
|
||||
pre-066 shape — pinning the clean-rollback guarantee;
|
||||
* up -> down -> up lands cleanly with no leftover-column conflict.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
import sqlalchemy as sa
|
||||
from alembic import command
|
||||
from alembic.config import Config
|
||||
|
||||
_MIGRATIONS_DIR = str(
|
||||
Path(__file__).resolve().parent.parent / "turnstone" / "core" / "storage" / "migrations"
|
||||
)
|
||||
|
||||
|
||||
def _alembic_cfg(db_path: Path) -> Config:
|
||||
cfg = Config()
|
||||
cfg.set_main_option("script_location", _MIGRATIONS_DIR)
|
||||
cfg.set_main_option("sqlalchemy.url", f"sqlite:///{db_path}")
|
||||
return cfg
|
||||
|
||||
|
||||
def _insert_pre066_task(engine: sa.Engine) -> None:
|
||||
with engine.begin() as conn:
|
||||
conn.execute(
|
||||
sa.text(
|
||||
"INSERT INTO scheduled_tasks "
|
||||
"(task_id, name, schedule_type, initial_message, created, updated) "
|
||||
"VALUES ('t1', 'Nightly', 'cron', 'run', "
|
||||
"'2026-01-01T00:00:00', '2026-01-01T00:00:00')"
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
class TestMigration066:
|
||||
def test_upgrade_adds_persona_and_project_columns(self, tmp_path: Path) -> None:
|
||||
db_path = tmp_path / "066-up.db"
|
||||
command.upgrade(_alembic_cfg(db_path), "066")
|
||||
|
||||
engine = sa.create_engine(f"sqlite:///{db_path}")
|
||||
try:
|
||||
cols = {c["name"] for c in sa.inspect(engine).get_columns("scheduled_tasks")}
|
||||
assert {"persona", "project_id"} <= cols
|
||||
finally:
|
||||
engine.dispose()
|
||||
|
||||
def test_preexisting_row_migrates_with_empty_default(self, tmp_path: Path) -> None:
|
||||
db_path = tmp_path / "066-default.db"
|
||||
cfg = _alembic_cfg(db_path)
|
||||
# Stop at 065, insert a pre-066 scheduled task, THEN upgrade to 066.
|
||||
command.upgrade(cfg, "065")
|
||||
engine = sa.create_engine(f"sqlite:///{db_path}")
|
||||
try:
|
||||
_insert_pre066_task(engine)
|
||||
command.upgrade(cfg, "066")
|
||||
with engine.connect() as conn:
|
||||
row = conn.execute(
|
||||
sa.text("SELECT persona, project_id FROM scheduled_tasks WHERE task_id = 't1'")
|
||||
).fetchone()
|
||||
assert row is not None
|
||||
assert row[0] == "" and row[1] == ""
|
||||
finally:
|
||||
engine.dispose()
|
||||
|
||||
def test_downgrade_removes_persona_and_project_columns(self, tmp_path: Path) -> None:
|
||||
db_path = tmp_path / "066-down.db"
|
||||
cfg = _alembic_cfg(db_path)
|
||||
command.upgrade(cfg, "066")
|
||||
command.downgrade(cfg, "065")
|
||||
|
||||
engine = sa.create_engine(f"sqlite:///{db_path}")
|
||||
try:
|
||||
cols = {c["name"] for c in sa.inspect(engine).get_columns("scheduled_tasks")}
|
||||
assert "persona" not in cols and "project_id" not in cols
|
||||
finally:
|
||||
engine.dispose()
|
||||
|
||||
def test_downgrade_then_upgrade_round_trip(self, tmp_path: Path) -> None:
|
||||
"""up -> down -> up must land cleanly (no leftover column conflict)."""
|
||||
db_path = tmp_path / "066-roundtrip.db"
|
||||
cfg = _alembic_cfg(db_path)
|
||||
command.upgrade(cfg, "066")
|
||||
command.downgrade(cfg, "065")
|
||||
command.upgrade(cfg, "066")
|
||||
|
||||
engine = sa.create_engine(f"sqlite:///{db_path}")
|
||||
try:
|
||||
cols = {c["name"] for c in sa.inspect(engine).get_columns("scheduled_tasks")}
|
||||
assert {"persona", "project_id"} <= cols
|
||||
finally:
|
||||
engine.dispose()
|
||||
@@ -167,6 +167,43 @@ class TestModelRegistry:
|
||||
with pytest.raises(ValueError, match="Unknown model alias"):
|
||||
reg.get_client("nonexistent")
|
||||
|
||||
def test_client_construction_failure_is_value_error(self) -> None:
|
||||
# Environment failures inside SDK construction (e.g. httpx raising
|
||||
# FileNotFoundError for a CA bundle deleted by a venv rebuild) must
|
||||
# surface as ValueError so routes answer 503-with-message instead
|
||||
# of an opaque 500.
|
||||
reg = self._make_registry()
|
||||
with (
|
||||
patch(
|
||||
"turnstone.core.model_registry.create_client",
|
||||
side_effect=FileNotFoundError(2, "No such file", "/gone/cacert.pem"),
|
||||
),
|
||||
pytest.raises(ValueError, match="'default'.*FileNotFoundError") as excinfo,
|
||||
):
|
||||
reg.get_client("default")
|
||||
assert isinstance(excinfo.value.__cause__, FileNotFoundError)
|
||||
# The message is echoed in 503 bodies: exception TYPE only — the
|
||||
# raw exception text can embed filesystem paths and must stay in
|
||||
# the server log.
|
||||
assert "/gone/cacert.pem" not in str(excinfo.value)
|
||||
assert "No such file" not in str(excinfo.value)
|
||||
# Nothing half-constructed may be cached — a later call with a
|
||||
# repaired environment must construct for real.
|
||||
assert "default" not in reg._clients
|
||||
|
||||
def test_client_construction_value_error_passes_through(self) -> None:
|
||||
# create_client's own misconfig ValueErrors already carry
|
||||
# remediation text and must not be double-wrapped.
|
||||
reg = self._make_registry()
|
||||
with (
|
||||
patch(
|
||||
"turnstone.core.model_registry.create_client",
|
||||
side_effect=ValueError("anthropic-compatible requires base_url"),
|
||||
),
|
||||
pytest.raises(ValueError, match="^anthropic-compatible requires base_url$"),
|
||||
):
|
||||
reg.get_client("default")
|
||||
|
||||
def test_shutdown(self) -> None:
|
||||
reg = self._make_registry()
|
||||
reg.get_client("default")
|
||||
|
||||
@@ -98,6 +98,98 @@ class TestLenAndClear:
|
||||
q = NudgeQueue()
|
||||
assert q.clear() == 0
|
||||
|
||||
def test_clear_channels_drops_only_matching(self):
|
||||
"""The abandoned-generation path drops advisory channels but must
|
||||
preserve ``"any"``-channel external events (watch fires,
|
||||
background-shell exits) in order."""
|
||||
q = NudgeQueue()
|
||||
q.enqueue("tool_error", "1", "tool")
|
||||
q.enqueue("watch_triggered", "2", "any")
|
||||
q.enqueue("correction", "3", "user")
|
||||
q.enqueue("background_shell_exit", "4", "any")
|
||||
assert q.clear_channels({"tool", "user"}) == 2
|
||||
assert q.pending() == [("watch_triggered", "2"), ("background_shell_exit", "4")]
|
||||
|
||||
def test_clear_channels_empty_returns_zero(self):
|
||||
q = NudgeQueue()
|
||||
assert q.clear_channels({"tool", "user"}) == 0
|
||||
|
||||
def test_demote_channel_retags_preserving_order_and_metadata(self):
|
||||
"""Cancel demotes 'any' → 'quiet': same entries, same order, same
|
||||
metadata/valid_until — only wake eligibility changes."""
|
||||
q = NudgeQueue()
|
||||
q.enqueue("watch_triggered", "w", "any", metadata={"watch_name": "ci"})
|
||||
q.enqueue("correction", "c", "user")
|
||||
q.enqueue("background_shell_exit", "b", "any", valid_until=lambda: True)
|
||||
assert q.demote_channel("any", "quiet") == 2
|
||||
assert q.pending(channel="any") == []
|
||||
assert q.pending(channel="quiet") == [
|
||||
("watch_triggered", "w"),
|
||||
("background_shell_exit", "b"),
|
||||
]
|
||||
# Metadata and valid_until ride the demotion; USER_DRAIN delivers.
|
||||
from turnstone.core.nudge_queue import USER_DRAIN, WAKE_PENDING
|
||||
|
||||
assert not q.has_pending(WAKE_PENDING - {"user"}) # no 'any' left
|
||||
drained = q.drain(USER_DRAIN)
|
||||
assert [(t, x, m) for t, x, m in drained] == [
|
||||
("watch_triggered", "w", {"watch_name": "ci"}),
|
||||
("correction", "c", None),
|
||||
("background_shell_exit", "b", None),
|
||||
]
|
||||
|
||||
def test_cap_channel_none_sees_demoted_entries(self):
|
||||
"""The watch soft cap counts across channels: entries a cancel
|
||||
demoted to 'quiet' still occupy the budget, and drop-oldest evicts
|
||||
the stalest regardless of channel."""
|
||||
q = NudgeQueue()
|
||||
q.enqueue("watch_triggered", "1", "any")
|
||||
q.demote_channel("any", "quiet")
|
||||
q.enqueue("watch_triggered", "2", "any")
|
||||
assert q.cap_at_or_drop_oldest("watch_triggered", 2, channel=None) is True
|
||||
assert q.pending() == [("watch_triggered", "2")]
|
||||
|
||||
def test_requeue_preserves_seq_for_chronology(self):
|
||||
"""A failed delivery gives entries back with their ORIGINAL seq, so
|
||||
a re-queued poll-4 still sorts before the poll-5 that arrived during
|
||||
the failed attempt — counters never run backwards."""
|
||||
q = NudgeQueue()
|
||||
q.enqueue("watch_triggered", "poll-4", "any")
|
||||
(drained_entry,) = q.drain_entries({"any"})
|
||||
q.enqueue("watch_triggered", "poll-5", "any") # newer event lands
|
||||
q.requeue(drained_entry, channel="quiet")
|
||||
entries = q.drain_entries({"any", "quiet"})
|
||||
entries.sort(key=lambda e: e.seq)
|
||||
assert [e.text for e in entries] == ["poll-4", "poll-5"]
|
||||
|
||||
def test_requeue_positions_by_seq_for_fifo_drains(self):
|
||||
"""Positioned insertion: plain (unsorted) drains also see the
|
||||
re-queued older entry first."""
|
||||
q = NudgeQueue()
|
||||
q.enqueue("a", "old", "quiet")
|
||||
(old_entry,) = q.drain_entries({"quiet"})
|
||||
q.enqueue("b", "new", "quiet")
|
||||
q.requeue(old_entry)
|
||||
assert [text for _t, text in q.pending()] == ["old", "new"]
|
||||
|
||||
def test_requeue_preserves_valid_until_and_metadata(self):
|
||||
alive = {"value": True}
|
||||
q = NudgeQueue()
|
||||
q.enqueue("n", "x", "any", valid_until=lambda: alive["value"], metadata={"k": 1})
|
||||
(entry,) = q.drain_entries({"any"})
|
||||
q.requeue(entry, channel="quiet")
|
||||
alive["value"] = False
|
||||
assert q.drain({"quiet"}) == [] # predicate survived the round-trip
|
||||
|
||||
def test_quiet_is_outside_the_wake_gate(self):
|
||||
from turnstone.core.nudge_queue import TOOL_DRAIN, USER_DRAIN, WAKE_PENDING
|
||||
|
||||
q = NudgeQueue()
|
||||
q.enqueue("background_shell_exit", "b", "quiet")
|
||||
assert not q.has_pending(WAKE_PENDING)
|
||||
assert q.has_pending(USER_DRAIN)
|
||||
assert q.has_pending(TOOL_DRAIN)
|
||||
|
||||
|
||||
class TestDropOldestByType:
|
||||
def test_drop_oldest_by_type_removes_earliest_match(self):
|
||||
|
||||
@@ -15,6 +15,7 @@ import pytest
|
||||
|
||||
from turnstone.core.oauth_ssrf import (
|
||||
OAuthSSRFError,
|
||||
OAuthSSRFPrivateAddressError,
|
||||
effective_port,
|
||||
is_localhost,
|
||||
validate_discovered_endpoint,
|
||||
@@ -86,6 +87,55 @@ class TestValidateUrlNoSSRF:
|
||||
):
|
||||
validate_url_no_ssrf("https://corp.example.com", allow_http=False)
|
||||
|
||||
def test_private_address_raises_distinct_subclass(self) -> None:
|
||||
# Callers with an operator opt-in (OIDC) catch the subclass to
|
||||
# append the remediation hint; plain OAuthSSRFError catches still work.
|
||||
with (
|
||||
patch("socket.getaddrinfo", return_value=self._PRIVATE_ADDR),
|
||||
pytest.raises(OAuthSSRFPrivateAddressError),
|
||||
):
|
||||
validate_url_no_ssrf("https://corp.example.com", allow_http=False)
|
||||
|
||||
def test_allow_private_accepts_rfc1918(self) -> None:
|
||||
with patch("socket.getaddrinfo", return_value=self._PRIVATE_ADDR):
|
||||
parsed = validate_url_no_ssrf(
|
||||
"https://auth.corp.example.com", allow_http=False, allow_private=True
|
||||
)
|
||||
assert parsed.hostname == "auth.corp.example.com"
|
||||
|
||||
def test_allow_private_accepts_cgnat(self) -> None:
|
||||
# 100.64/10 (RFC 6598, shared address space) — e.g. a tailnet-hosted IdP.
|
||||
with patch("socket.getaddrinfo", return_value=[(2, 1, 6, "", ("100.64.0.7", 0))]):
|
||||
validate_url_no_ssrf("https://idp.tail.example", allow_http=False, allow_private=True)
|
||||
|
||||
def test_allow_private_accepts_loopback_hostname(self) -> None:
|
||||
# A non-localhost hostname resolving to loopback (IdP behind a
|
||||
# local reverse proxy) is operator-trusted under the opt-in.
|
||||
with patch("socket.getaddrinfo", return_value=self._LOOPBACK_ADDR):
|
||||
validate_url_no_ssrf("https://auth.internal", allow_http=False, allow_private=True)
|
||||
|
||||
def test_allow_private_still_rejects_link_local(self) -> None:
|
||||
# Cloud metadata services live on link-local; no legitimate IdP does.
|
||||
with (
|
||||
patch("socket.getaddrinfo", return_value=[(2, 1, 6, "", ("169.254.169.254", 0))]),
|
||||
pytest.raises(OAuthSSRFError, match="refused even with private"),
|
||||
):
|
||||
validate_url_no_ssrf("https://md.example.com", allow_http=False, allow_private=True)
|
||||
|
||||
def test_allow_private_still_rejects_unspecified(self) -> None:
|
||||
# The message names the class so 0.0.0.0/:: rejections are unambiguous.
|
||||
with (
|
||||
patch("socket.getaddrinfo", return_value=[(2, 1, 6, "", ("0.0.0.0", 0))]),
|
||||
pytest.raises(OAuthSSRFError, match="unspecified"),
|
||||
):
|
||||
validate_url_no_ssrf("https://zero.example.com", allow_http=False, allow_private=True)
|
||||
|
||||
def test_allow_private_does_not_relax_https(self) -> None:
|
||||
with pytest.raises(OAuthSSRFError, match="must use HTTPS"):
|
||||
validate_url_no_ssrf(
|
||||
"http://auth.corp.example.com", allow_http=False, allow_private=True
|
||||
)
|
||||
|
||||
def test_rejects_unresolvable(self) -> None:
|
||||
import socket
|
||||
|
||||
@@ -122,6 +172,19 @@ class TestValidateDiscoveredEndpoint:
|
||||
trusted_endpoint_hosts=frozenset(),
|
||||
)
|
||||
|
||||
def test_allow_private_passes_through(self) -> None:
|
||||
# Same-origin endpoint on a private-resolving issuer host is accepted
|
||||
# when the operator opted in.
|
||||
issuer = urllib.parse.urlparse("https://auth.corp.example.com")
|
||||
with patch("socket.getaddrinfo", return_value=[(2, 1, 6, "", ("10.0.0.5", 0))]):
|
||||
validate_discovered_endpoint(
|
||||
"https://auth.corp.example.com/token",
|
||||
issuer,
|
||||
allow_http=False,
|
||||
trusted_endpoint_hosts=frozenset(),
|
||||
allow_private=True,
|
||||
)
|
||||
|
||||
def test_trusted_endpoint_host_passes(self) -> None:
|
||||
issuer = urllib.parse.urlparse("https://idp.example.com")
|
||||
with patch("socket.getaddrinfo", return_value=self._PUBLIC_ADDR):
|
||||
|
||||
@@ -90,6 +90,42 @@ class TestLoadOIDCConfig:
|
||||
assert cfg.scopes == "openid"
|
||||
assert cfg.provider_name == "Okta"
|
||||
|
||||
def test_load_oidc_config_allow_private_network_env(self, monkeypatch):
|
||||
monkeypatch.setenv("TURNSTONE_OIDC_ISSUER", "https://auth.internal.example")
|
||||
monkeypatch.setenv("TURNSTONE_OIDC_CLIENT_ID", "cid")
|
||||
monkeypatch.setenv("TURNSTONE_OIDC_CLIENT_SECRET", "csecret")
|
||||
monkeypatch.setenv("TURNSTONE_OIDC_ALLOW_PRIVATE_NETWORK", "true")
|
||||
|
||||
with patch("turnstone.core.config.load_config", return_value={}):
|
||||
cfg = load_oidc_config()
|
||||
|
||||
assert cfg.allow_private_network is True
|
||||
|
||||
def test_load_oidc_config_allow_private_network_toml(self, monkeypatch):
|
||||
monkeypatch.setenv("TURNSTONE_OIDC_ISSUER", "https://auth.internal.example")
|
||||
monkeypatch.setenv("TURNSTONE_OIDC_CLIENT_ID", "cid")
|
||||
monkeypatch.setenv("TURNSTONE_OIDC_CLIENT_SECRET", "csecret")
|
||||
monkeypatch.delenv("TURNSTONE_OIDC_ALLOW_PRIVATE_NETWORK", raising=False)
|
||||
|
||||
with patch(
|
||||
"turnstone.core.config.load_config",
|
||||
return_value={"allow_private_network": True},
|
||||
):
|
||||
cfg = load_oidc_config()
|
||||
|
||||
assert cfg.allow_private_network is True
|
||||
|
||||
def test_load_oidc_config_allow_private_network_default_off(self, monkeypatch):
|
||||
monkeypatch.setenv("TURNSTONE_OIDC_ISSUER", "https://auth.example.com")
|
||||
monkeypatch.setenv("TURNSTONE_OIDC_CLIENT_ID", "cid")
|
||||
monkeypatch.setenv("TURNSTONE_OIDC_CLIENT_SECRET", "csecret")
|
||||
monkeypatch.delenv("TURNSTONE_OIDC_ALLOW_PRIVATE_NETWORK", raising=False)
|
||||
|
||||
with patch("turnstone.core.config.load_config", return_value={}):
|
||||
cfg = load_oidc_config()
|
||||
|
||||
assert cfg.allow_private_network is False
|
||||
|
||||
def test_load_oidc_config_disabled_when_missing(self, monkeypatch):
|
||||
monkeypatch.delenv("TURNSTONE_OIDC_ISSUER", raising=False)
|
||||
monkeypatch.delenv("TURNSTONE_OIDC_CLIENT_ID", raising=False)
|
||||
@@ -345,6 +381,27 @@ class TestValidateIssuerURL:
|
||||
):
|
||||
validate_issuer_url("https://idp.example.com")
|
||||
|
||||
def test_private_address_hint_mentions_opt_in(self):
|
||||
"""The rejection message points the operator at allow_private_network."""
|
||||
with (
|
||||
patch("socket.getaddrinfo", return_value=[(2, 1, 6, "", ("10.0.0.5", 0))]),
|
||||
pytest.raises(OIDCError, match="allow_private_network"),
|
||||
):
|
||||
validate_issuer_url("https://auth.internal.example")
|
||||
|
||||
def test_allow_private_accepts_private_issuer(self):
|
||||
"""The opt-in accepts an issuer resolving to RFC 1918 space."""
|
||||
with patch("socket.getaddrinfo", return_value=[(2, 1, 6, "", ("10.0.0.5", 0))]):
|
||||
validate_issuer_url("https://auth.internal.example", allow_private=True)
|
||||
|
||||
def test_allow_private_still_rejects_link_local(self):
|
||||
"""Link-local (cloud metadata) is refused even with the opt-in."""
|
||||
with (
|
||||
patch("socket.getaddrinfo", return_value=[(2, 1, 6, "", ("169.254.169.254", 0))]),
|
||||
pytest.raises(OIDCError, match="refused even with private"),
|
||||
):
|
||||
validate_issuer_url("https://md.internal.example", allow_private=True)
|
||||
|
||||
def test_rejects_http_non_localhost(self):
|
||||
"""HTTP is rejected for non-localhost hosts."""
|
||||
with pytest.raises(OIDCError, match="must use HTTPS"):
|
||||
@@ -508,6 +565,20 @@ class TestValidateDiscoveredEndpoint:
|
||||
trusted_endpoint_hosts=frozenset(),
|
||||
)
|
||||
|
||||
def test_private_endpoint_hint_mentions_opt_in(self):
|
||||
"""A discovered endpoint resolving private carries the opt-in hint
|
||||
just like the issuer does — the remediation is the same knob."""
|
||||
with (
|
||||
patch("socket.getaddrinfo", return_value=[(2, 1, 6, "", ("10.0.0.5", 0))]),
|
||||
pytest.raises(OIDCError, match="allow_private_network"),
|
||||
):
|
||||
validate_discovered_endpoint(
|
||||
"https://idp.example.com/token",
|
||||
self._issuer(),
|
||||
allow_http=False,
|
||||
trusted_endpoint_hosts=frozenset(),
|
||||
)
|
||||
|
||||
def test_rejects_http_when_issuer_is_https(self):
|
||||
"""http:// discovered endpoint rejected when issuer is https://."""
|
||||
with (
|
||||
@@ -2166,6 +2237,58 @@ class TestDiscoverOIDC:
|
||||
|
||||
asyncio.run(_run())
|
||||
|
||||
def test_discover_oidc_private_issuer_rejected_by_default(self):
|
||||
"""Without the opt-in, a private-resolving issuer disables OIDC."""
|
||||
config = _make_config(
|
||||
issuer="https://auth.internal.example",
|
||||
authorization_endpoint="",
|
||||
token_endpoint="",
|
||||
userinfo_endpoint="",
|
||||
jwks_uri="",
|
||||
)
|
||||
|
||||
async def _run():
|
||||
with patch("socket.getaddrinfo", return_value=[(2, 1, 6, "", ("10.0.0.5", 0))]):
|
||||
result = await discover_oidc(config)
|
||||
assert result.enabled is False
|
||||
|
||||
asyncio.run(_run())
|
||||
|
||||
def test_discover_oidc_private_issuer_with_opt_in(self):
|
||||
"""allow_private_network=True lets a private-resolving IdP discover."""
|
||||
config = _make_config(
|
||||
issuer="https://auth.internal.example",
|
||||
allow_private_network=True,
|
||||
authorization_endpoint="",
|
||||
token_endpoint="",
|
||||
userinfo_endpoint="",
|
||||
jwks_uri="",
|
||||
)
|
||||
|
||||
discovery_doc = {
|
||||
"authorization_endpoint": "https://auth.internal.example/authorize",
|
||||
"token_endpoint": "https://auth.internal.example/token",
|
||||
"userinfo_endpoint": "https://auth.internal.example/userinfo",
|
||||
"jwks_uri": "https://auth.internal.example/jwks",
|
||||
}
|
||||
|
||||
mock_response = MagicMock()
|
||||
mock_response.json.return_value = discovery_doc
|
||||
mock_response.raise_for_status = MagicMock()
|
||||
|
||||
async def _run():
|
||||
client = _mock_async_client(lambda url: _async_return(mock_response))
|
||||
with (
|
||||
patch("socket.getaddrinfo", return_value=[(2, 1, 6, "", ("10.0.0.5", 0))]),
|
||||
patch("httpx.AsyncClient", return_value=client),
|
||||
):
|
||||
result = await discover_oidc(config)
|
||||
|
||||
assert result.enabled is True
|
||||
assert result.token_endpoint == "https://auth.internal.example/token"
|
||||
|
||||
asyncio.run(_run())
|
||||
|
||||
def test_discover_oidc_failure(self):
|
||||
"""Mock httpx error -> enabled=False returned."""
|
||||
config = _make_config(
|
||||
|
||||
@@ -0,0 +1,815 @@
|
||||
"""End-to-end coverage for the ``open_preview`` tool wiring.
|
||||
|
||||
Spans the seams the preview descriptor rides: preparer validation +
|
||||
approval posture, executor target resolution (mocked ``httpx`` for URLs,
|
||||
tmp files for paths, monkeypatched storage for attachments), the
|
||||
``_tool_previews`` side channel + live SSE event, the ``Turn.meta``
|
||||
round-trip, the ``/history`` projection, the storage reconstruct routing,
|
||||
and the auth scope of the serving route.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from types import SimpleNamespace
|
||||
from unittest.mock import MagicMock
|
||||
|
||||
from turnstone.core.session import ChatSession
|
||||
from turnstone.core.trajectory import Role, turn_from_dict, turn_to_dict
|
||||
|
||||
PNG_1x1 = (
|
||||
b"\x89PNG\r\n\x1a\n\x00\x00\x00\rIHDR\x00\x00\x00\x01\x00\x00\x00\x01"
|
||||
b"\x08\x06\x00\x00\x00\x1f\x15\xc4\x89\x00\x00\x00\rIDATx\x9cc\xfc\xcf"
|
||||
b"\xc0\xc0\xc0\x00\x00\x00\x05\x00\x01\xa5\xf6E@\x00\x00\x00\x00IEND\xaeB`\x82"
|
||||
)
|
||||
|
||||
|
||||
class _RecordingUI:
|
||||
"""SessionUI double that records tool_result calls (kwargs included)."""
|
||||
|
||||
def __init__(self):
|
||||
self.tool_results = []
|
||||
|
||||
def __getattr__(self, name):
|
||||
# Every other SessionUI hook is an inert no-op.
|
||||
def _noop(*args, **kwargs):
|
||||
return None
|
||||
|
||||
return _noop
|
||||
|
||||
def on_tool_result(self, call_id, name, output, **kwargs):
|
||||
self.tool_results.append((call_id, name, output, kwargs))
|
||||
|
||||
|
||||
def _make_session(**kwargs):
|
||||
defaults = dict(
|
||||
client=MagicMock(),
|
||||
model="test-model",
|
||||
ui=_RecordingUI(),
|
||||
instructions=None,
|
||||
temperature=0.5,
|
||||
max_tokens=4096,
|
||||
tool_timeout=5,
|
||||
)
|
||||
defaults.update(kwargs)
|
||||
return ChatSession(**defaults)
|
||||
|
||||
|
||||
def _fake_response(url, body, content_type):
|
||||
import httpx
|
||||
|
||||
resp = SimpleNamespace()
|
||||
# A real httpx.URL so the executor's userinfo-strip path runs unmocked.
|
||||
resp.url = httpx.URL(url)
|
||||
resp.content = body
|
||||
resp.text = body.decode("utf-8", errors="replace")
|
||||
resp.headers = {"content-type": content_type}
|
||||
resp.raise_for_status = lambda: None
|
||||
return resp
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Preparer
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestPrepareOpenPreview:
|
||||
def test_missing_target_errors(self):
|
||||
s = _make_session()
|
||||
item = s._prepare_open_preview("c1", {})
|
||||
assert item["error"].startswith("Error: missing target")
|
||||
|
||||
def test_invalid_kind_errors(self):
|
||||
s = _make_session()
|
||||
item = s._prepare_open_preview("c1", {"target": "a.txt", "kind": "hologram"})
|
||||
assert "kind must be one of" in item["error"]
|
||||
|
||||
def test_url_target_needs_approval(self):
|
||||
s = _make_session()
|
||||
item = s._prepare_open_preview("c1", {"target": "https://example.com/x"})
|
||||
assert item["needs_approval"] is True
|
||||
assert item["target_kind"] == "url"
|
||||
assert item["approval_label"] == "open_preview"
|
||||
assert "error" not in item
|
||||
|
||||
def test_private_url_blocked_pre_approval(self):
|
||||
s = _make_session()
|
||||
item = s._prepare_open_preview("c1", {"target": "http://169.254.169.254/meta"})
|
||||
assert "error" in item
|
||||
assert item["needs_approval"] is False
|
||||
|
||||
def test_path_target_runs_unprompted(self):
|
||||
s = _make_session()
|
||||
item = s._prepare_open_preview("c1", {"target": "~/notes.md"})
|
||||
assert item["needs_approval"] is False
|
||||
assert item["target_kind"] == "path"
|
||||
assert not item["path"].startswith("~")
|
||||
|
||||
def test_attachment_target(self):
|
||||
s = _make_session()
|
||||
item = s._prepare_open_preview("c1", {"target": "attachment:abc123"})
|
||||
assert item["needs_approval"] is False
|
||||
assert item["target_kind"] == "attachment"
|
||||
assert item["attachment_id"] == "abc123"
|
||||
empty = s._prepare_open_preview("c1", {"target": "attachment:"})
|
||||
assert "error" in empty
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Executor
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestExecOpenPreview:
|
||||
def test_url_html_builds_web_descriptor(self, monkeypatch):
|
||||
s = _make_session()
|
||||
body = b"<html><head><title>Acme Pricing</title></head><body>x</body></html>"
|
||||
monkeypatch.setattr(
|
||||
"turnstone.core.session.fetch_with_ssrf_guard",
|
||||
lambda url, **kw: _fake_response(url, body, "text/html; charset=utf-8"),
|
||||
)
|
||||
item = s._prepare_open_preview("c1", {"target": "https://acme.com/pricing"})
|
||||
call_id, msg = s._exec_open_preview(item)
|
||||
assert call_id == "c1"
|
||||
assert "Acme Pricing" in msg
|
||||
descriptor, att = s._tool_previews["c1"]
|
||||
assert descriptor["kind"] == "web"
|
||||
assert descriptor["title"] == "Acme Pricing"
|
||||
assert descriptor["source"] == "https://acme.com/pricing"
|
||||
assert descriptor["content_type"].startswith("text/html")
|
||||
assert att.kind == "preview"
|
||||
# The stored bytes gained a base for relative-asset resolution.
|
||||
assert b'<base href="https://acme.com/pricing">' in att.content
|
||||
# The live event carried the descriptor.
|
||||
results = s.ui.tool_results
|
||||
assert results and results[-1][3].get("preview") == descriptor
|
||||
|
||||
def test_url_userinfo_stripped_from_descriptor(self, monkeypatch):
|
||||
s = _make_session()
|
||||
body = b"<html><head></head><body>x</body></html>"
|
||||
monkeypatch.setattr(
|
||||
"turnstone.core.session.fetch_with_ssrf_guard",
|
||||
lambda url, **kw: _fake_response(url, body, "text/html"),
|
||||
)
|
||||
item = s._prepare_open_preview("c1", {"target": "https://user:sekret@acme.com/page"})
|
||||
s._exec_open_preview(item)
|
||||
descriptor, att = s._tool_previews["c1"]
|
||||
assert "sekret" not in descriptor["source"]
|
||||
assert "sekret" not in descriptor["title"]
|
||||
assert b"sekret" not in att.content # the injected <base href>
|
||||
|
||||
def test_redirect_into_private_space_blocked(self, monkeypatch):
|
||||
s = _make_session()
|
||||
|
||||
# The guarded fetch raises BEFORE requesting a private hop — the
|
||||
# executor's ValueError lane turns that into a tool error.
|
||||
def _blocked(url, **kw):
|
||||
raise ValueError("Blocked: URL resolves to private/internal address (169.254.169.254)")
|
||||
|
||||
monkeypatch.setattr("turnstone.core.session.fetch_with_ssrf_guard", _blocked)
|
||||
item = s._prepare_open_preview("c1", {"target": "https://innocent.example/"})
|
||||
_, msg = s._exec_open_preview(item)
|
||||
assert msg.startswith("Error: fetch failed: Blocked")
|
||||
assert "c1" not in s._tool_previews
|
||||
|
||||
def test_oversized_web_content_errors(self, monkeypatch):
|
||||
s = _make_session()
|
||||
big = b"<html>" + b"x" * (4 * 1024 * 1024 + 16) + b"</html>"
|
||||
monkeypatch.setattr(
|
||||
"turnstone.core.session.fetch_with_ssrf_guard",
|
||||
lambda url, **kw: _fake_response(url, big, "text/html"),
|
||||
)
|
||||
item = s._prepare_open_preview("c1", {"target": "https://example.com/big"})
|
||||
_, msg = s._exec_open_preview(item)
|
||||
assert msg.startswith("Error:")
|
||||
assert "too large" in msg
|
||||
|
||||
def test_url_pdf_over_10mb_previews_to_kind_cap(self, monkeypatch):
|
||||
# Review finding (PR #800): a flat 10 MB URL pre-check rejected PDFs
|
||||
# the 32 MiB pdf kind cap allows — the fetch ceiling must track the
|
||||
# widest kind cap and leave the per-kind caps as the authority.
|
||||
from turnstone.core.preview import PREVIEW_SIZE_CAPS
|
||||
|
||||
s = _make_session()
|
||||
body = b"%PDF-1.7\n" + b"a" * (12 * 1024 * 1024)
|
||||
seen = {}
|
||||
|
||||
def _capture(url, **kw):
|
||||
seen.update(kw)
|
||||
return _fake_response(url, body, "application/pdf")
|
||||
|
||||
monkeypatch.setattr("turnstone.core.session.fetch_with_ssrf_guard", _capture)
|
||||
item = s._prepare_open_preview("c1", {"target": "https://acme.com/report.pdf"})
|
||||
_, msg = s._exec_open_preview(item)
|
||||
assert not msg.startswith("Error:")
|
||||
descriptor, _ = s._tool_previews["c1"]
|
||||
assert descriptor["kind"] == "pdf"
|
||||
assert descriptor["size"] == len(body)
|
||||
assert seen["max_bytes"] == max(PREVIEW_SIZE_CAPS.values())
|
||||
|
||||
def test_path_image(self, tmp_path):
|
||||
s = _make_session()
|
||||
p = tmp_path / "chart.png"
|
||||
p.write_bytes(PNG_1x1)
|
||||
item = s._prepare_open_preview("c1", {"target": str(p)})
|
||||
_, msg = s._exec_open_preview(item)
|
||||
assert not msg.startswith("Error:")
|
||||
descriptor, att = s._tool_previews["c1"]
|
||||
assert descriptor["kind"] == "image"
|
||||
assert descriptor["content_type"] == "image/png"
|
||||
assert descriptor["title"] == "chart.png"
|
||||
assert att.content == PNG_1x1
|
||||
|
||||
def test_preview_blob_id_salted_out_of_upload_namespace(self, tmp_path):
|
||||
import hashlib
|
||||
|
||||
s = _make_session()
|
||||
p = tmp_path / "chart.png"
|
||||
p.write_bytes(PNG_1x1)
|
||||
item = s._prepare_open_preview("c1", {"target": str(p)})
|
||||
s._exec_open_preview(item)
|
||||
_, att = s._tool_previews["c1"]
|
||||
# Uploads are keyed bare sha256(body) and save_attachment freezes
|
||||
# `kind` at first insert — an unsalted preview of identical bytes
|
||||
# would collide with (or pre-empt) a real upload's row.
|
||||
assert att.attachment_id != hashlib.sha256(PNG_1x1).hexdigest()
|
||||
assert att.attachment_id == hashlib.sha256(b"preview:" + PNG_1x1).hexdigest()
|
||||
|
||||
def test_path_csv_is_table(self, tmp_path):
|
||||
s = _make_session()
|
||||
p = tmp_path / "results.csv"
|
||||
p.write_text("name,score\na,1\nb,2\n")
|
||||
item = s._prepare_open_preview("c1", {"target": str(p)})
|
||||
s._exec_open_preview(item)
|
||||
descriptor, _ = s._tool_previews["c1"]
|
||||
assert descriptor["kind"] == "table"
|
||||
assert descriptor["content_type"].startswith("text/csv")
|
||||
|
||||
def test_path_missing_errors(self):
|
||||
s = _make_session()
|
||||
item = s._prepare_open_preview("c1", {"target": "/nonexistent/nowhere.txt"})
|
||||
_, msg = s._exec_open_preview(item)
|
||||
assert msg.startswith("Error: file not found")
|
||||
|
||||
def test_path_binary_unpreviewable(self, tmp_path):
|
||||
s = _make_session()
|
||||
p = tmp_path / "blob.bin"
|
||||
p.write_bytes(b"\x00\x01\x02\x03" * 64)
|
||||
item = s._prepare_open_preview("c1", {"target": str(p)})
|
||||
_, msg = s._exec_open_preview(item)
|
||||
assert "not previewable" in msg
|
||||
|
||||
def test_attachment_target_requires_ws_reference(self, monkeypatch):
|
||||
s = _make_session(ws_id="ws-1")
|
||||
monkeypatch.setattr(
|
||||
"turnstone.core.memory.get_attachment",
|
||||
lambda aid: {"content": b"# doc", "mime_type": "text/markdown", "filename": "d.md"},
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
"turnstone.core.memory.attachment_referenced_in_ws",
|
||||
lambda aid, ws: False,
|
||||
)
|
||||
item = s._prepare_open_preview("c1", {"target": "attachment:deadbeef"})
|
||||
_, msg = s._exec_open_preview(item)
|
||||
assert msg.startswith("Error: attachment not found")
|
||||
|
||||
def test_attachment_target_happy_path(self, monkeypatch):
|
||||
s = _make_session(ws_id="ws-1")
|
||||
monkeypatch.setattr(
|
||||
"turnstone.core.memory.get_attachment",
|
||||
lambda aid: {"content": b"# doc", "mime_type": "text/markdown", "filename": "d.md"},
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
"turnstone.core.memory.attachment_referenced_in_ws",
|
||||
lambda aid, ws: True,
|
||||
)
|
||||
item = s._prepare_open_preview("c1", {"target": "attachment:deadbeef"})
|
||||
_, msg = s._exec_open_preview(item)
|
||||
assert not msg.startswith("Error:")
|
||||
descriptor, _ = s._tool_previews["c1"]
|
||||
assert descriptor["kind"] == "markdown"
|
||||
assert descriptor["title"] == "d.md"
|
||||
|
||||
def test_legacy_charset_table_stored_as_utf8(self, monkeypatch):
|
||||
# A latin-1 CSV attachment previews as a table, and the executor
|
||||
# transcodes it to UTF-8 at store time so "café" round-trips instead of
|
||||
# erroring "not previewable".
|
||||
s = _make_session(ws_id="ws-1")
|
||||
latin1_csv = "name,city\nRené,Montréal\n".encode("iso-8859-1")
|
||||
monkeypatch.setattr(
|
||||
"turnstone.core.memory.get_attachment",
|
||||
lambda aid: {
|
||||
"content": latin1_csv,
|
||||
"mime_type": "text/csv; charset=iso-8859-1",
|
||||
"filename": "people.csv",
|
||||
},
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
"turnstone.core.memory.attachment_referenced_in_ws",
|
||||
lambda aid, ws: True,
|
||||
)
|
||||
item = s._prepare_open_preview("c1", {"target": "attachment:deadbeef"})
|
||||
_, msg = s._exec_open_preview(item)
|
||||
assert not msg.startswith("Error:")
|
||||
descriptor, att = s._tool_previews["c1"]
|
||||
assert descriptor["kind"] == "table"
|
||||
assert descriptor["content_type"].startswith("text/csv")
|
||||
# Stored bytes are valid UTF-8 with the accented characters preserved.
|
||||
assert att.content.decode("utf-8") == "name,city\nRené,Montréal\n"
|
||||
|
||||
def test_title_override_wins(self, tmp_path):
|
||||
s = _make_session()
|
||||
p = tmp_path / "x.csv"
|
||||
p.write_text("a,b\n")
|
||||
item = s._prepare_open_preview("c1", {"target": str(p), "title": "Q3 numbers"})
|
||||
s._exec_open_preview(item)
|
||||
descriptor, _ = s._tool_previews["c1"]
|
||||
assert descriptor["title"] == "Q3 numbers"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Trajectory / history / storage seams
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestDescriptorSeams:
|
||||
DESCRIPTOR = {
|
||||
"kind": "web",
|
||||
"title": "T",
|
||||
"source": "https://a.io",
|
||||
"attachment_id": "abc",
|
||||
"content_type": "text/html; charset=utf-8",
|
||||
"size": 7,
|
||||
}
|
||||
|
||||
def test_turn_roundtrip(self):
|
||||
turn = turn_from_dict(
|
||||
{
|
||||
"role": "tool",
|
||||
"tool_call_id": "c1",
|
||||
"content": "Preview shown",
|
||||
"_preview": self.DESCRIPTOR,
|
||||
}
|
||||
)
|
||||
assert turn.meta.extra["preview"] == self.DESCRIPTOR
|
||||
out = turn_to_dict(turn)
|
||||
assert out["_preview"] == self.DESCRIPTOR
|
||||
|
||||
def test_history_projection_carries_preview(self):
|
||||
from turnstone.core.history_decoration import project_history_messages
|
||||
|
||||
msgs = [
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": "",
|
||||
"tool_calls": [
|
||||
{"id": "c1", "function": {"name": "open_preview", "arguments": "{}"}}
|
||||
],
|
||||
},
|
||||
{
|
||||
"role": "tool",
|
||||
"tool_call_id": "c1",
|
||||
"content": "Preview shown to the user: T (web, 7 bytes)",
|
||||
"_preview": self.DESCRIPTOR,
|
||||
},
|
||||
]
|
||||
history = project_history_messages(msgs)
|
||||
tool_entries = [h for h in history if h.get("role") == "tool"]
|
||||
assert tool_entries and tool_entries[0]["preview"] == self.DESCRIPTOR
|
||||
|
||||
def test_reconstruct_routes_tool_preview_meta(self):
|
||||
import json
|
||||
|
||||
from turnstone.core.storage._utils import reconstruct_turns
|
||||
|
||||
# Row layout per reconstruct_turns' unpack: (row_id, role, content,
|
||||
# tool_name, tool_call_id, provider_data, tool_calls_json, source,
|
||||
# event_id, is_error, meta).
|
||||
row = (
|
||||
1,
|
||||
"tool",
|
||||
"ok",
|
||||
"open_preview",
|
||||
"c1",
|
||||
None,
|
||||
None,
|
||||
None,
|
||||
7,
|
||||
0,
|
||||
json.dumps({"effect_status": "unknown", "preview": self.DESCRIPTOR}),
|
||||
)
|
||||
turns = reconstruct_turns([row], "ws-1", attachments_by_msg={})
|
||||
assert turns[0].role is Role.TOOL
|
||||
assert turns[0].meta.extra["preview"] == self.DESCRIPTOR
|
||||
assert turns[0].meta.extra["effect_status"] == "unknown"
|
||||
|
||||
def test_reconstruct_skips_preview_blob_refs(self):
|
||||
"""A preview blob on a tool row's ref-list must NOT become a content
|
||||
block — it is meta-addressed frontend content, and a content block
|
||||
would be materialized onto the wire on reload."""
|
||||
from turnstone.core.storage._utils import reconstruct_turns
|
||||
|
||||
row = (
|
||||
1,
|
||||
"tool",
|
||||
"ok",
|
||||
"open_preview",
|
||||
"c1",
|
||||
None,
|
||||
None,
|
||||
None,
|
||||
None,
|
||||
0,
|
||||
None,
|
||||
)
|
||||
atts = {
|
||||
1: [
|
||||
{
|
||||
"attachment_id": "abc",
|
||||
"kind": "preview",
|
||||
"filename": "preview-web",
|
||||
"mime_type": "text/html; charset=utf-8",
|
||||
"size_bytes": 7,
|
||||
},
|
||||
{
|
||||
"attachment_id": "img1",
|
||||
"kind": "image",
|
||||
"filename": "shot.png",
|
||||
"mime_type": "image/png",
|
||||
"size_bytes": 9,
|
||||
},
|
||||
]
|
||||
}
|
||||
turns = reconstruct_turns([row], "ws-1", attachments_by_msg=atts)
|
||||
kinds = [b.kind for b in turns[0].content if b.__class__.__name__ == "AttachmentRef"]
|
||||
# The vision lane still reconstructs; the preview blob does not.
|
||||
assert kinds == ["image"]
|
||||
|
||||
def test_preview_route_scope_is_read(self):
|
||||
from turnstone.core.auth import required_scope
|
||||
|
||||
assert required_scope("GET", "/v1/api/workstreams/ws1/attachments/abc/preview") == "read"
|
||||
assert (
|
||||
required_scope("GET", "/node/n1/v1/api/workstreams/ws1/attachments/abc/preview")
|
||||
== "read"
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# fetch_with_ssrf_guard — per-hop redirect screening (core/web.py)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class _FakeHop:
|
||||
"""client.stream() double: a context manager yielding chunked body bytes."""
|
||||
|
||||
def __init__(self, status, headers=None, body=b""):
|
||||
self.status_code = status
|
||||
self.headers = headers or {}
|
||||
self._chunks = body if isinstance(body, list) else [body]
|
||||
|
||||
def __enter__(self):
|
||||
return self
|
||||
|
||||
def __exit__(self, *a):
|
||||
return False
|
||||
|
||||
def iter_bytes(self):
|
||||
yield from self._chunks
|
||||
|
||||
|
||||
class _FakeClient:
|
||||
"""httpx.Client double: serves a scripted {url: response} table."""
|
||||
|
||||
calls: list[str] = []
|
||||
table: dict[str, _FakeHop] = {}
|
||||
|
||||
def __init__(self, **kwargs):
|
||||
pass
|
||||
|
||||
def __enter__(self):
|
||||
return self
|
||||
|
||||
def __exit__(self, *a):
|
||||
return False
|
||||
|
||||
def stream(self, method, url):
|
||||
_FakeClient.calls.append(url)
|
||||
return _FakeClient.table[url]
|
||||
|
||||
|
||||
class TestFetchWithSsrfGuard:
|
||||
def _wire(self, monkeypatch, table):
|
||||
_FakeClient.calls = []
|
||||
_FakeClient.table = table
|
||||
monkeypatch.setattr("turnstone.core.web.httpx.Client", _FakeClient)
|
||||
|
||||
def test_follows_public_redirect_chain(self, monkeypatch):
|
||||
from turnstone.core.web import fetch_with_ssrf_guard
|
||||
|
||||
self._wire(
|
||||
monkeypatch,
|
||||
{
|
||||
"https://a.example/": _FakeHop(302, {"location": "https://b.example/x"}),
|
||||
"https://b.example/x": _FakeHop(200, {}, body=b"landed"),
|
||||
},
|
||||
)
|
||||
monkeypatch.setattr("turnstone.core.web.check_ssrf", lambda url: None)
|
||||
resp = fetch_with_ssrf_guard("https://a.example/", timeout=5)
|
||||
assert resp.status_code == 200
|
||||
assert _FakeClient.calls == ["https://a.example/", "https://b.example/x"]
|
||||
|
||||
def test_private_hop_blocked_before_request(self, monkeypatch):
|
||||
import pytest
|
||||
|
||||
from turnstone.core.web import fetch_with_ssrf_guard
|
||||
|
||||
self._wire(
|
||||
monkeypatch,
|
||||
{
|
||||
"https://a.example/": _FakeHop(302, {"location": "http://169.254.169.254/latest"}),
|
||||
},
|
||||
)
|
||||
blocked = {"http://169.254.169.254/latest": "Blocked: private"}
|
||||
monkeypatch.setattr("turnstone.core.web.check_ssrf", lambda url: blocked.get(url))
|
||||
with pytest.raises(ValueError, match="Blocked: private"):
|
||||
fetch_with_ssrf_guard("https://a.example/", timeout=5)
|
||||
# The load-bearing assertion: the private hop was NEVER requested.
|
||||
assert _FakeClient.calls == ["https://a.example/"]
|
||||
|
||||
def test_relative_location_resolves_against_current(self, monkeypatch):
|
||||
from turnstone.core.web import fetch_with_ssrf_guard
|
||||
|
||||
self._wire(
|
||||
monkeypatch,
|
||||
{
|
||||
"https://a.example/start": _FakeHop(301, {"location": "/moved"}),
|
||||
"https://a.example/moved": _FakeHop(200, {}),
|
||||
},
|
||||
)
|
||||
monkeypatch.setattr("turnstone.core.web.check_ssrf", lambda url: None)
|
||||
resp = fetch_with_ssrf_guard("https://a.example/start", timeout=5)
|
||||
assert resp.status_code == 200
|
||||
# The realized response carries the FINAL hop's URL — open_preview's
|
||||
# descriptor source and stored <base href> both key off it.
|
||||
assert str(resp.url) == "https://a.example/moved"
|
||||
|
||||
def test_redirect_loop_capped(self, monkeypatch):
|
||||
import pytest
|
||||
|
||||
from turnstone.core.web import fetch_with_ssrf_guard
|
||||
|
||||
self._wire(
|
||||
monkeypatch,
|
||||
{"https://a.example/": _FakeHop(302, {"location": "https://a.example/"})},
|
||||
)
|
||||
monkeypatch.setattr("turnstone.core.web.check_ssrf", lambda url: None)
|
||||
with pytest.raises(ValueError, match="redirects"):
|
||||
fetch_with_ssrf_guard("https://a.example/", timeout=5)
|
||||
|
||||
def test_body_over_budget_aborts(self, monkeypatch):
|
||||
import pytest
|
||||
|
||||
from turnstone.core.web import fetch_with_ssrf_guard
|
||||
|
||||
self._wire(
|
||||
monkeypatch,
|
||||
{"https://a.example/": _FakeHop(200, {}, body=[b"aaaa", b"bbbb", b"cccc"])},
|
||||
)
|
||||
monkeypatch.setattr("turnstone.core.web.check_ssrf", lambda url: None)
|
||||
with pytest.raises(ValueError, match="fetch limit"):
|
||||
fetch_with_ssrf_guard("https://a.example/", timeout=5, max_bytes=10)
|
||||
|
||||
def test_redirect_hop_body_never_read(self, monkeypatch):
|
||||
from turnstone.core.web import fetch_with_ssrf_guard
|
||||
|
||||
class _BodyBomb(_FakeHop):
|
||||
def iter_bytes(self):
|
||||
raise AssertionError("redirect hop body must not be read")
|
||||
|
||||
self._wire(
|
||||
monkeypatch,
|
||||
{
|
||||
"https://a.example/": _BodyBomb(302, {"location": "https://b.example/x"}),
|
||||
"https://b.example/x": _FakeHop(200, {}, body=b"ok"),
|
||||
},
|
||||
)
|
||||
monkeypatch.setattr("turnstone.core.web.check_ssrf", lambda url: None)
|
||||
resp = fetch_with_ssrf_guard("https://a.example/", timeout=5)
|
||||
assert resp.status_code == 200
|
||||
assert resp.content == b"ok"
|
||||
|
||||
def test_stale_framing_headers_dropped(self, monkeypatch):
|
||||
from turnstone.core.web import fetch_with_ssrf_guard
|
||||
|
||||
self._wire(
|
||||
monkeypatch,
|
||||
{
|
||||
"https://a.example/": _FakeHop(
|
||||
200,
|
||||
{
|
||||
"content-encoding": "gzip",
|
||||
"content-length": "999",
|
||||
"content-type": "text/html; charset=utf-8",
|
||||
},
|
||||
body=b"<html>hi</html>",
|
||||
)
|
||||
},
|
||||
)
|
||||
monkeypatch.setattr("turnstone.core.web.check_ssrf", lambda url: None)
|
||||
resp = fetch_with_ssrf_guard("https://a.example/", timeout=5)
|
||||
# iter_bytes() hands the guard content-DECODED bytes — a surviving
|
||||
# content-encoding would make .text try to gunzip plain text, and the
|
||||
# upstream content-length no longer describes the body carried.
|
||||
assert "content-encoding" not in resp.headers
|
||||
assert resp.headers.get("content-length") != "999"
|
||||
assert resp.headers.get("content-type") == "text/html; charset=utf-8"
|
||||
assert resp.text == "<html>hi</html>"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Cancelled-batch synthesis — a staged preview whose descriptor already
|
||||
# reached the frontend must commit, not vanish (session.py review fix)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestCancelledBatchPreservesPreview:
|
||||
def test_synthesize_commits_staged_preview(self, monkeypatch):
|
||||
import json as _json
|
||||
|
||||
from turnstone.core.attachments import Attachment
|
||||
from turnstone.core.trajectory import Turn
|
||||
|
||||
s = _make_session(ws_id="ws-1")
|
||||
descriptor = {
|
||||
"kind": "web",
|
||||
"title": "T",
|
||||
"source": "https://a.io",
|
||||
"attachment_id": "abc",
|
||||
"content_type": "text/html; charset=utf-8",
|
||||
"size": 7,
|
||||
}
|
||||
att = Attachment(
|
||||
attachment_id="abc",
|
||||
filename="preview-web",
|
||||
mime_type="text/html; charset=utf-8",
|
||||
kind="preview",
|
||||
content=b"<p>x</p>",
|
||||
)
|
||||
s._tool_previews["c1"] = (descriptor, att)
|
||||
# Assistant turn with one UNANSWERED call — the cancel shape.
|
||||
s.messages.append(
|
||||
turn_from_dict(
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": "",
|
||||
"tool_calls": [
|
||||
{
|
||||
"id": "c1",
|
||||
"type": "function",
|
||||
"function": {"name": "open_preview", "arguments": "{}"},
|
||||
}
|
||||
],
|
||||
}
|
||||
)
|
||||
)
|
||||
s._msg_tokens.append(1)
|
||||
|
||||
saved = {}
|
||||
monkeypatch.setattr(
|
||||
"turnstone.core.session.save_message",
|
||||
lambda ws, role, content, name, **kw: (
|
||||
saved.update({"meta": kw.get("meta"), "row": 42}) or 42
|
||||
),
|
||||
)
|
||||
persisted = {}
|
||||
monkeypatch.setattr(
|
||||
ChatSession,
|
||||
"_persist_attachment_refs",
|
||||
lambda self, row_id, atts, origin="upload": persisted.update(
|
||||
{"row": row_id, "ids": [a.attachment_id for a in atts], "origin": origin}
|
||||
),
|
||||
)
|
||||
|
||||
s._synthesize_cancelled_results("Cancelled by user.")
|
||||
|
||||
# Side channel drained; descriptor + blob committed with the turn.
|
||||
assert "c1" not in s._tool_previews
|
||||
meta = _json.loads(saved["meta"])
|
||||
assert meta["preview"] == descriptor
|
||||
assert meta["effect_status"] == "unknown"
|
||||
assert persisted == {"row": 42, "ids": ["abc"], "origin": "tool"}
|
||||
# The in-memory synthesized turn carries the descriptor too.
|
||||
tool_turns = [t for t in s.messages if isinstance(t, Turn) and t.role is Role.TOOL]
|
||||
assert tool_turns and tool_turns[-1].meta.extra.get("preview") == descriptor
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# tools.allow_private_network — the self-hoster opt-in (admin Settings → Tools)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestAllowPrivateNetwork:
|
||||
def test_screen_public_url_passes(self):
|
||||
from turnstone.core.session import _screen_tool_url
|
||||
|
||||
err, private = _screen_tool_url("https://example.com/x", False)
|
||||
assert err is None and private is False
|
||||
|
||||
def test_screen_private_blocked_with_discoverable_hint(self):
|
||||
from turnstone.core.session import _screen_tool_url
|
||||
|
||||
err, private = _screen_tool_url("http://10.0.0.7/grafana", False)
|
||||
assert err is not None and private is False
|
||||
# The refusal teaches the knob (mirrors the oidc opt-in hint pattern).
|
||||
assert "tools.allow_private_network" in err
|
||||
assert "Settings" in err
|
||||
|
||||
def test_screen_private_allowed_when_opted_in(self):
|
||||
from turnstone.core.session import _screen_tool_url
|
||||
|
||||
err, private = _screen_tool_url("http://10.0.0.7/grafana", True)
|
||||
assert err is None and private is True
|
||||
|
||||
def test_screen_invalid_url_never_hints(self):
|
||||
from turnstone.core.session import _screen_tool_url
|
||||
|
||||
err, private = _screen_tool_url("http://", True)
|
||||
assert err is not None and private is False
|
||||
assert "allow_private_network" not in err
|
||||
|
||||
def test_bare_session_defaults_strict(self):
|
||||
# No ConfigStore (CLI / eval surface) → no admin opted in → strict.
|
||||
s = _make_session()
|
||||
assert s._allow_private_network() is False
|
||||
|
||||
def test_prepare_web_fetch_private_opted_in(self, monkeypatch):
|
||||
s = _make_session()
|
||||
monkeypatch.setattr(ChatSession, "_allow_private_network", lambda self: True)
|
||||
item = s._prepare_web_fetch(
|
||||
"c1", {"url": "http://192.168.1.50:3000/d/home", "question": "what is shown?"}
|
||||
)
|
||||
assert "error" not in item
|
||||
assert item["needs_approval"] is True # the human gate stays
|
||||
assert "(private network)" in item["header"]
|
||||
assert item["allow_private_origin"] is True
|
||||
|
||||
def test_prepare_open_preview_private_opted_in(self, monkeypatch):
|
||||
s = _make_session()
|
||||
monkeypatch.setattr(ChatSession, "_allow_private_network", lambda self: True)
|
||||
item = s._prepare_open_preview("c1", {"target": "http://192.168.1.50:3000/d/home"})
|
||||
assert "error" not in item
|
||||
assert item["needs_approval"] is True
|
||||
assert "(private network)" in item["header"]
|
||||
assert item["allow_private_origin"] is True
|
||||
|
||||
def test_prepare_private_still_blocked_by_default(self, monkeypatch):
|
||||
s = _make_session()
|
||||
monkeypatch.setattr(ChatSession, "_allow_private_network", lambda self: False)
|
||||
for prepare, args in (
|
||||
(s._prepare_web_fetch, {"url": "http://10.0.0.7/x", "question": "q"}),
|
||||
(s._prepare_open_preview, {"target": "http://10.0.0.7/x"}),
|
||||
):
|
||||
item = prepare("c1", args)
|
||||
assert "error" in item
|
||||
assert "tools.allow_private_network" in item["error"]
|
||||
|
||||
def test_executor_passes_private_origin_to_guard(self, monkeypatch):
|
||||
s = _make_session()
|
||||
monkeypatch.setattr(ChatSession, "_allow_private_network", lambda self: True)
|
||||
seen = {}
|
||||
|
||||
def _capture(url, **kw):
|
||||
seen.update(kw, url=url)
|
||||
return _fake_response(url, b"<html><head></head><body>x</body></html>", "text/html")
|
||||
|
||||
monkeypatch.setattr("turnstone.core.session.fetch_with_ssrf_guard", _capture)
|
||||
item = s._prepare_open_preview("c1", {"target": "http://10.0.0.7/status"})
|
||||
s._exec_open_preview(item)
|
||||
assert seen["allow_private_origin"] is True
|
||||
|
||||
def test_guard_skips_hop_screen_for_private_origin(self, monkeypatch):
|
||||
from turnstone.core.web import fetch_with_ssrf_guard
|
||||
|
||||
_FakeClient.calls = []
|
||||
_FakeClient.table = {
|
||||
"http://10.0.0.7/a": _FakeHop(302, {"location": "http://10.0.0.8/b"}),
|
||||
"http://10.0.0.8/b": _FakeHop(200, {}),
|
||||
}
|
||||
monkeypatch.setattr("turnstone.core.web.httpx.Client", _FakeClient)
|
||||
|
||||
def _explode(url):
|
||||
raise AssertionError("hop screening must be skipped for a private origin")
|
||||
|
||||
monkeypatch.setattr("turnstone.core.web.check_ssrf", _explode)
|
||||
resp = fetch_with_ssrf_guard("http://10.0.0.7/a", timeout=5, allow_private_origin=True)
|
||||
assert resp.status_code == 200
|
||||
assert _FakeClient.calls == ["http://10.0.0.7/a", "http://10.0.0.8/b"]
|
||||
|
||||
def test_registry_entry_shape(self):
|
||||
from turnstone.core.settings_registry import SETTINGS
|
||||
|
||||
d = SETTINGS["tools.allow_private_network"]
|
||||
assert d.type == "bool"
|
||||
assert d.default is False
|
||||
assert d.section == "tools"
|
||||
assert d.help # the admin form renders this — it must explain the caveat
|
||||
@@ -236,6 +236,32 @@ class TestCredentialLeakage:
|
||||
assert r.sanitized is not None
|
||||
assert "s3cretpw" not in r.sanitized
|
||||
|
||||
def test_sqlalchemy_driver_connection_string(self) -> None:
|
||||
# SQLAlchemy dialect+driver URLs must match — the bare-dialect
|
||||
# list alone leaked these (only +psycopg was enumerated).
|
||||
for url in (
|
||||
"postgresql+psycopg2://admin:s3cret_pass@db.internal:5432/prod",
|
||||
"postgresql+asyncpg://admin:s3cret_pass@db.internal/prod",
|
||||
"mysql+pymysql://admin:s3cret_pass@db.internal/prod",
|
||||
):
|
||||
r = evaluate_output(url)
|
||||
assert "connection_string_leak" in r.flags, url
|
||||
assert r.sanitized is not None, url
|
||||
assert "s3cret_pass" not in r.sanitized, url
|
||||
assert ":[REDACTED:password]@" in r.sanitized, url
|
||||
|
||||
def test_uppercase_scheme_connection_string(self) -> None:
|
||||
# RFC 3986 schemes are case-insensitive; an uppercase scheme must
|
||||
# not bypass redaction.
|
||||
for url in (
|
||||
"POSTGRESQL+PSYCOPG2://admin:s3cret_pass@db.internal/prod",
|
||||
"HTTPS://admin:s3cret_pass@api.internal/x",
|
||||
):
|
||||
r = evaluate_output(url)
|
||||
assert "connection_string_leak" in r.flags, url
|
||||
assert r.sanitized is not None, url
|
||||
assert "s3cret_pass" not in r.sanitized, url
|
||||
|
||||
def test_bearer_scheme_case_insensitive(self) -> None:
|
||||
# RFC 7235 scheme name is case-insensitive.
|
||||
r = evaluate_output("authorization: bearer eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiIxIn0.sig12345")
|
||||
|
||||
@@ -15,6 +15,7 @@ from turnstone.core.output_guard_judge import (
|
||||
OutputJudgeVerdict,
|
||||
_extract_json,
|
||||
)
|
||||
from turnstone.core.providers._protocol import ModelCapabilities
|
||||
|
||||
|
||||
def _make_provider(
|
||||
@@ -68,6 +69,76 @@ def _make_judge(
|
||||
return judge
|
||||
|
||||
|
||||
class TestCapabilityThreading:
|
||||
"""#823: the output-guard judge threads resolved capabilities to
|
||||
create_completion, like every other create_completion caller."""
|
||||
|
||||
@staticmethod
|
||||
def _recording_provider() -> tuple[Any, dict[str, Any]]:
|
||||
captured: dict[str, Any] = {}
|
||||
|
||||
def _cc(**kwargs: Any) -> Any:
|
||||
captured.update(kwargs)
|
||||
result = MagicMock()
|
||||
result.content = '{"risk_level": "none", "flags": []}'
|
||||
return result
|
||||
|
||||
provider = MagicMock()
|
||||
provider.provider_name = "openai"
|
||||
provider.get_capabilities = MagicMock(
|
||||
return_value=ModelCapabilities(context_window=200_000)
|
||||
)
|
||||
provider.create_completion = MagicMock(side_effect=_cc)
|
||||
return provider, captured
|
||||
|
||||
def test_fallback_threads_session_capabilities(self) -> None:
|
||||
provider, captured = self._recording_provider()
|
||||
sess_caps = ModelCapabilities(context_window=40_000, effort_passthrough=True)
|
||||
client = MagicMock(base_url="http://s", api_key="k")
|
||||
judge = OutputGuardJudge(
|
||||
config=JudgeConfig(output_guard_llm=True), # no alias → fallback
|
||||
session_provider=provider,
|
||||
session_client=client,
|
||||
session_model="m",
|
||||
session_capabilities=sess_caps,
|
||||
)
|
||||
judge._create_client = lambda: client # type: ignore[method-assign]
|
||||
assert judge._capabilities is sess_caps
|
||||
v = judge.evaluate("a small, safe output", func_name="bash", call_id="c1")
|
||||
assert v.succeeded
|
||||
assert captured["capabilities"] is sess_caps
|
||||
|
||||
def test_alias_merges_operator_capabilities(self) -> None:
|
||||
provider, captured = self._recording_provider()
|
||||
provider.get_capabilities = MagicMock(return_value=ModelCapabilities(supports_tools=True))
|
||||
cfg = MagicMock()
|
||||
cfg.context_window = 64_000
|
||||
cfg.capabilities = {"supports_tools": False}
|
||||
registry = MagicMock()
|
||||
registry.has_alias.return_value = True
|
||||
registry.resolve.return_value = (
|
||||
MagicMock(base_url="http://a", api_key="k"),
|
||||
"local-9b",
|
||||
cfg,
|
||||
)
|
||||
registry.get_provider.return_value = provider
|
||||
client = MagicMock(base_url="http://s", api_key="k")
|
||||
judge = OutputGuardJudge(
|
||||
config=JudgeConfig(output_guard_llm=True, output_guard_model="og"),
|
||||
session_provider=_make_provider(),
|
||||
session_client=client,
|
||||
session_model="m",
|
||||
session_capabilities=MagicMock(context_window=100_000),
|
||||
model_registry=registry,
|
||||
)
|
||||
judge._create_client = lambda: client # type: ignore[method-assign]
|
||||
assert judge._capabilities.supports_tools is False # operator override applied
|
||||
v = judge.evaluate("a small, safe output", func_name="bash", call_id="c1")
|
||||
assert v.succeeded
|
||||
assert captured["capabilities"] is judge._capabilities
|
||||
assert captured["capabilities"].supports_tools is False
|
||||
|
||||
|
||||
class TestVerdictDataclass:
|
||||
def test_default_verdict_with_no_error_succeeds(self) -> None:
|
||||
# A default OutputJudgeVerdict has risk_level='none' and error=''
|
||||
@@ -291,14 +362,16 @@ class TestOversizeGuard:
|
||||
session_provider=provider,
|
||||
session_client=MagicMock(base_url="http://test", api_key="k"),
|
||||
session_model="test-model",
|
||||
context_window=40_000, # the session's real window
|
||||
# The session's real window rides in the resolved caps the caller
|
||||
# passes; the guard must key off it, not provider.get_capabilities().
|
||||
session_capabilities=MagicMock(context_window=40_000),
|
||||
)
|
||||
assert judge._judge_context_window == 40_000
|
||||
|
||||
def test_zero_window_coerced_away_on_both_paths(self) -> None:
|
||||
"""A config.toml context_window=0 (present but unusable) must not zero
|
||||
the guard: coerce to the session window (alias path) / the default."""
|
||||
from turnstone.core.output_guard_judge import _DEFAULT_JUDGE_CONTEXT_WINDOW
|
||||
from turnstone.core.judge import _DEFAULT_JUDGE_CONTEXT_WINDOW
|
||||
|
||||
# Alias path: ModelConfig.context_window == 0 → session window.
|
||||
cfg = MagicMock()
|
||||
@@ -313,7 +386,7 @@ class TestOversizeGuard:
|
||||
session_client=MagicMock(base_url="http://s", api_key="s"),
|
||||
session_model="m",
|
||||
model_registry=registry,
|
||||
context_window=64_000,
|
||||
session_capabilities=MagicMock(context_window=64_000),
|
||||
)
|
||||
assert alias_judge._judge_context_window == 64_000
|
||||
|
||||
|
||||
@@ -0,0 +1,305 @@
|
||||
"""Unit tests for the preview-content policy module (``turnstone/core/preview.py``).
|
||||
|
||||
Pure-function coverage: kind resolution precedence (magic bytes → MIME hint →
|
||||
extension → UTF-8 fallback), the explicit ``kind`` override lanes, base-href
|
||||
injection, title extraction, and the per-MIME serving headers the route
|
||||
attaches. The tool executor and the HTTP route are covered separately
|
||||
(``test_open_preview_tool.py`` / ``test_server_attachments_endpoints.py``).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from turnstone.core.attachments import IMAGE_SIZE_CAP, PDF_SIZE_CAP, TEXT_DOC_SIZE_CAP
|
||||
from turnstone.core.preview import (
|
||||
PREVIEW_BLOB_KIND,
|
||||
PREVIEW_KINDS,
|
||||
PREVIEW_SERVE_MIMES,
|
||||
PREVIEW_SIZE_CAPS,
|
||||
build_preview_descriptor,
|
||||
inject_base_href,
|
||||
page_title,
|
||||
preview_response_headers,
|
||||
resolve_preview_kind,
|
||||
transcode_text,
|
||||
)
|
||||
|
||||
PNG_1x1 = (
|
||||
b"\x89PNG\r\n\x1a\n\x00\x00\x00\rIHDR\x00\x00\x00\x01\x00\x00\x00\x01"
|
||||
b"\x08\x06\x00\x00\x00\x1f\x15\xc4\x89\x00\x00\x00\rIDATx\x9cc\xfc\xcf"
|
||||
b"\xc0\xc0\xc0\x00\x00\x00\x05\x00\x01\xa5\xf6E@\x00\x00\x00\x00IEND\xaeB`\x82"
|
||||
)
|
||||
PDF_MIN = b"%PDF-1.4 fake body"
|
||||
HTML_DOC = b"<html><head><title>Acme Pricing</title></head><body>hi</body></html>"
|
||||
|
||||
|
||||
class TestResolvePreviewKind:
|
||||
def test_magic_bytes_win_over_everything(self):
|
||||
# A PNG claiming to be CSV by both MIME and extension is an image.
|
||||
assert resolve_preview_kind("text/csv", "data.csv", PNG_1x1) == ("image", "image/png")
|
||||
assert resolve_preview_kind("text/plain", "doc.txt", PDF_MIN) == (
|
||||
"pdf",
|
||||
"application/pdf",
|
||||
)
|
||||
|
||||
def test_mime_hint_html(self):
|
||||
kind, mime = resolve_preview_kind("text/html; charset=iso-8859-1", "page", HTML_DOC)
|
||||
assert kind == "web"
|
||||
assert mime == "text/html; charset=utf-8"
|
||||
|
||||
def test_mime_hint_families(self):
|
||||
assert resolve_preview_kind("text/csv", "x", b"a,b\n1,2")[0] == "table"
|
||||
assert resolve_preview_kind("application/json", "x", b"[]") == (
|
||||
"table",
|
||||
"application/json",
|
||||
)
|
||||
assert resolve_preview_kind("text/markdown", "x", b"# hi")[0] == "markdown"
|
||||
assert resolve_preview_kind("text/x-log", "x", b"line")[0] == "text"
|
||||
|
||||
def test_extension_fallback_when_no_mime(self):
|
||||
assert resolve_preview_kind("", "report.html", HTML_DOC)[0] == "web"
|
||||
assert resolve_preview_kind("", "data.tsv", b"a\tb")[0] == "table"
|
||||
assert resolve_preview_kind("", "notes.md", b"# t")[0] == "markdown"
|
||||
# URL tails strip query/fragment before the extension check.
|
||||
assert resolve_preview_kind("", "https://x.io/a.csv?dl=1#f", b"a,b")[0] == "table"
|
||||
|
||||
def test_utf8_text_fallback(self):
|
||||
assert resolve_preview_kind("", "LICENSE", b"MIT License") == (
|
||||
"text",
|
||||
"text/plain; charset=utf-8",
|
||||
)
|
||||
|
||||
def test_binary_is_not_previewable(self):
|
||||
assert resolve_preview_kind("", "blob.bin", b"\x00\x01\x02\x03" * 8) is None
|
||||
# Text-DECLARED binary is misdeclared, not previewable text.
|
||||
assert resolve_preview_kind("text/plain", "x", b"\x00\xff" * 8) is None
|
||||
assert resolve_preview_kind("application/octet-stream", "x", b"\x00" * 32) is None
|
||||
|
||||
def test_override_validates_bytes(self):
|
||||
# image override on non-image bytes fails rather than mislabeling.
|
||||
assert resolve_preview_kind("", "x", b"not an image", "image") is None
|
||||
assert resolve_preview_kind("", "x", PNG_1x1, "image") == ("image", "image/png")
|
||||
assert resolve_preview_kind("", "x", b"not a pdf", "pdf") is None
|
||||
# Text-family override on binary bytes fails.
|
||||
assert resolve_preview_kind("", "x", b"\x00\x01", "text") is None
|
||||
|
||||
def test_override_forces_view(self):
|
||||
# kind='text' on an HTML doc = view source.
|
||||
assert resolve_preview_kind("text/html", "p.html", HTML_DOC, "text")[0] == "text"
|
||||
# kind='table' keeps the real payload type for the client parser.
|
||||
assert resolve_preview_kind("application/json", "d", b"[1]", "table") == (
|
||||
"table",
|
||||
"application/json",
|
||||
)
|
||||
assert resolve_preview_kind("", "d.tsv", b"a\tb", "table") == (
|
||||
"table",
|
||||
"text/tab-separated-values; charset=utf-8",
|
||||
)
|
||||
assert resolve_preview_kind("", "d.txt", b"a,b", "table") == (
|
||||
"table",
|
||||
"text/csv; charset=utf-8",
|
||||
)
|
||||
|
||||
def test_unknown_override_rejected(self):
|
||||
assert resolve_preview_kind("text/plain", "x", b"hi", "hologram") is None
|
||||
|
||||
|
||||
class TestHtmlHelpers:
|
||||
def test_base_href_inserted_after_head(self):
|
||||
out = inject_base_href("<html><head><meta x></head></html>", "https://a.io/p/q")
|
||||
assert out.startswith('<html><head><base href="https://a.io/p/q">')
|
||||
|
||||
def test_base_href_prepended_without_head(self):
|
||||
out = inject_base_href("<p>bare</p>", "https://a.io/")
|
||||
assert out.startswith('<base href="https://a.io/">')
|
||||
|
||||
def test_existing_base_untouched(self):
|
||||
doc = '<head><base href="https://original/"></head>'
|
||||
assert inject_base_href(doc, "https://other/") == doc
|
||||
|
||||
def test_base_href_attribute_escaped(self):
|
||||
out = inject_base_href("<head></head>", 'https://a.io/"><script>x</script>')
|
||||
assert "<script>" not in out
|
||||
assert ""><script>" in out
|
||||
|
||||
def test_page_title_extraction(self):
|
||||
assert page_title(HTML_DOC.decode()) == "Acme Pricing"
|
||||
assert page_title("<title>a & b\n c</title>") == "a & b c"
|
||||
assert page_title("<p>no title</p>") is None
|
||||
assert page_title("<title></title>") is None
|
||||
|
||||
|
||||
_LOCKED_HTML_CSP = (
|
||||
"sandbox; default-src 'none'; style-src 'unsafe-inline'; img-src data:; font-src data:"
|
||||
)
|
||||
|
||||
|
||||
class TestServingPolicy:
|
||||
def test_html_default_locks_out_remote_assets(self):
|
||||
# Default (no opt-in): sandboxed AND off the network — inline styling +
|
||||
# data-URI images render, but the page can fetch nothing, so previewing
|
||||
# never discloses the viewer to the origin site.
|
||||
h = preview_response_headers("text/html", "page.html")
|
||||
assert h["Content-Security-Policy"] == _LOCKED_HTML_CSP
|
||||
assert h["X-Content-Type-Options"] == "nosniff"
|
||||
assert h["Cache-Control"] == "private, no-store"
|
||||
assert h["Content-Disposition"].startswith("inline;")
|
||||
|
||||
def test_html_assets_opt_in_gets_bare_sandbox_csp(self):
|
||||
# allow_remote_assets=True drops back to the bare sandbox so the page's
|
||||
# own images / CSS load.
|
||||
h = preview_response_headers("text/html", "page.html", allow_remote_assets=True)
|
||||
assert h["Content-Security-Policy"] == "sandbox"
|
||||
assert h["X-Content-Type-Options"] == "nosniff"
|
||||
|
||||
def test_assets_flag_does_not_touch_non_html_kinds(self):
|
||||
for mime in ("application/pdf", "image/png", "text/csv", "text/plain"):
|
||||
assert preview_response_headers(
|
||||
mime, "f", allow_remote_assets=True
|
||||
) == preview_response_headers(mime, "f")
|
||||
|
||||
def test_pdf_gets_no_csp(self):
|
||||
h = preview_response_headers("application/pdf", "doc.pdf")
|
||||
assert "Content-Security-Policy" not in h
|
||||
assert h["X-Content-Type-Options"] == "nosniff"
|
||||
|
||||
def test_other_kinds_keep_full_csp(self):
|
||||
for mime in ("image/png", "text/csv", "text/plain"):
|
||||
h = preview_response_headers(mime, "f")
|
||||
assert h["Content-Security-Policy"] == "default-src 'none'; sandbox"
|
||||
|
||||
def test_filename_header_injection_stripped(self):
|
||||
h = preview_response_headers("text/plain", 'a"\r\nX-Evil: 1')
|
||||
assert "\r" not in h["Content-Disposition"]
|
||||
assert "\n" not in h["Content-Disposition"]
|
||||
assert '"' not in h["Content-Disposition"].split("filename=")[1].strip('"')
|
||||
|
||||
def test_serve_allowlist_covers_every_stored_kind(self):
|
||||
for mime in (
|
||||
"text/html",
|
||||
"application/pdf",
|
||||
"image/png",
|
||||
"image/webp",
|
||||
"text/csv",
|
||||
"text/tab-separated-values",
|
||||
"application/json",
|
||||
"text/markdown",
|
||||
"text/plain",
|
||||
):
|
||||
assert mime in PREVIEW_SERVE_MIMES
|
||||
|
||||
def test_caps_reuse_attachment_constants(self):
|
||||
assert PREVIEW_SIZE_CAPS["image"] == IMAGE_SIZE_CAP
|
||||
assert PREVIEW_SIZE_CAPS["pdf"] == PDF_SIZE_CAP
|
||||
assert PREVIEW_SIZE_CAPS["text"] == TEXT_DOC_SIZE_CAP
|
||||
assert set(PREVIEW_SIZE_CAPS) == set(PREVIEW_KINDS)
|
||||
|
||||
def test_blob_kind_is_outside_model_vocabulary(self):
|
||||
assert PREVIEW_BLOB_KIND not in ("image", "text", "pdf", "audio")
|
||||
|
||||
def test_descriptor_shape(self):
|
||||
d = build_preview_descriptor(
|
||||
kind="web",
|
||||
title="T",
|
||||
source="https://a.io",
|
||||
attachment_id="abc",
|
||||
content_type="text/html; charset=utf-8",
|
||||
size=7,
|
||||
)
|
||||
assert d == {
|
||||
"kind": "web",
|
||||
"title": "T",
|
||||
"source": "https://a.io",
|
||||
"attachment_id": "abc",
|
||||
"content_type": "text/html; charset=utf-8",
|
||||
"size": 7,
|
||||
}
|
||||
|
||||
|
||||
class TestReviewHardening:
|
||||
"""Pins for the review-round fixes (2026-07-07)."""
|
||||
|
||||
def test_filename_folds_to_latin1_safe_ascii(self):
|
||||
# Starlette encodes header values latin-1; em dashes / CJK titles
|
||||
# must fold, not 500 the serving route.
|
||||
h = preview_response_headers("text/html", "Docs — v1.7 日本語.html")
|
||||
h["Content-Disposition"].encode("latin-1") # must not raise
|
||||
h2 = preview_response_headers("text/plain", "——")
|
||||
h2["Content-Disposition"].encode("latin-1")
|
||||
assert (
|
||||
'filename="preview"' in h2["Content-Disposition"]
|
||||
or "filename=" in h2["Content-Disposition"]
|
||||
)
|
||||
|
||||
def test_base_href_never_precedes_doctype(self):
|
||||
doc = "<!DOCTYPE html><body>no head</body>"
|
||||
out = inject_base_href(doc, "https://a.io/")
|
||||
assert out.startswith("<!DOCTYPE html>")
|
||||
assert '<base href="https://a.io/">' in out
|
||||
# <html> without <head> also keeps document order.
|
||||
doc2 = "<!doctype html><html lang=en><body>x</body></html>"
|
||||
out2 = inject_base_href(doc2, "https://a.io/")
|
||||
assert out2.startswith("<!doctype html><html lang=en>")
|
||||
assert out2.index("<base") > out2.index("<html")
|
||||
|
||||
def test_legacy_charset_web_pages_stay_previewable(self):
|
||||
# windows-1252 / iso-8859-1 bytes are not UTF-8; web kind must not
|
||||
# reject them (the executor transcodes at store time).
|
||||
latin1_html = "<html><body>café</body></html>".encode("latin-1")
|
||||
assert resolve_preview_kind("text/html; charset=iso-8859-1", "p", latin1_html) == (
|
||||
"web",
|
||||
"text/html; charset=utf-8",
|
||||
)
|
||||
# Extension lane and explicit override agree.
|
||||
assert resolve_preview_kind("", "page.html", latin1_html)[0] == "web"
|
||||
assert resolve_preview_kind("", "page.bin", latin1_html, "web")[0] == "web"
|
||||
# Non-web text kinds now transcode too — a declared text/csv MIME on
|
||||
# legacy-charset bytes is previewable (was strict-UTF-8-only before).
|
||||
assert resolve_preview_kind("text/csv", "d.csv", latin1_html) == (
|
||||
"table",
|
||||
"text/csv; charset=utf-8",
|
||||
)
|
||||
# …but binary declared as text (a NUL byte) is still rejected.
|
||||
assert resolve_preview_kind("text/csv", "d.csv", b"\x00\x01\x02" * 8) is None
|
||||
|
||||
|
||||
class TestLegacyCharsetText:
|
||||
"""Text-family kinds transcode legacy charsets at store time; only the
|
||||
undeclared fallback lane stays strict UTF-8 (2026-07-07 follow-up)."""
|
||||
|
||||
def test_declared_latin1_csv_is_a_table(self):
|
||||
latin1_csv = "name,city\nRené,Montréal\n".encode("iso-8859-1")
|
||||
# MIME hint carrying the charset.
|
||||
assert resolve_preview_kind("text/csv; charset=iso-8859-1", "d", latin1_csv) == (
|
||||
"table",
|
||||
"text/csv; charset=utf-8",
|
||||
)
|
||||
# Extension lane and explicit override agree — all "declared text".
|
||||
assert resolve_preview_kind("", "data.csv", latin1_csv)[0] == "table"
|
||||
assert resolve_preview_kind("", "data.bin", latin1_csv, "table")[0] == "table"
|
||||
|
||||
def test_declared_text_nul_byte_still_binary(self):
|
||||
# The ladder never fails, so the NUL check is the only binary gate left
|
||||
# for declared text — it must hold in every declared lane.
|
||||
nul = b"a,b\n1,\x00\n"
|
||||
assert resolve_preview_kind("text/csv", "d.csv", nul) is None
|
||||
assert resolve_preview_kind("", "d.csv", nul) is None
|
||||
assert resolve_preview_kind("", "d", nul, "table") is None
|
||||
|
||||
def test_undeclared_non_utf8_still_rejected(self):
|
||||
# No MIME hint, no text-family extension, no override: the bare
|
||||
# fallback lane stays strict UTF-8 — cp1252+replace would otherwise
|
||||
# classify arbitrary binary as text.
|
||||
assert resolve_preview_kind("", "mystery", b"caf\xe9 nonsense \xff\xfe") is None
|
||||
|
||||
def test_transcode_ladder_rungs(self):
|
||||
# (a) charset= parameter honored.
|
||||
assert transcode_text("café".encode("iso-8859-1"), "text/csv; charset=iso-8859-1") == "café"
|
||||
# (b) UTF-8 when the charset is absent / unknown.
|
||||
assert transcode_text("héllo".encode(), "text/plain") == "héllo"
|
||||
assert transcode_text("héllo".encode(), "text/plain; charset=made-up") == "héllo"
|
||||
# (c) cp1252 fallback rung: smart quotes are invalid UTF-8 (the shape a
|
||||
# legacy .txt with no charset takes — empty mime hint), decoded via the
|
||||
# last rung rather than erroring.
|
||||
smart = b"he said \x93hi\x94"
|
||||
out = transcode_text(smart, "")
|
||||
assert "“" in out and "”" in out
|
||||
@@ -0,0 +1,214 @@
|
||||
"""Static guards for the preview pane frontend (shared_static/preview.js and
|
||||
its wiring through conversation.js / interactive.js / shell.js).
|
||||
|
||||
Same posture as ``test_shell_js.py``: Python-side string-presence assertions
|
||||
that catch the silent one-line regression (a renamed export, a dropped
|
||||
sandbox attribute, a de-registered pane type). Parse + sink + var guards for
|
||||
``preview.js`` itself live in ``test_shell_js.py``'s bundle sweeps.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
_ROOT = Path(__file__).resolve().parent.parent
|
||||
_SHARED = _ROOT / "turnstone/shared_static"
|
||||
_PANE_JS = _SHARED / "pane.js"
|
||||
_PREVIEW_JS = _SHARED / "preview.js"
|
||||
_CONVERSATION_JS = _SHARED / "conversation.js"
|
||||
_INTERACTIVE_JS = _SHARED / "interactive.js"
|
||||
_SHELL_JS = _SHARED / "shell.js"
|
||||
_PREVIEW_CSS = _SHARED / "preview.css"
|
||||
_UI_INDEX = _ROOT / "turnstone/ui/static/index.html"
|
||||
_CONSOLE_INDEX = _ROOT / "turnstone/console/static/index.html"
|
||||
|
||||
|
||||
def _read(p: Path) -> str:
|
||||
return p.read_text(encoding="utf-8")
|
||||
|
||||
|
||||
class TestPreviewPaneModule:
|
||||
def test_factory_exported(self) -> None:
|
||||
assert "export function createPreviewPane" in _read(_PREVIEW_JS)
|
||||
|
||||
def test_web_iframe_is_fully_sandboxed(self) -> None:
|
||||
"""The web renderer must keep the empty-sandbox attribute — every
|
||||
capability (scripts, same-origin, forms, popups) stays off. Dropping
|
||||
or loosening it turns fetched pages into live documents."""
|
||||
body = _read(_PREVIEW_JS)
|
||||
assert 'frame.setAttribute("sandbox", "")' in body
|
||||
assert 'frame.setAttribute("referrerpolicy", "no-referrer")' in body
|
||||
|
||||
def test_pdf_iframe_is_not_sandboxed(self) -> None:
|
||||
"""Deliberate asymmetry: Chromium's PDF viewer refuses to paint in a
|
||||
sandboxed context. The renderer comment carries the rationale; this
|
||||
pins that renderPdf never gained a sandbox attribute by copy-paste."""
|
||||
body = _read(_PREVIEW_JS)
|
||||
pdf_fn = body.split("const renderPdf")[1].split("const renderImage")[0]
|
||||
assert "sandbox" not in pdf_fn or "No sandbox attribute" in pdf_fn
|
||||
|
||||
def test_content_loads_through_authfetch_probe(self) -> None:
|
||||
"""src-loaded kinds preflight with a probe request (authFetch of
|
||||
?probe=1), NOT a HEAD. The console reverse proxy forwards a HEAD as a
|
||||
full GET, so a real HEAD would drag the whole blob across the hop just
|
||||
to discard it; the probe still surfaces the persist race + auth
|
||||
failures as a typed error card and rides the 401-refresh retry a bare
|
||||
iframe/img src can't."""
|
||||
body = _read(_PREVIEW_JS)
|
||||
assert "authFetch(probeUrl)" in body
|
||||
assert "probe=1" in body
|
||||
# The old full-GET HEAD preflight is gone.
|
||||
assert 'method: "HEAD"' not in body
|
||||
|
||||
def test_markdown_uses_the_sanctioned_html_lane(self) -> None:
|
||||
body = _read(_PREVIEW_JS)
|
||||
assert "setSafeHtml(doc, renderMarkdown(text))" in body
|
||||
|
||||
def test_markdown_runs_vendor_post_pass(self) -> None:
|
||||
"""The pane runs renderer.js's post-render pass (hljs token coloring +
|
||||
mermaid) like the conversation pane — dropping it silently regresses
|
||||
code highlighting and diagram rendering in previews."""
|
||||
body = _read(_PREVIEW_JS)
|
||||
assert "postRenderMarkdown(" in body
|
||||
|
||||
def test_remote_assets_toggle_is_default_off(self) -> None:
|
||||
"""The remote-assets opt-in defaults OFF: a previewed page must not
|
||||
contact its origin site until the user asks. Pins the label / tooltip
|
||||
copy and the sticky-boolean initializer."""
|
||||
body = _read(_PREVIEW_JS)
|
||||
assert "Load remote images & styles" in body
|
||||
assert "Off keeps this preview from contacting the site" in body
|
||||
assert "pane._assetsOn = false" in body
|
||||
|
||||
def test_assets_flag_only_rides_behind_toggle(self) -> None:
|
||||
"""assets=1 reaches the URL only when the per-pane toggle is on."""
|
||||
body = _read(_PREVIEW_JS)
|
||||
assert "assets=1" in body
|
||||
assert "pane._assetsOn" in body
|
||||
|
||||
def test_history_is_bounded(self) -> None:
|
||||
assert "HISTORY_CAP" in _read(_PREVIEW_JS)
|
||||
|
||||
def test_table_renderer_caps_rows(self) -> None:
|
||||
assert "TABLE_ROW_CAP" in _read(_PREVIEW_JS)
|
||||
|
||||
def test_url_builder_encodes_path_parts(self) -> None:
|
||||
body = _read(_PREVIEW_JS)
|
||||
assert "encodeURIComponent(ws)" in body
|
||||
assert 'encodeURIComponent(descriptor.attachment_id || "")' in body
|
||||
|
||||
|
||||
class TestTranscriptChip:
|
||||
def test_chip_builder_exported(self) -> None:
|
||||
assert "export function buildPreviewChip" in _read(_CONVERSATION_JS)
|
||||
|
||||
def test_live_path_gates_auto_open_on_focus(self) -> None:
|
||||
"""A backgrounded session must not commandeer the split — the live
|
||||
path auto-opens only while the originating pane is focused; the chip
|
||||
is the deliberate reopen everywhere else."""
|
||||
body = _read(_INTERACTIVE_JS)
|
||||
assert "if (this._host.isFocused(this)) this._host.onPreview(preview);" in body
|
||||
|
||||
def test_replay_path_renders_chip_without_auto_open(self) -> None:
|
||||
body = _read(_INTERACTIVE_JS)
|
||||
# The replay branch builds the chip…
|
||||
assert "buildPreviewChip(msg.preview" in body
|
||||
# …and the auto-open call appears exactly once (the live path).
|
||||
assert body.count("this._host.onPreview(preview)") == 1
|
||||
|
||||
def test_tool_result_event_passes_preview(self) -> None:
|
||||
assert "evt.preview," in _read(_INTERACTIVE_JS)
|
||||
|
||||
def test_host_bridge_carries_transport_ctx(self) -> None:
|
||||
"""The preview pane fetches blobs from the ORIGINATING workstream
|
||||
through the same node proxy — the bridge must pass both base and
|
||||
wsId, not just the descriptor."""
|
||||
body = _read(_INTERACTIVE_JS)
|
||||
assert "window.TS_SHELL.openPreview(descriptor, { base: base, wsId: wsId })" in body
|
||||
|
||||
|
||||
class TestShellWiring:
|
||||
def test_pane_type_registered(self) -> None:
|
||||
body = _read(_SHELL_JS)
|
||||
assert 'pm.registerType("preview"' in body
|
||||
assert "createPreviewPane" in body
|
||||
|
||||
def test_opens_beside_the_conversation(self) -> None:
|
||||
"""openPaneBeside is the load-bearing gesture — the preview coexists
|
||||
with the conversation that spawned it instead of replacing it."""
|
||||
body = _read(_SHELL_JS)
|
||||
assert 'pm.openPaneBeside("preview")' in body
|
||||
|
||||
def test_seam_exported_on_ts_shell(self) -> None:
|
||||
assert "openPreview," in _read(_SHELL_JS)
|
||||
|
||||
|
||||
class TestStylesheets:
|
||||
def test_both_surfaces_link_preview_css(self) -> None:
|
||||
for page in (_UI_INDEX, _CONSOLE_INDEX):
|
||||
assert "/shared/preview.css" in _read(page), page.name
|
||||
|
||||
def test_stylesheet_uses_ds_tokens_not_legacy_vars(self) -> None:
|
||||
"""conv-* card rule: DS tokens only — chat.css legacy vars
|
||||
(--green/--red/--fg) must not creep into the new sheet."""
|
||||
body = _read(_PREVIEW_CSS)
|
||||
assert "var(--ink-" in body
|
||||
assert "var(--hair)" in body
|
||||
for legacy in ("var(--green)", "var(--red)", "var(--fg)"):
|
||||
assert legacy not in body
|
||||
|
||||
|
||||
class TestEphemeralDismiss:
|
||||
"""The preview is an ephemeral pane: dismissing its split cell CLOSES it
|
||||
(tab and content gone) instead of parking an orphan tab whose only reopen
|
||||
is the transcript chip. Regression guard for the pane/tab desync."""
|
||||
|
||||
def test_preview_pane_is_ephemeral(self) -> None:
|
||||
"""createPreviewPane must flag the pane ephemeral — the whole fix keys
|
||||
off this bit."""
|
||||
body = _read(_PREVIEW_JS)
|
||||
assert "ephemeral: true" in body, "the preview pane must declare itself ephemeral"
|
||||
|
||||
def test_shellpane_carries_the_ephemeral_flag(self) -> None:
|
||||
body = _read(_PANE_JS)
|
||||
assert "this.ephemeral = opts.ephemeral || false;" in body, (
|
||||
"ShellPane must accept and default the ephemeral flag"
|
||||
)
|
||||
|
||||
def test_cell_chip_closes_ephemeral_pane_outright(self) -> None:
|
||||
"""In a split the ✕ chip normally HIDES the cell (closeCell); for an
|
||||
ephemeral pane it must fall through to close() — the `!pane.ephemeral`
|
||||
guard is what routes it there. Pin BOTH the guard and where the
|
||||
skipped case lands (the else), or gutting the else regresses the fix
|
||||
while the guard string survives verbatim."""
|
||||
body = _read(_PANE_JS)
|
||||
assert "if (this._layout && this._leafFor(pane.id) && !pane.ephemeral)" in body, (
|
||||
"the cell chip must skip closeCell for an ephemeral pane"
|
||||
)
|
||||
assert "else this.close(pane.id);" in body, (
|
||||
"the skipped (ephemeral / single-pane) case must land on close()"
|
||||
)
|
||||
|
||||
def test_cell_chip_signals_destruction_for_ephemeral(self) -> None:
|
||||
"""The glyph/label must not lie: an ephemeral pane's split chip reads
|
||||
as a destructive close (✕ + danger hover + 'Close pane'), never the
|
||||
reversible '− / Hide from split'."""
|
||||
body = _read(_PANE_JS)
|
||||
assert "const destroys = !multi || pane.ephemeral;" in body, (
|
||||
"chip mode must treat ephemeral panes as destructive even in a split"
|
||||
)
|
||||
|
||||
def test_unsplit_closes_ephemeral_non_survivors(self) -> None:
|
||||
"""Collapsing the split from the OTHER pane must not orphan the preview
|
||||
either — unsplit closes ephemeral panes it isn't keeping."""
|
||||
body = _read(_PANE_JS)
|
||||
assert "const keep = this._activeId;" in body, (
|
||||
"the unsplit survivor must be the FOCUSED pane — the filter's "
|
||||
"`id !== keep` guard is only correct if keep is _activeId"
|
||||
)
|
||||
assert "for (const id of doomed) this.close(id);" in body, (
|
||||
"unsplit must destroy ephemeral panes it does not keep"
|
||||
)
|
||||
assert "return id !== keep && p && p.ephemeral;" in body, (
|
||||
"unsplit must spare the focused survivor and non-ephemeral panes"
|
||||
)
|
||||
@@ -277,6 +277,41 @@ class TestConvertMessagesReasoningReplay:
|
||||
types = [it.get("type") for it in items]
|
||||
assert "reasoning" not in types
|
||||
|
||||
def test_agent_shaped_turn_pairs_reasoning_with_restored_call_ids(
|
||||
self, provider: OpenAIResponsesProvider
|
||||
) -> None:
|
||||
# The sub-agent wire shape (native lane carried, minted ids already
|
||||
# restored to the provider originals by the lowering map): the stored
|
||||
# reasoning item rides immediately before the function_call rebuilt
|
||||
# from the SAME original call id, and the function_call_output pairs
|
||||
# to it — the ordering + id agreement the Responses API requires when
|
||||
# replaying reasoning across an agent's own tool loop.
|
||||
messages = [
|
||||
{"role": "user", "content": "go"},
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": None,
|
||||
"tool_calls": [{"id": "call_orig1", "function": {"name": "f", "arguments": "{}"}}],
|
||||
"_provider_content": [
|
||||
{"type": "reasoning", "id": "rs_1", "summary": [], "encrypted_content": "enc"},
|
||||
{
|
||||
"type": "function_call",
|
||||
"call_id": "call_orig1",
|
||||
"name": "f",
|
||||
"arguments": "{}",
|
||||
},
|
||||
],
|
||||
},
|
||||
{"role": "tool", "tool_call_id": "call_orig1", "content": "out"},
|
||||
]
|
||||
_, items = provider._convert_messages(messages, replay_reasoning_to_model=True)
|
||||
types = [it.get("type") for it in items]
|
||||
assert types == ["message", "reasoning", "function_call", "function_call_output"]
|
||||
assert items[1]["id"] == "rs_1"
|
||||
assert items[1]["encrypted_content"] == "enc"
|
||||
assert items[2]["call_id"] == "call_orig1"
|
||||
assert items[3]["call_id"] == "call_orig1"
|
||||
|
||||
def test_no_reasoning_items_when_provider_content_lacks_reasoning(
|
||||
self, provider: OpenAIResponsesProvider
|
||||
) -> None:
|
||||
|
||||
+629
-2
@@ -16,6 +16,7 @@ from turnstone.core.providers._openai_common import (
|
||||
apply_cache_retention,
|
||||
apply_temperature_and_effort,
|
||||
apply_tool_search,
|
||||
extract_usage,
|
||||
format_citations,
|
||||
lookup_openai_capabilities,
|
||||
sanitize_messages,
|
||||
@@ -1902,6 +1903,201 @@ class TestGoogleProviderFidelity:
|
||||
assert len(cleaned) == 2
|
||||
assert cleaned[0]["content"] == "hello"
|
||||
|
||||
def test_prepare_messages_swap_cannot_resurrect_malformed_arguments(self) -> None:
|
||||
# The raw fidelity dicts carry the model's ORIGINAL arguments string;
|
||||
# the sanitized top-level mirror is what the swap replaces. A raw
|
||||
# dict whose arguments are malformed must be legalized during the
|
||||
# swap (thought_signature and id untouched) — otherwise every replay
|
||||
# resurrects the malformed string the upstream sanitize pass fixed.
|
||||
from turnstone.core.providers._google import GoogleProvider
|
||||
|
||||
prov = GoogleProvider()
|
||||
msgs = [
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": "",
|
||||
"tool_calls": [
|
||||
# Mirror already legalized upstream.
|
||||
{"id": "c1", "type": "function", "function": {"name": "f", "arguments": "{}"}},
|
||||
{
|
||||
"id": "c2",
|
||||
"type": "function",
|
||||
"function": {"name": "g", "arguments": '{"ok": 1}'},
|
||||
},
|
||||
],
|
||||
"_provider_content": [
|
||||
{
|
||||
"id": "c1",
|
||||
"type": "function",
|
||||
# Raw, unterminated — the model's original output.
|
||||
"function": {"name": "f", "arguments": '{"path": "/tmp'},
|
||||
"thought_signature": "sig123",
|
||||
},
|
||||
{
|
||||
"id": "c2",
|
||||
"type": "function",
|
||||
"function": {"name": "g", "arguments": '{"ok": 1}'},
|
||||
"thought_signature": "sig456",
|
||||
},
|
||||
],
|
||||
},
|
||||
{"role": "tool", "tool_call_id": "c1", "content": "ok"},
|
||||
{"role": "tool", "tool_call_id": "c2", "content": "ok"},
|
||||
]
|
||||
cleaned = prov._prepare_messages(msgs)
|
||||
tcs = cleaned[0]["tool_calls"]
|
||||
assert tcs[0]["function"]["arguments"] == "{}" # legalized
|
||||
assert tcs[0]["thought_signature"] == "sig123" # fidelity preserved
|
||||
assert tcs[0]["id"] == "c1"
|
||||
# The valid sibling passes through byte-identical.
|
||||
assert tcs[1]["function"]["arguments"] == '{"ok": 1}'
|
||||
assert tcs[1]["thought_signature"] == "sig456"
|
||||
|
||||
def test_prepare_messages_swap_serializes_dict_arguments(self) -> None:
|
||||
# The internal-shape case the shared legalize helper handles: a raw
|
||||
# fidelity dict whose arguments landed as an unserialized dict is
|
||||
# json.dumps'd — content preserved, not collapsed to "{}".
|
||||
from turnstone.core.providers._google import GoogleProvider
|
||||
|
||||
prov = GoogleProvider()
|
||||
msgs = [
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": "",
|
||||
"tool_calls": [
|
||||
{"id": "c1", "type": "function", "function": {"name": "f", "arguments": "{}"}},
|
||||
],
|
||||
"_provider_content": [
|
||||
{
|
||||
"id": "c1",
|
||||
"type": "function",
|
||||
"function": {"name": "f", "arguments": {"path": "/tmp/x"}},
|
||||
"thought_signature": "sig1",
|
||||
},
|
||||
],
|
||||
},
|
||||
{"role": "tool", "tool_call_id": "c1", "content": "ok"},
|
||||
]
|
||||
cleaned = prov._prepare_messages(msgs)
|
||||
tc = cleaned[0]["tool_calls"][0]
|
||||
assert json.loads(tc["function"]["arguments"]) == {"path": "/tmp/x"}
|
||||
assert tc["thought_signature"] == "sig1"
|
||||
|
||||
def test_prepare_messages_blank_id_raw_row_keeps_sanitized_mirror(self) -> None:
|
||||
# A historical fidelity row whose raw dict carries a blank id (saved
|
||||
# before the capture-time blank-id gate existed): swapping it in
|
||||
# would resurrect the blank id on every replay, so the swap is
|
||||
# skipped and the sanitized mirror — with its back-filled id — stays.
|
||||
from turnstone.core.providers._google import GoogleProvider
|
||||
|
||||
prov = GoogleProvider()
|
||||
msgs = [
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": "",
|
||||
"tool_calls": [
|
||||
{
|
||||
"id": "call_backfilled",
|
||||
"type": "function",
|
||||
"function": {"name": "f", "arguments": "{}"},
|
||||
},
|
||||
],
|
||||
"_provider_content": [
|
||||
{
|
||||
"id": "",
|
||||
"type": "function",
|
||||
"function": {"name": "f", "arguments": "{}"},
|
||||
"thought_signature": "sig",
|
||||
},
|
||||
],
|
||||
},
|
||||
{"role": "tool", "tool_call_id": "call_backfilled", "content": "ok"},
|
||||
]
|
||||
cleaned = prov._prepare_messages(msgs)
|
||||
tc = cleaned[0]["tool_calls"][0]
|
||||
assert tc["id"] == "call_backfilled" # mirror kept, raw lane not swapped
|
||||
assert "thought_signature" not in tc
|
||||
|
||||
def test_prepare_messages_ignores_non_dict_provider_content_elements(self) -> None:
|
||||
# A corrupted persisted lane with a non-dict element must not crash
|
||||
# the request build.
|
||||
from turnstone.core.providers._google import GoogleProvider
|
||||
|
||||
prov = GoogleProvider()
|
||||
msgs = [
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": "x",
|
||||
"_provider_content": ["garbage-string"],
|
||||
},
|
||||
]
|
||||
cleaned = prov._prepare_messages(msgs)
|
||||
assert cleaned[0]["content"] == "x"
|
||||
assert "_provider_content" not in cleaned[0]
|
||||
|
||||
def test_prepare_messages_partial_lane_keeps_sanitized_mirror(self) -> None:
|
||||
# A partially-corrupted lane (one valid raw dict + one garbage
|
||||
# element) must not swap a SHORTER list over the mirror — that would
|
||||
# drop a mirrored call whose tool result remains in history and
|
||||
# orphan it. The sanitized mirror stays.
|
||||
from turnstone.core.providers._google import GoogleProvider
|
||||
|
||||
prov = GoogleProvider()
|
||||
msgs = [
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": "",
|
||||
"tool_calls": [
|
||||
{
|
||||
"id": "call_A",
|
||||
"type": "function",
|
||||
"function": {"name": "f", "arguments": "{}"},
|
||||
},
|
||||
{
|
||||
"id": "call_B",
|
||||
"type": "function",
|
||||
"function": {"name": "g", "arguments": "{}"},
|
||||
},
|
||||
],
|
||||
"_provider_content": [
|
||||
{
|
||||
"id": "call_A",
|
||||
"type": "function",
|
||||
"function": {"name": "f", "arguments": "{}"},
|
||||
"thought_signature": "sig",
|
||||
},
|
||||
"garbage-string",
|
||||
],
|
||||
},
|
||||
{"role": "tool", "tool_call_id": "call_A", "content": "ok"},
|
||||
{"role": "tool", "tool_call_id": "call_B", "content": "ok"},
|
||||
]
|
||||
cleaned = prov._prepare_messages(msgs)
|
||||
ids = [tc["id"] for tc in cleaned[0]["tool_calls"]]
|
||||
assert ids == ["call_A", "call_B"] # mirror kept — no orphaned call_B
|
||||
|
||||
def test_prepare_messages_swap_passes_non_dict_function_through(self) -> None:
|
||||
# A degenerate fidelity block with function=None must pass through
|
||||
# untouched (the prior behaviour), not raise.
|
||||
from turnstone.core.providers._google import GoogleProvider
|
||||
|
||||
prov = GoogleProvider()
|
||||
msgs = [
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": "x",
|
||||
"tool_calls": [
|
||||
{"id": "c1", "type": "function", "function": {"name": "f", "arguments": "{}"}},
|
||||
],
|
||||
"_provider_content": [
|
||||
{"id": "c1", "type": "function", "function": None},
|
||||
],
|
||||
},
|
||||
{"role": "tool", "tool_call_id": "c1", "content": "ok"},
|
||||
]
|
||||
cleaned = prov._prepare_messages(msgs)
|
||||
assert cleaned[0]["tool_calls"][0]["function"] is None
|
||||
|
||||
def test_non_streaming_captures_provider_blocks(self) -> None:
|
||||
from turnstone.core.providers._google import GoogleProvider
|
||||
|
||||
@@ -2212,6 +2408,39 @@ class TestOpenAIParameterGating:
|
||||
assert "temperature" not in kwargs
|
||||
assert kwargs["reasoning_effort"] == "medium" # fell back from unsupported "low"
|
||||
|
||||
def test_gpt56_sol_max_effort_and_temperature(self) -> None:
|
||||
"""GPT-5.6 (Sol / bare alias): 1M context + tool search; accepts
|
||||
the NEW "max" reasoning effort verbatim (first commercial OpenAI
|
||||
model to use it); temperature only at reasoning_effort="none"."""
|
||||
caps = lookup_openai_capabilities("gpt-5.6")
|
||||
assert caps.context_window == 1050000
|
||||
assert caps.supports_tool_search is True
|
||||
assert caps.supports_vision is True
|
||||
assert "max" in caps.reasoning_effort_values
|
||||
kwargs: dict[str, Any] = {}
|
||||
apply_temperature_and_effort(kwargs, caps, temperature=0.7, reasoning_effort="max")
|
||||
assert "temperature" not in kwargs
|
||||
assert kwargs["reasoning_effort"] == "max"
|
||||
none_kwargs: dict[str, Any] = {}
|
||||
apply_temperature_and_effort(none_kwargs, caps, temperature=0.7, reasoning_effort="none")
|
||||
assert none_kwargs["temperature"] == 0.7
|
||||
assert none_kwargs["reasoning_effort"] == "none"
|
||||
|
||||
def test_gpt56_sol_id_resolves_by_prefix(self) -> None:
|
||||
"""The explicit "gpt-5.6-sol" id and dated Sol snapshots inherit
|
||||
the Sol/alias row (incl. "max") by longest-prefix match."""
|
||||
assert "max" in lookup_openai_capabilities("gpt-5.6-sol").reasoning_effort_values
|
||||
assert "max" in lookup_openai_capabilities("gpt-5.6-2026-07-09").reasoning_effort_values
|
||||
|
||||
def test_gpt56_terra_luna_support_max_effort(self) -> None:
|
||||
"""Every GPT-5.6 tier accepts the documented "max" effort."""
|
||||
for tier in ("gpt-5.6-terra", "gpt-5.6-luna"):
|
||||
caps = lookup_openai_capabilities(tier)
|
||||
assert "max" in caps.reasoning_effort_values, tier
|
||||
kwargs: dict[str, Any] = {}
|
||||
apply_temperature_and_effort(kwargs, caps, temperature=0.7, reasoning_effort="max")
|
||||
assert kwargs["reasoning_effort"] == "max", tier
|
||||
|
||||
|
||||
class TestAnthropicOrphanedToolUse:
|
||||
"""Verify _convert_messages synthesizes tool_results for orphaned tool_use."""
|
||||
@@ -3140,6 +3369,80 @@ class TestAnthropicProviderBlocks:
|
||||
assert assistant_msg["role"] == "assistant"
|
||||
assert assistant_msg["content"] == [{"type": "text", "text": "Hi there"}]
|
||||
|
||||
def test_agent_native_lane_with_restore_map_is_wire_consistent(self) -> None:
|
||||
"""The sub-agent wire shape: an assistant Turn carrying the provider-
|
||||
native lane, its minted tool id restored to the provider original by
|
||||
the lowering map. The native blocks replay verbatim (thinking +
|
||||
signature untouched) and the native tool_use id, the top-level
|
||||
mirror, and the tool_result all agree."""
|
||||
from turnstone.core.lowering import restore_provider_tool_ids
|
||||
from turnstone.core.trajectory import (
|
||||
ProviderNative,
|
||||
ToolCall,
|
||||
Turn,
|
||||
dicts_from_turns,
|
||||
)
|
||||
|
||||
thinking = {"type": "thinking", "thinking": "look first", "signature": "sig_1"}
|
||||
tool_use = {"type": "tool_use", "id": "toolu_01X", "name": "f", "input": {}}
|
||||
minted = "task-1::r1s1::toolu_01X"
|
||||
turns = [
|
||||
Turn.user("go"),
|
||||
Turn.assistant(
|
||||
"using f",
|
||||
tool_calls=(ToolCall(id=minted, name="f", arguments="{}"),),
|
||||
native=ProviderNative(
|
||||
producer="anthropic",
|
||||
blocks=(thinking, {"type": "text", "text": "using f"}, tool_use),
|
||||
),
|
||||
),
|
||||
Turn.tool(minted, "out"),
|
||||
]
|
||||
wire = restore_provider_tool_ids(dicts_from_turns(turns), {minted: "toolu_01X"})
|
||||
_, converted = self.provider._convert_messages(wire, replay_reasoning_to_model=True)
|
||||
assistant = converted[1]
|
||||
assert [b["type"] for b in assistant["content"]] == ["thinking", "text", "tool_use"]
|
||||
assert assistant["content"][0]["signature"] == "sig_1"
|
||||
assert assistant["content"][2]["id"] == "toolu_01X"
|
||||
tool_results = [b for b in converted[2]["content"] if b.get("type") == "tool_result"]
|
||||
assert tool_results and tool_results[0]["tool_use_id"] == "toolu_01X"
|
||||
|
||||
def test_agent_native_lane_without_restore_map_orphans_the_result(self) -> None:
|
||||
"""Documents why the id map is a PREREQUISITE of carrying the native
|
||||
lane, not hygiene: without it the tool_result arrives with the minted
|
||||
id, matches no native tool_use, and the converter drops it as an
|
||||
orphan — leaving an unanswered tool_use on the wire (a provider
|
||||
rejection)."""
|
||||
from turnstone.core.trajectory import (
|
||||
ProviderNative,
|
||||
ToolCall,
|
||||
Turn,
|
||||
dicts_from_turns,
|
||||
)
|
||||
|
||||
tool_use = {"type": "tool_use", "id": "toolu_01X", "name": "f", "input": {}}
|
||||
minted = "task-1::r1s1::toolu_01X"
|
||||
turns = [
|
||||
Turn.user("go"),
|
||||
Turn.assistant(
|
||||
"",
|
||||
tool_calls=(ToolCall(id=minted, name="f", arguments="{}"),),
|
||||
native=ProviderNative(producer="anthropic", blocks=(tool_use,)),
|
||||
),
|
||||
Turn.tool(minted, "out"),
|
||||
]
|
||||
_, converted = self.provider._convert_messages(
|
||||
dicts_from_turns(turns), replay_reasoning_to_model=True
|
||||
)
|
||||
all_results = [
|
||||
b
|
||||
for m in converted
|
||||
if isinstance(m.get("content"), list)
|
||||
for b in m["content"]
|
||||
if isinstance(b, dict) and b.get("type") == "tool_result"
|
||||
]
|
||||
assert all_results == []
|
||||
|
||||
def test_block_to_dict_with_model_dump(self) -> None:
|
||||
"""_block_to_dict uses model_dump(exclude_none=True) when available."""
|
||||
from turnstone.core.providers._anthropic import _block_to_dict
|
||||
@@ -3461,6 +3764,31 @@ class TestModelCapabilitiesToolSearch:
|
||||
caps = ModelCapabilities()
|
||||
assert caps.supports_tool_search is False
|
||||
|
||||
def test_public_positional_prefix_remains_stable(self) -> None:
|
||||
"""New optional fields must not shift the exported constructor's existing slots."""
|
||||
caps = ModelCapabilities(
|
||||
100000,
|
||||
10000,
|
||||
False,
|
||||
False,
|
||||
False,
|
||||
"max_tokens",
|
||||
"manual",
|
||||
"thinking",
|
||||
"reasoning_effort",
|
||||
True,
|
||||
("low",),
|
||||
("low",),
|
||||
"low",
|
||||
True,
|
||||
True,
|
||||
True,
|
||||
True,
|
||||
)
|
||||
assert caps.supports_web_search is True
|
||||
assert caps.supports_tool_search is True
|
||||
assert caps.supports_vision is True
|
||||
|
||||
|
||||
class TestMidConversationSystemCapability:
|
||||
"""supports_mid_conversation_system — NextOpus (claude-opus-4-8) only."""
|
||||
@@ -3909,8 +4237,50 @@ class TestOpenAIPromptCaching:
|
||||
def setup_method(self) -> None:
|
||||
self.provider = OpenAIProvider()
|
||||
|
||||
def test_cache_retention_set_for_gpt5(self) -> None:
|
||||
"""GPT-5.x models get prompt_cache_retention=24h."""
|
||||
@pytest.mark.parametrize("model", ("gpt-5.5-local-lora", "gpt-5.6-local-lora"))
|
||||
def test_chat_compat_streaming_omits_commercial_cache_params(self, model: str) -> None:
|
||||
"""A local model name must not activate commercial OpenAI cache controls."""
|
||||
client = MagicMock()
|
||||
client.chat.completions.create.return_value = iter(())
|
||||
|
||||
list(
|
||||
self.provider.create_streaming(
|
||||
client=client,
|
||||
model=model,
|
||||
messages=[{"role": "user", "content": "hi"}],
|
||||
)
|
||||
)
|
||||
|
||||
sent = client.chat.completions.create.call_args.kwargs
|
||||
assert "prompt_cache_retention" not in sent
|
||||
assert "prompt_cache_options" not in sent
|
||||
|
||||
@pytest.mark.parametrize("model", ("gpt-5.5-local-lora", "gpt-5.6-local-lora"))
|
||||
def test_chat_compat_completion_omits_commercial_cache_params(self, model: str) -> None:
|
||||
"""The non-streaming local lane has the same cache-parameter isolation."""
|
||||
response = MagicMock()
|
||||
response.choices = [
|
||||
MagicMock(
|
||||
message=MagicMock(content="hello", tool_calls=None, annotations=None),
|
||||
finish_reason="stop",
|
||||
)
|
||||
]
|
||||
response.usage = None
|
||||
client = MagicMock()
|
||||
client.chat.completions.create.return_value = response
|
||||
|
||||
self.provider.create_completion(
|
||||
client=client,
|
||||
model=model,
|
||||
messages=[{"role": "user", "content": "hi"}],
|
||||
)
|
||||
|
||||
sent = client.chat.completions.create.call_args.kwargs
|
||||
assert "prompt_cache_retention" not in sent
|
||||
assert "prompt_cache_options" not in sent
|
||||
|
||||
def test_cache_retention_set_for_pre_gpt56_models(self) -> None:
|
||||
"""Pre-5.6 GPT-5 models retain the legacy 24-hour cache policy."""
|
||||
for model in (
|
||||
"gpt-5",
|
||||
"gpt-5.1",
|
||||
@@ -3925,6 +4295,15 @@ class TestOpenAIPromptCaching:
|
||||
kwargs: dict[str, Any] = {}
|
||||
apply_cache_retention(kwargs, model)
|
||||
assert kwargs.get("prompt_cache_retention") == "24h", f"Failed for {model}"
|
||||
assert "prompt_cache_options" not in kwargs
|
||||
|
||||
def test_gpt56_uses_prompt_cache_options(self) -> None:
|
||||
"""GPT-5.6 uses the replacement cache API introduced in SDK 2.45."""
|
||||
for model in ("gpt-5.6", "gpt-5.6-sol", "gpt-5.6-terra", "gpt-5.6-luna"):
|
||||
kwargs: dict[str, Any] = {}
|
||||
apply_cache_retention(kwargs, model)
|
||||
assert kwargs.get("prompt_cache_options") == {"ttl": "30m"}, model
|
||||
assert "prompt_cache_retention" not in kwargs
|
||||
|
||||
def test_cache_retention_not_set_for_non_gpt5(self) -> None:
|
||||
"""Non-GPT-5 models do not get cache retention."""
|
||||
@@ -3932,6 +4311,38 @@ class TestOpenAIPromptCaching:
|
||||
kwargs: dict[str, Any] = {}
|
||||
apply_cache_retention(kwargs, model)
|
||||
assert "prompt_cache_retention" not in kwargs, f"Unexpected retention for {model}"
|
||||
assert "prompt_cache_options" not in kwargs, f"Unexpected options for {model}"
|
||||
|
||||
def test_cache_write_tokens_from_responses_usage(self) -> None:
|
||||
"""GPT-5.6 cache writes flow into normalized usage accounting."""
|
||||
usage = MagicMock()
|
||||
usage.prompt_tokens = None
|
||||
usage.input_tokens = 100
|
||||
usage.completion_tokens = None
|
||||
usage.output_tokens = 20
|
||||
usage.total_tokens = 120
|
||||
usage.prompt_tokens_details = None
|
||||
usage.input_tokens_details = MagicMock(cached_tokens=30, cache_write_tokens=70)
|
||||
|
||||
normalized = extract_usage(usage)
|
||||
|
||||
assert normalized is not None
|
||||
assert normalized.cache_read_tokens == 30
|
||||
assert normalized.cache_creation_tokens == 70
|
||||
|
||||
def test_cache_write_tokens_from_chat_usage(self) -> None:
|
||||
"""The Chat Completions usage shape reports the same cache-write metric."""
|
||||
usage = MagicMock()
|
||||
usage.prompt_tokens = 100
|
||||
usage.completion_tokens = 20
|
||||
usage.total_tokens = 120
|
||||
usage.prompt_tokens_details = MagicMock(cached_tokens=30, cache_write_tokens=70)
|
||||
|
||||
normalized = extract_usage(usage)
|
||||
|
||||
assert normalized is not None
|
||||
assert normalized.cache_read_tokens == 30
|
||||
assert normalized.cache_creation_tokens == 70
|
||||
|
||||
def test_streaming_cached_tokens_from_usage(self) -> None:
|
||||
"""Streaming usage extracts cached_tokens from prompt_tokens_details."""
|
||||
@@ -4093,6 +4504,78 @@ class TestOpenAIResponsesProvider:
|
||||
assert caps.supports_tool_search is True
|
||||
|
||||
|
||||
class TestOpenAIChatReasoningCapture:
|
||||
"""Non-streaming ``create_completion`` surfaces the Chat-Completions
|
||||
lane's non-canonical reasoning (vLLM ``--reasoning-parser``, llama.cpp
|
||||
``reasoning_format``) as ``CompletionResult.reasoning`` — the twin of the
|
||||
streaming path's ``reasoning_delta`` extraction, same attribute pair and
|
||||
precedence."""
|
||||
|
||||
@staticmethod
|
||||
def _client(*, reasoning: Any = None, reasoning_content: Any = None) -> MagicMock:
|
||||
msg = MagicMock()
|
||||
msg.content = "ok"
|
||||
msg.tool_calls = None
|
||||
msg.annotations = None
|
||||
msg.reasoning = reasoning
|
||||
msg.reasoning_content = reasoning_content
|
||||
choice = MagicMock()
|
||||
choice.message = msg
|
||||
choice.finish_reason = "stop"
|
||||
resp = MagicMock()
|
||||
resp.choices = [choice]
|
||||
resp.usage = None
|
||||
client = MagicMock()
|
||||
client.chat.completions.create.return_value = resp
|
||||
return client
|
||||
|
||||
def _complete(self, client: MagicMock):
|
||||
provider = OpenAIChatCompletionsProvider()
|
||||
return provider.create_completion(
|
||||
client=client, model="m", messages=[{"role": "user", "content": "hi"}]
|
||||
)
|
||||
|
||||
def test_reasoning_content_captured(self) -> None:
|
||||
result = self._complete(self._client(reasoning_content="thought text"))
|
||||
assert result.reasoning == "thought text"
|
||||
|
||||
def test_reasoning_attribute_takes_precedence(self) -> None:
|
||||
result = self._complete(self._client(reasoning="direct", reasoning_content="parsed"))
|
||||
assert result.reasoning == "direct"
|
||||
|
||||
def test_absent_reasoning_is_empty(self) -> None:
|
||||
result = self._complete(self._client())
|
||||
assert result.reasoning == ""
|
||||
|
||||
def test_non_string_reasoning_collapses_to_empty(self) -> None:
|
||||
# A server surfacing a structured reasoning object (not text) must not
|
||||
# leak a non-str into the result.
|
||||
result = self._complete(self._client(reasoning={"odd": True}))
|
||||
assert result.reasoning == ""
|
||||
|
||||
def test_structured_reasoning_does_not_shadow_reasoning_content(self) -> None:
|
||||
# A truthy non-string in ``reasoning`` must not shadow valid text in
|
||||
# ``reasoning_content`` — the first non-empty STRING wins.
|
||||
result = self._complete(
|
||||
self._client(reasoning={"content": "structured"}, reasoning_content="parsed text")
|
||||
)
|
||||
assert result.reasoning == "parsed text"
|
||||
|
||||
def test_streaming_delta_shares_the_same_guard(self) -> None:
|
||||
# The streaming twin: a structured object in ``reasoning`` must not
|
||||
# leak into reasoning_delta (it would TypeError the session's
|
||||
# ``"".join`` accumulator) nor shadow the parsed string.
|
||||
provider = OpenAIChatCompletionsProvider()
|
||||
chunk = _openai_stream_chunk(
|
||||
reasoning={"content": "structured"}, # type: ignore[arg-type] — the hostile input under test
|
||||
reasoning_content="parsed text",
|
||||
finish_reason="stop",
|
||||
)
|
||||
chunks = list(provider._iter_stream(iter([chunk])))
|
||||
assert any(c.reasoning_delta == "parsed text" for c in chunks)
|
||||
assert all(isinstance(c.reasoning_delta, str) for c in chunks)
|
||||
|
||||
|
||||
class TestResponsesMessageConversion:
|
||||
"""Tests for _convert_messages — Chat Completions format to Responses API."""
|
||||
|
||||
@@ -4316,6 +4799,122 @@ class TestResponsesParamBuilding:
|
||||
)
|
||||
assert kwargs["store"] is False
|
||||
|
||||
def _build(self, caps: ModelCapabilities, reasoning_effort: str = "medium") -> dict[str, Any]:
|
||||
return self.provider._build_kwargs(
|
||||
model="gpt-5.6-sol",
|
||||
messages=[{"role": "user", "content": "Hi"}],
|
||||
tools=None,
|
||||
max_tokens=4096,
|
||||
temperature=0.5,
|
||||
reasoning_effort=reasoning_effort,
|
||||
deferred_names=None,
|
||||
capabilities=caps,
|
||||
)
|
||||
|
||||
def test_verbosity_emitted_under_text_when_supported(self) -> None:
|
||||
"""Operator-declared verbosity nests under text.verbosity (never
|
||||
top-level, which 400s on the Responses API)."""
|
||||
kwargs = self._build(ModelCapabilities(supports_verbosity=True, verbosity="low"))
|
||||
assert kwargs["text"] == {"verbosity": "low"}
|
||||
|
||||
def test_verbosity_omitted_when_unsupported(self) -> None:
|
||||
"""A verbosity value on a model that doesn't support it is dropped."""
|
||||
kwargs = self._build(ModelCapabilities(supports_verbosity=False, verbosity="low"))
|
||||
assert "text" not in kwargs
|
||||
|
||||
def test_verbosity_omitted_when_value_empty(self) -> None:
|
||||
"""Supported but unset (the default) → nothing sent, server default."""
|
||||
kwargs = self._build(ModelCapabilities(supports_verbosity=True, verbosity=""))
|
||||
assert "text" not in kwargs
|
||||
|
||||
def test_pro_mode_folds_into_reasoning(self) -> None:
|
||||
"""reasoning.mode='pro' rides alongside the effort in one dict."""
|
||||
caps = ModelCapabilities(
|
||||
supports_pro_mode=True,
|
||||
reasoning_mode="pro",
|
||||
reasoning_effort_values=("low", "medium", "high"),
|
||||
)
|
||||
kwargs = self._build(caps, reasoning_effort="high")
|
||||
assert kwargs["reasoning"] == {"effort": "high", "mode": "pro"}
|
||||
|
||||
def test_pro_mode_rejected_when_unsupported(self) -> None:
|
||||
"""A pro mode on a model without reasoning-mode support is dropped."""
|
||||
caps = ModelCapabilities(
|
||||
supports_pro_mode=False,
|
||||
reasoning_mode="pro",
|
||||
reasoning_effort_values=("low", "medium", "high"),
|
||||
)
|
||||
kwargs = self._build(caps, reasoning_effort="high")
|
||||
assert kwargs["reasoning"] == {"effort": "high"}
|
||||
|
||||
def test_pro_mode_without_effort_sends_mode_only(self) -> None:
|
||||
"""No declared effort (param omitted) but pro mode set → the
|
||||
reasoning dict carries mode alone (effort defaults server-side)."""
|
||||
caps = ModelCapabilities(supports_pro_mode=True, reasoning_mode="pro")
|
||||
kwargs = self._build(caps, reasoning_effort="medium")
|
||||
assert kwargs["reasoning"] == {"mode": "pro"}
|
||||
|
||||
def test_standard_reasoning_mode_is_accepted(self) -> None:
|
||||
"""The SDK's explicit standard mode is valid even though omission is equivalent."""
|
||||
caps = ModelCapabilities(
|
||||
supports_pro_mode=True,
|
||||
reasoning_mode="standard",
|
||||
reasoning_effort_values=("low", "medium", "high"),
|
||||
)
|
||||
kwargs = self._build(caps, reasoning_effort="high")
|
||||
assert kwargs["reasoning"] == {"effort": "high", "mode": "standard"}
|
||||
|
||||
def test_verbosity_unknown_value_dropped(self) -> None:
|
||||
"""A verbosity outside {low,medium,high} is dropped, not sent — an
|
||||
operator typo must not 400 every request."""
|
||||
kwargs = self._build(ModelCapabilities(supports_verbosity=True, verbosity="verbose"))
|
||||
assert "text" not in kwargs
|
||||
|
||||
def test_pro_mode_unknown_value_dropped(self) -> None:
|
||||
"""An unknown reasoning_mode is dropped; a valid effort still rides."""
|
||||
caps = ModelCapabilities(
|
||||
supports_pro_mode=True,
|
||||
reasoning_mode="ultra",
|
||||
reasoning_effort_values=("low", "medium", "high"),
|
||||
)
|
||||
kwargs = self._build(caps, reasoning_effort="high")
|
||||
assert kwargs["reasoning"] == {"effort": "high"}
|
||||
|
||||
def test_verbosity_non_string_value_dropped(self) -> None:
|
||||
"""Malformed operator JSON must not crash request construction."""
|
||||
kwargs = self._build(ModelCapabilities(supports_verbosity=True, verbosity=["low"]))
|
||||
assert "text" not in kwargs
|
||||
|
||||
def test_pro_mode_non_string_value_dropped(self) -> None:
|
||||
"""Malformed operator JSON must not crash request construction."""
|
||||
caps = ModelCapabilities(
|
||||
supports_pro_mode=True,
|
||||
reasoning_mode=["pro"],
|
||||
reasoning_effort_values=("low", "medium", "high"),
|
||||
)
|
||||
kwargs = self._build(caps, reasoning_effort="high")
|
||||
assert kwargs["reasoning"] == {"effort": "high"}
|
||||
|
||||
def test_gpt56_terra_max_reaches_responses_wire(self) -> None:
|
||||
"""Terra sends the documented max effort on the actual Responses path."""
|
||||
kwargs = self.provider._build_kwargs(
|
||||
model="gpt-5.6-terra",
|
||||
messages=[{"role": "user", "content": "Hi"}],
|
||||
tools=None,
|
||||
max_tokens=4096,
|
||||
temperature=0.5,
|
||||
reasoning_effort="max",
|
||||
deferred_names=None,
|
||||
)
|
||||
assert kwargs["reasoning"] == {"effort": "max"}
|
||||
|
||||
def test_gpt56_verbosity_and_pro_flags(self) -> None:
|
||||
"""Every GPT-5.6 tier supports verbosity and pro reasoning mode."""
|
||||
for tier in ("gpt-5.6", "gpt-5.6-sol", "gpt-5.6-terra", "gpt-5.6-luna"):
|
||||
caps = lookup_openai_capabilities(tier)
|
||||
assert caps.supports_verbosity is True
|
||||
assert caps.supports_pro_mode is True
|
||||
|
||||
def _kwargs_with(self, tools: list[dict[str, Any]], caps: ModelCapabilities) -> dict[str, Any]:
|
||||
return self.provider._build_kwargs(
|
||||
model="gpt-5.4",
|
||||
@@ -4367,6 +4966,34 @@ class TestResponsesParamBuilding:
|
||||
)
|
||||
assert kwargs["prompt_cache_retention"] == "24h"
|
||||
|
||||
def test_compat_responses_omits_commercial_cache_params(self) -> None:
|
||||
provider = type(self.provider)(compat=True)
|
||||
for model in ("gpt-5.5-local-lora", "gpt-5.6-local-lora"):
|
||||
kwargs = provider._build_kwargs(
|
||||
model=model,
|
||||
messages=[{"role": "user", "content": "Hi"}],
|
||||
tools=None,
|
||||
max_tokens=4096,
|
||||
temperature=0.5,
|
||||
reasoning_effort="medium",
|
||||
deferred_names=None,
|
||||
)
|
||||
assert "prompt_cache_retention" not in kwargs, model
|
||||
assert "prompt_cache_options" not in kwargs, model
|
||||
|
||||
def test_cache_options_for_gpt56(self) -> None:
|
||||
kwargs = self.provider._build_kwargs(
|
||||
model="gpt-5.6-sol",
|
||||
messages=[{"role": "user", "content": "Hi"}],
|
||||
tools=None,
|
||||
max_tokens=4096,
|
||||
temperature=0.5,
|
||||
reasoning_effort="medium",
|
||||
deferred_names=None,
|
||||
)
|
||||
assert kwargs["prompt_cache_options"] == {"ttl": "30m"}
|
||||
assert "prompt_cache_retention" not in kwargs
|
||||
|
||||
def test_instructions_from_system_messages(self) -> None:
|
||||
kwargs = self.provider._build_kwargs(
|
||||
model="gpt-5.4",
|
||||
|
||||
@@ -1549,3 +1549,365 @@ def test_streaming_apply_marks_buffer_only_on_success() -> None:
|
||||
assert ".catch(function (e) {" in body[chain_at : chain_at + 3500], (
|
||||
"every mermaid chain link must settle back to fulfilled"
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Renderer containment escapes (frontend-render-containment-brief)
|
||||
#
|
||||
# The renderer protects structural blocks with in-band NUL-framed sentinels
|
||||
# (NUL + two-letter-tag + index + NUL, e.g. code-block 0 -> chr(0)+"CB0"+chr(0)).
|
||||
# escapeHtml preserves U+0000, so model/tool text carrying such a sequence used
|
||||
# to FORGE a sentinel: the shared restore pass rewrote every match, duplicating
|
||||
# or relocating a protected block (B1), printing literal "undefined" for an
|
||||
# out-of-range index (B2), or injecting a restored span across a container (B3).
|
||||
# Fix 1 strips U+0000 (NUL) at the TOP-LEVEL render entry only, so no forged
|
||||
# NUL survives to frame a sentinel while generated (recursive-frame) sentinels
|
||||
# are left intact. Only NUL is stripped — every other control byte survives so
|
||||
# code fences show pasted source verbatim. Inputs build NUL via chr(0) (never
|
||||
# a literal escape) per the brief.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
_NUL = chr(0)
|
||||
|
||||
|
||||
def test_forged_code_block_sentinel_does_not_duplicate_block() -> None:
|
||||
"""B1: prose carrying a forged ``chr(0)+CB0+chr(0)`` used to make the
|
||||
shared restore pass emit the protected code block a SECOND time (content
|
||||
spoofing / relocation). Stripping NUL at the entry neutralises the
|
||||
forgery: exactly one code block, no leaked sentinel."""
|
||||
md = "```python\nprint('hi')\n```\n\nprose " + _NUL + "CB0" + _NUL + " end"
|
||||
out = _render(md)
|
||||
assert out.count("<pre>") == 1, "forged CB sentinel duplicated the block:\n" + out
|
||||
assert out.count("print(") == 1
|
||||
assert _NUL not in out, "raw NUL / forged sentinel leaked into output"
|
||||
|
||||
|
||||
def test_forged_out_of_range_sentinel_does_not_print_undefined() -> None:
|
||||
"""B2: ``chr(0)+IC7+chr(0)`` with no inline codes used to restore
|
||||
``inlineCodes[7]`` -> literal ``undefined`` in the rendered text. After
|
||||
the entry strip the forged framing is gone, so no ``undefined`` appears."""
|
||||
out = _render("text " + _NUL + "IC7" + _NUL + " tail")
|
||||
assert "undefined" not in out, "out-of-range forged sentinel printed 'undefined':\n" + out
|
||||
assert _NUL not in out
|
||||
|
||||
|
||||
def test_control_strip_preserves_legit_fence_and_inline() -> None:
|
||||
"""Fix 1 must not disturb legitimately generated sentinels: a normal
|
||||
fence and inline-code span still render after the entry strip (the strip
|
||||
only removes caller-supplied control chars, which are never valid data)."""
|
||||
out = _render("Here is `inline` and a block:\n\n```py\nx = 1\n```")
|
||||
assert "<code>inline</code>" in out
|
||||
assert "<pre><code" in out
|
||||
assert "x = 1" in out
|
||||
assert _NUL not in out
|
||||
|
||||
|
||||
def test_strip_removes_only_nul_preserving_other_control_bytes() -> None:
|
||||
"""The entry strip removes ONLY NUL (the sentinel-framing byte), so a code
|
||||
fence still shows pasted control bytes (terminal output, ANSI escapes)
|
||||
verbatim. Stripping the whole C0/DEL range would silently corrupt code
|
||||
samples; only NUL can forge a sentinel."""
|
||||
esc = chr(27) # ANSI escape — legitimate in pasted terminal output
|
||||
out = _render("```\nbefore " + esc + "[0m after " + _NUL + " end\n```")
|
||||
assert esc in out, "ESC (0x1b) must survive inside a code fence:\n" + repr(out)
|
||||
assert _NUL not in out, "NUL must still be stripped (sentinel-framing byte)"
|
||||
assert "before " in out and " end" in out
|
||||
|
||||
|
||||
def test_forged_inline_sentinel_not_injected_inside_fence() -> None:
|
||||
"""B3: a forged ``chr(0)+IC0+chr(0)`` placed inside a real code fence
|
||||
used to be substituted AFTER the fence was restored (CB restores before
|
||||
IC), injecting a real ``<code>`` span into the ``<pre>``. With a genuine
|
||||
inline-code span present (so inlineCodes[0] exists), the forged reference
|
||||
must NOT clone it into the code block."""
|
||||
md = "`real`\n\n```text\nbefore " + _NUL + "IC0" + _NUL + " after\n```"
|
||||
out = _render(md)
|
||||
assert out.count("<code>real</code>") == 1, "forged IC sentinel injected into <pre>:\n" + out
|
||||
assert _NUL not in out
|
||||
assert "before IC0 after" in out, "fence body should show the inert forged tag as text"
|
||||
|
||||
|
||||
def test_nul_strip_scoped_to_top_level_call() -> None:
|
||||
"""Structural pin for the PLAUSIBLE placement refinement: the NUL strip
|
||||
lives inside the ``_fnDepth === 0`` guard of the exported wrapper, NOT in
|
||||
``_renderMarkdownBody`` (which runs at every recursion depth). An
|
||||
unconditional strip would shred the generated sentinels that recursive
|
||||
``<details>``/footnote frames legitimately carry — foreclosing the
|
||||
recursive-frame fix. Recursion must reach raw text with its sentinels."""
|
||||
body = _RENDERER_JS.read_text(encoding="utf-8")
|
||||
assert "_NUL_STRIP_RE" in body
|
||||
wrapper = body.index("export function renderMarkdown(text)")
|
||||
body_fn = body.index("function _renderMarkdownBody(text)")
|
||||
seg = body[wrapper:body_fn]
|
||||
guard_at = seg.index("_fnDepth === 0")
|
||||
strip_at = seg.index("_NUL_STRIP_RE", guard_at)
|
||||
incr_at = seg.index("_fnDepth++")
|
||||
assert guard_at < strip_at < incr_at, (
|
||||
"the NUL strip must run inside the top-level (_fnDepth === 0) "
|
||||
"guard, before the depth increment"
|
||||
)
|
||||
assert "_NUL_STRIP_RE" not in body[body_fn:], (
|
||||
"strip must not live in _renderMarkdownBody (would run at every depth)"
|
||||
)
|
||||
|
||||
|
||||
def test_recursive_frame_degrades_without_literal_undefined() -> None:
|
||||
"""Fix 2 floor for the NEW-1 residual: a recursive render frame
|
||||
(``<details>`` body, footnote definition) whose fresh block arrays cannot
|
||||
resolve an outer-scope sentinel must NOT print the literal word
|
||||
``undefined``. The restore callbacks return the (inert) matched sentinel
|
||||
instead. (This asserts only the ``undefined`` floor — Fix 5 is what makes
|
||||
the body actually render; the raw sentinel that the node harness preserves
|
||||
here is dropped by a real browser's tokenizer.)"""
|
||||
details = _render("<details>\n<summary>x</summary>\n\n```py\nsecret_code()\n```\n\n</details>")
|
||||
assert "undefined" not in details, "code-in-<details> printed 'undefined':\n" + details
|
||||
footnote = _render("See[^1].\n\n[^1]: a `snippet` ok")
|
||||
assert "undefined" not in footnote, "inline-code-in-footnote printed 'undefined':\n" + footnote
|
||||
|
||||
|
||||
def test_standalone_code_block_not_wrapped_in_paragraph() -> None:
|
||||
"""Fix 6 (NEW-3): code blocks need the ``<p>SENTINEL</p>`` unwrap variant
|
||||
that DT/BQ/MB/TB already have. Without it a lone fenced block emits
|
||||
``<p><pre>…</pre></p>``, which a real browser splits into a stray empty
|
||||
``<p>`` before the ``<pre>``. The unwrap removes the wrapping paragraph."""
|
||||
out = _render("```py\nx = 1\n```")
|
||||
assert "<pre><code" in out
|
||||
assert "<p><pre>" not in out, "code block still wrapped in a paragraph:\n" + out
|
||||
assert out.strip().startswith("<pre>"), "code block should not be paragraph-wrapped:\n" + out
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Fix 3 — blockquote-in-fence (B4): fence protection must run before (and
|
||||
# mask) the line-based blockquote pass, with the fence open anchored to line
|
||||
# start so a blockquoted fence (`> ```) is NOT matched at column > 0.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_blockquote_inside_fence_not_extracted() -> None:
|
||||
"""B4 (the common one, no special chars): ``> `` lines INSIDE a code
|
||||
fence used to be scooped out by the blockquote pre-pass (which ran first)
|
||||
and rendered as a real ``<blockquote>`` nested in ``<pre><code>`` — a
|
||||
shell transcript or quoted-email code block would sprout a headline. The
|
||||
fence pass now runs first and masks the region."""
|
||||
out = _render("```text\nplain\n> quoted\nafter\n```")
|
||||
assert "<blockquote>" not in out, "blockquote extracted from inside a fence:\n" + out
|
||||
assert "<pre><code" in out
|
||||
assert "> quoted" in out, "the quoted line must stay literal (escaped) code:\n" + out
|
||||
|
||||
|
||||
def test_blockquoted_fence_renders_as_code() -> None:
|
||||
"""A fence nested inside a blockquote (``> ```` ``) must still render as a
|
||||
code block WITHIN the ``<blockquote>``. Anchoring the fence open to line
|
||||
start means it is not matched at column > 0, so the blockquote pass
|
||||
extracts the ``> `` run and its recursive render handles the fence. (Pins
|
||||
that we did not over-correct by simply hoisting the fence pass — which
|
||||
would have swallowed the blockquoted fence as ``undefined``.)"""
|
||||
out = _render("> ```\n> code\n> ```")
|
||||
assert "<blockquote>" in out
|
||||
assert "<pre><code>code</code></pre>" in out, "blockquoted fence lost its code:\n" + out
|
||||
assert "undefined" not in out
|
||||
assert _NUL not in out
|
||||
|
||||
|
||||
def test_indented_fence_still_renders_as_code() -> None:
|
||||
"""The open anchor allows arbitrary leading indent, so a legitimately
|
||||
indented fence (e.g. under a list item) still renders as code rather than a
|
||||
paragraph of literal backticks. (A bare ``^`` anchor would drop it; the
|
||||
deeper 4-space-indent case is pinned separately.)"""
|
||||
out = _render(" ```py\n x = 1\n ```")
|
||||
assert "<pre><code" in out, "indented fence dropped (not rendered as code):\n" + out
|
||||
assert "x = 1" in out
|
||||
|
||||
|
||||
def test_indented_fence_close_leaves_no_trailing_whitespace_line() -> None:
|
||||
"""An indented closing line's leading spaces must NOT survive as a trailing
|
||||
whitespace-only line inside the code block: the content strip removes a
|
||||
trailing newline PLUS any indent the close dragged into the capture (a
|
||||
`` ``` `` closed at column 0 is unaffected). Copilot review, PR #804."""
|
||||
out = _render(" ```py\n x = 1\n ```")
|
||||
m = re.search(r"<code[^>]*>(.*?)</code>", out, re.S)
|
||||
assert m, "no <code> block:\n" + out
|
||||
assert m.group(1) == " x = 1", "indented fence close left a trailing whitespace line: " + repr(
|
||||
m.group(1)
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Fix 4 — <details> open anchored to line start (B5). The details pass ran
|
||||
# with an unanchored open, so a `<details>` mentioned mid-line inside inline
|
||||
# code matched across the backtick spans and swallowed the DT sentinel /
|
||||
# lost the content between them.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_inline_code_details_tag_not_consumed_by_details_pass() -> None:
|
||||
"""B5: ``Use `<details>` then `</details>` to fold`` must render two
|
||||
inline-code spans of the literal tags — NOT a real <details> element with
|
||||
the text between the spans swallowed."""
|
||||
out = _render("Use `<details>` then `</details>` to fold.")
|
||||
assert "<details>" in out, "opening <details> tag not shown as literal code:\n" + out
|
||||
assert "</details>" in out, "closing </details> tag not shown as literal code:\n" + out
|
||||
assert "<details>" not in out, "a real <details> element was wrongly created:\n" + out
|
||||
assert out.count("<code>") == 2, "expected two inline-code spans:\n" + out
|
||||
|
||||
|
||||
def test_block_details_still_renders() -> None:
|
||||
"""No-regression: a genuine multi-line <details> block (at line start)
|
||||
still renders as a real disclosure element."""
|
||||
out = _render("<details>\n<summary>More</summary>\n\nBody text here.\n\n</details>")
|
||||
assert "<details><summary>More</summary>" in out
|
||||
assert "Body text here." in out
|
||||
|
||||
|
||||
def test_oneline_details_still_renders() -> None:
|
||||
"""No-regression: the common one-line form must survive the open anchor
|
||||
(anchoring the CLOSE too would break this — do not)."""
|
||||
out = _render("<details><summary>x</summary>y</details>")
|
||||
assert "<details><summary>x</summary>" in out
|
||||
assert "y" in out and out.rstrip().endswith("</details>")
|
||||
|
||||
|
||||
def test_details_inside_fence_stays_literal() -> None:
|
||||
"""Lock the behavior Fix 5a must preserve: a <details> shown INSIDE a code
|
||||
fence is masked by the (earlier) fence pass and must stay literal escaped
|
||||
code, never extracted into a real element."""
|
||||
out = _render("```html\n<details><summary>s</summary>x</details>\n```")
|
||||
assert "<pre><code" in out
|
||||
assert "<details>" in out, "details-in-fence should be literal code:\n" + out
|
||||
assert "<details>" not in out, "details inside a fence was wrongly extracted:\n" + out
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Fix 5 (NEW-1) — recursive-frame content loss. renderMarkdown recurses for
|
||||
# <details> bodies and footnote definitions. When those bodies were extracted
|
||||
# AFTER the fence/inline-code/math passes, they carried outer-scope sentinels
|
||||
# that the recursive call — with fresh, empty block arrays — could not resolve,
|
||||
# so a code block / inline code / math inside them rendered as `undefined` (or,
|
||||
# after the Fix 2 floor, an inert `CB0`/`IC0` sentinel) — silent content loss.
|
||||
# The structural fix extracts <details> from RAW markdown (before fence/inline
|
||||
# protection, fence-aware) and collects footnote definitions before the inline
|
||||
# passes, so each recursion sees raw content.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_code_block_in_details_renders_code() -> None:
|
||||
"""NEW-1 (a), the headline case: a fenced code block inside <details> must
|
||||
render the CODE, not `undefined` and not an inert `CB0` sentinel."""
|
||||
out = _render("<details>\n<summary>x</summary>\n\n```py\nsecret_code()\n```\n\n</details>")
|
||||
assert "secret_code()" in out, "code inside <details> was lost:\n" + out
|
||||
assert "<pre><code" in out and 'class="language-py"' in out
|
||||
assert "undefined" not in out
|
||||
assert _NUL not in out, "a raw sentinel leaked (recursion did not see raw markdown):\n" + out
|
||||
|
||||
|
||||
def test_blockquote_in_details_renders() -> None:
|
||||
"""NEW-1 generalises to any recursive block: a blockquote inside <details>
|
||||
must render as a real <blockquote>, not a lost/inert sentinel."""
|
||||
out = _render("<details>\n<summary>x</summary>\n\n> quoted\n\n</details>")
|
||||
assert "<blockquote>" in out, "blockquote inside <details> was lost:\n" + out
|
||||
assert "quoted" in out
|
||||
assert _NUL not in out
|
||||
|
||||
|
||||
def test_inline_code_in_footnote_renders() -> None:
|
||||
"""NEW-1 (b): inline code in a footnote definition must render as a real
|
||||
<code> span in the footnote section, not `undefined`/`IC0`."""
|
||||
out = _render("See[^1].\n\n[^1]: uses `code` here")
|
||||
assert "<code>code</code>" in out, "inline code in footnote def was lost:\n" + out
|
||||
assert "undefined" not in out
|
||||
assert _NUL not in out
|
||||
|
||||
|
||||
def test_math_in_footnote_renders() -> None:
|
||||
r"""NEW-1 (b), math variant: display/inline math in a footnote definition
|
||||
must reach KaTeX, not restore to `undefined`/`MB0`."""
|
||||
out = _render("See[^1].\n\n[^1]: with \\(x^2\\) inline")
|
||||
assert '<span class="katex">' in out, "math in footnote def was lost:\n" + out
|
||||
assert "undefined" not in out
|
||||
assert _NUL not in out
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Review round-1 regression pins: the details pass runs AFTER fence protection
|
||||
# (fence-masking, not offset math, provides fence-awareness), and both the
|
||||
# fence and details opens allow arbitrary leading indent.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_details_close_tag_shown_in_fenced_example_does_not_close_block() -> None:
|
||||
"""A `</details>` shown as example code inside a fence must NOT close the
|
||||
real disclosure early. Because the fence pass runs first and masks the
|
||||
example as a sentinel, the details close matches only the real trailing
|
||||
tag; the fenced example renders as literal code inside the block."""
|
||||
md = "<details>\n<summary>s</summary>\n\n```html\n</details>\n```\n\n</details>"
|
||||
out = _render(md)
|
||||
assert '<pre><code class="language-html">' in out, "fenced example was swallowed:\n" + out
|
||||
assert "</details>" in out, "example </details> should be literal code:\n" + out
|
||||
assert out.strip().startswith("<details><summary>s</summary>"), out
|
||||
assert out.rstrip().endswith("</details>"), "real block closed early / stray text:\n" + out
|
||||
assert _NUL not in out
|
||||
|
||||
|
||||
def test_deeply_indented_fence_renders_as_code() -> None:
|
||||
"""A fence indented 4+ spaces (as when nested under a list item) still
|
||||
tokenises as a code block — the open anchor allows arbitrary indent, so we
|
||||
don't regress deeply-nested code samples to literal backticks."""
|
||||
out = _render(" ```py\n x = 1\n ```")
|
||||
assert "<pre><code" in out, "deeply-indented fence dropped:\n" + out
|
||||
assert "x = 1" in out
|
||||
|
||||
|
||||
def test_fence_on_list_marker_line_renders_as_code() -> None:
|
||||
"""A code fence that OPENS on the same line as a list marker (`- ```py`)
|
||||
still tokenises as a code block inside the list item. The open matches
|
||||
after an optional list marker, which is re-emitted before the sentinel so
|
||||
the list pass still sees the item. Regression guard: a bare `^[ \\t]*`
|
||||
anchor (no list-marker allowance) destroyed the block and leaked the raw
|
||||
backticks + language tag as text."""
|
||||
for src in ["- ```py\n print(1)\n ```", "1. ```py\n print(1)\n ```"]:
|
||||
out = _render(src)
|
||||
assert "<pre><code" in out, "list-marker-line fence dropped:\n" + repr(src) + "\n" + out
|
||||
assert "print(1)" in out
|
||||
assert "```py" not in out, "raw fence backticks leaked as text:\n" + out
|
||||
assert "<li>" in out, "list structure lost:\n" + out
|
||||
|
||||
|
||||
def test_nested_list_fence_stays_nested() -> None:
|
||||
"""A fenced code block as a NESTED sub-item keeps its nesting level: the
|
||||
fence pass re-emits the leading indent before the sentinel, so the list
|
||||
pass still reads the sub-item's indentation. Regression guard: dropping
|
||||
the indent flattened the code block to a top-level sibling of the parent."""
|
||||
out = _render("- parent\n - ```py\n code\n ```")
|
||||
assert "parent" in out
|
||||
assert "<pre><code" in out and "```py" not in out
|
||||
assert out.count("<ul>") == 2, "nested list fence flattened to a sibling:\n" + out
|
||||
|
||||
|
||||
def test_big_ordered_marker_fence_is_protected() -> None:
|
||||
r"""A fence opening on a 10+ digit ordered-list marker line is still
|
||||
protected — the marker alternation uses ``\d+``, matching the list pass,
|
||||
not a capped ``\d{1,9}`` that would leave the fence unprotected."""
|
||||
out = _render("1234567890. ```py\ncode\n```")
|
||||
assert "<pre><code" in out, "big ordered-marker fence leaked as text:\n" + out
|
||||
assert "```py" not in out
|
||||
|
||||
|
||||
def test_fenced_block_in_footnote_renders_in_footnote() -> None:
|
||||
"""A fenced code block continuing a footnote definition renders INSIDE the
|
||||
footnote section (the fence pass re-emits the 2-space indent the
|
||||
continuation scan needs; the restore round-trip then resolves it there)."""
|
||||
out = _render("See[^1].\n\n[^1]: note\n ```py\n x=1\n ```")
|
||||
assert 'class="footnotes"' in out
|
||||
assert out.find("<pre") > out.find('class="footnotes"'), (
|
||||
"fenced code in a footnote rendered outside the footnote section:\n" + out
|
||||
)
|
||||
assert "x=1" in out
|
||||
|
||||
|
||||
def test_indented_details_is_extracted() -> None:
|
||||
"""An indented `<details>` (e.g. under a list item) is still extracted into
|
||||
a real disclosure element — the open anchor allows leading whitespace,
|
||||
while a mid-line `<details>` inside inline code still is not (B5)."""
|
||||
out = _render(" <details><summary>x</summary>y</details>")
|
||||
assert "<details><summary>x</summary>" in out, "indented <details> not extracted:\n" + out
|
||||
assert "y" in out
|
||||
|
||||
@@ -176,6 +176,206 @@ class TestScheduleAPI:
|
||||
assert resp.status_code == 400
|
||||
assert "future" in resp.json()["error"].lower()
|
||||
|
||||
@staticmethod
|
||||
def _seed_persona(storage, name="researcher", kinds=None):
|
||||
storage.create_persona(
|
||||
{
|
||||
"persona_id": f"id-{name}",
|
||||
"name": name,
|
||||
"display_name": name.title(),
|
||||
"description": "",
|
||||
"base_prompt": "You are a test persona.",
|
||||
"applies_to_kinds": kinds or ["interactive"],
|
||||
}
|
||||
)
|
||||
|
||||
def test_create_with_persona_and_project(self, client, storage):
|
||||
self._seed_persona(storage)
|
||||
# Owned by the authenticated admin (created_by) → attachable.
|
||||
storage.create_project("proj_1", "My Project", "test-admin")
|
||||
resp = client.post(
|
||||
"/v1/api/admin/schedules",
|
||||
json=_cron_payload(persona="researcher", project_id="proj_1"),
|
||||
)
|
||||
assert resp.status_code == 200, resp.text
|
||||
data = resp.json()
|
||||
assert data["persona"] == "researcher"
|
||||
assert data["project_id"] == "proj_1"
|
||||
|
||||
def test_create_defaults_persona_project_empty(self, client):
|
||||
resp = client.post("/v1/api/admin/schedules", json=_cron_payload())
|
||||
assert resp.status_code == 200
|
||||
data = resp.json()
|
||||
assert data["persona"] == ""
|
||||
assert data["project_id"] == ""
|
||||
|
||||
def test_create_unknown_persona_rejected(self, client):
|
||||
resp = client.post(
|
||||
"/v1/api/admin/schedules",
|
||||
json=_cron_payload(persona="ghost"),
|
||||
)
|
||||
assert resp.status_code == 400
|
||||
assert "persona" in resp.json()["error"].lower()
|
||||
|
||||
def test_create_persona_wrong_kind_rejected(self, client, storage):
|
||||
# A coordinator-only persona is refused — schedules only ever dispatch
|
||||
# interactive workstreams, so the picker/validation are kind-scoped.
|
||||
self._seed_persona(storage, name="orchestrator", kinds=["coordinator"])
|
||||
resp = client.post(
|
||||
"/v1/api/admin/schedules",
|
||||
json=_cron_payload(persona="orchestrator"),
|
||||
)
|
||||
assert resp.status_code == 400
|
||||
|
||||
def test_create_unattachable_project_rejected(self, client, storage):
|
||||
# A private project owned by someone else — the admin isn't a member.
|
||||
storage.create_project("proj_x", "Theirs", "someone-else", visibility="private")
|
||||
resp = client.post(
|
||||
"/v1/api/admin/schedules",
|
||||
json=_cron_payload(project_id="proj_x"),
|
||||
)
|
||||
assert resp.status_code == 403
|
||||
|
||||
def test_update_persona_and_project(self, client, storage):
|
||||
self._seed_persona(storage, name="scribe")
|
||||
storage.create_project("proj_2", "Proj Two", "test-admin")
|
||||
task_id = client.post("/v1/api/admin/schedules", json=_cron_payload()).json()["task_id"]
|
||||
resp = client.put(
|
||||
f"/v1/api/admin/schedules/{task_id}",
|
||||
json={"persona": "scribe", "project_id": "proj_2"},
|
||||
)
|
||||
assert resp.status_code == 200, resp.text
|
||||
data = client.get(f"/v1/api/admin/schedules/{task_id}").json()
|
||||
assert data["persona"] == "scribe"
|
||||
assert data["project_id"] == "proj_2"
|
||||
|
||||
@staticmethod
|
||||
def _legacy_task(storage, task_id="legacy"):
|
||||
"""A schedule from before the created_by fix — created_by is ''."""
|
||||
storage.create_scheduled_task(
|
||||
task_id=task_id,
|
||||
name="Legacy",
|
||||
description="",
|
||||
schedule_type="cron",
|
||||
cron_expr="0 9 * * *",
|
||||
at_time="",
|
||||
target_mode="auto",
|
||||
model="",
|
||||
initial_message="go",
|
||||
auto_approve=False,
|
||||
auto_approve_tools=[],
|
||||
created_by="",
|
||||
next_run="2099-01-01T09:00:00",
|
||||
)
|
||||
|
||||
def test_update_assign_project_heals_empty_created_by(self, client, storage):
|
||||
# Assigning a project to an orphaned schedule adopts the editing admin
|
||||
# as owner so the attach — and every future dispatch — has an identity.
|
||||
self._legacy_task(storage)
|
||||
storage.create_project("proj_heal", "Heal", "test-admin")
|
||||
resp = client.put(
|
||||
"/v1/api/admin/schedules/legacy",
|
||||
json={"project_id": "proj_heal"},
|
||||
)
|
||||
assert resp.status_code == 200, resp.text
|
||||
row = storage.get_scheduled_task("legacy")
|
||||
assert row["project_id"] == "proj_heal"
|
||||
assert row["created_by"] == "test-admin"
|
||||
|
||||
def test_update_denied_project_does_not_heal_created_by(self, client, storage):
|
||||
# Healing must not become an attach bypass: a project the editing admin
|
||||
# can't reach is still 403, and created_by/project stay untouched.
|
||||
self._legacy_task(storage, task_id="legacy2")
|
||||
storage.create_project("proj_other", "Other", "someone-else", visibility="private")
|
||||
resp = client.put(
|
||||
"/v1/api/admin/schedules/legacy2",
|
||||
json={"project_id": "proj_other"},
|
||||
)
|
||||
assert resp.status_code == 403
|
||||
row = storage.get_scheduled_task("legacy2")
|
||||
assert row["created_by"] == ""
|
||||
assert row["project_id"] == ""
|
||||
|
||||
def test_update_project_keeps_existing_owner(self, client, storage):
|
||||
# A schedule that already has a real owner is NOT re-owned by an editing
|
||||
# admin — created_by is only adopted for the orphaned "" case.
|
||||
self._seed_persona(storage, name="researcher")
|
||||
storage.create_scheduled_task(
|
||||
task_id="owned",
|
||||
name="Owned",
|
||||
description="",
|
||||
schedule_type="cron",
|
||||
cron_expr="0 9 * * *",
|
||||
at_time="",
|
||||
target_mode="auto",
|
||||
model="",
|
||||
initial_message="go",
|
||||
auto_approve=False,
|
||||
auto_approve_tools=[],
|
||||
created_by="original-owner",
|
||||
next_run="2099-01-01T09:00:00",
|
||||
)
|
||||
# A public project the original owner (and anyone) can attach to.
|
||||
storage.create_project("proj_pub", "Pub", "someone-else", visibility="public")
|
||||
resp = client.put(
|
||||
"/v1/api/admin/schedules/owned",
|
||||
json={"project_id": "proj_pub"},
|
||||
)
|
||||
assert resp.status_code == 200, resp.text
|
||||
row = storage.get_scheduled_task("owned")
|
||||
assert row["project_id"] == "proj_pub"
|
||||
assert row["created_by"] == "original-owner"
|
||||
|
||||
def test_update_unchanged_persona_skips_revalidation(self, client, storage):
|
||||
# A persona disabled after creation must not block editing other fields
|
||||
# when the shelf resends the unchanged slug (it still fails at dispatch).
|
||||
self._seed_persona(storage, name="researcher")
|
||||
task_id = client.post(
|
||||
"/v1/api/admin/schedules", json=_cron_payload(persona="researcher")
|
||||
).json()["task_id"]
|
||||
storage.update_persona("id-researcher", enabled=False)
|
||||
resp = client.put(
|
||||
f"/v1/api/admin/schedules/{task_id}",
|
||||
json={"name": "Renamed", "persona": "researcher"},
|
||||
)
|
||||
assert resp.status_code == 200, resp.text
|
||||
assert resp.json()["name"] == "Renamed"
|
||||
assert resp.json()["persona"] == "researcher"
|
||||
|
||||
def test_update_unchanged_project_skips_regate(self, client, storage):
|
||||
# Project attach isn't re-gated when unchanged, so a project deleted (or
|
||||
# membership lost) out from under the schedule doesn't block edits.
|
||||
storage.create_project("proj_keep", "Keep", "test-admin")
|
||||
task_id = client.post(
|
||||
"/v1/api/admin/schedules", json=_cron_payload(project_id="proj_keep")
|
||||
).json()["task_id"]
|
||||
storage.delete_project("proj_keep") # a re-gate would now 400
|
||||
resp = client.put(
|
||||
f"/v1/api/admin/schedules/{task_id}",
|
||||
json={"name": "Renamed", "project_id": "proj_keep"},
|
||||
)
|
||||
assert resp.status_code == 200, resp.text
|
||||
assert resp.json()["project_id"] == "proj_keep"
|
||||
|
||||
def test_update_ignores_created_by_in_body(self, client, storage):
|
||||
# created_by is never sourced from the request body — a spoofed value
|
||||
# in the PUT payload is ignored (only the heal path from auth writes it).
|
||||
task_id = client.post("/v1/api/admin/schedules", json=_cron_payload()).json()["task_id"]
|
||||
client.put(
|
||||
f"/v1/api/admin/schedules/{task_id}",
|
||||
json={"name": "X", "created_by": "attacker"},
|
||||
)
|
||||
row = storage.get_scheduled_task(task_id)
|
||||
assert row["created_by"] == "test-admin"
|
||||
|
||||
def test_update_unknown_persona_rejected(self, client):
|
||||
task_id = client.post("/v1/api/admin/schedules", json=_cron_payload()).json()["task_id"]
|
||||
resp = client.put(
|
||||
f"/v1/api/admin/schedules/{task_id}",
|
||||
json={"persona": "ghost"},
|
||||
)
|
||||
assert resp.status_code == 400
|
||||
|
||||
def test_get_schedule(self, client):
|
||||
create_resp = client.post("/v1/api/admin/schedules", json=_cron_payload())
|
||||
task_id = create_resp.json()["task_id"]
|
||||
|
||||
@@ -47,9 +47,44 @@ class TestScheduledTaskCRUD:
|
||||
assert result["enabled"] == 1
|
||||
assert result["created_by"] == "u_admin"
|
||||
assert result["next_run"] == "2099-01-01T09:00:00"
|
||||
# persona/project default to "" — empty means "kind default" / "no
|
||||
# project", resolved late at dispatch (mirrors empty model/skill).
|
||||
assert result["persona"] == ""
|
||||
assert result["project_id"] == ""
|
||||
assert "created" in result
|
||||
assert "updated" in result
|
||||
|
||||
def test_create_with_persona_and_project(self, db):
|
||||
db.create_scheduled_task(**_make_task_kwargs(persona="researcher", project_id="proj_42"))
|
||||
result = db.get_scheduled_task("task_001")
|
||||
assert result is not None
|
||||
assert result["persona"] == "researcher"
|
||||
assert result["project_id"] == "proj_42"
|
||||
|
||||
def test_update_persona_and_project(self, db):
|
||||
db.create_scheduled_task(**_make_task_kwargs())
|
||||
assert db.update_scheduled_task("task_001", persona="scribe", project_id="proj_9")
|
||||
updated = db.get_scheduled_task("task_001")
|
||||
assert updated is not None
|
||||
assert updated["persona"] == "scribe"
|
||||
assert updated["project_id"] == "proj_9"
|
||||
# Clearing back to defaults is a first-class update, not a no-op.
|
||||
assert db.update_scheduled_task("task_001", persona="", project_id="")
|
||||
cleared = db.get_scheduled_task("task_001")
|
||||
assert cleared is not None
|
||||
assert cleared["persona"] == ""
|
||||
assert cleared["project_id"] == ""
|
||||
|
||||
def test_update_created_by(self, db):
|
||||
# created_by is allow-listed for update so the API can adopt an orphaned
|
||||
# ("") schedule's owner. Exercised here so the Postgres backend covers
|
||||
# the write too (the API test is SQLite-pinned).
|
||||
db.create_scheduled_task(**_make_task_kwargs(created_by=""))
|
||||
assert db.update_scheduled_task("task_001", created_by="adopted")
|
||||
row = db.get_scheduled_task("task_001")
|
||||
assert row is not None
|
||||
assert row["created_by"] == "adopted"
|
||||
|
||||
def test_get_nonexistent(self, db):
|
||||
assert db.get_scheduled_task("no_such_task") is None
|
||||
|
||||
|
||||
@@ -156,6 +156,52 @@ class TestSchedulerTick:
|
||||
assert run_kwargs["status"] == "dispatched"
|
||||
assert run_kwargs["ws_id"] == "ws_abc123"
|
||||
|
||||
def test_dispatch_passes_persona_and_project(self, mocks):
|
||||
"""persona + project_id ride to create_workstream; created_by becomes
|
||||
the user_id the node gates the project attach against."""
|
||||
collector, storage = mocks
|
||||
|
||||
task = _make_task(persona="researcher", project_id="proj_42")
|
||||
storage.list_due_tasks.return_value = [task]
|
||||
collector.get_nodes.return_value = ([_make_node()], 1)
|
||||
collector.get_node_detail.return_value = {"server_url": "http://node-001:8080"}
|
||||
|
||||
scheduler = TaskScheduler(collector, storage)
|
||||
with patch(
|
||||
"turnstone.console.scheduler.TurnstoneServer.create_workstream",
|
||||
return_value=_mock_create_response(),
|
||||
) as mock_create:
|
||||
scheduler._tick()
|
||||
|
||||
mock_create.assert_called_once()
|
||||
call_kwargs = mock_create.call_args[1]
|
||||
assert call_kwargs["persona"] == "researcher"
|
||||
assert call_kwargs["project_id"] == "proj_42"
|
||||
assert call_kwargs["user_id"] == "u_admin"
|
||||
|
||||
def test_dispatch_defaults_persona_project_empty(self, mocks):
|
||||
"""A task row without persona/project keys dispatches with empty
|
||||
strings — the node then resolves the current kind default / no attach."""
|
||||
collector, storage = mocks
|
||||
|
||||
task = _make_task()
|
||||
task.pop("persona", None)
|
||||
task.pop("project_id", None)
|
||||
storage.list_due_tasks.return_value = [task]
|
||||
collector.get_nodes.return_value = ([_make_node()], 1)
|
||||
collector.get_node_detail.return_value = {"server_url": "http://node-001:8080"}
|
||||
|
||||
scheduler = TaskScheduler(collector, storage)
|
||||
with patch(
|
||||
"turnstone.console.scheduler.TurnstoneServer.create_workstream",
|
||||
return_value=_mock_create_response(),
|
||||
) as mock_create:
|
||||
scheduler._tick()
|
||||
|
||||
call_kwargs = mock_create.call_args[1]
|
||||
assert call_kwargs["persona"] == ""
|
||||
assert call_kwargs["project_id"] == ""
|
||||
|
||||
def test_dispatch_pool_mode(self, mocks):
|
||||
collector, storage = mocks
|
||||
|
||||
|
||||
@@ -337,6 +337,8 @@ async def test_create_schedule():
|
||||
schedule_type="cron",
|
||||
initial_message="Run nightly checks",
|
||||
cron_expr="0 2 * * *",
|
||||
persona="researcher",
|
||||
project_id="proj_1",
|
||||
)
|
||||
assert resp.task_id == "t1"
|
||||
body = captured_body[0]
|
||||
@@ -344,6 +346,8 @@ async def test_create_schedule():
|
||||
assert body["schedule_type"] == "cron"
|
||||
assert body["cron_expr"] == "0 2 * * *"
|
||||
assert body["initial_message"] == "Run nightly checks"
|
||||
assert body["persona"] == "researcher"
|
||||
assert body["project_id"] == "proj_1"
|
||||
# Optional fields with defaults should not appear when not set
|
||||
assert "description" not in body
|
||||
assert "model" not in body
|
||||
|
||||
@@ -380,12 +380,16 @@ async def test_create_workstream_extended_params():
|
||||
auto_approve_tools="read_file,write_file",
|
||||
user_id="u42",
|
||||
ws_id="ws_custom",
|
||||
persona="researcher",
|
||||
project_id="proj_9",
|
||||
)
|
||||
assert captured_body["name"] == "ext"
|
||||
assert captured_body["initial_message"] == "hi"
|
||||
assert captured_body["auto_approve_tools"] == "read_file,write_file"
|
||||
assert captured_body["user_id"] == "u42"
|
||||
assert captured_body["ws_id"] == "ws_custom"
|
||||
assert captured_body["persona"] == "researcher"
|
||||
assert captured_body["project_id"] == "proj_9"
|
||||
|
||||
|
||||
@pytest.mark.anyio
|
||||
@@ -406,3 +410,5 @@ async def test_create_workstream_omits_empty_params():
|
||||
assert "auto_approve_tools" not in captured_body
|
||||
assert "user_id" not in captured_body
|
||||
assert "ws_id" not in captured_body
|
||||
assert "persona" not in captured_body
|
||||
assert "project_id" not in captured_body
|
||||
|
||||
@@ -296,6 +296,24 @@ class TestGetContent:
|
||||
assert "default-src 'none'" in resp.headers.get("content-security-policy", "")
|
||||
assert resp.headers.get("content-disposition", "").startswith("inline;")
|
||||
|
||||
def test_get_content_non_latin1_filename_does_not_500(self, app_client):
|
||||
# Starlette encodes header values as latin-1 and raises on anything
|
||||
# else; an uploaded filename with CJK / em dashes must fold to an
|
||||
# ASCII-safe Content-Disposition rather than 500 the serving route.
|
||||
# Mirrors preview_response_headers' latin-1 fold.
|
||||
client, _ = app_client
|
||||
aid = _upload(client, "ws-A", "userA", "文書 — v1.md", b"x", "text/markdown")
|
||||
resp = client.get(
|
||||
f"/v1/api/workstreams/ws-A/attachments/{aid}/content",
|
||||
headers=_auth("userA"),
|
||||
)
|
||||
assert resp.status_code == 200
|
||||
assert resp.content == b"x"
|
||||
# Non-ASCII folded to '?', ASCII kept — pinning the value proves the
|
||||
# fold actually ran and the header is latin-1 clean (all codepoints
|
||||
# < 0x80), not merely that the route didn't crash.
|
||||
assert resp.headers["content-disposition"] == 'inline; filename="?? ? v1.md"'
|
||||
|
||||
def test_get_content_forces_text_plain_for_text_kinds(self, app_client):
|
||||
# Uploading an HTML-ish file as text/html must NOT be served back
|
||||
# with Content-Type: text/html from our origin (XSS vector).
|
||||
@@ -459,6 +477,11 @@ class TestSendMessageAttachments:
|
||||
session = MagicMock()
|
||||
session._cancel_event = threading.Event()
|
||||
session.queue_message = MagicMock()
|
||||
# A bare Mock's auto-created ``_nudge_queue`` (truthy, has_pending
|
||||
# truthy, no-op deliver) turns the worker-exit wake backstop into an
|
||||
# endless respawn loop; declare this a stub session WITHOUT a queue
|
||||
# so the wake gate's stub-guard bails.
|
||||
session._nudge_queue = None
|
||||
captured: dict = {}
|
||||
|
||||
def fake_send(message, attachments=None, send_id=None):
|
||||
@@ -480,6 +503,7 @@ class TestSendMessageAttachments:
|
||||
ws.session = session
|
||||
ws.worker_thread = None
|
||||
ws._worker_running = False
|
||||
ws._closed = False # a bare Mock attr is truthy → send() would refuse
|
||||
ws._lock = threading.RLock()
|
||||
mgr.get.return_value = ws
|
||||
return captured, session
|
||||
@@ -658,6 +682,9 @@ class TestQueuedSendWithAttachments:
|
||||
session = MagicMock()
|
||||
session._cancel_event = threading.Event()
|
||||
session.queue_message = fake_queue_message
|
||||
# Stub session without a NudgeQueue — see _wire_ws for why a bare
|
||||
# Mock queue would feed the exit backstop an endless wake loop.
|
||||
session._nudge_queue = None
|
||||
|
||||
ui = MagicMock()
|
||||
ui._ws_lock = threading.Lock()
|
||||
@@ -675,6 +702,7 @@ class TestQueuedSendWithAttachments:
|
||||
ws.session = session
|
||||
ws.worker_thread = worker
|
||||
ws._worker_running = True
|
||||
ws._closed = False # a bare Mock attr is truthy → send() would refuse
|
||||
ws._lock = threading.RLock()
|
||||
mgr.get.return_value = ws
|
||||
return captured
|
||||
@@ -738,6 +766,7 @@ class TestBusyWorkerAttachments:
|
||||
ws.ui = ui
|
||||
ws.session = session
|
||||
ws.worker_thread = worker
|
||||
ws._closed = False # a bare Mock attr is truthy → send() would refuse
|
||||
ws._lock = threading.RLock()
|
||||
mgr.get.return_value = ws
|
||||
return ws, session
|
||||
@@ -1032,3 +1061,156 @@ class TestTextToSpeech:
|
||||
body = resp.json()
|
||||
assert body["error"] == "Speech synthesis backend failed"
|
||||
assert "internal-host" not in body["error"]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# GET /preview — the renderable serving route (preview pane)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _seed_committed(ws_id: str, kind: str, mime: str, body: bytes, filename: str) -> str:
|
||||
"""Commit a blob the way the open_preview fold does: content-addressed
|
||||
save + a tool row whose ref-list names it (the serving ownership gate)."""
|
||||
import hashlib
|
||||
|
||||
from turnstone.core.memory import save_attachment, save_message, set_message_attachments
|
||||
|
||||
aid = hashlib.sha256(b"preview:" + body).hexdigest()
|
||||
save_attachment(aid, filename, mime, len(body), kind, body, "tool")
|
||||
row_id = save_message(ws_id, "tool", "Preview shown", "open_preview", tool_call_id="c1")
|
||||
assert row_id is not None
|
||||
set_message_attachments(ws_id, row_id, [aid])
|
||||
return aid
|
||||
|
||||
|
||||
class TestGetPreview:
|
||||
def test_html_default_serves_locked_down_csp(self, app_client):
|
||||
client, _ = app_client
|
||||
body = b'<html><head><base href="https://acme.com/"></head><body>x</body></html>'
|
||||
aid = _seed_committed("ws-A", "preview", "text/html; charset=utf-8", body, "preview-web")
|
||||
resp = client.get(
|
||||
f"/v1/api/workstreams/ws-A/attachments/{aid}/preview",
|
||||
headers=_auth("userA"),
|
||||
)
|
||||
assert resp.status_code == 200
|
||||
assert resp.headers["content-type"].startswith("text/html")
|
||||
assert resp.content == body
|
||||
# Default (no ?assets): renderable but off the network — sandboxed,
|
||||
# inline styling + data-URI images only, so previewing discloses
|
||||
# nothing to the origin site.
|
||||
assert resp.headers.get("content-security-policy") == (
|
||||
"sandbox; default-src 'none'; style-src 'unsafe-inline'; img-src data:; font-src data:"
|
||||
)
|
||||
assert resp.headers.get("x-content-type-options") == "nosniff"
|
||||
assert resp.headers.get("content-disposition", "").startswith("inline;")
|
||||
assert resp.headers.get("cache-control") == "private, no-store"
|
||||
|
||||
def test_html_assets_flag_serves_bare_sandbox(self, app_client):
|
||||
# ?assets=1 is the per-pane opt-in: drop back to the bare sandbox so
|
||||
# the page's own images / CSS load.
|
||||
client, _ = app_client
|
||||
body = b"<html><head></head><body>x</body></html>"
|
||||
aid = _seed_committed("ws-A", "preview", "text/html; charset=utf-8", body, "preview-web")
|
||||
resp = client.get(
|
||||
f"/v1/api/workstreams/ws-A/attachments/{aid}/preview?assets=1",
|
||||
headers=_auth("userA"),
|
||||
)
|
||||
assert resp.status_code == 200
|
||||
assert resp.headers.get("content-security-policy") == "sandbox"
|
||||
|
||||
def test_pdf_served_without_csp(self, app_client):
|
||||
client, _ = app_client
|
||||
aid = _seed_committed("ws-A", "preview", "application/pdf", b"%PDF-1.4 x", "d.pdf")
|
||||
resp = client.get(
|
||||
f"/v1/api/workstreams/ws-A/attachments/{aid}/preview",
|
||||
headers=_auth("userA"),
|
||||
)
|
||||
assert resp.status_code == 200
|
||||
assert resp.headers["content-type"].startswith("application/pdf")
|
||||
# Chromium's viewer refuses sandboxed contexts — the route omits CSP.
|
||||
assert "content-security-policy" not in resp.headers
|
||||
|
||||
def test_image_keeps_full_csp(self, app_client):
|
||||
client, _ = app_client
|
||||
aid = _seed_committed("ws-A", "preview", "image/png", PNG_1x1, "chart.png")
|
||||
resp = client.get(
|
||||
f"/v1/api/workstreams/ws-A/attachments/{aid}/preview",
|
||||
headers=_auth("userA"),
|
||||
)
|
||||
assert resp.status_code == 200
|
||||
assert "default-src 'none'" in resp.headers.get("content-security-policy", "")
|
||||
|
||||
def test_non_renderable_mime_415(self, app_client):
|
||||
client, _ = app_client
|
||||
aid = _seed_committed("ws-A", "audio", "audio/wav", WAV_12, "a.wav")
|
||||
resp = client.get(
|
||||
f"/v1/api/workstreams/ws-A/attachments/{aid}/preview",
|
||||
headers=_auth("userA"),
|
||||
)
|
||||
assert resp.status_code == 415
|
||||
|
||||
def test_uploaded_attachment_also_previews(self, app_client):
|
||||
# An UPLOADED image (committed via the normal user lane) renders
|
||||
# through /preview too — the pane serves attachment: targets.
|
||||
client, _ = app_client
|
||||
aid = _seed_committed("ws-A", "image", "image/png", PNG_1x1, "up.png")
|
||||
resp = client.get(
|
||||
f"/v1/api/workstreams/ws-A/attachments/{aid}/preview",
|
||||
headers=_auth("userA"),
|
||||
)
|
||||
assert resp.status_code == 200
|
||||
assert resp.headers["content-type"].startswith("image/png")
|
||||
|
||||
def test_unreferenced_id_404(self, app_client):
|
||||
client, _ = app_client
|
||||
aid = _seed_committed("ws-A", "preview", "text/html", b"<p>x</p>", "p")
|
||||
resp = client.get(
|
||||
f"/v1/api/workstreams/ws-B/attachments/{aid}/preview",
|
||||
headers=_auth("userB"),
|
||||
)
|
||||
assert resp.status_code == 404
|
||||
|
||||
def test_probe_returns_204_with_hardening_headers(self, app_client):
|
||||
# The pane preflights src-loaded kinds with ?probe=1 instead of HEAD:
|
||||
# the console reverse proxy forwards a HEAD as a full GET, so a real
|
||||
# HEAD would drag the whole blob across the hop just to discard it. The
|
||||
# probe runs the ownership + renderable-type gates and returns the real
|
||||
# response's hardening headers with an empty body.
|
||||
client, _ = app_client
|
||||
body = b"<html><head></head><body>x</body></html>"
|
||||
aid = _seed_committed("ws-A", "preview", "text/html; charset=utf-8", body, "preview-web")
|
||||
resp = client.get(
|
||||
f"/v1/api/workstreams/ws-A/attachments/{aid}/preview?probe=1",
|
||||
headers=_auth("userA"),
|
||||
)
|
||||
assert resp.status_code == 204
|
||||
assert resp.content == b""
|
||||
# Same hardening headers the real GET would carry (the probe answers
|
||||
# "will the load paint?"): the html CSP is present.
|
||||
assert resp.headers.get("content-security-policy") == (
|
||||
"sandbox; default-src 'none'; style-src 'unsafe-inline'; img-src data:; font-src data:"
|
||||
)
|
||||
assert resp.headers.get("x-content-type-options") == "nosniff"
|
||||
|
||||
def test_probe_composes_with_assets_flag(self, app_client):
|
||||
# ?probe=1&assets=1 → 204 whose headers reflect the assets opt-in.
|
||||
client, _ = app_client
|
||||
body = b"<html><head></head><body>x</body></html>"
|
||||
aid = _seed_committed("ws-A", "preview", "text/html; charset=utf-8", body, "preview-web")
|
||||
resp = client.get(
|
||||
f"/v1/api/workstreams/ws-A/attachments/{aid}/preview?probe=1&assets=1",
|
||||
headers=_auth("userA"),
|
||||
)
|
||||
assert resp.status_code == 204
|
||||
assert resp.headers.get("content-security-policy") == "sandbox"
|
||||
|
||||
def test_probe_non_renderable_mime_still_415(self, app_client):
|
||||
# A probe must answer "will the real load succeed?" — a non-renderable
|
||||
# blob 415s exactly as the real GET would, before any 204.
|
||||
client, _ = app_client
|
||||
aid = _seed_committed("ws-A", "audio", "audio/wav", WAV_12, "a.wav")
|
||||
resp = client.get(
|
||||
f"/v1/api/workstreams/ws-A/attachments/{aid}/preview?probe=1",
|
||||
headers=_auth("userA"),
|
||||
)
|
||||
assert resp.status_code == 415
|
||||
|
||||
@@ -434,6 +434,91 @@ class TestCreateMultipart:
|
||||
# pane's rehydrate can't observe it as still-staged.
|
||||
assert get_attachment_buffer().get(aid, ws_id=ws_id, user_id="userA") is None
|
||||
|
||||
def test_create_raced_by_live_worker_keeps_attachments_staged(self, app_client, monkeypatch):
|
||||
"""The enqueue branch (caller-supplied ws_id raced by a concurrent
|
||||
/send claiming the worker first) can't deliver attachments through
|
||||
the interjection seam — they must REMAIN STAGED so the composer
|
||||
still shows them and the user's next send delivers them, while the
|
||||
message text itself rides the queue."""
|
||||
from turnstone.core import session_worker
|
||||
from turnstone.core.attachment_buffer import get_attachment_buffer
|
||||
|
||||
client, _sessions, _gq = app_client
|
||||
queued: list[str] = []
|
||||
|
||||
def _record_queue(self, text, *a, **k):
|
||||
queued.append(text)
|
||||
return ("", "normal", "msg-x")
|
||||
|
||||
monkeypatch.setattr(_FakeSession, "queue_message", _record_queue)
|
||||
|
||||
def _live_worker_send(ws, *, enqueue, run, thread_name=None):
|
||||
enqueue() # a worker already owns the ws — reuse path
|
||||
return True
|
||||
|
||||
monkeypatch.setattr(session_worker, "send", _live_worker_send)
|
||||
|
||||
meta = {"name": "raced", "initial_message": "look at this file"}
|
||||
resp = client.post(
|
||||
"/v1/api/workstreams/new",
|
||||
data={"meta": json.dumps(meta)},
|
||||
files=[("file", ("notes.md", b"# hello\n", "text/markdown"))],
|
||||
headers=_auth("userA"),
|
||||
)
|
||||
assert resp.status_code == 200, resp.text
|
||||
ws_id = resp.json()["ws_id"]
|
||||
aid = resp.json()["attachment_ids"][0]
|
||||
|
||||
assert queued == ["look at this file"] # text preserved via the queue
|
||||
# NOT drained: the upload stays staged, recoverable on the next send.
|
||||
assert get_attachment_buffer().get(aid, ws_id=ws_id, user_id="userA") is not None
|
||||
# Delivered path → no dropped-message marker on the response.
|
||||
assert "initial_message_status" not in resp.json()
|
||||
|
||||
def test_create_raced_queue_full_reports_dropped_message(self, app_client, monkeypatch):
|
||||
"""``queue.Full`` on the raced enqueue path must not read as
|
||||
success: it propagates out of ``_enqueue_init`` into
|
||||
``session_worker.send``'s backpressure branch (→ ``False``), and
|
||||
the create response carries ``initial_message_status:
|
||||
"queue_full"`` instead of a bare 200 implying the first message
|
||||
was delivered. Attachments stay staged for the retry."""
|
||||
import queue as _queue
|
||||
|
||||
from turnstone.core import session_worker
|
||||
from turnstone.core.attachment_buffer import get_attachment_buffer
|
||||
|
||||
client, _sessions, _gq = app_client
|
||||
|
||||
def _full_queue(self, *a, **k):
|
||||
raise _queue.Full
|
||||
|
||||
monkeypatch.setattr(_FakeSession, "queue_message", _full_queue)
|
||||
|
||||
def _live_worker_send(ws, *, enqueue, run, thread_name=None):
|
||||
# Mirror the real send()'s reuse-path backpressure contract:
|
||||
# queue.Full → False, never a raise to the caller.
|
||||
try:
|
||||
enqueue()
|
||||
except _queue.Full:
|
||||
return False
|
||||
return True
|
||||
|
||||
monkeypatch.setattr(session_worker, "send", _live_worker_send)
|
||||
|
||||
meta = {"name": "raced-full", "initial_message": "look at this file"}
|
||||
resp = client.post(
|
||||
"/v1/api/workstreams/new",
|
||||
data={"meta": json.dumps(meta)},
|
||||
files=[("file", ("notes.md", b"# hello\n", "text/markdown"))],
|
||||
headers=_auth("userA"),
|
||||
)
|
||||
assert resp.status_code == 200, resp.text
|
||||
body = resp.json()
|
||||
assert body["initial_message_status"] == "queue_full"
|
||||
# Attachments untouched — the composer chips survive for the retry.
|
||||
aid = body["attachment_ids"][0]
|
||||
assert get_attachment_buffer().get(aid, ws_id=body["ws_id"], user_id="userA") is not None
|
||||
|
||||
def test_create_with_attachments_no_initial_message_keeps_staged(self, app_client):
|
||||
import hashlib
|
||||
|
||||
@@ -527,3 +612,28 @@ class TestCreateJsonStillWorks:
|
||||
assert data["ws_id"]
|
||||
# New optional field, but always emitted (empty list when absent)
|
||||
assert data["attachment_ids"] == []
|
||||
|
||||
def test_initial_message_routes_through_session_worker_send(self, app_client):
|
||||
"""The initial-message worker is dispatched via
|
||||
``session_worker.send`` (not an inlined ``threading.Thread``) so it
|
||||
inherits the ownership-clear wake backstop. Patching the module
|
||||
attribute captures the wiring without spawning a thread — server.py
|
||||
calls ``session_worker.send`` as a module attribute even from its
|
||||
local import."""
|
||||
from unittest.mock import patch
|
||||
|
||||
client, _sessions, _gq = app_client
|
||||
with patch("turnstone.core.session_worker.send", return_value=True) as mock_send:
|
||||
resp = client.post(
|
||||
"/v1/api/workstreams/new",
|
||||
json={"name": "init-dispatch", "initial_message": "go"},
|
||||
headers=_auth("userA"),
|
||||
)
|
||||
assert resp.status_code == 200, resp.text
|
||||
assert mock_send.call_count == 1
|
||||
kwargs = mock_send.call_args.kwargs
|
||||
assert kwargs["thread_name"].startswith("ws-init-")
|
||||
# ``run`` is the init closure the shared dispatcher spawns; the
|
||||
# dead-by-construction ``enqueue`` branch is still wired (loudly).
|
||||
assert callable(kwargs["run"])
|
||||
assert callable(kwargs["enqueue"])
|
||||
|
||||
+716
-3
@@ -1002,6 +1002,27 @@ class TestEvaluateIntentProjection:
|
||||
assert fa["edits"][0]["near_line"] == 42
|
||||
assert fa["replace_all"] is False
|
||||
|
||||
# -- bash: backgrounding is part of the intent (#817) -------------------
|
||||
|
||||
def test_bash_background_projects_run_in_background(self) -> None:
|
||||
"""The judge must know a bash command will run detached — a
|
||||
backgrounded server/miner is a different intent than a bounded run.
|
||||
Built via the real preparer so the prepared item can't silently drop
|
||||
the flag before the projection reads it."""
|
||||
session = _make_session()
|
||||
item = session._prepare_bash(
|
||||
"c1", {"command": "python -m http.server 8000", "run_in_background": True}
|
||||
)
|
||||
fa = _project_func_args(item)
|
||||
assert fa["run_in_background"] is True
|
||||
assert fa["command"] == "python -m http.server 8000"
|
||||
|
||||
def test_bash_foreground_projects_run_in_background_false(self) -> None:
|
||||
session = _make_session()
|
||||
item = session._prepare_bash("c1", {"command": "echo hi"})
|
||||
fa = _project_func_args(item)
|
||||
assert fa["run_in_background"] is False
|
||||
|
||||
# -- skills: the dead-assignment bug -----------------------------------
|
||||
|
||||
def test_skills_create_projection_is_not_empty(self) -> None:
|
||||
@@ -2429,9 +2450,582 @@ class TestAgentChildRegistration:
|
||||
parent_call_id="task-1",
|
||||
)
|
||||
|
||||
# Sub-agent tool ids are namespaced by the parent so the UI registry
|
||||
# can't collide across concurrent task agents (local sequential ids).
|
||||
session.ui.note_agent_child.assert_called_once_with("task-1::call_1", "task-1")
|
||||
# Sub-agent tool ids are minted ``{parent}::r{run}s{step}::{provider_id}``
|
||||
# so the UI registry can't collide across concurrent task agents, across
|
||||
# turns within one agent (local sequential ids like "call_0"), or across
|
||||
# runs whose PARENT id was itself reused.
|
||||
session.ui.note_agent_child.assert_called_once_with("task-1::r1s1::call_1", "task-1")
|
||||
|
||||
def test_cross_turn_reused_provider_ids_stay_distinct(self):
|
||||
# A local provider reuses "call_0" verbatim every response. The minted
|
||||
# id carries a per-agent step sequence, so the registry, the wire, the
|
||||
# recall projection, and the cancel ledger all see two DISTINCT calls.
|
||||
# Pre-mint both mapped to "task-1::call_0": the live card collapsed the
|
||||
# rows (bug-3) while FIFO recall kept them apart — the two disagreed on
|
||||
# identical input.
|
||||
from turnstone.core.providers._openai_chat import OpenAIChatCompletionsProvider
|
||||
|
||||
session = _make_session()
|
||||
session._provider = OpenAIChatCompletionsProvider()
|
||||
session.ui.note_agent_child = MagicMock()
|
||||
|
||||
call_count = [0]
|
||||
|
||||
def fake_create(**_kwargs):
|
||||
call_count[0] += 1
|
||||
resp = MagicMock()
|
||||
choice = MagicMock()
|
||||
if call_count[0] <= 2:
|
||||
choice.finish_reason = "tool_calls"
|
||||
tc = MagicMock()
|
||||
tc.id = "call_0" # reused verbatim across turns
|
||||
tc.function.name = "read_file"
|
||||
tc.function.arguments = f'{{"path": "/tmp/f{call_count[0]}"}}'
|
||||
choice.message.tool_calls = [tc]
|
||||
choice.message.content = None
|
||||
else:
|
||||
choice.finish_reason = "stop"
|
||||
choice.message.tool_calls = None
|
||||
choice.message.content = "done"
|
||||
resp.choices = [choice]
|
||||
resp.usage = MagicMock(prompt_tokens=10, completion_tokens=5)
|
||||
return resp
|
||||
|
||||
session.client.chat.completions.create = fake_create
|
||||
|
||||
def fake_prepare(tc_dict, **_kwargs):
|
||||
n = call_count[0]
|
||||
return {
|
||||
"call_id": tc_dict["id"],
|
||||
"func_name": "read_file",
|
||||
"needs_approval": False,
|
||||
"execute": lambda p, n=n: (p["call_id"], f"contents-{n}"),
|
||||
}
|
||||
|
||||
agent_turns = [Turn.user("x")]
|
||||
with patch.object(session, "_prepare_tool", side_effect=fake_prepare):
|
||||
session._run_agent(
|
||||
agent_turns,
|
||||
tools=[{"type": "function", "function": {"name": "read_file"}}],
|
||||
label="task",
|
||||
parent_call_id="task-1",
|
||||
)
|
||||
|
||||
# Registry: two registrations, distinct minted ids, same parent.
|
||||
assert [c.args for c in session.ui.note_agent_child.call_args_list] == [
|
||||
("task-1::r1s1::call_0", "task-1"),
|
||||
("task-1::r1s2::call_0", "task-1"),
|
||||
]
|
||||
# Recall projection: two steps, each paired to its OWN result.
|
||||
steps = ChatSession._project_agent_steps(agent_turns)
|
||||
assert [s["id"] for s in steps] == ["task-1::r1s1::call_0", "task-1::r1s2::call_0"]
|
||||
assert [s["output"] for s in steps] == ["contents-1", "contents-2"]
|
||||
# Cancel ledger agrees: both calls answered, no in-flight gap.
|
||||
issued, first_gap = ChatSession._cancel_ledger(agent_turns)
|
||||
assert issued == [("read_file", True), ("read_file", True)]
|
||||
assert first_gap is None
|
||||
|
||||
@staticmethod
|
||||
def _reusing_provider(session, tool_turns: int = 1):
|
||||
"""Fake create() reissuing id "call_0" for ``tool_turns`` turns, then
|
||||
stopping — the local-server id-reuse shape. Returns the counter."""
|
||||
call_count = [0]
|
||||
|
||||
def fake_create(**_kwargs):
|
||||
call_count[0] += 1
|
||||
resp = MagicMock()
|
||||
choice = MagicMock()
|
||||
if call_count[0] <= tool_turns:
|
||||
choice.finish_reason = "tool_calls"
|
||||
tc = MagicMock()
|
||||
tc.id = "call_0"
|
||||
tc.function.name = "read_file"
|
||||
tc.function.arguments = '{"path": "/tmp/x"}'
|
||||
choice.message.tool_calls = [tc]
|
||||
choice.message.content = None
|
||||
else:
|
||||
choice.finish_reason = "stop"
|
||||
choice.message.tool_calls = None
|
||||
choice.message.content = "done"
|
||||
resp.choices = [choice]
|
||||
resp.usage = MagicMock(prompt_tokens=10, completion_tokens=5)
|
||||
return resp
|
||||
|
||||
session.client.chat.completions.create = fake_create
|
||||
return call_count
|
||||
|
||||
def test_parent_id_reuse_across_runs_mints_distinct_child_ids(self):
|
||||
# A local provider reuses "call_0" for the PARENT task_agent call too:
|
||||
# two sequential runs share parent_call_id "call_0". The session-level
|
||||
# run counter keeps their minted CHILD ids distinct — with only the
|
||||
# per-run step seq (the intermediate fix, before the run counter) both
|
||||
# runs minted "call_0::s1::call_0" and the second agent's sub-tool
|
||||
# steps grafted onto the first agent's DOM rows.
|
||||
#
|
||||
# SCOPE: this fixes child (sub-tool) ids only. The parent CARD still
|
||||
# keys on the raw reused parent id ("call_0") — stash_agent_trajectory,
|
||||
# _tool_status, the card's own data-call-id row — so two runs with the
|
||||
# same parent id still alias at the card level. Parent ids are
|
||||
# main-loop ids; de-colliding them is the main-loop id-hygiene
|
||||
# follow-up, not this change.
|
||||
from turnstone.core.providers._openai_chat import OpenAIChatCompletionsProvider
|
||||
|
||||
session = _make_session()
|
||||
session._provider = OpenAIChatCompletionsProvider()
|
||||
session.ui.note_agent_child = MagicMock()
|
||||
|
||||
def fake_prepare(tc_dict, **_kwargs):
|
||||
return {
|
||||
"call_id": tc_dict["id"],
|
||||
"func_name": "read_file",
|
||||
"needs_approval": False,
|
||||
"execute": lambda p: (p["call_id"], "contents"),
|
||||
}
|
||||
|
||||
minted: list[str] = []
|
||||
for _run in range(2):
|
||||
self._reusing_provider(session)
|
||||
with patch.object(session, "_prepare_tool", side_effect=fake_prepare):
|
||||
session._run_agent(
|
||||
[Turn.user("x")],
|
||||
tools=[{"type": "function", "function": {"name": "read_file"}}],
|
||||
label="task",
|
||||
parent_call_id="call_0",
|
||||
)
|
||||
minted.append(session.ui.note_agent_child.call_args.args[0])
|
||||
|
||||
assert minted == ["call_0::r1s1::call_0", "call_0::r2s1::call_0"]
|
||||
assert len(set(minted)) == 2
|
||||
|
||||
def test_agent_wire_restores_provider_ids_and_sanitizes_args(self):
|
||||
# The agent seam bypasses the main-loop wire prep and builds its own
|
||||
# history, so it runs its own validity passes. Drive one tool turn
|
||||
# whose call carries a minted "::" id (mapped back to the provider's
|
||||
# own id on the wire) and malformed non-object arguments (a strict
|
||||
# renderer json.loads and 400s them), then assert the REPLAY request
|
||||
# the second _api_call sends carries the PROVIDER-ORIGINAL id on both
|
||||
# the call and its result, and object-shaped arguments. The internal
|
||||
# id keeps the minted "::" form.
|
||||
import json as _json
|
||||
|
||||
from turnstone.core.providers._openai_chat import OpenAIChatCompletionsProvider
|
||||
|
||||
session = _make_session()
|
||||
session._provider = OpenAIChatCompletionsProvider()
|
||||
session.ui.note_agent_child = MagicMock()
|
||||
|
||||
seen_messages: list[list[dict]] = []
|
||||
call_count = [0]
|
||||
|
||||
def fake_create(**kwargs):
|
||||
seen_messages.append(kwargs.get("messages") or [])
|
||||
call_count[0] += 1
|
||||
resp = MagicMock()
|
||||
choice = MagicMock()
|
||||
if call_count[0] == 1:
|
||||
choice.finish_reason = "tool_calls"
|
||||
tc = MagicMock()
|
||||
tc.id = "call_0"
|
||||
tc.function.name = "read_file"
|
||||
# Malformed: unterminated JSON with a non-"length" finish
|
||||
# reason — the sanitize pass's reason to exist.
|
||||
tc.function.arguments = '{"path": "/tmp/x"'
|
||||
choice.message.tool_calls = [tc]
|
||||
choice.message.content = None
|
||||
else:
|
||||
choice.finish_reason = "stop"
|
||||
choice.message.tool_calls = None
|
||||
choice.message.content = "done"
|
||||
resp.choices = [choice]
|
||||
resp.usage = MagicMock(prompt_tokens=10, completion_tokens=5)
|
||||
return resp
|
||||
|
||||
session.client.chat.completions.create = fake_create
|
||||
|
||||
def fake_prepare(tc_dict, **_kwargs):
|
||||
return {
|
||||
"call_id": tc_dict["id"],
|
||||
"func_name": "read_file",
|
||||
"needs_approval": False,
|
||||
"execute": lambda p: (p["call_id"], "contents"),
|
||||
}
|
||||
|
||||
with patch.object(session, "_prepare_tool", side_effect=fake_prepare):
|
||||
session._run_agent(
|
||||
[Turn.user("x")],
|
||||
tools=[{"type": "function", "function": {"name": "read_file"}}],
|
||||
label="task",
|
||||
parent_call_id="task-1",
|
||||
)
|
||||
|
||||
# Internal id (registry) keeps the minted "::" form.
|
||||
internal = session.ui.note_agent_child.call_args.args[0]
|
||||
assert internal == "task-1::r1s1::call_0"
|
||||
# The SECOND request replays the tool turn: the wire carries the
|
||||
# provider's own id, consistent between the call and its result (the
|
||||
# shape the provider-native tool_use block also holds, so a native
|
||||
# replay and a rebuild agree); arguments are legalized to a JSON
|
||||
# object.
|
||||
replay = seen_messages[1]
|
||||
wire_calls = [tc for m in replay if m.get("tool_calls") for tc in m["tool_calls"]]
|
||||
wire_results = [m for m in replay if m.get("role") == "tool"]
|
||||
assert wire_calls and wire_results
|
||||
assert wire_calls[0]["id"] == "call_0"
|
||||
assert wire_results[0]["tool_call_id"] == "call_0"
|
||||
assert isinstance(_json.loads(wire_calls[0]["function"]["arguments"]), dict)
|
||||
|
||||
def test_agent_carries_native_lane_and_replays_thinking_anthropic(self):
|
||||
# The load-bearing fidelity pin: a thinking-model agent's SECOND
|
||||
# request must carry the prior assistant turn's native lane verbatim
|
||||
# — thinking block and signature untouched — with the provider's own
|
||||
# tool_use id agreeing across the native block, the restored
|
||||
# top-level mirror, and the tool_result. Pre-native-lane, the seam
|
||||
# rebuilt the turn from content + tool_calls and the model re-reasoned
|
||||
# from scratch every tool turn (and commercial Anthropic rejects a
|
||||
# thinking-enabled tool_use turn without its thinking block).
|
||||
from turnstone.core.providers._anthropic import AnthropicProvider
|
||||
|
||||
class _Block:
|
||||
def __init__(self, **d):
|
||||
self._d = d
|
||||
for k, v in d.items():
|
||||
setattr(self, k, v)
|
||||
|
||||
def model_dump(self, **_kw):
|
||||
return dict(self._d)
|
||||
|
||||
session = _make_session()
|
||||
session._provider = AnthropicProvider()
|
||||
session.ui.note_agent_child = MagicMock()
|
||||
|
||||
seen: list[dict] = []
|
||||
call_count = [0]
|
||||
|
||||
def fake_stream(**kwargs):
|
||||
seen.append(kwargs)
|
||||
call_count[0] += 1
|
||||
resp = MagicMock()
|
||||
if call_count[0] == 1:
|
||||
resp.content = [
|
||||
_Block(type="thinking", thinking="check the file first", signature="sig_v1"),
|
||||
_Block(type="text", text="reading"),
|
||||
_Block(type="tool_use", id="toolu_01AB", name="read_file", input={"path": "x"}),
|
||||
]
|
||||
resp.stop_reason = "tool_use"
|
||||
else:
|
||||
resp.content = [_Block(type="text", text="done")]
|
||||
resp.stop_reason = "end_turn"
|
||||
resp.usage = None
|
||||
mgr = MagicMock()
|
||||
mgr.__enter__ = MagicMock(
|
||||
return_value=MagicMock(get_final_message=MagicMock(return_value=resp))
|
||||
)
|
||||
mgr.__exit__ = MagicMock(return_value=False)
|
||||
return mgr
|
||||
|
||||
session.client.messages.stream = fake_stream
|
||||
|
||||
def fake_prepare(tc_dict, **_kwargs):
|
||||
return {
|
||||
"call_id": tc_dict["id"],
|
||||
"func_name": "read_file",
|
||||
"needs_approval": False,
|
||||
"execute": lambda p: (p["call_id"], "contents"),
|
||||
}
|
||||
|
||||
with (
|
||||
patch.object(session, "_prepare_tool", side_effect=fake_prepare),
|
||||
patch.object(session, "_resolve_replay_reasoning_to_model", return_value=True),
|
||||
):
|
||||
session._run_agent(
|
||||
[Turn.user("x")],
|
||||
tools=[{"type": "function", "function": {"name": "read_file", "parameters": {}}}],
|
||||
label="task",
|
||||
parent_call_id="task-1",
|
||||
)
|
||||
|
||||
# Internal key stays minted — the nesting registry saw the "::" id.
|
||||
assert session.ui.note_agent_child.call_args.args[0] == "task-1::r1s1::toolu_01AB"
|
||||
# Second request: the assistant wire turn IS the native lane.
|
||||
replay = seen[1]["messages"]
|
||||
assistant = next(
|
||||
m for m in replay if m["role"] == "assistant" and isinstance(m.get("content"), list)
|
||||
)
|
||||
kinds = [b.get("type") for b in assistant["content"]]
|
||||
assert kinds == ["thinking", "text", "tool_use"]
|
||||
assert assistant["content"][0]["thinking"] == "check the file first"
|
||||
assert assistant["content"][0]["signature"] == "sig_v1" # byte-untouched
|
||||
assert assistant["content"][2]["id"] == "toolu_01AB" # provider-original
|
||||
tool_results = [
|
||||
b
|
||||
for m in replay
|
||||
if m["role"] == "user" and isinstance(m.get("content"), list)
|
||||
for b in m["content"]
|
||||
if isinstance(b, dict) and b.get("type") == "tool_result"
|
||||
]
|
||||
assert tool_results and tool_results[0]["tool_use_id"] == "toolu_01AB"
|
||||
|
||||
def test_agent_blank_provider_id_skips_native_lane(self):
|
||||
# A server that leaves a tool-call id blank gets a uuid back-fill in
|
||||
# the tool_calls mirror (_ensure_tool_call_ids) — but the native
|
||||
# tool_use block keeps the blank id verbatim. Carrying the lane for
|
||||
# that turn would replay a native tool_use whose id matches no
|
||||
# tool_result (Anthropic orphans the result and 400s). The shared
|
||||
# builder must drop the whole Messages-shaped lane for exactly that
|
||||
# turn (a residual thinking block would REPLACE the rebuilt content
|
||||
# and lose the tool_use) and fall back to the rebuild path, where
|
||||
# every wire representation uses the back-filled id consistently.
|
||||
from turnstone.core.providers._anthropic import AnthropicProvider
|
||||
|
||||
class _Block:
|
||||
def __init__(self, **d):
|
||||
self._d = d
|
||||
for k, v in d.items():
|
||||
setattr(self, k, v)
|
||||
|
||||
def model_dump(self, **_kw):
|
||||
return dict(self._d)
|
||||
|
||||
session = _make_session()
|
||||
session._provider = AnthropicProvider()
|
||||
session.ui.note_agent_child = MagicMock()
|
||||
|
||||
seen: list[dict] = []
|
||||
call_count = [0]
|
||||
|
||||
def fake_stream(**kwargs):
|
||||
seen.append(kwargs)
|
||||
call_count[0] += 1
|
||||
resp = MagicMock()
|
||||
if call_count[0] == 1:
|
||||
resp.content = [
|
||||
_Block(type="thinking", thinking="hm", signature="sig_b"),
|
||||
# Blank provider id — the back-fill case.
|
||||
_Block(type="tool_use", id="", name="read_file", input={"path": "x"}),
|
||||
]
|
||||
resp.stop_reason = "tool_use"
|
||||
else:
|
||||
resp.content = [_Block(type="text", text="done")]
|
||||
resp.stop_reason = "end_turn"
|
||||
resp.usage = None
|
||||
mgr = MagicMock()
|
||||
mgr.__enter__ = MagicMock(
|
||||
return_value=MagicMock(get_final_message=MagicMock(return_value=resp))
|
||||
)
|
||||
mgr.__exit__ = MagicMock(return_value=False)
|
||||
return mgr
|
||||
|
||||
session.client.messages.stream = fake_stream
|
||||
|
||||
def fake_prepare(tc_dict, **_kwargs):
|
||||
return {
|
||||
"call_id": tc_dict["id"],
|
||||
"func_name": "read_file",
|
||||
"needs_approval": False,
|
||||
"execute": lambda p: (p["call_id"], "contents"),
|
||||
}
|
||||
|
||||
turns = [Turn.user("x")]
|
||||
with (
|
||||
patch.object(session, "_prepare_tool", side_effect=fake_prepare),
|
||||
patch.object(session, "_resolve_replay_reasoning_to_model", return_value=True),
|
||||
):
|
||||
session._run_agent(
|
||||
turns,
|
||||
tools=[{"type": "function", "function": {"name": "read_file", "parameters": {}}}],
|
||||
label="task",
|
||||
parent_call_id="task-1",
|
||||
)
|
||||
|
||||
# The back-filled turn carries NO native lane.
|
||||
assert turns[1].native is None
|
||||
# The replay request rebuilds the turn: tool_use and tool_result agree
|
||||
# on the back-filled uuid — no blank id, no orphan.
|
||||
replay = seen[1]["messages"]
|
||||
tool_uses = [
|
||||
b
|
||||
for m in replay
|
||||
if m["role"] == "assistant" and isinstance(m.get("content"), list)
|
||||
for b in m["content"]
|
||||
if isinstance(b, dict) and b.get("type") == "tool_use"
|
||||
]
|
||||
tool_results = [
|
||||
b
|
||||
for m in replay
|
||||
if m["role"] == "user" and isinstance(m.get("content"), list)
|
||||
for b in m["content"]
|
||||
if isinstance(b, dict) and b.get("type") == "tool_result"
|
||||
]
|
||||
assert tool_uses and tool_results
|
||||
assert tool_uses[0]["id"] # non-blank (uuid back-fill, restored)
|
||||
assert tool_results[0]["tool_use_id"] == tool_uses[0]["id"]
|
||||
|
||||
def test_agent_blank_provider_id_keeps_synthesized_reasoning(self):
|
||||
# The over-drop guard: a Chat-Completions server that BOTH leaves
|
||||
# tool-call ids blank AND surfaces reasoning_content (llama.cpp,
|
||||
# older vLLM) must still get its reasoning carried — the blank-id
|
||||
# gate drops only the blocks a back-fill desyncs, and the
|
||||
# synthesized reasoning_text lane has no client tool blocks at all.
|
||||
from types import SimpleNamespace
|
||||
|
||||
from turnstone.core.providers._openai_chat import OpenAIChatCompletionsProvider
|
||||
from turnstone.core.providers._openai_common import OPENAI_COMPAT_DEFAULT
|
||||
|
||||
session = _make_session()
|
||||
session._provider = OpenAIChatCompletionsProvider()
|
||||
session._model_alias = "loc"
|
||||
session._registry = MagicMock()
|
||||
session._registry.resolve_agent_alias.return_value = None
|
||||
session._registry.resolve_agent_effort.return_value = None
|
||||
session._registry.get_config.return_value = SimpleNamespace(
|
||||
server_compat={"server_type": "vllm"}, replay_reasoning_to_model=True
|
||||
)
|
||||
session.ui.note_agent_child = MagicMock()
|
||||
|
||||
call_count = [0]
|
||||
|
||||
def fake_create(**kwargs):
|
||||
call_count[0] += 1
|
||||
resp = MagicMock()
|
||||
choice = MagicMock()
|
||||
if call_count[0] == 1:
|
||||
choice.finish_reason = "tool_calls"
|
||||
tc = MagicMock()
|
||||
tc.id = "" # blank — the back-fill case
|
||||
tc.function.name = "read_file"
|
||||
tc.function.arguments = '{"path": "x"}'
|
||||
choice.message.tool_calls = [tc]
|
||||
choice.message.content = None
|
||||
choice.message.reasoning = None
|
||||
choice.message.reasoning_content = "work it out"
|
||||
else:
|
||||
choice.finish_reason = "stop"
|
||||
choice.message.tool_calls = None
|
||||
choice.message.content = "done"
|
||||
choice.message.reasoning = None
|
||||
choice.message.reasoning_content = None
|
||||
resp.choices = [choice]
|
||||
resp.usage = MagicMock(prompt_tokens=1, completion_tokens=1)
|
||||
return resp
|
||||
|
||||
session.client.chat.completions.create = fake_create
|
||||
|
||||
def fake_prepare(tc_dict, **_kwargs):
|
||||
return {
|
||||
"call_id": tc_dict["id"],
|
||||
"func_name": "read_file",
|
||||
"needs_approval": False,
|
||||
"execute": lambda p: (p["call_id"], "contents"),
|
||||
}
|
||||
|
||||
turns = [Turn.user("x")]
|
||||
with (
|
||||
patch.object(session, "_prepare_tool", side_effect=fake_prepare),
|
||||
patch.object(session, "_resolve_capabilities", return_value=OPENAI_COMPAT_DEFAULT),
|
||||
patch.object(session, "_provider_extra_params", return_value={}),
|
||||
):
|
||||
session._run_agent(
|
||||
turns,
|
||||
tools=[{"type": "function", "function": {"name": "read_file"}}],
|
||||
label="task",
|
||||
parent_call_id="task-1",
|
||||
)
|
||||
|
||||
# Reasoning survives the blank-id turn.
|
||||
assert turns[1].native is not None
|
||||
assert [b["type"] for b in turns[1].native.blocks] == ["reasoning_text"]
|
||||
assert turns[1].native.blocks[0]["text"] == "work it out"
|
||||
|
||||
def test_agent_synthesizes_reasoning_and_attaches_vllm_replay_field(self):
|
||||
# Chat-Completions lane (vLLM): non-streaming ``reasoning_content`` is
|
||||
# captured into CompletionResult.reasoning, synthesized into the agent
|
||||
# turn's native lane as a ``reasoning_text`` block by the SAME
|
||||
# finalize helper the main loop uses — source-tagged from the AGENT
|
||||
# alias — and replayed on the next request as vLLM's non-standard
|
||||
# ``reasoning`` field (Phase 5 at the agent seam; the internal
|
||||
# ``_provider_content`` key itself never reaches the wire).
|
||||
from types import SimpleNamespace
|
||||
|
||||
from turnstone.core.providers._openai_chat import OpenAIChatCompletionsProvider
|
||||
from turnstone.core.providers._openai_common import OPENAI_COMPAT_DEFAULT
|
||||
|
||||
session = _make_session()
|
||||
session._provider = OpenAIChatCompletionsProvider()
|
||||
session._model_alias = "loc-qwen"
|
||||
session._registry = MagicMock()
|
||||
session._registry.resolve_agent_alias.return_value = None
|
||||
session._registry.resolve_agent_effort.return_value = None
|
||||
session._registry.get_config.return_value = SimpleNamespace(
|
||||
server_compat={"server_type": "vllm"}, replay_reasoning_to_model=True
|
||||
)
|
||||
session.ui.note_agent_child = MagicMock()
|
||||
|
||||
seen_messages: list[list[dict]] = []
|
||||
call_count = [0]
|
||||
|
||||
def fake_create(**kwargs):
|
||||
seen_messages.append(kwargs.get("messages") or [])
|
||||
call_count[0] += 1
|
||||
resp = MagicMock()
|
||||
choice = MagicMock()
|
||||
if call_count[0] == 1:
|
||||
choice.finish_reason = "tool_calls"
|
||||
tc = MagicMock()
|
||||
tc.id = "call_0"
|
||||
tc.function.name = "read_file"
|
||||
tc.function.arguments = '{"path": "x"}'
|
||||
choice.message.tool_calls = [tc]
|
||||
choice.message.content = None
|
||||
choice.message.reasoning = None
|
||||
choice.message.reasoning_content = "scan the repo first"
|
||||
else:
|
||||
choice.finish_reason = "stop"
|
||||
choice.message.tool_calls = None
|
||||
choice.message.content = "done"
|
||||
choice.message.reasoning = None
|
||||
choice.message.reasoning_content = None
|
||||
resp.choices = [choice]
|
||||
resp.usage = MagicMock(prompt_tokens=10, completion_tokens=5)
|
||||
return resp
|
||||
|
||||
session.client.chat.completions.create = fake_create
|
||||
|
||||
def fake_prepare(tc_dict, **_kwargs):
|
||||
return {
|
||||
"call_id": tc_dict["id"],
|
||||
"func_name": "read_file",
|
||||
"needs_approval": False,
|
||||
"execute": lambda p: (p["call_id"], "contents"),
|
||||
}
|
||||
|
||||
turns = [Turn.user("x")]
|
||||
with (
|
||||
patch.object(session, "_prepare_tool", side_effect=fake_prepare),
|
||||
patch.object(session, "_resolve_capabilities", return_value=OPENAI_COMPAT_DEFAULT),
|
||||
patch.object(session, "_provider_extra_params", return_value={}),
|
||||
):
|
||||
session._run_agent(
|
||||
turns,
|
||||
tools=[{"type": "function", "function": {"name": "read_file"}}],
|
||||
label="task",
|
||||
parent_call_id="task-1",
|
||||
)
|
||||
|
||||
# The agent Turn carries the synthesized native lane, source-tagged
|
||||
# via the agent alias (alias threading through the shared helper).
|
||||
assistant_turn = turns[1]
|
||||
assert assistant_turn.native is not None
|
||||
assert assistant_turn.native.producer == "openai-compatible"
|
||||
assert assistant_turn.native.blocks == (
|
||||
{"type": "reasoning_text", "text": "scan the repo first", "source": "vllm"},
|
||||
)
|
||||
# The replay request carries the vLLM ``reasoning`` field on the
|
||||
# assistant turn; the internal ``_provider_content`` key is stripped
|
||||
# by the provider's sanitize before the wire.
|
||||
replay = seen_messages[1]
|
||||
assistant_wire = next(m for m in replay if m.get("role") == "assistant")
|
||||
assert assistant_wire.get("reasoning") == "scan the repo first"
|
||||
assert "_provider_content" not in assistant_wire
|
||||
|
||||
|
||||
class TestRunAgentDenialMessage:
|
||||
@@ -2612,6 +3206,9 @@ class TestProjectAgentSteps:
|
||||
def test_colliding_ids_paired_fifo_not_last_wins(self):
|
||||
# A local provider reuses id "call_0" across turns; FIFO pairing gives
|
||||
# each call its OWN result, not last-wins (which would show out-B twice).
|
||||
# Parented runs can no longer produce this input (_run_agent mints
|
||||
# unique ids), but the FIFO stays as honest pairing for input a mint
|
||||
# never touched — an unparented run, or turns constructed directly.
|
||||
from turnstone.core.trajectory import ToolCall, Turn
|
||||
|
||||
turns = [
|
||||
@@ -5086,6 +5683,55 @@ class TestMetacognitiveBuffers:
|
||||
assert sys_turn["_source"] == "tool_error"
|
||||
assert sys_turn["content"] == "you hit an error; check memory"
|
||||
|
||||
def test_denial_nudge_queues_on_tool_channel(self, tmp_db):
|
||||
"""A denial responds to the tool batch the user just rejected — the
|
||||
producer must queue it on the TOOL channel so it drains through
|
||||
``_collect_advisories`` alongside the denied results (the same seam
|
||||
tool_error / repeat use), not sit on the user channel until the next
|
||||
user-message seam — by which point the model has already reacted to
|
||||
the denial without the nudge.
|
||||
|
||||
Drives the REAL ``_execute_tools`` two-phase gate with real
|
||||
``_nudges_enabled`` / ``should_nudge`` gating; only the prepare
|
||||
step and the UI approval are stubbed."""
|
||||
from turnstone.core.metacognition import format_nudge
|
||||
|
||||
session = _make_session()
|
||||
# ``should_nudge`` skips the very first message — give the session
|
||||
# the natural pre-batch shape (user turn + assistant tool-call turn).
|
||||
session.messages.append(turn_from_dict({"role": "user", "content": "do the thing"}))
|
||||
session.messages.append(turn_from_dict({"role": "assistant", "content": "calling"}))
|
||||
|
||||
item = {
|
||||
"call_id": "call_1",
|
||||
"func_name": "notify",
|
||||
"needs_approval": True,
|
||||
# Must NOT run — a denied tool never executes.
|
||||
"execute": lambda p: (p["call_id"], "EXECUTED — must not happen"),
|
||||
}
|
||||
with (
|
||||
patch.object(session, "_safe_prepare_tool", return_value=item),
|
||||
patch.object(session.ui, "approve_tools", return_value=(False, "use /tmp instead")),
|
||||
patch.object(session, "_visible_memory_count", return_value=0),
|
||||
):
|
||||
tool_calls = [
|
||||
{
|
||||
"id": "call_1",
|
||||
"type": "function",
|
||||
"function": {"name": "notify", "arguments": "{}"},
|
||||
}
|
||||
]
|
||||
results, feedback = session._execute_tools(tool_calls)
|
||||
|
||||
# The denied item surfaced the operator's feedback as its result…
|
||||
assert results == [("call_1", "Denied by user: use /tmp instead")]
|
||||
assert feedback is None
|
||||
# …and the denial nudge is queued on the TOOL channel, so the same
|
||||
# batch's ``_collect_advisories`` drain delivers it; nothing defers
|
||||
# to the next user turn.
|
||||
assert session._nudge_queue.pending(channel="tool") == [("denial", format_nudge("denial"))]
|
||||
assert session._nudge_queue.pending(channel="user") == []
|
||||
|
||||
def test_queued_message_appends_system_turn_after_tool_batch(self, tmp_db):
|
||||
"""A queued message arriving during a tool batch becomes a
|
||||
first-class ``{"role": "system", "_source": "user_interjection"}``
|
||||
@@ -6984,6 +7630,73 @@ def test_utility_completion_defers_temperature_to_session():
|
||||
assert kw2["temperature"] == 0.9 # explicit override still honored
|
||||
|
||||
|
||||
def test_web_fetch_extraction_inherits_session_max_tokens_and_effort():
|
||||
"""web_fetch's extraction call must inherit the session/registry max_tokens
|
||||
and reasoning_effort rather than forcing constants. Hard-coding
|
||||
max_tokens=8192 / reasoning_effort="low" broke local-inference models whose
|
||||
registry entry advertises a tighter output limit or a reasoning config the
|
||||
forced values fought — this lane now behaves like the main turn."""
|
||||
from unittest.mock import patch
|
||||
|
||||
from turnstone.core.providers._protocol import CompletionResult
|
||||
|
||||
session = _make_session(max_tokens=512, reasoning_effort="high")
|
||||
|
||||
resp = MagicMock()
|
||||
resp.raise_for_status.return_value = None
|
||||
resp.headers = {"content-type": "text/plain"}
|
||||
resp.text = "The page body that holds the answer."
|
||||
|
||||
with (
|
||||
patch("turnstone.core.session.fetch_with_ssrf_guard", return_value=resp),
|
||||
patch.object(
|
||||
session,
|
||||
"_utility_completion",
|
||||
return_value=CompletionResult(content="Extracted answer."),
|
||||
) as uc,
|
||||
):
|
||||
call_id, answer = session._exec_web_fetch({"call_id": "c1", "url": "https://example.com/"})
|
||||
|
||||
assert call_id == "c1"
|
||||
assert answer == "Extracted answer."
|
||||
_, kw = uc.call_args
|
||||
# 512 < context_window // 4 (8192), so the tighter session value passes
|
||||
# through unclamped — inheritance, not the old hard-coded 8192.
|
||||
assert kw["max_tokens"] == 512
|
||||
assert kw["reasoning_effort"] == "high" # session value, not the old "low"
|
||||
|
||||
|
||||
def test_web_fetch_extraction_caps_max_tokens_to_window_reserve():
|
||||
"""The extraction request is capped to the ~25% window slice Phase 2
|
||||
reserves (``context_window // 4``), matching the main turn's response
|
||||
reserve — so a large operator ``max_tokens`` on a small-context local
|
||||
model can't push prompt + output past the window."""
|
||||
from unittest.mock import patch
|
||||
|
||||
from turnstone.core.providers._protocol import CompletionResult
|
||||
|
||||
# context_window=8192 -> reserve 2048; the session budget is far larger.
|
||||
session = _make_session(max_tokens=16384, context_window=8192)
|
||||
|
||||
resp = MagicMock()
|
||||
resp.raise_for_status.return_value = None
|
||||
resp.headers = {"content-type": "text/plain"}
|
||||
resp.text = "The page body that holds the answer."
|
||||
|
||||
with (
|
||||
patch("turnstone.core.session.fetch_with_ssrf_guard", return_value=resp),
|
||||
patch.object(
|
||||
session,
|
||||
"_utility_completion",
|
||||
return_value=CompletionResult(content="Extracted answer."),
|
||||
) as uc,
|
||||
):
|
||||
session._exec_web_fetch({"call_id": "c1", "url": "https://example.com/"})
|
||||
|
||||
_, kw = uc.call_args
|
||||
assert kw["max_tokens"] == 2048 # context_window // 4, not the 16384 session value
|
||||
|
||||
|
||||
def test_record_aux_usage_skips_when_usage_missing():
|
||||
"""A provider that reports no usage object must not emit a phantom
|
||||
zero-token row."""
|
||||
|
||||
@@ -37,7 +37,7 @@ async def _stub(_request: Request) -> JSONResponse:
|
||||
|
||||
def _attach() -> AttachmentHandlers:
|
||||
return AttachmentHandlers(
|
||||
upload=_stub, list=_stub, get_content=_stub, thumbnail=_stub, delete=_stub
|
||||
upload=_stub, list=_stub, get_content=_stub, thumbnail=_stub, preview=_stub, delete=_stub
|
||||
)
|
||||
|
||||
|
||||
|
||||
@@ -361,3 +361,94 @@ class TestResolveServerType:
|
||||
session._registry = BrokenRegistry()
|
||||
session._model_alias = "x"
|
||||
assert session._resolve_server_type() == ""
|
||||
|
||||
|
||||
class TestFinalizeProviderBlocks:
|
||||
"""Direct unit tests for the shared native-lane builder
|
||||
``ChatSession._finalize_provider_blocks`` — in particular the
|
||||
``had_blank_ids`` gate (a uuid back-fill reaches only the tool_calls
|
||||
mirror, so blocks that would replay the blank id must be dropped while
|
||||
the reasoning lane survives)."""
|
||||
|
||||
def test_passthrough_without_blank_ids(self) -> None:
|
||||
session = _make_session()
|
||||
blocks = [
|
||||
{"type": "thinking", "thinking": "x", "signature": "s"},
|
||||
{"type": "tool_use", "id": "toolu_1", "name": "f", "input": {}},
|
||||
]
|
||||
out = session._finalize_provider_blocks(blocks, [], has_tool_calls=True)
|
||||
assert out is blocks
|
||||
|
||||
def test_no_tool_calls_strips_orphan_client_blocks(self) -> None:
|
||||
session = _make_session()
|
||||
blocks = [
|
||||
{"type": "thinking", "thinking": "x", "signature": "s"},
|
||||
{"type": "tool_use", "id": "toolu_1", "name": "f", "input": {}},
|
||||
]
|
||||
out = session._finalize_provider_blocks(blocks, [], has_tool_calls=False)
|
||||
assert [b["type"] for b in out] == ["thinking"]
|
||||
|
||||
def test_blank_ids_drop_messages_shaped_lane_entirely(self) -> None:
|
||||
# Anthropic-shaped lane with a blank-id tool_use: only reasoning_text
|
||||
# may survive a blank-id turn, so the whole Messages-shaped lane goes
|
||||
# — on that translator a surviving native lane REPLACES the rebuilt
|
||||
# content, so a lane missing its tool_use would orphan the mirror's
|
||||
# calls.
|
||||
session = _make_session()
|
||||
blocks = [
|
||||
{"type": "thinking", "thinking": "x", "signature": "s"},
|
||||
{"type": "text", "text": "using f"},
|
||||
{"type": "tool_use", "id": "", "name": "f", "input": {}},
|
||||
]
|
||||
out = session._finalize_provider_blocks(blocks, [], has_tool_calls=True, had_blank_ids=True)
|
||||
assert out == []
|
||||
|
||||
def test_blank_ids_drop_asymmetric_thinking_lane_without_tool_blocks(self) -> None:
|
||||
# Asymmetric capture (thinking/text present, tool_use absent, mirror
|
||||
# blank-id): the rule is total — no client block needs to be present
|
||||
# for the Messages-shaped lane to be dropped on a blank-id turn.
|
||||
session = _make_session()
|
||||
blocks = [
|
||||
{"type": "thinking", "thinking": "x", "signature": "s"},
|
||||
{"type": "text", "text": "t"},
|
||||
]
|
||||
out = session._finalize_provider_blocks(blocks, [], has_tool_calls=True, had_blank_ids=True)
|
||||
assert out == []
|
||||
|
||||
def test_blank_ids_drop_responses_reasoning_items(self) -> None:
|
||||
# Responses reasoning items pair with their original sibling items;
|
||||
# on a blank-id turn the function_call siblings are rebuilt from the
|
||||
# back-filled mirror, so the reasoning items must go too.
|
||||
session = _make_session()
|
||||
blocks = [
|
||||
{"type": "reasoning", "id": "rs_1", "summary": [], "encrypted_content": "enc"},
|
||||
{"type": "function_call", "call_id": "", "name": "f", "arguments": "{}"},
|
||||
]
|
||||
out = session._finalize_provider_blocks(blocks, [], has_tool_calls=True, had_blank_ids=True)
|
||||
assert out == []
|
||||
|
||||
def test_blank_ids_keep_only_reasoning_text(self) -> None:
|
||||
# Google-shaped lane: the raw function dict (blank id) is dropped;
|
||||
# the synthesized reasoning_text block survives — it carries no id
|
||||
# and is shape-invalid on the Messages translator by design, and the
|
||||
# Google swap simply finds no function blocks and keeps the
|
||||
# sanitized mirror.
|
||||
session = _make_session()
|
||||
blocks = [
|
||||
{"id": "", "type": "function", "function": {"name": "f", "arguments": "{}"}},
|
||||
]
|
||||
out = session._finalize_provider_blocks(
|
||||
blocks, ["thinking text"], has_tool_calls=True, had_blank_ids=True
|
||||
)
|
||||
assert [b["type"] for b in out] == ["reasoning_text"]
|
||||
assert out[0]["text"] == "thinking text"
|
||||
|
||||
def test_blank_ids_without_client_blocks_keep_the_lane(self) -> None:
|
||||
# llama.cpp / older vLLM: blank tool ids AND loose reasoning text,
|
||||
# but no client tool blocks at all — nothing can desync, so the
|
||||
# synthesized reasoning lane must be kept (the over-drop case).
|
||||
session = _make_session()
|
||||
out = session._finalize_provider_blocks(
|
||||
[], ["step by step"], has_tool_calls=True, had_blank_ids=True
|
||||
)
|
||||
assert [b["type"] for b in out] == ["reasoning_text"]
|
||||
|
||||
@@ -39,6 +39,22 @@ def _make_ui(ws_id: str = "ws-1", user_id: str = "u1") -> _ConcreteUI:
|
||||
return _ConcreteUI(ws_id=ws_id, user_id=user_id)
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _per_token_flush(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""Force per-token flushes (batch window 0) for this whole file.
|
||||
|
||||
These tests pin per-emit invariants — seq advance, mid-stream
|
||||
inflight buffer state, snapshot atomicity — that predate emit-time
|
||||
token batching and remain the contract AT each flush boundary;
|
||||
window 0 makes every token its own flush, which is exactly the
|
||||
emit shape they were written against. Batching cadence itself
|
||||
(window/size coalescing, pending-batch visibility, flush-before-
|
||||
non-token ordering) is pinned in ``test_sse_token_batching.py``.
|
||||
"""
|
||||
|
||||
monkeypatch.setattr("turnstone.core.session_ui_base._TOKEN_BATCH_WINDOW_SECS", 0.0)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Listener fan-out
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
The shared worker dispatch is load-bearing for both the interactive
|
||||
``/v1/api/workstreams/{ws_id}/send`` HTTP handler and the coordinator
|
||||
``CoordinatorAdapter.send`` path. Tests cover the four invariants the
|
||||
``CoordinatorAdapter.send`` path. Tests cover the five invariants the
|
||||
module must hold:
|
||||
|
||||
* live worker → enqueue, no thread spawn
|
||||
@@ -10,10 +10,14 @@ module must hold:
|
||||
* concurrent ``send`` calls produce exactly one worker thread
|
||||
(Stage 1 bug-1 — the racy ``Thread.is_alive()`` gate stays caught)
|
||||
* ``_worker_running`` cleared in ``finally`` even on uncaught exception
|
||||
* ownership-clear wake backstop: a worker exiting with USER_DRAIN
|
||||
nudges queued on an IDLE workstream spawns the wake send that the
|
||||
IDLE fan-out (which ran on this worker's own thread) had to drop
|
||||
|
||||
Callers pass no-arg closures, so this module never touches
|
||||
``ws.session`` — keeps the contract narrow and lets watch-style
|
||||
dispatchers drive a session that isn't installed on ``ws``.
|
||||
Callers pass no-arg closures, so dispatch never touches ``ws.session``;
|
||||
the exit backstop only PEEKS it defensively (``getattr`` for
|
||||
``_nudge_queue``, bail on stubs) — watch-style dispatchers can still
|
||||
drive a session that isn't installed on ``ws``.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -22,8 +26,10 @@ import queue
|
||||
import threading
|
||||
from typing import Any
|
||||
|
||||
from tests._helpers import wait_until as _wait_until
|
||||
from turnstone.core import session_worker
|
||||
from turnstone.core.workstream import Workstream
|
||||
from turnstone.core.nudge_queue import USER_DRAIN, NudgeQueue
|
||||
from turnstone.core.workstream import Workstream, WorkstreamState
|
||||
|
||||
|
||||
class _SendSession:
|
||||
@@ -140,6 +146,41 @@ def test_enqueue_unexpected_exception_returns_false_logged() -> None:
|
||||
assert ws._worker_running is True
|
||||
|
||||
|
||||
def test_closed_workstream_refused_no_spawn() -> None:
|
||||
"""Authoritative closed-check: ``close()`` sets ``_closed`` under
|
||||
``ws._lock``, so a wake (or send) racing it must be refused HERE —
|
||||
the wake gate's lockless peek can go stale, and a spawn past this
|
||||
point would run a full unattended turn (inference, tool calls,
|
||||
storage writes) on a workstream whose ``ws_closed`` already fired.
|
||||
"""
|
||||
session = _SendSession()
|
||||
ws = _make_ws(session)
|
||||
ws._closed = True
|
||||
|
||||
ok = _send_message(ws, session, "hello")
|
||||
|
||||
assert ok is False
|
||||
assert session.send_calls == []
|
||||
assert session.queue_calls == []
|
||||
assert ws.worker_thread is None
|
||||
assert ws._worker_running is False
|
||||
|
||||
|
||||
def test_closed_workstream_refused_on_reuse_path_too() -> None:
|
||||
"""The refusal precedes the enqueue branch: no interjection is queued
|
||||
onto a session whose workstream is already closed."""
|
||||
session = _SendSession()
|
||||
ws = _make_ws(session)
|
||||
ws._worker_running = True
|
||||
ws._closed = True
|
||||
|
||||
ok = _send_message(ws, session, "hello")
|
||||
|
||||
assert ok is False
|
||||
assert session.queue_calls == []
|
||||
assert ws._worker_running is True # untouched — not ours to clear
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# _worker_running lifecycle
|
||||
# ---------------------------------------------------------------------------
|
||||
@@ -301,6 +342,141 @@ def test_thread_name_explicit_override() -> None:
|
||||
ws.worker_thread.join(timeout=2.0)
|
||||
|
||||
|
||||
class _WakeCapableSession(_SendSession):
|
||||
"""Adds the ChatSession surface the exit backstop peeks at."""
|
||||
|
||||
def __init__(self, **kwargs: Any) -> None:
|
||||
super().__init__(**kwargs)
|
||||
self._nudge_queue = NudgeQueue()
|
||||
self.deliver_calls = 0
|
||||
self.deliver_thread_names: list[str] = []
|
||||
self.delivered = threading.Event()
|
||||
|
||||
def deliver_wake_nudge_from_queue(self) -> None:
|
||||
# Mirror the real contract: the wake drains its own queue, so
|
||||
# the wake worker's OWN exit backstop sees nothing pending and
|
||||
# the chain converges instead of spawning wakes forever.
|
||||
self.deliver_calls += 1
|
||||
self.deliver_thread_names.append(threading.current_thread().name)
|
||||
self._nudge_queue.drain(USER_DRAIN)
|
||||
self.delivered.set()
|
||||
|
||||
|
||||
class TestWorkerExitWakeBackstop:
|
||||
"""A worker exiting while its (idle) workstream has USER_DRAIN
|
||||
nudges queued spawns the wake send the IDLE fan-out had to drop.
|
||||
|
||||
Production shape being modelled: ``set_state(IDLE)`` fires its
|
||||
subscribers on the worker thread from inside ``run()`` —
|
||||
``CoordinatorIdleObserver`` enqueues ``idle_children``, then
|
||||
``IdleNudgeWatcher``'s wake dispatch lands on the reuse path
|
||||
(this very worker still owns the flag) and no-ops. The enqueue
|
||||
inside ``run`` below stands in for that observer enqueue.
|
||||
"""
|
||||
|
||||
def test_worker_exit_delivers_pending_wake(self) -> None:
|
||||
session = _WakeCapableSession()
|
||||
ws = _make_ws(session)
|
||||
assert ws.state is WorkstreamState.IDLE # dataclass default
|
||||
|
||||
def run() -> None:
|
||||
# What the IDLE fan-out's observer does, on this thread.
|
||||
session._nudge_queue.enqueue("idle_children", "kids waiting", "any")
|
||||
|
||||
ok = session_worker.send(ws, enqueue=lambda: None, run=run)
|
||||
assert ok is True
|
||||
|
||||
# The wake is delivered on a fresh wake-named worker thread…
|
||||
assert session.delivered.wait(timeout=2.0), (
|
||||
"exit backstop did not deliver the pending nudge"
|
||||
)
|
||||
assert session.deliver_thread_names[0].startswith("wake-nudge-")
|
||||
# …after which the wake worker's own exit backstop sees an empty
|
||||
# queue and the chain converges: flag at rest, exactly one deliver.
|
||||
_wait_until(lambda: ws._worker_running is False)
|
||||
assert session.deliver_calls == 1
|
||||
assert len(session._nudge_queue) == 0
|
||||
|
||||
def test_worker_exit_no_wake_when_queue_empty(self) -> None:
|
||||
session = _WakeCapableSession()
|
||||
ws = _make_ws(session)
|
||||
|
||||
ok = session_worker.send(ws, enqueue=lambda: None, run=lambda: None)
|
||||
assert ok is True
|
||||
original = ws.worker_thread
|
||||
assert original is not None
|
||||
original.join(timeout=2.0)
|
||||
|
||||
assert ws.worker_thread is original # no wake spawned
|
||||
assert session.deliver_calls == 0
|
||||
assert ws._worker_running is False
|
||||
|
||||
def test_worker_exit_no_wake_for_stub_session_without_queue(self) -> None:
|
||||
"""The narrow-contract escape hatch: a session without a
|
||||
``_nudge_queue`` (watch-style stubs) is skipped by the shared
|
||||
wake gate's own defensive peek — no AttributeError, no wake."""
|
||||
session = _SendSession()
|
||||
ws = _make_ws(session)
|
||||
|
||||
ok = _send_message(ws, session, "hello")
|
||||
assert ok is True
|
||||
original = ws.worker_thread
|
||||
assert original is not None
|
||||
original.join(timeout=2.0)
|
||||
|
||||
assert ws.worker_thread is original
|
||||
assert ws._worker_running is False
|
||||
|
||||
def test_worker_exit_no_wake_when_state_not_idle(self) -> None:
|
||||
"""An ERROR exit stays parked for the operator — pending nudges
|
||||
wait for the next real interaction rather than burning
|
||||
unattended inference on a failed session."""
|
||||
session = _WakeCapableSession()
|
||||
ws = _make_ws(session)
|
||||
|
||||
def run() -> None:
|
||||
session._nudge_queue.enqueue("idle_children", "kids waiting", "any")
|
||||
ws.state = WorkstreamState.ERROR
|
||||
|
||||
ok = session_worker.send(ws, enqueue=lambda: None, run=run)
|
||||
assert ok is True
|
||||
original = ws.worker_thread
|
||||
assert original is not None
|
||||
original.join(timeout=2.0)
|
||||
|
||||
assert ws.worker_thread is original
|
||||
assert session.deliver_calls == 0
|
||||
assert len(session._nudge_queue) == 1 # still queued for later seams
|
||||
|
||||
def test_abandoned_worker_does_not_run_wake_backstop(self) -> None:
|
||||
"""Only the owner retries: an abandoned worker (successor claimed
|
||||
the flag) finishing late must not spawn a wake — the successor's
|
||||
own exit runs the backstop."""
|
||||
send_gate = threading.Event()
|
||||
session = _WakeCapableSession(send_gate=send_gate)
|
||||
ws = _make_ws(session)
|
||||
|
||||
ok = _send_message(ws, session, "hello")
|
||||
assert ok is True
|
||||
abandoned = ws.worker_thread
|
||||
assert abandoned is not None
|
||||
|
||||
session._nudge_queue.enqueue("idle_children", "kids waiting", "any")
|
||||
sentinel = threading.Thread(target=lambda: None, name="successor")
|
||||
with ws._lock:
|
||||
ws.worker_thread = sentinel
|
||||
ws._worker_running = True
|
||||
|
||||
send_gate.set()
|
||||
abandoned.join(timeout=3.0)
|
||||
assert not abandoned.is_alive()
|
||||
|
||||
# No wake spawned by the abandoned thread; ownership intact.
|
||||
assert ws.worker_thread is sentinel
|
||||
assert session.deliver_calls == 0
|
||||
assert ws._worker_running is True
|
||||
|
||||
|
||||
def test_does_not_deadlock_when_run_briefly_grabs_ws_lock() -> None:
|
||||
"""Sanity check: ``run`` is invoked OUTSIDE ``ws._lock``. A worker
|
||||
body that briefly takes the lock (e.g. to update worker state)
|
||||
|
||||
+12
-6
@@ -49,6 +49,7 @@ _ESM_BUNDLES = [
|
||||
_SHARED / "composer_queue.js",
|
||||
_SHARED / "interactive.js",
|
||||
_SHARED / "conversation.js",
|
||||
_SHARED / "preview.js",
|
||||
_SHARED / "redact_credentials.js",
|
||||
]
|
||||
|
||||
@@ -69,6 +70,7 @@ _ESM_NO_VAR_BUNDLES = [
|
||||
_SHARED / "auth.js",
|
||||
_SHARED / "interactive.js",
|
||||
_SHARED / "conversation.js",
|
||||
_SHARED / "preview.js",
|
||||
_SHARED / "redact_credentials.js",
|
||||
]
|
||||
|
||||
@@ -864,16 +866,20 @@ def test_pane_manager_split_engine() -> None:
|
||||
assert "_restoreLayout(data)" in pane and "seen.has(d.paneId)" in pane
|
||||
# the visible-but-unfocused tab marker
|
||||
assert 'classList.toggle("shown"' in pane
|
||||
# per-pane ✕: split mode hides ONE cell keeping the tab (closeCell);
|
||||
# single-pane it closes the pane (withheld from non-closable) — the click
|
||||
# decides at click time, the label tracks the mode. Manager-injected into
|
||||
# the pane SECTION (content untouched), removed via _clearCellStyle.
|
||||
# per-pane ✕: split mode hides ONE cell keeping the tab (closeCell), EXCEPT
|
||||
# an ephemeral pane which closes outright; single-pane it closes the pane
|
||||
# (withheld from non-closable) — the click decides at click time, the label
|
||||
# tracks the mode. Manager-injected into the pane SECTION (content
|
||||
# untouched), removed via _clearCellStyle. Ephemeral-dismiss behaviour has
|
||||
# its own deep coverage in test_preview_js.py::TestEphemeralDismiss.
|
||||
assert "closeCell(paneId)" in pane and "_refreshCellChips()" in pane
|
||||
assert 'b.className = "cell-unsplit"' in pane
|
||||
assert '"Close pane"' in pane, "the single-pane chip mode"
|
||||
# mode-DISTINCT glyphs (designer P1: identical signifier + locus with a
|
||||
# reversible/destructive divergence is a mode-error trap)
|
||||
assert 'b.textContent = multi ? "−" : "✕"' in pane
|
||||
# reversible/destructive divergence is a mode-error trap) — a click that
|
||||
# cannot be a reversible cell-hide shows ✕, else −.
|
||||
assert "const destroys = !multi || pane.ephemeral;" in pane
|
||||
assert 'b.textContent = destroys ? "✕" : "−"' in pane
|
||||
assert '"cell-unsplit--close"' in pane
|
||||
assert "this._removeCellChip(pane)" in pane
|
||||
# open-beside: the coordinator child-link placement (split right of the
|
||||
|
||||
@@ -0,0 +1,140 @@
|
||||
"""Static + runtime guards for the shared SSE overflow-recovery helper.
|
||||
|
||||
``turnstone/shared_static/sse_overflow.js`` is the client half of the SSE
|
||||
overflow recovery — the storm-guard threshold, the cooldown-ladder constants,
|
||||
and the two pure helpers (``overflowWindowTripped`` / ``degradedCooldownStep``)
|
||||
— extracted so BOTH the interactive pane (``shared_static/interactive.js``) and
|
||||
the coordinator pane (``console/static/coordinator/coordinator.js``) share one
|
||||
source of truth for the trip math instead of drifting copies. The panes keep
|
||||
their own transport/DOM glue; only the pure core lives here.
|
||||
|
||||
Like the rest of the WebUI the module has no JS test framework, so these are
|
||||
Python-side string-presence assertions plus two ``node`` runtime probes that
|
||||
execute the extracted pure functions — the storm-guard math is the part the
|
||||
design review marked UNCONFIRMED, so it gets run, not just string-pinned.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import re
|
||||
import subprocess
|
||||
import tempfile
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
_ROOT = Path(__file__).resolve().parent.parent
|
||||
_SSE_OVERFLOW = _ROOT / "turnstone/shared_static/sse_overflow.js"
|
||||
|
||||
|
||||
def test_module_exports_constants_and_pure_helpers() -> None:
|
||||
"""The single source of truth exports the five tuning constants and the two
|
||||
pure helpers. Both panes import these by name (pinned in their own suites),
|
||||
so a rename here is a breaking change that must surface loudly."""
|
||||
body = _SSE_OVERFLOW.read_text(encoding="utf-8")
|
||||
for const, value in (
|
||||
("OVERFLOW_TRIP_COUNT", "3"),
|
||||
("OVERFLOW_TRIP_WINDOW_MS", "60000"),
|
||||
("DEGRADED_COOLDOWN_BASE_MS", "15000"),
|
||||
("DEGRADED_COOLDOWN_MAX_MS", "120000"),
|
||||
("DEGRADED_COOLDOWN_RESET_MS", "300000"),
|
||||
):
|
||||
assert f"export const {const} = {value};" in body, f"missing export const {const}"
|
||||
assert "export function overflowWindowTripped(" in body
|
||||
assert "export function degradedCooldownStep(" in body
|
||||
|
||||
|
||||
def test_overflow_window_tripped_runtime() -> None:
|
||||
"""Runtime probe for the limiter's rolling-window helper — the storm-guard
|
||||
math is the part of Fix A the design review marked UNCONFIRMED, so it gets
|
||||
executed, not just string-pinned: prunes stale entries in place, trips at
|
||||
exactly K-in-window, and does not trip for closes spread wider than the
|
||||
window."""
|
||||
body = _SSE_OVERFLOW.read_text(encoding="utf-8")
|
||||
m = re.search(
|
||||
r"^export function overflowWindowTripped\(times, nowMs, count, windowMs\) \{.*?^\}",
|
||||
body,
|
||||
re.S | re.M,
|
||||
)
|
||||
assert m is not None, "overflowWindowTripped not found (keep it a module-level export)"
|
||||
harness = (
|
||||
m.group(0)
|
||||
+ "\n"
|
||||
+ "// trips at exactly count-in-window\n"
|
||||
+ "let t = [1000, 2000, 3000];\n"
|
||||
+ "if (!overflowWindowTripped(t, 3000, 3, 60000)) throw new Error('K-in-window must trip');\n"
|
||||
+ "// stale entries prune in place and prevent the trip\n"
|
||||
+ "t = [1000, 2000, 70000];\n"
|
||||
+ "if (overflowWindowTripped(t, 70000, 3, 60000)) throw new Error('stale entries must not trip');\n"
|
||||
+ "if (JSON.stringify(t) !== '[70000]') throw new Error('prune in place failed: ' + JSON.stringify(t));\n"
|
||||
+ "// boundary: an entry exactly windowMs old is still counted\n"
|
||||
+ "t = [10000, 70000];\n"
|
||||
+ "if (!overflowWindowTripped(t, 70000, 2, 60000)) throw new Error('boundary entry must count');\n"
|
||||
+ "// below threshold never trips\n"
|
||||
+ "t = [];\n"
|
||||
+ "if (overflowWindowTripped(t, 1, 1, 60000) !== false) throw new Error('empty must not trip');\n"
|
||||
)
|
||||
with tempfile.NamedTemporaryFile(mode="w", suffix=".mjs", delete=False) as f:
|
||||
f.write(harness)
|
||||
tmp = f.name
|
||||
try:
|
||||
proc = subprocess.run(["node", tmp], capture_output=True, text=True, timeout=15)
|
||||
except FileNotFoundError:
|
||||
pytest.skip("node binary not available on PATH")
|
||||
finally:
|
||||
os.unlink(tmp)
|
||||
assert proc.returncode == 0, (
|
||||
f"overflowWindowTripped runtime probe failed. stdout={proc.stdout!r} stderr={proc.stderr!r}"
|
||||
)
|
||||
|
||||
|
||||
def test_degraded_cooldown_ladder_escalates_and_resets_runtime() -> None:
|
||||
"""Review finding [0] regression: the degraded-catchup cooldown ladder must
|
||||
actually ESCALATE across consecutive trips (15→30→60→120s, capped) and reset
|
||||
to base only after a genuine quiet gap. The original bug cleared the
|
||||
overflow-window array in the trip handler, so the empty-window check reset
|
||||
the cooldown to base on every storm's first overflow and the doubling never
|
||||
took effect. The fix keys the ladder off a last-trip timestamp via the pure
|
||||
degradedCooldownStep helper, exercised here directly."""
|
||||
body = _SSE_OVERFLOW.read_text(encoding="utf-8")
|
||||
m = re.search(
|
||||
r"^export function degradedCooldownStep\(.*?\) \{.*?^\}",
|
||||
body,
|
||||
re.S | re.M,
|
||||
)
|
||||
assert m is not None, "degradedCooldownStep not found (keep it a module-level export)"
|
||||
harness = (
|
||||
m.group(0)
|
||||
+ "\n"
|
||||
+ "const BASE=15000, MAX=120000, RESET=300000;\n"
|
||||
+ "function assert(c,msg){ if(!c) throw new Error(msg); }\n"
|
||||
+ "// First trip: gap since lastTrip(0) exceeds RESET -> base, next doubles.\n"
|
||||
+ "let s = degradedCooldownStep(BASE, 0, 1000000, BASE, MAX, RESET);\n"
|
||||
+ "assert(s.cooldown===15000, 'first trip cooldown '+s.cooldown);\n"
|
||||
+ "assert(s.nextCooldownMs===30000, 'first next '+s.nextCooldownMs);\n"
|
||||
+ "// Second trip recurs within RESET -> escalates (uses the doubled prev).\n"
|
||||
+ "s = degradedCooldownStep(30000, 1000000, 1030000, BASE, MAX, RESET);\n"
|
||||
+ "assert(s.cooldown===30000, 'second trip must ESCALATE not reset, got '+s.cooldown);\n"
|
||||
+ "assert(s.nextCooldownMs===60000, 'second next '+s.nextCooldownMs);\n"
|
||||
+ "// Third + fourth keep escalating and cap at MAX.\n"
|
||||
+ "s = degradedCooldownStep(60000, 1030000, 1060000, BASE, MAX, RESET);\n"
|
||||
+ "assert(s.cooldown===60000 && s.nextCooldownMs===120000, 'third '+JSON.stringify(s));\n"
|
||||
+ "s = degradedCooldownStep(120000, 1060000, 1090000, BASE, MAX, RESET);\n"
|
||||
+ "assert(s.cooldown===120000 && s.nextCooldownMs===120000, 'fourth must cap at MAX '+JSON.stringify(s));\n"
|
||||
+ "// A quiet gap longer than RESET resets the ladder to base.\n"
|
||||
+ "s = degradedCooldownStep(120000, 1090000, 1090000+RESET+1, BASE, MAX, RESET);\n"
|
||||
+ "assert(s.cooldown===15000, 'quiet gap must reset to base, got '+s.cooldown);\n"
|
||||
)
|
||||
with tempfile.NamedTemporaryFile(mode="w", suffix=".mjs", delete=False) as f:
|
||||
f.write(harness)
|
||||
tmp = f.name
|
||||
try:
|
||||
proc = subprocess.run(["node", tmp], capture_output=True, text=True, timeout=15)
|
||||
except FileNotFoundError:
|
||||
pytest.skip("node binary not available on PATH")
|
||||
finally:
|
||||
os.unlink(tmp)
|
||||
assert proc.returncode == 0, (
|
||||
f"degradedCooldownStep escalation probe failed. stderr={proc.stderr!r}"
|
||||
)
|
||||
@@ -20,6 +20,7 @@ The browser-side guard for the ``onerror`` close pattern lives in
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import queue
|
||||
import threading
|
||||
from types import SimpleNamespace as SimpleNS
|
||||
from typing import Any
|
||||
@@ -199,17 +200,17 @@ def test_event_id_monotonic_under_concurrent_writers() -> None:
|
||||
|
||||
|
||||
def test_event_id_does_not_skip_when_listener_queue_full() -> None:
|
||||
"""If a slow listener's queue is full, the per-listener
|
||||
``put_nowait`` is silently dropped — but the counter must NOT
|
||||
skip. A subsequently-registered listener with
|
||||
``Last-Event-ID=0`` must see ALL the ids from the buffer
|
||||
(1..N), not a sparse subset. Pre-bug-class: moving the
|
||||
id-increment inside the per-listener loop would create phantom
|
||||
"gaps" the truncation detector would misread."""
|
||||
"""If a slow listener's queue is full, the per-listener put is
|
||||
rejected (the first rejection poisons the listener; later ones are
|
||||
latch refusals) — but the counter must NOT skip. A subsequently-
|
||||
registered listener with ``Last-Event-ID=0`` must see ALL the ids
|
||||
from the buffer (1..N), not a sparse subset. Pre-bug-class:
|
||||
moving the id-increment inside the per-listener loop would create
|
||||
phantom "gaps" the truncation detector would misread."""
|
||||
ui = _make_ui()
|
||||
slow_lq = ui._register_listener(maxsize=1)
|
||||
slow_lq.put_nowait({"placeholder": True}) # full immediately
|
||||
# Fire 10 events — 9 will hit queue.Full and be suppressed.
|
||||
# Fire 10 events — none can land in the full/poisoned queue.
|
||||
for i in range(10):
|
||||
ui._enqueue({"type": "tool_started", "name": f"t{i}"})
|
||||
# Replay from id=0 — fresh listener gets all 10, ids 1..10 dense.
|
||||
@@ -261,12 +262,17 @@ def test_cross_thread_writer_and_replay_observer_consistent() -> None:
|
||||
)
|
||||
|
||||
|
||||
def test_event_id_persists_across_turn_boundaries() -> None:
|
||||
def test_event_id_persists_across_turn_boundaries(monkeypatch: Any) -> None:
|
||||
"""Resetting ``_event_id`` to 0 at turn boundaries would silently
|
||||
mis-replay a long-lived SSE subscriber whose ``Last-Event-ID``
|
||||
was from a prior turn. Mirrors the pre-existing
|
||||
``test_inflight_seq_monotonic_across_turn_boundaries`` invariant
|
||||
on the snap_seq side, extended to the buffer/replay side."""
|
||||
on the snap_seq side, extended to the buffer/replay side.
|
||||
|
||||
Batch window forced to 0 (per-token flush) — this test pins id
|
||||
numbering across turn boundaries, not the batching cadence."""
|
||||
|
||||
monkeypatch.setattr("turnstone.core.session_ui_base._TOKEN_BATCH_WINDOW_SECS", 0.0)
|
||||
ui = _make_ui()
|
||||
ui.on_content_token("turn-N tok1 ")
|
||||
ui.on_content_token("turn-N tok2 ")
|
||||
@@ -305,7 +311,7 @@ def test_replay_ok_skips_in_progress_snapshot_path() -> None:
|
||||
assert snap["seq"] >= 1
|
||||
|
||||
|
||||
def test_truncated_path_snapshot_captures_real_snap_seq() -> None:
|
||||
def test_truncated_path_snapshot_captures_real_snap_seq(monkeypatch: Any) -> None:
|
||||
"""Regression for PR #542 review comment 1 (Copilot, low-confidence).
|
||||
|
||||
On the truncated path the caller used to set ``snap_seq=0``, which
|
||||
@@ -321,9 +327,13 @@ def test_truncated_path_snapshot_captures_real_snap_seq() -> None:
|
||||
``register_listener_with_replay`` under the same nested-lock
|
||||
acquire as the listener registration + buffer slice + counter
|
||||
read, so ``snap_seq`` returned in the snapshot is the exact
|
||||
high-water mark the snapshot text corresponds to."""
|
||||
high-water mark the snapshot text corresponds to.
|
||||
|
||||
Batch window forced to 0 so each token is its own ring entry —
|
||||
the truncation scenario needs 10 distinct buffered events."""
|
||||
import collections
|
||||
|
||||
monkeypatch.setattr("turnstone.core.session_ui_base._TOKEN_BATCH_WINDOW_SECS", 0.0)
|
||||
ui = _make_ui()
|
||||
ui._event_buffer = collections.deque(maxlen=3)
|
||||
# Fire enough events to trigger truncation on reconnect with a
|
||||
@@ -362,15 +372,15 @@ def test_snap_seq_high_water_mark_holds_under_writer_race() -> None:
|
||||
double-render.
|
||||
|
||||
The race window in plain Python is narrow (a few bytecodes
|
||||
between lock release and the ``_enqueue`` call), so a pure
|
||||
barrier-based race rarely hits it. This test injects a
|
||||
deterministic sleep into ``_enqueue`` via monkey-patch to
|
||||
widen the window enough to be reliably observed under the
|
||||
pre-fix code path — AND to be reliably AVOIDED under the
|
||||
post-fix code path (because the post-fix
|
||||
``on_content_token`` calls ``_enqueue`` while still holding
|
||||
``_ws_lock``, so the snapshot reader can't acquire
|
||||
``_ws_lock`` until the writer is fully done).
|
||||
between lock release and the emit call), so a pure barrier-based
|
||||
race rarely hits it. This test injects a deterministic sleep
|
||||
into ``_enqueue_direct`` — the inner emit point the token
|
||||
batcher's flush calls under ``_ws_lock`` — to widen the window
|
||||
enough to be reliably observed if the flush's inflight-append
|
||||
and enqueue are ever split across ``_ws_lock`` sections, AND to
|
||||
be reliably AVOIDED under the correct code path (the flush holds
|
||||
``_ws_lock`` across both, so the snapshot reader can't acquire
|
||||
it until the writer is fully done).
|
||||
"""
|
||||
import queue
|
||||
import threading
|
||||
@@ -378,20 +388,20 @@ def test_snap_seq_high_water_mark_holds_under_writer_race() -> None:
|
||||
|
||||
ui = _make_ui()
|
||||
marker = "RACE-MARKER"
|
||||
original_enqueue = ui._enqueue
|
||||
original_direct = ui._enqueue_direct
|
||||
|
||||
# Widen the race window: sleep just BEFORE the original
|
||||
# ``_enqueue`` runs (which is where ``_event_id`` would advance).
|
||||
# Post-fix this sleep happens while the writer still holds
|
||||
# ``_ws_lock`` — readers block. Pre-fix the writer has
|
||||
# released ``_ws_lock`` before reaching this monkey-patch, so
|
||||
# the reader gets a clean window to capture an inconsistent
|
||||
# ``(inflight, _event_id)`` pair.
|
||||
def slow_enqueue(data: dict[str, Any]) -> None:
|
||||
# Widen the race window: sleep just BEFORE the inner emit runs
|
||||
# (which is where ``_event_id`` advances). Under the correct
|
||||
# locking this sleep happens while the writer still holds
|
||||
# ``_ws_lock`` — readers block. If the flush ever releases
|
||||
# ``_ws_lock`` before its enqueue, the reader gets a clean
|
||||
# window to capture an inconsistent ``(inflight, _event_id)``
|
||||
# pair and the invariant below trips.
|
||||
def slow_direct(data: dict[str, Any]) -> int:
|
||||
time.sleep(0.05) # 50 ms — orders of magnitude wider than the GIL switch interval
|
||||
return original_enqueue(data)
|
||||
return original_direct(data)
|
||||
|
||||
ui._enqueue = slow_enqueue # type: ignore[method-assign]
|
||||
ui._enqueue_direct = slow_direct # type: ignore[method-assign]
|
||||
|
||||
snap_box: dict[str, Any] = {}
|
||||
writer_done = threading.Event()
|
||||
@@ -572,10 +582,15 @@ def test_handler_emits_retry_on_first_yield() -> None:
|
||||
assert 2500 <= retry <= 4500, f"retry {retry} outside jitter band [2500, 4500]"
|
||||
|
||||
|
||||
def test_handler_replay_ok_skips_snapshot_emits_id() -> None:
|
||||
def test_handler_replay_ok_skips_snapshot_emits_id(monkeypatch: Any) -> None:
|
||||
"""``Last-Event-ID`` + buffer covers gap → emit buffered events
|
||||
with SSE ``id:`` field, SKIP the in-progress snapshot (it would
|
||||
double-render content the buffered events already carry)."""
|
||||
double-render content the buffered events already carry).
|
||||
|
||||
Batch window forced to 0 so the two tokens are two ring entries
|
||||
(the assertion wants two distinct ``id:`` lines)."""
|
||||
|
||||
monkeypatch.setattr("turnstone.core.session_ui_base._TOKEN_BATCH_WINDOW_SECS", 0.0)
|
||||
ui = _make_ui()
|
||||
ui.on_content_token("hello ")
|
||||
ui.on_content_token("world")
|
||||
@@ -590,13 +605,17 @@ def test_handler_replay_ok_skips_snapshot_emits_id() -> None:
|
||||
assert "id: 2" in blob, f"missing id: 2 in:\n{blob}"
|
||||
|
||||
|
||||
def test_handler_truncated_emits_envelope_then_snapshot() -> None:
|
||||
def test_handler_truncated_emits_envelope_then_snapshot(monkeypatch: Any) -> None:
|
||||
"""Stale ``Last-Event-ID`` + buffer too short → emit
|
||||
``replay_truncated`` envelope, THEN fall through to the
|
||||
fresh-style replay (state_change + in_progress_snapshot) as the
|
||||
recovery floor."""
|
||||
recovery floor.
|
||||
|
||||
Batch window forced to 0 so each token is its own ring entry —
|
||||
the truncation scenario needs the deque to evict."""
|
||||
import collections
|
||||
|
||||
monkeypatch.setattr("turnstone.core.session_ui_base._TOKEN_BATCH_WINDOW_SECS", 0.0)
|
||||
ui = _make_ui()
|
||||
ui._event_buffer = collections.deque(maxlen=3)
|
||||
for i in range(10):
|
||||
@@ -717,3 +736,382 @@ def test_handler_replay_ok_does_not_resurface_last_error(monkeypatch: Any) -> No
|
||||
ui.on_content_token("hi") # one buffered event so Last-Event-ID=0 → replay_ok
|
||||
_, blob = _drain_handler_yields(ui, headers={"Last-Event-ID": "0"}, state="error", max_yields=8)
|
||||
assert "boom" not in blob
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Poison-on-overflow (Fix A) — queue.Full stops being a silent drop
|
||||
# ---------------------------------------------------------------------------
|
||||
#
|
||||
# Silent per-listener drops at queue.Full left permanent holes BELOW the
|
||||
# client's advancing lastEventId (scattered interleaved drops once the
|
||||
# queue saturates), which reconnect-with-replay can never heal (the slice
|
||||
# is ``eid > last_event_id`` only). The listener queue now latches
|
||||
# ``poisoned`` atomically at the FIRST rejected put and refuses every
|
||||
# later put, freezing its contents as a contiguous prefix; the drain
|
||||
# loop closes the stream (after an id-less ``stream_overflow`` frame)
|
||||
# and the native EventSource reconnect replays the contiguous tail.
|
||||
#
|
||||
# Poisoning at the first full (not after N) is load-bearing: any
|
||||
# delivered-while-dropping window advances lastEventId past interior
|
||||
# holes -> permanent gap even after a "successful" reconnect.
|
||||
|
||||
|
||||
def _fake_live_request(*, path_params: dict[str, str] | None = None) -> Request:
|
||||
"""A request whose ``receive()`` never resolves, so
|
||||
``is_disconnected()`` stays ``False`` — the poison check, not
|
||||
disconnect detection, must be what terminates the drain loop."""
|
||||
scope = {
|
||||
"type": "http",
|
||||
"method": "GET",
|
||||
"headers": [],
|
||||
"path": "/events",
|
||||
"raw_path": b"/events",
|
||||
"query_string": b"",
|
||||
"path_params": path_params or {},
|
||||
"app": MagicMock(),
|
||||
}
|
||||
|
||||
async def _recv() -> dict[str, Any]:
|
||||
await asyncio.Event().wait() # pends forever
|
||||
return {"type": "http.disconnect"} # unreachable
|
||||
|
||||
return Request(scope, receive=_recv)
|
||||
|
||||
|
||||
def test_listener_queue_poisons_at_first_full_and_refuses_after() -> None:
|
||||
"""The first rejected put latches ``poisoned`` (atomically, under
|
||||
the queue's own mutex) and every later put is refused even if the
|
||||
consumer frees slots — otherwise a racing consumer pop would let a
|
||||
later event land BEHIND the hole and the drain would deliver past
|
||||
it, advancing lastEventId beyond an unreplayable gap."""
|
||||
ui = _make_ui()
|
||||
lq = ui._register_listener(maxsize=2)
|
||||
ui._enqueue({"type": "a"})
|
||||
ui._enqueue({"type": "b"})
|
||||
assert getattr(lq, "poisoned", None) is False
|
||||
ui._enqueue({"type": "c"}) # first overflow -> latch
|
||||
assert lq.poisoned is True
|
||||
lq.get_nowait() # consumer frees a slot
|
||||
ui._enqueue({"type": "d"}) # must be refused — queue contents frozen
|
||||
leftover = []
|
||||
while True:
|
||||
try:
|
||||
leftover.append(lq.get_nowait())
|
||||
except queue.Empty:
|
||||
break
|
||||
assert [ev["type"] for ev in leftover] == ["b"], (
|
||||
"a post-poison put landed in the freed slot — interior hole"
|
||||
)
|
||||
# The ring is untouched by listener poisoning: ids stay dense.
|
||||
_, replay, status, _, _, _ = ui.register_listener_with_replay(0)
|
||||
assert status == "replay_ok"
|
||||
assert [ev["_event_id"] for ev in replay] == [1, 2, 3, 4]
|
||||
|
||||
|
||||
def test_poisoned_gap_is_contiguous_tail_fully_replayable() -> None:
|
||||
"""Recovery math at the poison instant: delivered ids form a
|
||||
contiguous prefix, the ring holds everything, and a reconnect with
|
||||
``Last-Event-ID = <last delivered>`` replays exactly the missing
|
||||
tail — no duplicate, no loss, no off-by-one."""
|
||||
ui = _make_ui()
|
||||
lq = ui._register_listener(maxsize=3)
|
||||
for i in range(5):
|
||||
ui._enqueue({"type": "tool_started", "name": f"t{i}"})
|
||||
# Queue froze at [1,2,3]; 4 latched poison; 5 was refused.
|
||||
delivered = []
|
||||
while True:
|
||||
try:
|
||||
delivered.append(lq.get_nowait()["_event_id"])
|
||||
except queue.Empty:
|
||||
break
|
||||
assert delivered == [1, 2, 3]
|
||||
_, replay, status, lost, _, _ = ui.register_listener_with_replay(delivered[-1])
|
||||
assert status == "replay_ok"
|
||||
assert lost == 0
|
||||
assert [ev["_event_id"] for ev in replay] == [4, 5]
|
||||
assert delivered + [ev["_event_id"] for ev in replay] == [1, 2, 3, 4, 5]
|
||||
|
||||
|
||||
def test_poison_isolated_to_slow_listener() -> None:
|
||||
"""One slow tab must not degrade its siblings: the healthy listener
|
||||
keeps receiving every event after the slow one is poisoned (and the
|
||||
poisoned one stops consuming fan-out puts entirely)."""
|
||||
ui = _make_ui()
|
||||
slow = ui._register_listener(maxsize=1)
|
||||
healthy = ui._register_listener(maxsize=100)
|
||||
for i in range(6):
|
||||
ui._enqueue({"type": "tool_started", "name": f"t{i}"})
|
||||
assert slow.poisoned is True
|
||||
got = []
|
||||
while True:
|
||||
try:
|
||||
got.append(healthy.get_nowait()["name"])
|
||||
except queue.Empty:
|
||||
break
|
||||
assert got == [f"t{i}" for i in range(6)]
|
||||
|
||||
|
||||
def test_drain_loop_closes_with_overflow_frame_on_poison() -> None:
|
||||
"""Once its queue is poisoned the drain loop must terminate the SSE
|
||||
response — discarding the queued backlog (the replay covers it) —
|
||||
after yielding a final id-less ``stream_overflow`` frame so the
|
||||
client can count overflow closes (reconnect-limiter + the
|
||||
drop-vs-render-wedge field instrumentation) without advancing
|
||||
``lastEventId`` past the gap."""
|
||||
from turnstone.core.session_ui_base import _DEFAULT_LISTENER_QUEUE_MAX
|
||||
|
||||
ui = _make_ui()
|
||||
handler = _wire_events_handler(ui)
|
||||
req = _fake_live_request(path_params={"ws_id": ui.ws_id})
|
||||
|
||||
async def _run() -> list[Any]:
|
||||
resp = await handler(req)
|
||||
agen = resp.body_iterator
|
||||
yields = [await agen.__anext__()] # retry frame
|
||||
yields.append(await agen.__anext__()) # synthetic state_change
|
||||
# Overflow the registered listener's queue: cap fills, +1 poisons.
|
||||
for i in range(_DEFAULT_LISTENER_QUEUE_MAX + 1):
|
||||
ui._enqueue({"type": "info", "message": f"m{i}"})
|
||||
yields.append(await agen.__anext__()) # overflow frame, then close
|
||||
try:
|
||||
extra = await agen.__anext__()
|
||||
except StopAsyncIteration:
|
||||
extra = None
|
||||
yields.append(extra)
|
||||
return yields
|
||||
|
||||
yields = asyncio.run(_run())
|
||||
assert yields[-1] is None, "drain loop kept yielding after poison"
|
||||
overflow = yields[-2]
|
||||
assert isinstance(overflow, dict)
|
||||
assert "stream_overflow" in overflow["data"]
|
||||
assert "id" not in overflow, (
|
||||
"the overflow frame must not carry an SSE id — advancing "
|
||||
"lastEventId here would strand the dropped gap below the cursor"
|
||||
)
|
||||
# The 500-event backlog was discarded, not delivered: nothing
|
||||
# between the synthetic replay and the overflow frame.
|
||||
assert all("m0" not in str(y) for y in yields)
|
||||
|
||||
|
||||
def test_drain_loop_delivers_until_poison_then_stops_before_backlog() -> None:
|
||||
"""Pre-poison delivery works normally; at poison the loop closes
|
||||
BEFORE delivering the queued backlog (check precedes the blocking
|
||||
get), so the client's lastEventId freezes at the contiguous prefix
|
||||
and reconnect replays everything else."""
|
||||
from turnstone.core.session_ui_base import _DEFAULT_LISTENER_QUEUE_MAX
|
||||
|
||||
ui = _make_ui()
|
||||
handler = _wire_events_handler(ui)
|
||||
req = _fake_live_request(path_params={"ws_id": ui.ws_id})
|
||||
|
||||
async def _run() -> tuple[list[Any], Any, Any]:
|
||||
resp = await handler(req)
|
||||
agen = resp.body_iterator
|
||||
head = [await agen.__anext__(), await agen.__anext__()] # retry + state
|
||||
ui._enqueue({"type": "info", "message": "live-1"})
|
||||
live = await agen.__anext__()
|
||||
for i in range(_DEFAULT_LISTENER_QUEUE_MAX + 1):
|
||||
ui._enqueue({"type": "info", "message": f"m{i}"})
|
||||
tail = await agen.__anext__()
|
||||
try:
|
||||
await agen.__anext__()
|
||||
closed = False
|
||||
except StopAsyncIteration:
|
||||
closed = True
|
||||
return head, live, (tail, closed)
|
||||
|
||||
_, live, (tail, closed) = asyncio.run(_run())
|
||||
assert "live-1" in live["data"]
|
||||
assert "stream_overflow" in tail["data"]
|
||||
assert closed, "generator must return right after the overflow frame"
|
||||
|
||||
|
||||
def test_overflow_reconnect_replays_full_gap_through_handler() -> None:
|
||||
"""End-to-end recovery shape: after an overflow close, a reconnect
|
||||
carrying the pre-poison ``Last-Event-ID`` replays the whole gap via
|
||||
``replay_ok`` — the poisoned stream lost nothing durable."""
|
||||
ui = _make_ui()
|
||||
lq = ui._register_listener(maxsize=3)
|
||||
for i in range(5):
|
||||
ui._enqueue({"type": "tool_started", "name": f"t{i}"})
|
||||
delivered_ids = []
|
||||
while True:
|
||||
try:
|
||||
delivered_ids.append(lq.get_nowait()["_event_id"])
|
||||
except queue.Empty:
|
||||
break
|
||||
ui._unregister_listener(lq) # what the drain loop's finally does
|
||||
_, blob = _drain_handler_yields(
|
||||
ui, headers={"Last-Event-ID": str(delivered_ids[-1])}, max_yields=6
|
||||
)
|
||||
assert "replay_truncated" not in blob
|
||||
assert "t3" in blob
|
||||
assert "t4" in blob
|
||||
|
||||
|
||||
def test_listener_queue_basic_put_get_semantics() -> None:
|
||||
"""Stdlib-drift canary for ``_ListenerQueue.put_nowait``'s
|
||||
reimplementation against ``queue.Queue``'s documented extension
|
||||
surface (``mutex`` / ``_qsize`` / ``_put`` / ``unfinished_tasks`` /
|
||||
``not_empty``): normal put/get round-trips work, FIFO order holds,
|
||||
a blocked ``get(timeout=...)`` is woken by a put (the
|
||||
``not_empty.notify`` path the drain loop's executor get relies on),
|
||||
and the poison latch engages exactly at the first rejected put."""
|
||||
from turnstone.core.session_ui_base import _ListenerQueue
|
||||
|
||||
q = _ListenerQueue(maxsize=2)
|
||||
q.put_nowait({"n": 1})
|
||||
q.put_nowait({"n": 2})
|
||||
assert q.qsize() == 2
|
||||
try:
|
||||
q.put_nowait({"n": 3})
|
||||
raise AssertionError("third put must raise queue.Full")
|
||||
except queue.Full:
|
||||
pass
|
||||
assert q.poisoned is True
|
||||
assert q.get_nowait()["n"] == 1 # FIFO preserved
|
||||
try:
|
||||
q.put_nowait({"n": 4})
|
||||
raise AssertionError("post-poison put must be refused")
|
||||
except queue.Full:
|
||||
pass
|
||||
assert q.get_nowait()["n"] == 2
|
||||
|
||||
# A blocked get() must be woken by a concurrent put_nowait — the
|
||||
# notify path the events handler's executor get depends on.
|
||||
fresh = _ListenerQueue(maxsize=2)
|
||||
got: list[dict[str, Any]] = []
|
||||
|
||||
def _getter() -> None:
|
||||
got.append(fresh.get(timeout=5))
|
||||
|
||||
t = threading.Thread(target=_getter)
|
||||
t.start()
|
||||
fresh.put_nowait({"n": 42})
|
||||
t.join(timeout=5)
|
||||
assert not t.is_alive(), "get(timeout) never woke — not_empty.notify broken"
|
||||
assert got == [{"n": 42}]
|
||||
|
||||
|
||||
def test_closing_queue_unwinds_clean_not_overflow_when_poisoned() -> None:
|
||||
"""Review finding [1]: a ws closing/evicting while a slow pane's
|
||||
queue is full must unwind as a CLEAN close, not a false
|
||||
``stream_overflow``. The poison latch rejects the in-band
|
||||
``ws_closed`` sentinel, so ``mark_closing`` carries the signal
|
||||
out-of-band and the drain loop honours it BEFORE the poison check —
|
||||
otherwise a clean close of a slow consumer is mis-reported as a
|
||||
send-overflow (polluting the client's drop-vs-wedge counter and
|
||||
tripping its reconnect limiter on a ws that is simply gone)."""
|
||||
ui = _make_ui()
|
||||
handler = _wire_events_handler(ui)
|
||||
req = _fake_live_request(path_params={"ws_id": ui.ws_id})
|
||||
|
||||
async def _run() -> tuple[Any, bool]:
|
||||
resp = await handler(req)
|
||||
agen = resp.body_iterator
|
||||
await agen.__anext__() # retry frame
|
||||
await agen.__anext__() # synthetic state_change
|
||||
# Overflow the listener queue so it poisons, exactly as a slow
|
||||
# consumer would, THEN close the ws (evict/delete/close path).
|
||||
from turnstone.core.session_ui_base import _DEFAULT_LISTENER_QUEUE_MAX
|
||||
|
||||
for i in range(_DEFAULT_LISTENER_QUEUE_MAX + 1):
|
||||
ui._enqueue({"type": "info", "message": f"m{i}"})
|
||||
assert ui._listeners, "listener should still be registered pre-close"
|
||||
lq = ui._listeners[0]
|
||||
assert lq.poisoned is True
|
||||
# Simulate _broadcast_ws_closed_to_listeners' out-of-band flag.
|
||||
lq.mark_closing()
|
||||
try:
|
||||
frame = await agen.__anext__()
|
||||
closed = False
|
||||
except StopAsyncIteration:
|
||||
frame = None
|
||||
closed = True
|
||||
return frame, closed
|
||||
|
||||
frame, closed = asyncio.run(_run())
|
||||
assert closed, "closing queue must end the stream"
|
||||
assert frame is None, f"closing ws must NOT emit a stream_overflow frame; got {frame!r}"
|
||||
|
||||
|
||||
def test_broadcast_ws_closed_marks_closing_on_poisoned_queue() -> None:
|
||||
"""The teardown broadcaster must set the out-of-band ``closing``
|
||||
flag even when the queue is poisoned/full (its in-band ``ws_closed``
|
||||
put is refused by the poison latch). Pins the wiring finding [1]
|
||||
depends on: ``mark_closing`` is called for every listener."""
|
||||
from turnstone.core.adapters._ui_cleanup import _broadcast_ws_closed_to_listeners
|
||||
|
||||
ui = _make_ui()
|
||||
lq = ui._register_listener(maxsize=2)
|
||||
ui._enqueue({"type": "a"})
|
||||
ui._enqueue({"type": "b"})
|
||||
ui._enqueue({"type": "c"}) # overflow -> poison
|
||||
assert lq.poisoned is True
|
||||
assert lq.closing is False
|
||||
_broadcast_ws_closed_to_listeners(ui)
|
||||
assert lq.closing is True, "teardown must flag the poisoned queue closing"
|
||||
# Broadcaster clears the listener list (no re-fire on a closed ws).
|
||||
assert ui._listeners == []
|
||||
|
||||
|
||||
def test_healthy_queue_close_still_delivers_ws_closed_sentinel() -> None:
|
||||
"""The out-of-band flag must not regress the normal path: a
|
||||
non-full queue still receives the in-band ``ws_closed`` sentinel
|
||||
(so a drain loop blocked in ``get`` wakes immediately) AND gets the
|
||||
``closing`` flag."""
|
||||
from turnstone.core.adapters._ui_cleanup import _broadcast_ws_closed_to_listeners
|
||||
|
||||
ui = _make_ui()
|
||||
lq = ui._register_listener(maxsize=100)
|
||||
_broadcast_ws_closed_to_listeners(ui)
|
||||
assert lq.closing is True
|
||||
drained = []
|
||||
while True:
|
||||
try:
|
||||
drained.append(lq.get_nowait())
|
||||
except queue.Empty:
|
||||
break
|
||||
assert {ev["type"] for ev in drained} == {"ws_closed"}
|
||||
|
||||
|
||||
def test_healthy_closing_queue_drains_tail_before_close() -> None:
|
||||
"""Review round-2 finding [0]: a healthy (non-poisoned) client that is
|
||||
momentarily behind must still receive its queued tail — the turn's
|
||||
final content batch + ``stream_end`` — at ws teardown. A close has no
|
||||
reconnect+replay, so dropping that tail truncates the last assistant
|
||||
message permanently. The ``closing`` flag must therefore NOT
|
||||
short-circuit the FIFO drain for a healthy queue (an earlier revision
|
||||
checked it at the top of the loop and did exactly that); the in-band
|
||||
``ws_closed`` sentinel — which fits, the queue isn't full — closes the
|
||||
stream AFTER the drain delivers everything."""
|
||||
from turnstone.core.adapters._ui_cleanup import _broadcast_ws_closed_to_listeners
|
||||
|
||||
ui = _make_ui()
|
||||
handler = _wire_events_handler(ui)
|
||||
req = _fake_live_request(path_params={"ws_id": ui.ws_id})
|
||||
|
||||
async def _run() -> list[str]:
|
||||
resp = await handler(req)
|
||||
agen = resp.body_iterator
|
||||
await agen.__anext__() # retry frame
|
||||
await agen.__anext__() # synthetic state_change
|
||||
# Enqueue the turn's tail into a HEALTHY (roomy) queue, then close
|
||||
# the ws while those events are still undrained.
|
||||
ui.on_content_token("final answer")
|
||||
ui.on_stream_end()
|
||||
_broadcast_ws_closed_to_listeners(ui) # mark_closing + ws_closed sentinel
|
||||
out: list[str] = []
|
||||
while True:
|
||||
try:
|
||||
frame = await agen.__anext__()
|
||||
except StopAsyncIteration:
|
||||
break
|
||||
out.append(frame["data"] if isinstance(frame, dict) else str(frame))
|
||||
return out
|
||||
|
||||
blob = "\n".join(asyncio.run(_run()))
|
||||
assert "final answer" in blob, "healthy closing queue dropped its content tail"
|
||||
assert "stream_end" in blob, "healthy closing queue dropped stream_end"
|
||||
assert "stream_overflow" not in blob, "a healthy close must not emit an overflow frame"
|
||||
|
||||
@@ -0,0 +1,391 @@
|
||||
"""Emit-time micro-batching of content / reasoning tokens (SSE Fix B).
|
||||
|
||||
At local-inference rates (500+ tok/s) the per-delta ``_enqueue`` was the
|
||||
load that overflowed listener queues (silent drops -> corrupted panes).
|
||||
:meth:`SessionUIBase.on_content_token` / :meth:`on_reasoning_token` now
|
||||
coalesce fragments over a small window and enqueue ONE event per batch.
|
||||
|
||||
The two conditions that make batching safe are pinned here because each
|
||||
was a verified corruption mode in the design review:
|
||||
|
||||
- **Condition 1 — atomic flush.** The pending accumulator is invisible
|
||||
to snapshot readers; the flush appends to the inflight buffers AND
|
||||
enqueues the batched event inside one ``_ws_lock`` section, so
|
||||
``snap_seq`` stays a true high-water mark for the snapshot text. If
|
||||
inflight were appended per-token while enqueueing per-batch, a
|
||||
snapshot straddling the batch would double-render (the batch arrives
|
||||
with ``_seq > snap_seq`` carrying already-snapshotted text; the client
|
||||
has no content dedup — its ``content`` case is a blind ``+=``).
|
||||
- **Condition 2 — every non-token emit flushes first.** ``stream_end``
|
||||
/ ``tool_*`` / ``state_change`` bypass the batcher; if one overtook a
|
||||
pending batch, the client would reset its streaming refs and the late
|
||||
batch would paint into a NEW assistant bubble (the split/duplicate
|
||||
look). The flush lives at the top of ``_enqueue`` itself so every
|
||||
emit path — base-class, subclass, and route-level — is covered.
|
||||
|
||||
Negative-test discipline: the double-render tests fail if the flush's
|
||||
inflight-append + enqueue are split across ``_ws_lock`` sections, and
|
||||
the ordering tests fail if the ``_enqueue`` choke-point flush is
|
||||
removed — each was reverted-and-verified during development.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import queue
|
||||
import threading
|
||||
import time
|
||||
from typing import Any
|
||||
|
||||
import pytest
|
||||
|
||||
from turnstone.core.session_ui_base import _TOKEN_BATCH_WINDOW_SECS, SessionUIBase
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Helpers
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class _ConcreteUI(SessionUIBase):
|
||||
"""Minimal concrete subclass for direct UI tests."""
|
||||
|
||||
|
||||
def _make_ui(ws_id: str = "ws-batch") -> _ConcreteUI:
|
||||
return _ConcreteUI(ws_id=ws_id, user_id="u1")
|
||||
|
||||
|
||||
def _drain(lq: queue.Queue[dict[str, Any]]) -> list[dict[str, Any]]:
|
||||
out: list[dict[str, Any]] = []
|
||||
while True:
|
||||
try:
|
||||
out.append(lq.get_nowait())
|
||||
except queue.Empty:
|
||||
return out
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def wide_window(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""Make the batch window effectively infinite so tests control the
|
||||
flush points explicitly (via non-token emits / size cap) and a slow
|
||||
CI machine can't turn one expected batch into two."""
|
||||
monkeypatch.setattr("turnstone.core.session_ui_base._TOKEN_BATCH_WINDOW_SECS", 60.0)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Coalescing shape — one fresh id per batch, first fragment immediate
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_fast_tokens_coalesce_into_single_batch_event(wide_window: None) -> None:
|
||||
"""N tokens inside one window -> the first flushes immediately (the
|
||||
time-to-first-token protection), the rest coalesce into ONE enqueued
|
||||
event whose text is the concatenation, carrying one fresh
|
||||
``_event_id`` / ``_seq``."""
|
||||
ui = _make_ui()
|
||||
lq = ui._register_listener()
|
||||
for i in range(6):
|
||||
ui.on_content_token(f"t{i}")
|
||||
ui.on_stream_end()
|
||||
events = _drain(lq)
|
||||
content = [ev for ev in events if ev["type"] == "content"]
|
||||
assert [ev["text"] for ev in content] == ["t0", "t1t2t3t4t5"]
|
||||
# One fresh id per batch, and the token-event dedup tag rides it.
|
||||
for ev in content:
|
||||
assert isinstance(ev["_event_id"], int)
|
||||
assert ev["_seq"] == ev["_event_id"]
|
||||
# The batch is one ring entry too — ids stay dense (no reserved
|
||||
# per-token ids leak into the ring numbering).
|
||||
_, replay, status, _, _, _ = ui.register_listener_with_replay(0)
|
||||
assert status == "replay_ok"
|
||||
assert [ev["_event_id"] for ev in replay] == list(range(1, len(replay) + 1))
|
||||
|
||||
|
||||
def test_zero_window_flushes_every_token_individually(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""With the window forced to zero every token arrives past the
|
||||
window boundary and flushes on its own — batching self-disables
|
||||
with no behaviour change vs the pre-batching emit shape."""
|
||||
monkeypatch.setattr("turnstone.core.session_ui_base._TOKEN_BATCH_WINDOW_SECS", 0.0)
|
||||
ui = _make_ui()
|
||||
lq = ui._register_listener()
|
||||
for i in range(4):
|
||||
ui.on_content_token(f"t{i}")
|
||||
events = [ev for ev in _drain(lq) if ev["type"] == "content"]
|
||||
assert [ev["text"] for ev in events] == ["t0", "t1", "t2", "t3"]
|
||||
|
||||
|
||||
def test_slow_tokens_flush_individually_at_default_window() -> None:
|
||||
"""Tokens arriving slower than the real (unpatched) window each
|
||||
flush individually — pins the constant's scale: a human-readable
|
||||
typewriter stream must not regress to visible 25 ms batching
|
||||
artifacts, and a mid-turn stall must not hold tokens hostage."""
|
||||
ui = _make_ui()
|
||||
lq = ui._register_listener()
|
||||
for i in range(3):
|
||||
time.sleep(_TOKEN_BATCH_WINDOW_SECS + 0.01)
|
||||
ui.on_content_token(f"t{i}")
|
||||
events = [ev for ev in _drain(lq) if ev["type"] == "content"]
|
||||
assert [ev["text"] for ev in events] == ["t0", "t1", "t2"]
|
||||
|
||||
|
||||
def test_batch_size_cap_triggers_flush(wide_window: None, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""A pending batch reaching the size cap flushes without waiting for
|
||||
the window — bounds worst-case batch size (and client repaint cost)
|
||||
at fast rates."""
|
||||
monkeypatch.setattr("turnstone.core.session_ui_base._TOKEN_BATCH_MAX_CHARS", 8)
|
||||
ui = _make_ui()
|
||||
lq = ui._register_listener()
|
||||
ui.on_content_token("x") # immediate first flush
|
||||
ui.on_content_token("aaaa") # pending (4 < 8)
|
||||
ui.on_content_token("bbbb") # 8 >= 8 -> flush
|
||||
events = [ev for ev in _drain(lq) if ev["type"] == "content"]
|
||||
assert [ev["text"] for ev in events] == ["x", "aaaabbbb"]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Condition 1 — snapshot readers never double-render across a batch
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_snapshot_mid_batch_sees_only_flushed_text_no_double_render(
|
||||
wide_window: None,
|
||||
) -> None:
|
||||
"""A snapshot taken between two tokens of a pending batch must
|
||||
exclude the pending text (it has no event id yet), and the later
|
||||
flush must arrive with ``_seq > snap_seq`` so the client renders
|
||||
each character exactly once: snapshot text + post-``snap_seq`` live
|
||||
events == the full stream, no overlap."""
|
||||
ui = _make_ui()
|
||||
ui.on_content_token("aa") # immediate first flush
|
||||
ui.on_content_token("bb") # pending — invisible to snapshots
|
||||
lq, snap = ui.register_listener_with_in_progress_snapshot()
|
||||
assert snap["content"] == "aa", (
|
||||
"pending batch text leaked into the snapshot — the flush must be "
|
||||
"the only writer of the inflight buffers"
|
||||
)
|
||||
ui.on_stream_end() # flushes the pending batch, then stream_end
|
||||
live = [ev for ev in _drain(lq) if ev["type"] == "content" and ev["_seq"] > snap["seq"]]
|
||||
assert "".join(ev["text"] for ev in live) == "bb"
|
||||
assert snap["content"] + "".join(ev["text"] for ev in live) == "aabb"
|
||||
# The flush wrote the inflight buffer too — the NEXT snapshotter
|
||||
# sees the full text (nothing stranded in the accumulator).
|
||||
_, snap2 = ui.register_listener_with_in_progress_snapshot()
|
||||
assert snap2["content"] == "aabb"
|
||||
|
||||
|
||||
def test_replay_registration_mid_batch_no_double_render(wide_window: None) -> None:
|
||||
"""Same straddle through the ``Last-Event-ID`` reconnect path: the
|
||||
replay slice must not contain the pending batch (not enqueued yet),
|
||||
and the post-registration flush lands exactly once in the live
|
||||
queue."""
|
||||
ui = _make_ui()
|
||||
ui.on_content_token("aa") # immediate flush -> event id 1
|
||||
ui.on_content_token("bb") # pending
|
||||
lq, replay, status, _, _, snap = ui.register_listener_with_replay(1)
|
||||
assert status == "replay_ok"
|
||||
assert replay == [], "pending batch must not appear in the replay slice"
|
||||
ui.on_stream_end()
|
||||
live = [ev for ev in _drain(lq) if ev["type"] == "content"]
|
||||
assert "".join(ev["text"] for ev in live) == "bb"
|
||||
# Full-stream integrity for a fresh reconnect afterwards.
|
||||
_, replay2, _, _, _, _ = ui.register_listener_with_replay(0)
|
||||
assert "".join(ev["text"] for ev in replay2 if ev["type"] == "content") == "aabb"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Condition 2 — every non-token emit flushes the pending batch first
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_stream_end_flushes_pending_batch_before_itself(wide_window: None) -> None:
|
||||
"""``stream_end`` resets the client's streaming refs; a batch
|
||||
arriving after it would paint into a NEW assistant bubble. The
|
||||
flush must therefore precede ``stream_end`` on the wire (strictly
|
||||
smaller event id, earlier queue position)."""
|
||||
ui = _make_ui()
|
||||
lq = ui._register_listener()
|
||||
ui.on_content_token("aa")
|
||||
ui.on_content_token("bb") # pending
|
||||
ui.on_stream_end()
|
||||
events = _drain(lq)
|
||||
types = [ev["type"] for ev in events]
|
||||
assert types == ["content", "content", "stream_end"]
|
||||
assert events[1]["text"] == "bb"
|
||||
assert events[1]["_event_id"] < events[2]["_event_id"]
|
||||
|
||||
|
||||
def test_direct_enqueue_flushes_pending_batch_first(wide_window: None) -> None:
|
||||
"""The flush lives at the ``_enqueue`` choke point, so even
|
||||
route-level / subclass emits (``state_change``, ``cancelled``,
|
||||
``clear_ui``) deliver the pending batch first — not just the
|
||||
``on_*`` helpers."""
|
||||
ui = _make_ui()
|
||||
lq = ui._register_listener()
|
||||
ui.on_content_token("aa")
|
||||
ui.on_content_token("bb") # pending
|
||||
ui._enqueue({"type": "state_change", "state": "idle"})
|
||||
events = _drain(lq)
|
||||
assert [ev["type"] for ev in events] == ["content", "content", "state_change"]
|
||||
assert events[1]["text"] == "bb"
|
||||
|
||||
|
||||
def test_tool_and_status_emits_flush_pending_batch(wide_window: None) -> None:
|
||||
"""Representative non-token ``on_*`` emitters (tool output chunk,
|
||||
status) deliver a pending batch before their own event."""
|
||||
ui = _make_ui()
|
||||
lq = ui._register_listener()
|
||||
ui.on_content_token("aa")
|
||||
ui.on_content_token("bb") # pending
|
||||
ui.on_tool_output_chunk("call-1", "chunk")
|
||||
ui.on_content_token("cc") # immediate? No — window is wide and the
|
||||
# flush just ran, so this pends; the status emit must deliver it.
|
||||
ui.on_status({"prompt_tokens": 1, "completion_tokens": 2}, 1000, "med")
|
||||
events = _drain(lq)
|
||||
types = [ev["type"] for ev in events]
|
||||
assert types == ["content", "content", "tool_output_chunk", "content", "status"]
|
||||
assert events[1]["text"] == "bb"
|
||||
assert events[3]["text"] == "cc"
|
||||
|
||||
|
||||
def test_reasoning_batches_and_kind_switch_flushes(wide_window: None) -> None:
|
||||
"""Reasoning batches like content (own accumulator semantics), and a
|
||||
kind switch flushes the other kind first so wire order preserves
|
||||
arrival order between the two token streams."""
|
||||
ui = _make_ui()
|
||||
lq = ui._register_listener()
|
||||
ui.on_reasoning_token("r0") # immediate first flush
|
||||
ui.on_reasoning_token("r1") # pending
|
||||
ui.on_content_token("c0") # must flush the reasoning batch first
|
||||
ui.on_stream_end()
|
||||
events = _drain(lq)
|
||||
reasoning = [ev for ev in events if ev["type"] == "reasoning"]
|
||||
content = [ev for ev in events if ev["type"] == "content"]
|
||||
assert "".join(ev["text"] for ev in reasoning) == "r0r1"
|
||||
assert "".join(ev["text"] for ev in content) == "c0"
|
||||
assert max(ev["_event_id"] for ev in reasoning) < min(ev["_event_id"] for ev in content)
|
||||
# Reasoning landed in ITS inflight buffer, content in its own.
|
||||
_, snap = ui.register_listener_with_in_progress_snapshot()
|
||||
assert snap["reasoning"] == "r0r1"
|
||||
assert snap["content"] == "c0"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Buffer-cap and turn-boundary semantics under batching
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_inflight_cap_respected_and_stream_continues_past_cap(
|
||||
wide_window: None, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
"""The 512 KiB inflight cap applies to the batched append exactly as
|
||||
it did per-token: check-before-append (bounded overshoot), and the
|
||||
live stream keeps flowing past the cap — the cap bounds the
|
||||
snapshot, it is NOT a stop-streaming signal."""
|
||||
monkeypatch.setattr("turnstone.core.session_ui_base._MAX_TURN_CONTENT_CHARS", 6)
|
||||
ui = _make_ui()
|
||||
lq = ui._register_listener()
|
||||
ui.on_content_token("aaaa") # immediate flush; inflight size 4 < 6
|
||||
ui.on_content_token("bbbb") # pending
|
||||
ui.on_stream_end() # flush appends (4 < 6 -> append; size 8)
|
||||
ui.on_content_token("cccc") # immediate flush; 8 >= 6 -> NOT appended
|
||||
ui.on_stream_end()
|
||||
live = [ev for ev in _drain(lq) if ev["type"] == "content"]
|
||||
assert [ev["text"] for ev in live] == ["aaaa", "bbbb", "cccc"], (
|
||||
"live stream must continue past the inflight cap"
|
||||
)
|
||||
_, snap = ui.register_listener_with_in_progress_snapshot()
|
||||
assert snap["content"] == "aaaabbbb", (
|
||||
"snapshot text is capped (check-before-append overshoot only)"
|
||||
)
|
||||
|
||||
|
||||
def test_on_turn_start_discards_stale_pending(wide_window: None) -> None:
|
||||
"""``on_turn_start`` covers the crashed-prior-``send()`` case; a
|
||||
stale pending batch from that crash must be DISCARDED (never
|
||||
enqueued), not painted into the new turn's bubble."""
|
||||
ui = _make_ui()
|
||||
lq = ui._register_listener()
|
||||
ui.on_content_token("aa")
|
||||
ui.on_content_token("stale") # pending, then the send crashes
|
||||
ui.on_turn_start()
|
||||
ui.on_stream_end()
|
||||
live = [ev for ev in _drain(lq) if ev["type"] == "content"]
|
||||
assert [ev["text"] for ev in live] == ["aa"]
|
||||
_, snap = ui.register_listener_with_in_progress_snapshot()
|
||||
assert snap["content"] == ""
|
||||
|
||||
|
||||
def test_on_turn_committed_flushes_pending_before_reset(wide_window: None) -> None:
|
||||
"""``on_turn_committed`` runs after the assistant message committed;
|
||||
any pending text is part of that committed message, so it flushes
|
||||
(live view + ring stay complete) BEFORE the inflight reset."""
|
||||
ui = _make_ui()
|
||||
lq = ui._register_listener()
|
||||
ui.on_content_token("aa")
|
||||
ui.on_content_token("bb") # pending
|
||||
ui.on_turn_committed()
|
||||
live = [ev for ev in _drain(lq) if ev["type"] == "content"]
|
||||
assert [ev["text"] for ev in live] == ["aa", "bb"]
|
||||
_, snap = ui.register_listener_with_in_progress_snapshot()
|
||||
assert snap["content"] == "", "inflight reset still runs after the flush"
|
||||
|
||||
|
||||
def test_idle_state_payload_includes_pending_batch(wide_window: None) -> None:
|
||||
"""``snapshot_and_consume_state_payload('idle')`` is the cancel /
|
||||
error chokepoint that drains the turn-content accumulator; a pending
|
||||
batch must flush into it first so the dashboard payload carries the
|
||||
full turn."""
|
||||
ui = _make_ui()
|
||||
lq = ui._register_listener()
|
||||
ui.on_content_token("aa")
|
||||
ui.on_content_token("bb") # pending
|
||||
payload = ui.snapshot_and_consume_state_payload("idle")
|
||||
assert payload["content"] == "aabb"
|
||||
live = [ev for ev in _drain(lq) if ev["type"] == "content"]
|
||||
assert "".join(ev["text"] for ev in live) == "aabb"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Concurrency — flush choke point vs a concurrent snapshot reader
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_concurrent_snapshots_never_double_render_batched_stream(
|
||||
wide_window: None,
|
||||
) -> None:
|
||||
"""Hammer test for Condition 1: a writer streams batched tokens
|
||||
while a reader repeatedly registers snapshot listeners; for every
|
||||
snapshot, snapshot-text + post-``snap_seq`` live events must equal
|
||||
the full stream exactly once (no overlap, no gap) — the invariant
|
||||
that breaks if the inflight append and the batch enqueue are ever
|
||||
split across ``_ws_lock`` sections."""
|
||||
ui = _make_ui()
|
||||
n = 200
|
||||
done = threading.Event()
|
||||
|
||||
def _writer() -> None:
|
||||
for i in range(n):
|
||||
ui.on_content_token(f"[{i}]")
|
||||
ui.on_stream_end()
|
||||
done.set()
|
||||
|
||||
results: list[tuple[str, int, queue.Queue[dict[str, Any]]]] = []
|
||||
|
||||
def _reader() -> None:
|
||||
while not done.is_set():
|
||||
lq, snap = ui.register_listener_with_in_progress_snapshot()
|
||||
results.append((snap["content"], snap["seq"], lq))
|
||||
|
||||
w = threading.Thread(target=_writer)
|
||||
r = threading.Thread(target=_reader)
|
||||
w.start()
|
||||
r.start()
|
||||
w.join()
|
||||
r.join()
|
||||
|
||||
full = "".join(f"[{i}]" for i in range(n))
|
||||
for snap_content, snap_seq, lq in results:
|
||||
live = [ev for ev in _drain(lq) if ev["type"] == "content" and ev["_seq"] > snap_seq]
|
||||
rebuilt = snap_content + "".join(ev["text"] for ev in live)
|
||||
assert rebuilt == full, (
|
||||
f"client view diverged: snapshot({len(snap_content)} chars) + "
|
||||
f"{len(live)} live events != full stream"
|
||||
)
|
||||
@@ -404,3 +404,29 @@ class TestParametrizedKind:
|
||||
assert len(rows) == 1
|
||||
assert rows[0]["content"] == payload
|
||||
assert rows[0]["kind"] == kind
|
||||
|
||||
|
||||
class TestGetAttachmentsExcludeKinds:
|
||||
def test_exclude_kinds_filters_at_the_query(self, backend):
|
||||
"""Preview-pane blobs ride ref-lists only for GC + the serving gate;
|
||||
the reconstruct loader excludes them so a history load never pulls
|
||||
their multi-MB content just to discard it."""
|
||||
backend.register_workstream("ws-ex")
|
||||
blob = _hash(b"<html>big page</html>")
|
||||
img = _hash(PNG_1x1)
|
||||
backend.save_attachment(
|
||||
blob,
|
||||
"preview-web",
|
||||
"text/html; charset=utf-8",
|
||||
21,
|
||||
"preview",
|
||||
b"<html>big page</html>",
|
||||
"tool",
|
||||
)
|
||||
backend.save_attachment(
|
||||
img, "shot.png", "image/png", len(PNG_1x1), "image", PNG_1x1, "tool"
|
||||
)
|
||||
rows = backend.get_attachments([blob, img], exclude_kinds=("preview",))
|
||||
assert [r["attachment_id"] for r in rows] == [img]
|
||||
# Default stays unfiltered — the serving route still resolves previews.
|
||||
assert {r["attachment_id"] for r in backend.get_attachments([blob, img])} == {blob, img}
|
||||
|
||||
@@ -4,6 +4,7 @@ from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
|
||||
import pytest
|
||||
import sqlalchemy as sa
|
||||
|
||||
from turnstone.core.storage._schema import workstreams
|
||||
@@ -297,9 +298,9 @@ class TestLoadMessagesLimit:
|
||||
captured: list[list[str]] = []
|
||||
orig = backend.get_attachments
|
||||
|
||||
def _spy(ids):
|
||||
def _spy(ids, exclude_kinds=()):
|
||||
captured.append(sorted(ids))
|
||||
return orig(ids)
|
||||
return orig(ids, exclude_kinds=exclude_kinds)
|
||||
|
||||
# Tail-N=5 fetches only the 5 newest rows (all plain) — the
|
||||
# attachment row is excluded, so NO blob fetch is issued.
|
||||
@@ -576,6 +577,48 @@ class TestSearch:
|
||||
results = backend.search_history_recent(limit=1)
|
||||
assert len(results) == 1
|
||||
|
||||
def test_search_history_survives_oversized_row(self, backend):
|
||||
# A multi-MB row of mostly-unique words: on PostgreSQL its full
|
||||
# tsvector exceeds the 1MB hard limit, which used to abort every
|
||||
# search_history scan ("string is too long for tsvector") — one
|
||||
# giant tool dump silently killed history recall entirely.
|
||||
backend.register_workstream("s1")
|
||||
giant = "gargantuan beacon " + " ".join(f"w{i}" for i in range(300_000))
|
||||
assert len(giant) > 2_000_000
|
||||
backend.save_message("s1", "tool", giant)
|
||||
backend.save_message("s1", "user", "hello world")
|
||||
|
||||
results = backend.search_history("hello")
|
||||
assert any("hello" in str(r[3]) for r in results)
|
||||
|
||||
# The oversized row itself stays findable by its head.
|
||||
results = backend.search_history("gargantuan beacon")
|
||||
assert any("gargantuan" in str(r[3]) for r in results)
|
||||
|
||||
def test_search_history_fts_error_falls_back_to_ilike(self, request, backend, monkeypatch):
|
||||
# PostgreSQL only: a failed FTS statement aborts the connection's
|
||||
# autobegun transaction, and the ILIKE fallback runs on that same
|
||||
# connection — without a rollback first it dies with
|
||||
# InFailedSqlTransaction instead of returning results.
|
||||
if request.config.getoption("--storage-backend") != "postgresql":
|
||||
pytest.skip("exercises PostgreSQL aborted-transaction fallback")
|
||||
|
||||
backend.register_workstream("s1")
|
||||
backend.save_message("s1", "user", "hello fallback world")
|
||||
|
||||
real_execute = sa.engine.Connection.execute
|
||||
|
||||
def failing_fts_execute(self, statement, *args, **kwargs):
|
||||
if "to_tsvector" in str(statement):
|
||||
# A genuine server-side error, so the transaction is aborted
|
||||
# exactly as when to_tsvector rejects a row.
|
||||
return real_execute(self, sa.text("SELECT 1/0"))
|
||||
return real_execute(self, statement, *args, **kwargs)
|
||||
|
||||
monkeypatch.setattr(sa.engine.Connection, "execute", failing_fts_execute)
|
||||
results = backend.search_history("fallback")
|
||||
assert any("fallback" in str(r[3]) for r in results)
|
||||
|
||||
|
||||
# -- Workstream operations -----------------------------------------------------
|
||||
|
||||
|
||||
@@ -60,11 +60,11 @@ class TestToolsMetadata:
|
||||
"""Validate the metadata extracted from JSON files."""
|
||||
|
||||
def test_tool_count(self):
|
||||
# 16 interactive tools + 12 coordinator-only tools.
|
||||
assert len(TOOLS) == 28
|
||||
# 19 interactive tools + 12 coordinator-only tools.
|
||||
assert len(TOOLS) == 31
|
||||
|
||||
def test_task_agent_tools_count(self):
|
||||
assert len(TASK_AGENT_TOOLS) == 11
|
||||
assert len(TASK_AGENT_TOOLS) == 13
|
||||
|
||||
def test_coordinator_tools_count(self):
|
||||
from turnstone.core.tools import COORDINATOR_TOOLS
|
||||
@@ -109,6 +109,12 @@ class TestToolsMetadata:
|
||||
"web_fetch",
|
||||
"web_search",
|
||||
"notify",
|
||||
# Background-shell follow-ups: ``bash_output`` is read-only;
|
||||
# ``kill_shell`` only signals process groups the session itself
|
||||
# spawned via an approved bash call — strictly risk-reducing,
|
||||
# so gating cleanup behind approval adds friction, not safety.
|
||||
"bash_output",
|
||||
"kill_shell",
|
||||
# Coordinator read-only tools (no-mutation, safe to auto-approve):
|
||||
"inspect_workstream",
|
||||
"list_workstreams",
|
||||
@@ -126,7 +132,10 @@ class TestToolsMetadata:
|
||||
"edit_file": "old_string",
|
||||
"web_fetch": "url",
|
||||
"web_search": "query",
|
||||
"open_preview": "target",
|
||||
"task_agent": "prompt",
|
||||
"bash_output": "id",
|
||||
"kill_shell": "id",
|
||||
"memory": "name",
|
||||
"recall": "query",
|
||||
"notify": "message",
|
||||
|
||||
+725
-2
@@ -2,11 +2,15 @@
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import threading
|
||||
import time
|
||||
from datetime import UTC, datetime
|
||||
from typing import Any
|
||||
from unittest.mock import MagicMock
|
||||
|
||||
import pytest
|
||||
|
||||
from tests._helpers import wait_until
|
||||
from turnstone.core.watch import (
|
||||
WatchRunner,
|
||||
build_watch_reminder,
|
||||
@@ -545,8 +549,12 @@ class TestWatchRunner:
|
||||
assert runner.get_dispatch_fn("ws-1") is fn
|
||||
# Unknown ws → None.
|
||||
assert runner.get_dispatch_fn("ws-missing") is None
|
||||
# After removal → None.
|
||||
runner.remove_dispatch_fn("ws-1")
|
||||
# Owner-checked removal: a non-owner's teardown must not remove a
|
||||
# still-live registration (restore shell vs reopened pane).
|
||||
runner.remove_dispatch_fn("ws-1", owner=MagicMock())
|
||||
assert runner.get_dispatch_fn("ws-1") is fn
|
||||
# The owner (or a blind removal) does remove it.
|
||||
runner.remove_dispatch_fn("ws-1", owner=fn)
|
||||
assert runner.get_dispatch_fn("ws-1") is None
|
||||
|
||||
def test_run_command_success(self):
|
||||
@@ -566,3 +574,718 @@ class TestWatchRunner:
|
||||
output, code = runner._run_command("sleep 30")
|
||||
assert "timed out" in output.lower()
|
||||
assert code == -1
|
||||
|
||||
|
||||
def _watch_row(**over: Any) -> dict[str, Any]:
|
||||
"""A firing watch row (condition matches ``echo hello``); override
|
||||
fields per test."""
|
||||
row: dict[str, Any] = {
|
||||
"watch_id": "abc123",
|
||||
"ws_id": "ws-1",
|
||||
"name": "test-watch",
|
||||
"command": "echo hello",
|
||||
"stop_on": '"hello" in output',
|
||||
"max_polls": 100,
|
||||
"poll_count": 0,
|
||||
"last_output": None,
|
||||
"interval_secs": 60,
|
||||
"created": datetime.now(UTC).strftime("%Y-%m-%dT%H:%M:%S"),
|
||||
}
|
||||
row.update(over)
|
||||
return row
|
||||
|
||||
|
||||
def _slow_restore(runner: WatchRunner, ws_id: str, calls: list[str], lock: threading.Lock) -> Any:
|
||||
"""Restore_fn stand-in: record the call, register a dispatch fn (as the
|
||||
real restore does via ``set_watch_runner``), and sleep briefly so a
|
||||
concurrent second caller is guaranteed to be waiting on ``_restore_lock``
|
||||
when we return.
|
||||
"""
|
||||
with lock:
|
||||
calls.append(ws_id)
|
||||
time.sleep(0.05)
|
||||
fn = MagicMock()
|
||||
runner.set_dispatch_fn(ws_id, fn)
|
||||
return fn
|
||||
|
||||
|
||||
class TestWatchRunnerDeliveryRetry:
|
||||
"""Delivery failure HOLDS the built reminder and re-delivers it on a
|
||||
later tick — never re-running the command, so a transient stop_on match
|
||||
isn't lost — bounded by ``MAX_DELIVERY_ATTEMPTS``. Until delivery lands
|
||||
the row commits only ``next_poll`` plus the fire's durable poll charge:
|
||||
no baseline advance and no deactivation of a fire the model never saw,
|
||||
while a restart mid-hold (which re-runs the command) stays bounded by
|
||||
``max_polls``."""
|
||||
|
||||
def _make_runner(self, storage: Any, **kwargs: Any) -> WatchRunner:
|
||||
return WatchRunner(
|
||||
storage=storage,
|
||||
node_id="test-node",
|
||||
check_interval=0.1,
|
||||
tool_timeout=5,
|
||||
**kwargs,
|
||||
)
|
||||
|
||||
def test_delivery_failure_holds_reminder_and_defers_row(self):
|
||||
storage = MagicMock()
|
||||
storage.update_watch.return_value = True
|
||||
# No dispatch fn registered, no restore_fn → delivery fails.
|
||||
runner = self._make_runner(storage)
|
||||
|
||||
runner._poll_watch(_watch_row())
|
||||
|
||||
# Row commit is the retry cadence + this fire's durable poll charge
|
||||
# — the fire stays fully retryable, and a restart mid-hold (which
|
||||
# re-runs the command) stays bounded by max_polls.
|
||||
storage.update_watch.assert_called_once()
|
||||
args, kwargs = storage.update_watch.call_args
|
||||
assert args[0] == "abc123"
|
||||
assert set(kwargs) == {"next_poll", "poll_count"}
|
||||
assert kwargs["poll_count"] == 1 # charged durably at hold time
|
||||
assert kwargs["next_poll"] # advanced, not cleared
|
||||
# The reminder is HELD for re-delivery; the row is NOT marked
|
||||
# terminal-dispatched (the model never saw it).
|
||||
with runner._pending_delivery_lock:
|
||||
assert "abc123" in runner._pending_delivery
|
||||
assert runner._pending_delivery["abc123"]["attempts"] == 1
|
||||
with runner._terminal_dispatched_lock:
|
||||
assert "abc123" not in runner._terminal_dispatched
|
||||
|
||||
def test_redelivery_uses_held_reminder_without_rerunning_command(self):
|
||||
storage = MagicMock()
|
||||
storage.update_watch.return_value = True
|
||||
runner = self._make_runner(storage)
|
||||
|
||||
# Poll 1: fails → holds. Capture the exact held reminder object.
|
||||
runner._poll_watch(_watch_row())
|
||||
with runner._pending_delivery_lock:
|
||||
held = runner._pending_delivery["abc123"]["reminder"]
|
||||
|
||||
# ws restored: register a fn, and make _run_command explode so the
|
||||
# test proves re-delivery does NOT re-run the command.
|
||||
dispatch_fn = MagicMock()
|
||||
runner.set_dispatch_fn("ws-1", dispatch_fn)
|
||||
runner._run_command = MagicMock( # type: ignore[method-assign]
|
||||
side_effect=AssertionError("command must not re-run on re-delivery")
|
||||
)
|
||||
|
||||
runner._poll_watch(_watch_row())
|
||||
|
||||
# Delivered the SAME held reminder; command untouched; committed
|
||||
# terminal with the ORIGINAL fire's poll_count; hold cleared.
|
||||
dispatch_fn.assert_called_once()
|
||||
assert dispatch_fn.call_args[0][0] is held
|
||||
runner._run_command.assert_not_called()
|
||||
_a, kwargs = storage.update_watch.call_args
|
||||
assert kwargs["active"] is False
|
||||
assert kwargs["poll_count"] == 1 # retries consumed no ADDITIONAL budget
|
||||
with runner._pending_delivery_lock:
|
||||
assert "abc123" not in runner._pending_delivery
|
||||
|
||||
def test_transient_exhaustion_keeps_watch_active(self):
|
||||
# A purely transient cause (no fn, no restore → returns False) that
|
||||
# outlasts the attempt budget must NOT silently deactivate the watch
|
||||
# — it drops the held reminder, charges ONE poll to the max_polls
|
||||
# budget, and leaves the watch active to re-fire on its next
|
||||
# interval. The baseline (last_output) stays uncommitted so a
|
||||
# delta-style stop_on re-fires on the change the model never saw.
|
||||
from turnstone.core.watch import MAX_DELIVERY_ATTEMPTS
|
||||
|
||||
storage = MagicMock()
|
||||
storage.update_watch.return_value = True
|
||||
runner = self._make_runner(storage) # always fails, transiently
|
||||
|
||||
# Poll 1 fires + holds (attempts=1); polls 2..MAX bump attempts, the
|
||||
# MAX-th hitting the exhaustion ceiling.
|
||||
for _ in range(MAX_DELIVERY_ATTEMPTS):
|
||||
runner._poll_watch(_watch_row())
|
||||
|
||||
# Hold dropped, but the watch was NEVER deactivated — no active=False
|
||||
# commit anywhere; the final commit charges the poll and re-schedules
|
||||
# (no last_output → the fire re-detects next cycle).
|
||||
with runner._pending_delivery_lock:
|
||||
assert "abc123" not in runner._pending_delivery
|
||||
deactivations = [
|
||||
c for c in storage.update_watch.call_args_list if c.kwargs.get("active") is False
|
||||
]
|
||||
assert deactivations == []
|
||||
_a, kwargs = storage.update_watch.call_args # last commit
|
||||
assert set(kwargs) == {"poll_count", "next_poll"}
|
||||
assert kwargs["poll_count"] == 1 # one poll charged to the budget
|
||||
|
||||
def test_transient_exhaustion_with_budget_spent_deactivates(self):
|
||||
# The keep-alive-on-transient behavior is bounded by the watch's own
|
||||
# max_polls budget: once poll_count reaches it, exhaustion commits
|
||||
# the held (deactivating) update instead of re-running the command
|
||||
# every interval forever against an unreachable workstream.
|
||||
from turnstone.core.watch import MAX_DELIVERY_ATTEMPTS
|
||||
|
||||
storage = MagicMock()
|
||||
storage.update_watch.return_value = True
|
||||
runner = self._make_runner(storage) # always fails, transiently
|
||||
|
||||
for _ in range(MAX_DELIVERY_ATTEMPTS):
|
||||
runner._poll_watch(_watch_row(max_polls=1)) # budget spent on fire 1
|
||||
|
||||
with runner._pending_delivery_lock:
|
||||
assert "abc123" not in runner._pending_delivery
|
||||
_a, kwargs = storage.update_watch.call_args # last commit
|
||||
assert kwargs["active"] is False # deactivated: budget spent
|
||||
assert kwargs["poll_count"] == 1
|
||||
|
||||
def test_held_delivery_retries_on_capped_cadence_not_interval(self):
|
||||
# Re-delivery is a cheap in-memory dispatch — a daily watch whose
|
||||
# fire hit a busy restore slot must retry within
|
||||
# DELIVERY_RETRY_CAP_SECS, not sit on the reminder for 24 h.
|
||||
from turnstone.core.watch import DELIVERY_RETRY_CAP_SECS
|
||||
|
||||
storage = MagicMock()
|
||||
storage.update_watch.return_value = True
|
||||
runner = self._make_runner(storage)
|
||||
|
||||
runner._poll_watch(_watch_row(interval_secs=86_400))
|
||||
|
||||
_a, kwargs = storage.update_watch.call_args
|
||||
assert set(kwargs) == {"next_poll", "poll_count"}
|
||||
retry_at = datetime.strptime(kwargs["next_poll"], "%Y-%m-%dT%H:%M:%S").replace(tzinfo=UTC)
|
||||
delta = (retry_at - datetime.now(UTC)).total_seconds()
|
||||
assert 0 < delta <= DELIVERY_RETRY_CAP_SECS + 5 # capped, not 86400
|
||||
|
||||
def test_permanent_unrestorable_deactivates_immediately(self):
|
||||
# A permanent failure (restore raises WatchWorkstreamUnrestorable,
|
||||
# e.g. corrupt persona stamp) deactivates the watch on the FIRST
|
||||
# fire — no held reminder, no waiting out the attempt budget.
|
||||
from turnstone.core.watch import WatchWorkstreamUnrestorable
|
||||
|
||||
storage = MagicMock()
|
||||
storage.update_watch.return_value = True
|
||||
restore_fn = MagicMock(side_effect=WatchWorkstreamUnrestorable("ws-1"))
|
||||
runner = self._make_runner(storage, restore_fn=restore_fn)
|
||||
|
||||
runner._poll_watch(_watch_row())
|
||||
|
||||
restore_fn.assert_called_once_with("ws-1") # not retried 5×
|
||||
_a, kwargs = storage.update_watch.call_args
|
||||
assert kwargs["active"] is False # deactivated now
|
||||
with runner._pending_delivery_lock:
|
||||
assert "abc123" not in runner._pending_delivery # nothing held
|
||||
# The admission slot is released even on the raising path.
|
||||
with runner._restore_lock:
|
||||
assert "ws-1" not in runner._restoring
|
||||
|
||||
def test_pending_cleared_on_already_dispatched_retry(self):
|
||||
# Constructs the id-in-both-sets state DIRECTLY: no current path
|
||||
# produces it (_redeliver_pending clears the hold before its
|
||||
# commit), but the already-dispatched branch deactivates the row —
|
||||
# after which it never re-lists — so it is the last line of
|
||||
# defense against any such hold leaking forever. Pin that it
|
||||
# clears the hold alongside the retry-deactivate.
|
||||
storage = MagicMock()
|
||||
storage.update_watch.return_value = True
|
||||
runner = self._make_runner(storage)
|
||||
with runner._terminal_dispatched_lock:
|
||||
runner._terminal_dispatched.add("abc123")
|
||||
runner._stash_pending_delivery("abc123", {"text": "x"}, {"active": False}, attempts=1)
|
||||
|
||||
runner._poll_watch(_watch_row()) # hits the already_dispatched branch
|
||||
|
||||
with runner._pending_delivery_lock:
|
||||
assert "abc123" not in runner._pending_delivery
|
||||
with runner._terminal_dispatched_lock:
|
||||
assert "abc123" not in runner._terminal_dispatched
|
||||
|
||||
def test_restore_capacity_full_defers_without_restoring(self):
|
||||
# When MAX_CONCURRENT_RESTORES restores are already in flight, a
|
||||
# new evicted-ws poll must DEFER (return False, hold) rather than
|
||||
# block a poll slot — and must not start a restore.
|
||||
from turnstone.core.watch import MAX_CONCURRENT_RESTORES
|
||||
|
||||
storage = MagicMock()
|
||||
storage.update_watch.return_value = True
|
||||
restore_fn = MagicMock(return_value=MagicMock())
|
||||
runner = self._make_runner(storage, restore_fn=restore_fn)
|
||||
# Saturate the restore admission with other in-flight ws_ids.
|
||||
with runner._restore_lock:
|
||||
for i in range(MAX_CONCURRENT_RESTORES):
|
||||
runner._restoring[f"other-{i}"] = time.monotonic()
|
||||
|
||||
result = runner._dispatch_result("ws-evicted", {"text": "x"}, "w1")
|
||||
|
||||
assert result is False # deferred
|
||||
restore_fn.assert_not_called() # no restore admitted
|
||||
|
||||
def test_race_won_admission_delivers_outside_restore_lock(self):
|
||||
# A dispatch fn registered between the fast-path miss and the
|
||||
# admission check must be delivered WITHOUT holding _restore_lock:
|
||||
# the closure can block (ws._lock, wake-thread spawn), and running
|
||||
# it under the lock serialises every restore admission on the node
|
||||
# behind one delivery.
|
||||
storage = MagicMock()
|
||||
storage.update_watch.return_value = True
|
||||
restore_fn = MagicMock(return_value=None)
|
||||
runner = self._make_runner(storage, restore_fn=restore_fn)
|
||||
|
||||
lock_free_during_dispatch: list[bool] = []
|
||||
|
||||
def probe(reminder: dict[str, Any], watch_id: str) -> None:
|
||||
ok = runner._restore_lock.acquire(blocking=False)
|
||||
lock_free_during_dispatch.append(ok)
|
||||
if ok:
|
||||
runner._restore_lock.release()
|
||||
|
||||
real_try = runner._try_dispatch_fn
|
||||
calls = {"n": 0}
|
||||
|
||||
def fake_try(ws_id: str, reminder: dict[str, Any], watch_id: str) -> bool | None:
|
||||
calls["n"] += 1
|
||||
if calls["n"] == 1:
|
||||
# Simulate a restore completing between the fast path and
|
||||
# the admission check: the fn appears "while we waited".
|
||||
runner.set_dispatch_fn("ws-1", probe)
|
||||
return None
|
||||
return real_try(ws_id, reminder, watch_id)
|
||||
|
||||
runner._try_dispatch_fn = fake_try # type: ignore[method-assign]
|
||||
|
||||
result = runner._dispatch_result("ws-1", {"text": "x"}, "w1")
|
||||
|
||||
assert result is True # race-won fn delivered
|
||||
assert lock_free_during_dispatch == [True] # ...outside the lock
|
||||
restore_fn.assert_not_called() # no restore admitted for a live fn
|
||||
|
||||
def test_restore_returning_none_holds(self):
|
||||
storage = MagicMock()
|
||||
storage.update_watch.return_value = True
|
||||
restore_fn = MagicMock(return_value=None) # e.g. all slots active
|
||||
runner = self._make_runner(storage, restore_fn=restore_fn)
|
||||
|
||||
runner._poll_watch(_watch_row())
|
||||
|
||||
restore_fn.assert_called_once_with("ws-1")
|
||||
_a, kwargs = storage.update_watch.call_args
|
||||
assert set(kwargs) == {"next_poll", "poll_count"}
|
||||
with runner._pending_delivery_lock:
|
||||
assert "abc123" in runner._pending_delivery
|
||||
|
||||
def test_live_fn_raise_holds_without_restoring(self):
|
||||
# A registered fn that RAISES means the ws is live; we must NOT fall
|
||||
# through to restore (that would spawn a duplicate session on a live
|
||||
# conversation). The reminder is held for re-delivery instead.
|
||||
storage = MagicMock()
|
||||
storage.update_watch.return_value = True
|
||||
restore_fn = MagicMock()
|
||||
runner = self._make_runner(storage, restore_fn=restore_fn)
|
||||
runner.set_dispatch_fn("ws-1", MagicMock(side_effect=RuntimeError("stale closure")))
|
||||
|
||||
runner._poll_watch(_watch_row())
|
||||
|
||||
restore_fn.assert_not_called()
|
||||
with runner._pending_delivery_lock:
|
||||
assert "abc123" in runner._pending_delivery
|
||||
_a, kwargs = storage.update_watch.call_args
|
||||
assert set(kwargs) == {"next_poll", "poll_count"}
|
||||
|
||||
def test_forget_terminal_dispatched_clears_held_reminder(self):
|
||||
# User-cancel takes the row out of the due view; its held reminder
|
||||
# must be dropped too or it would leak (never re-polled).
|
||||
storage = MagicMock()
|
||||
storage.update_watch.return_value = True
|
||||
runner = self._make_runner(storage)
|
||||
runner._poll_watch(_watch_row())
|
||||
with runner._pending_delivery_lock:
|
||||
assert "abc123" in runner._pending_delivery
|
||||
|
||||
runner.forget_terminal_dispatched("abc123")
|
||||
|
||||
with runner._pending_delivery_lock:
|
||||
assert "abc123" not in runner._pending_delivery
|
||||
|
||||
def test_abandon_write_failure_keeps_hold_for_write_retry(self):
|
||||
# Reads-succeed/writes-fail storage (e.g. disk-full SQLite): a
|
||||
# failed abandon commit must keep the hold so the next tick retries
|
||||
# the WRITE via the redeliver path — the clear-first order let the
|
||||
# row re-list into a fresh COMMAND RUN every attempt-budget cycle,
|
||||
# forever, with the poll budget never advancing.
|
||||
from turnstone.core.watch import WatchWorkstreamUnrestorable
|
||||
|
||||
storage = MagicMock()
|
||||
storage.update_watch.return_value = True
|
||||
runner = self._make_runner(storage)
|
||||
runner._poll_watch(_watch_row()) # fire → hold (write still OK here)
|
||||
with runner._pending_delivery_lock:
|
||||
pending = dict(runner._pending_delivery["abc123"])
|
||||
|
||||
runner._restore_fn = MagicMock( # type: ignore[assignment]
|
||||
side_effect=WatchWorkstreamUnrestorable("ws-1")
|
||||
)
|
||||
storage.update_watch.side_effect = RuntimeError("disk full")
|
||||
|
||||
with pytest.raises(RuntimeError):
|
||||
runner._redeliver_pending(_watch_row(), pending)
|
||||
|
||||
with runner._pending_delivery_lock:
|
||||
assert "abc123" in runner._pending_delivery # hold survived
|
||||
|
||||
def test_unrestorable_abandon_write_failure_keeps_hold(self):
|
||||
# Fresh-fire permanent failure whose deactivation write fails must
|
||||
# not strand the still-active row into a fresh command run every
|
||||
# tick: the stash routes the next tick into the redeliver path,
|
||||
# which retries the WRITE — never the command.
|
||||
from turnstone.core.watch import WatchWorkstreamUnrestorable
|
||||
|
||||
storage = MagicMock()
|
||||
storage.update_watch.side_effect = RuntimeError("disk full")
|
||||
restore_fn = MagicMock(side_effect=WatchWorkstreamUnrestorable("ws-1"))
|
||||
runner = self._make_runner(storage, restore_fn=restore_fn)
|
||||
|
||||
with pytest.raises(RuntimeError):
|
||||
runner._poll_watch(_watch_row())
|
||||
|
||||
with runner._pending_delivery_lock:
|
||||
assert "abc123" in runner._pending_delivery # hold survived
|
||||
|
||||
# Next tick: the write retries and lands; the command never re-runs.
|
||||
runner._run_command = MagicMock( # type: ignore[method-assign]
|
||||
side_effect=AssertionError("command must not re-run")
|
||||
)
|
||||
storage.update_watch.side_effect = None
|
||||
storage.update_watch.return_value = True
|
||||
|
||||
runner._poll_watch(_watch_row())
|
||||
|
||||
runner._run_command.assert_not_called()
|
||||
_a, kwargs = storage.update_watch.call_args
|
||||
assert kwargs["active"] is False # deactivation landed on the retry
|
||||
with runner._pending_delivery_lock:
|
||||
assert "abc123" not in runner._pending_delivery
|
||||
|
||||
def test_exhaustion_write_failure_keeps_hold_for_write_retry(self):
|
||||
# Same pathology on the transient-exhaustion branch: the charge
|
||||
# commit failing must keep the hold (write retried next tick), not
|
||||
# drop it into a fresh command cycle with the budget never durable.
|
||||
from turnstone.core.watch import MAX_DELIVERY_ATTEMPTS
|
||||
|
||||
storage = MagicMock()
|
||||
storage.update_watch.return_value = True
|
||||
runner = self._make_runner(storage) # no fn, no restore → transient
|
||||
runner._poll_watch(_watch_row()) # fire → hold
|
||||
with runner._pending_delivery_lock:
|
||||
runner._pending_delivery["abc123"]["attempts"] = MAX_DELIVERY_ATTEMPTS - 1
|
||||
pending = dict(runner._pending_delivery["abc123"])
|
||||
|
||||
storage.update_watch.side_effect = RuntimeError("disk full")
|
||||
|
||||
with pytest.raises(RuntimeError):
|
||||
runner._redeliver_pending(_watch_row(), pending)
|
||||
|
||||
with runner._pending_delivery_lock:
|
||||
assert "abc123" in runner._pending_delivery # hold survived
|
||||
|
||||
|
||||
class TestWatchRunnerCancelRace:
|
||||
"""User-cancel racing the poll pool: the delivery paths re-check the
|
||||
row's active state so a cancelled watch can neither deliver nor leak a
|
||||
held reminder, and the tick sweep mops up the one interleaving the
|
||||
point checks can't reach (a stash landing after the cancel path's
|
||||
``forget_terminal_dispatched`` already cleared)."""
|
||||
|
||||
def _make_runner(self, storage: Any, **kwargs: Any) -> WatchRunner:
|
||||
return WatchRunner(
|
||||
storage=storage,
|
||||
node_id="test-node",
|
||||
check_interval=0.1,
|
||||
tool_timeout=5,
|
||||
**kwargs,
|
||||
)
|
||||
|
||||
def test_hold_dropped_when_watch_cancelled_mid_fire(self):
|
||||
# Cancel lands while the fire's command is running: the hold path
|
||||
# re-checks the row and DROPS instead of stashing — an inactive row
|
||||
# never re-lists, so a stash here would leak for the process
|
||||
# lifetime with nothing ever retrying or clearing it.
|
||||
storage = MagicMock()
|
||||
storage.update_watch.return_value = True
|
||||
storage.is_watch_active.return_value = False
|
||||
runner = self._make_runner(storage) # no fn, no restore → would hold
|
||||
|
||||
runner._poll_watch(_watch_row())
|
||||
|
||||
with runner._pending_delivery_lock:
|
||||
assert "abc123" not in runner._pending_delivery
|
||||
# No retry-cadence commit either — the row already left the view.
|
||||
storage.update_watch.assert_not_called()
|
||||
|
||||
def test_redelivery_dropped_when_watch_cancelled(self):
|
||||
# Cancel lands between the due listing and the redelivery dispatch:
|
||||
# deliver nothing (the model must not act on — nor a restore be
|
||||
# spawned for — a watch the user just cancelled) and drop the hold.
|
||||
storage = MagicMock()
|
||||
storage.update_watch.return_value = True
|
||||
runner = self._make_runner(storage)
|
||||
runner._poll_watch(_watch_row()) # fails → holds (row still active)
|
||||
with runner._pending_delivery_lock:
|
||||
assert "abc123" in runner._pending_delivery
|
||||
|
||||
storage.is_watch_active.return_value = False # user cancels
|
||||
dispatch_fn = MagicMock()
|
||||
runner.set_dispatch_fn("ws-1", dispatch_fn) # ws even came back live
|
||||
|
||||
runner._poll_watch(_watch_row())
|
||||
|
||||
dispatch_fn.assert_not_called()
|
||||
with runner._pending_delivery_lock:
|
||||
assert "abc123" not in runner._pending_delivery
|
||||
# The only row write remains the initial hold's cadence commit —
|
||||
# no terminal commit lands over the cancel's row state.
|
||||
assert storage.update_watch.call_count == 1
|
||||
|
||||
def test_tick_sweeps_cancelled_holds(self):
|
||||
# The residual interleaving: a stash that landed AFTER the cancel's
|
||||
# forget_terminal_dispatched cleared (its active re-check passed
|
||||
# just before the cancel's row write). The sweep drops it within
|
||||
# one tick.
|
||||
storage = MagicMock()
|
||||
storage.update_watch.return_value = True
|
||||
storage.list_due_watches.return_value = []
|
||||
runner = self._make_runner(storage)
|
||||
runner._stash_pending_delivery("abc123", {"text": "x"}, {"active": False}, attempts=1)
|
||||
storage.is_watch_active.return_value = False # row already cancelled
|
||||
|
||||
runner._tick()
|
||||
|
||||
with runner._pending_delivery_lock:
|
||||
assert "abc123" not in runner._pending_delivery
|
||||
|
||||
def test_tick_sweep_keeps_active_holds(self):
|
||||
storage = MagicMock()
|
||||
storage.update_watch.return_value = True
|
||||
storage.list_due_watches.return_value = []
|
||||
runner = self._make_runner(storage)
|
||||
runner._stash_pending_delivery("abc123", {"text": "x"}, {"active": False}, attempts=1)
|
||||
storage.is_watch_active.return_value = True
|
||||
|
||||
runner._tick()
|
||||
|
||||
with runner._pending_delivery_lock:
|
||||
assert "abc123" in runner._pending_delivery
|
||||
|
||||
def test_active_checks_bias_toward_delivery_on_storage_error(self):
|
||||
# is_watch_active RAISING must not drop a fire: the sweep keeps the
|
||||
# hold and the delivery paths proceed (bounded by their own attempt
|
||||
# and poll budgets) — a storage blip is not a cancellation.
|
||||
storage = MagicMock()
|
||||
storage.update_watch.return_value = True
|
||||
storage.list_due_watches.return_value = []
|
||||
storage.is_watch_active.side_effect = RuntimeError("storage down")
|
||||
runner = self._make_runner(storage)
|
||||
runner._stash_pending_delivery("abc123", {"text": "x"}, {"active": False}, attempts=1)
|
||||
|
||||
runner._tick() # sweep: biased active → kept
|
||||
|
||||
with runner._pending_delivery_lock:
|
||||
assert "abc123" in runner._pending_delivery
|
||||
|
||||
|
||||
class TestWatchRunnerRestoreSerialization:
|
||||
"""Two watches on ONE evicted workstream, polled concurrently, must
|
||||
trigger the restore path at most once — otherwise each spawns a live
|
||||
auto-approved session racing writes into one conversation history."""
|
||||
|
||||
def test_concurrent_same_ws_restores_once(self):
|
||||
storage = MagicMock()
|
||||
storage.update_watch.return_value = True
|
||||
restore_calls: list[str] = []
|
||||
calls_lock = threading.Lock()
|
||||
|
||||
runner = WatchRunner(
|
||||
storage=storage,
|
||||
node_id="n",
|
||||
check_interval=0.1,
|
||||
tool_timeout=5,
|
||||
restore_fn=lambda ws_id: _slow_restore(runner, ws_id, restore_calls, calls_lock),
|
||||
)
|
||||
|
||||
reminder = {"type": "watch_triggered", "text": "x"}
|
||||
barrier = threading.Barrier(2)
|
||||
results: list[bool] = []
|
||||
results_lock = threading.Lock()
|
||||
|
||||
def call(wid: str) -> None:
|
||||
barrier.wait(timeout=2.0)
|
||||
ok = runner._dispatch_result("ws-shared", reminder, wid)
|
||||
with results_lock:
|
||||
results.append(ok)
|
||||
|
||||
threads = [threading.Thread(target=call, args=(f"w{i}",)) for i in range(2)]
|
||||
for t in threads:
|
||||
t.start()
|
||||
for t in threads:
|
||||
t.join(timeout=3.0)
|
||||
|
||||
# Exactly one restore ran; the winner delivered (True) and the other
|
||||
# DEFERRED (False, holds + re-delivers next tick) rather than blocking
|
||||
# its poll slot on the in-flight restore or restoring a second time.
|
||||
assert restore_calls == ["ws-shared"]
|
||||
assert sorted(results) == [False, True]
|
||||
# Admission slot released after the restore.
|
||||
with runner._restore_lock:
|
||||
assert "ws-shared" not in runner._restoring
|
||||
|
||||
def test_wedged_restore_admissions_defer_and_alert(self, caplog):
|
||||
# Admission entries older than RESTORE_STALL_ALERT_SECS are alerted
|
||||
# on but NEVER evicted: the wedged poll thread's pool slot is never
|
||||
# released, so reclaiming its admission would just readmit a restore
|
||||
# that can wedge another pool thread on the same cause — trading
|
||||
# this capped degraded state (restores blocked, polling intact) for
|
||||
# total poll-pool collapse. New restores keep deferring; the error
|
||||
# log is the operator's restart signal.
|
||||
from turnstone.core.watch import RESTORE_STALL_ALERT_SECS
|
||||
|
||||
storage = MagicMock()
|
||||
storage.update_watch.return_value = True
|
||||
restore_fn = MagicMock(return_value=MagicMock())
|
||||
runner = WatchRunner(
|
||||
storage=storage,
|
||||
node_id="n",
|
||||
check_interval=0.1,
|
||||
tool_timeout=5,
|
||||
restore_fn=restore_fn,
|
||||
)
|
||||
stalled_at = time.monotonic() - RESTORE_STALL_ALERT_SECS - 1
|
||||
with runner._restore_lock:
|
||||
runner._restoring["wedged-1"] = stalled_at
|
||||
runner._restoring["wedged-2"] = stalled_at # both slots wedged
|
||||
|
||||
with caplog.at_level("ERROR"):
|
||||
result = runner._dispatch_result("ws-new", {"text": "x"}, "w1")
|
||||
|
||||
assert result is False # wedged capacity stays consumed → defer
|
||||
restore_fn.assert_not_called()
|
||||
with runner._restore_lock:
|
||||
assert "wedged-1" in runner._restoring
|
||||
assert "wedged-2" in runner._restoring
|
||||
assert any("watch_runner.restore_admission_wedged" in r.message for r in caplog.records)
|
||||
|
||||
|
||||
class TestWatchRunnerConcurrency:
|
||||
"""The tick thread only enumerates due rows; polls run on bounded
|
||||
daemon threads. Pins: genuine concurrency, per-watch in-flight
|
||||
dedup, saturation leaving rows due (not dropped), and ``stop``
|
||||
draining in-flight polls."""
|
||||
|
||||
def _make_runner(self, rows: list[dict[str, Any]], **kwargs: Any) -> WatchRunner:
|
||||
storage = MagicMock()
|
||||
storage.update_watch.return_value = True
|
||||
storage.list_due_watches.return_value = rows
|
||||
return WatchRunner(
|
||||
storage=storage,
|
||||
node_id="test-node",
|
||||
check_interval=0.1,
|
||||
tool_timeout=5,
|
||||
**kwargs,
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
def _wait_in_flight_empty(runner: WatchRunner, timeout: float = 3.0) -> None:
|
||||
def _drained() -> bool:
|
||||
with runner._in_flight_lock:
|
||||
return not runner._in_flight
|
||||
|
||||
wait_until(_drained, timeout=timeout)
|
||||
|
||||
def test_tick_polls_concurrently(self):
|
||||
rows = [_watch_row(watch_id=f"w{i}", ws_id=f"ws-{i}") for i in range(3)]
|
||||
runner = self._make_runner(rows)
|
||||
|
||||
all_in = threading.Event()
|
||||
release = threading.Event()
|
||||
barrier = threading.Barrier(3)
|
||||
|
||||
def fake_poll(_row: dict[str, Any]) -> None:
|
||||
# All three poll threads must be inside simultaneously for the
|
||||
# barrier to trip — serial execution would deadlock here (and
|
||||
# fail via the barrier timeout instead).
|
||||
barrier.wait(timeout=2.0)
|
||||
all_in.set()
|
||||
release.wait(timeout=2.0)
|
||||
|
||||
runner._poll_watch = fake_poll # type: ignore[method-assign]
|
||||
runner._tick()
|
||||
|
||||
assert all_in.wait(timeout=2.0), "polls did not run concurrently"
|
||||
release.set()
|
||||
self._wait_in_flight_empty(runner)
|
||||
|
||||
def test_tick_skips_in_flight_watch(self):
|
||||
rows = [_watch_row(watch_id="w0", ws_id="ws-0")]
|
||||
runner = self._make_runner(rows)
|
||||
polled: list[str] = []
|
||||
runner._poll_watch = lambda row: polled.append(row["watch_id"]) # type: ignore[method-assign]
|
||||
|
||||
# Simulate a slow poll from a previous tick still running.
|
||||
with runner._in_flight_lock:
|
||||
runner._in_flight.add("w0")
|
||||
|
||||
runner._tick()
|
||||
|
||||
assert polled == []
|
||||
# The foreign in-flight entry was not clobbered by the skip.
|
||||
with runner._in_flight_lock:
|
||||
assert "w0" in runner._in_flight
|
||||
|
||||
def test_tick_saturation_leaves_rows_due(self):
|
||||
rows = [_watch_row(watch_id=f"w{i}", ws_id=f"ws-{i}") for i in range(2)]
|
||||
runner = self._make_runner(rows, max_concurrent_polls=1)
|
||||
|
||||
started = threading.Event()
|
||||
release = threading.Event()
|
||||
polled: list[str] = []
|
||||
|
||||
def fake_poll(row: dict[str, Any]) -> None:
|
||||
polled.append(row["watch_id"])
|
||||
started.set()
|
||||
release.wait(timeout=2.0)
|
||||
|
||||
runner._poll_watch = fake_poll # type: ignore[method-assign]
|
||||
runner._tick()
|
||||
assert started.wait(timeout=2.0)
|
||||
|
||||
# Only the first row got a slot this tick; the second stays due
|
||||
# for the next tick rather than being dropped.
|
||||
assert polled == ["w0"]
|
||||
release.set()
|
||||
self._wait_in_flight_empty(runner)
|
||||
|
||||
# Next tick (slot free again) picks up the remaining row.
|
||||
runner._storage.list_due_watches.return_value = [rows[1]]
|
||||
runner._tick()
|
||||
self._wait_in_flight_empty(runner)
|
||||
assert polled == ["w0", "w1"]
|
||||
|
||||
def test_stop_waits_for_in_flight_polls(self):
|
||||
rows = [_watch_row(watch_id="w0", ws_id="ws-0")]
|
||||
runner = self._make_runner(rows)
|
||||
|
||||
started = threading.Event()
|
||||
release = threading.Event()
|
||||
|
||||
def fake_poll(_row: dict[str, Any]) -> None:
|
||||
started.set()
|
||||
release.wait(timeout=3.0)
|
||||
|
||||
runner._poll_watch = fake_poll # type: ignore[method-assign]
|
||||
runner._tick()
|
||||
assert started.wait(timeout=2.0)
|
||||
|
||||
stopper = threading.Thread(target=runner.stop, daemon=True)
|
||||
stopper.start()
|
||||
# stop() must be draining (poll still pinned), not returned.
|
||||
time.sleep(0.15)
|
||||
assert stopper.is_alive(), "stop() returned while a poll was in flight"
|
||||
|
||||
release.set()
|
||||
stopper.join(timeout=3.0)
|
||||
assert not stopper.is_alive()
|
||||
with runner._in_flight_lock:
|
||||
assert not runner._in_flight
|
||||
|
||||
@@ -54,9 +54,11 @@ def _make_session_for_dispatch(**kwargs: Any) -> ChatSession:
|
||||
return ChatSession(**defaults)
|
||||
|
||||
|
||||
def _register_runner(session: ChatSession) -> tuple[Any, Any]:
|
||||
def _register_runner(session: ChatSession, wake_fn: Any = None) -> tuple[Any, Any]:
|
||||
"""Attach a minimal stub ``WatchRunner`` to *session* and return the
|
||||
``(runner, dispatch_fn)`` pair captured by ``set_dispatch_fn``.
|
||||
``wake_fn`` rides through to ``set_watch_runner`` (default ``None``
|
||||
matches the pre-wake wiring most tests here exercise).
|
||||
"""
|
||||
captured: dict[str, Any] = {}
|
||||
|
||||
@@ -65,7 +67,7 @@ def _register_runner(session: ChatSession) -> tuple[Any, Any]:
|
||||
captured["fn"] = fn
|
||||
|
||||
runner = _StubRunner()
|
||||
session.set_watch_runner(runner)
|
||||
session.set_watch_runner(runner, wake_fn=wake_fn)
|
||||
return runner, captured["fn"]
|
||||
|
||||
|
||||
@@ -199,9 +201,9 @@ class TestSoftCap:
|
||||
# Oldest ("body-0") gone; newest ("overflow") present.
|
||||
assert "body-0" not in bodies
|
||||
assert "overflow" in bodies
|
||||
# Warning logged.
|
||||
assert any("watch_dispatch.queue_full" in r.message for r in caplog.records), (
|
||||
"expected a watch_dispatch.queue_full warning record"
|
||||
# Warning logged (the shared external-event rail owns the event now).
|
||||
assert any("external_event.queue_full" in r.message for r in caplog.records), (
|
||||
"expected an external_event.queue_full warning record"
|
||||
)
|
||||
|
||||
def test_dispatch_soft_cap_does_not_evict_other_types(self, tmp_db):
|
||||
@@ -400,3 +402,216 @@ class TestMetadataPropagation:
|
||||
assert len(snapshot) == 1
|
||||
_nt, _text, meta = snapshot[0]
|
||||
assert meta is None
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Wake trigger
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestWakeFn:
|
||||
"""``set_watch_runner``'s optional ``wake_fn`` fires once per enqueued
|
||||
dispatch — AFTER the entry lands — so a watch firing on an
|
||||
already-idle workstream (no IDLE transition for the
|
||||
``IdleNudgeWatcher`` to observe) can spawn the wake worker that
|
||||
drains it. Failures are contained: the enqueue must survive a
|
||||
raising ``wake_fn``, because a propagated raise would abort
|
||||
``WatchRunner._poll_watch`` before the watch-row update commits and
|
||||
re-fire the same reminder every subsequent tick.
|
||||
"""
|
||||
|
||||
def test_wake_fn_called_after_enqueue(self, tmp_db):
|
||||
session = _make_session_for_dispatch()
|
||||
depth_at_wake: list[int] = []
|
||||
_runner, dispatch = _register_runner(
|
||||
session, wake_fn=lambda: depth_at_wake.append(len(session._nudge_queue))
|
||||
)
|
||||
|
||||
dispatch(_reminder("watch fired body"), "watch-1")
|
||||
|
||||
# Fired exactly once, and the entry was already queued when it ran
|
||||
# — the wake worker's drain must be able to see the fresh entry.
|
||||
assert depth_at_wake == [1]
|
||||
|
||||
def test_wake_fn_not_called_when_payload_sanitizes_empty(self, tmp_db):
|
||||
"""A fire whose payload strips to nothing enqueues nothing — and
|
||||
must not wake anything either (a wake with an empty queue would
|
||||
just spawn a worker that no-ops at the drain guard)."""
|
||||
session = _make_session_for_dispatch()
|
||||
wake = MagicMock()
|
||||
_runner, dispatch = _register_runner(session, wake_fn=wake)
|
||||
|
||||
dispatch(_reminder("\x07\x0b\x7f"), "watch-1")
|
||||
|
||||
assert len(session._nudge_queue) == 0
|
||||
wake.assert_not_called()
|
||||
|
||||
def test_wake_fn_exception_is_contained(self, tmp_db, caplog):
|
||||
session = _make_session_for_dispatch()
|
||||
wake = MagicMock(side_effect=RuntimeError("boom"))
|
||||
_runner, dispatch = _register_runner(session, wake_fn=wake)
|
||||
|
||||
with caplog.at_level("WARNING"):
|
||||
dispatch(_reminder("body"), "watch-1") # must not raise
|
||||
|
||||
# Entry survived; the failure surfaced as a warning, not a raise
|
||||
# up into the poll loop.
|
||||
assert len(session._nudge_queue) == 1
|
||||
assert any("external_event.wake_failed" in r.message for r in caplog.records), (
|
||||
"expected an external_event.wake_failed warning record"
|
||||
)
|
||||
|
||||
|
||||
class _RecordingRunner:
|
||||
"""Stub WatchRunner recording registration/removal order. Mirrors the
|
||||
production owner-checked removal semantics — the resume tail passes
|
||||
``owner`` and peeks ``get_dispatch_fn`` before re-registering."""
|
||||
|
||||
def __init__(self) -> None:
|
||||
self.events: list[tuple[str, str]] = []
|
||||
self.fns: dict[str, Any] = {}
|
||||
|
||||
def set_dispatch_fn(self, ws_id: str, fn: Any) -> None:
|
||||
self.events.append(("set", ws_id))
|
||||
self.fns[ws_id] = fn
|
||||
|
||||
def get_dispatch_fn(self, ws_id: str) -> Any:
|
||||
return self.fns.get(ws_id)
|
||||
|
||||
def remove_dispatch_fn(self, ws_id: str, owner: Any = None) -> None:
|
||||
if owner is not None and self.fns.get(ws_id) is not owner:
|
||||
return
|
||||
self.events.append(("remove", ws_id))
|
||||
self.fns.pop(ws_id, None)
|
||||
|
||||
|
||||
class TestResumeReRegistration:
|
||||
"""A non-fork ``resume()`` rebinds ``_ws_id``; the dispatch
|
||||
registration must FOLLOW that identity — otherwise watches stamped
|
||||
with the adopted id never find the live session, and every fire
|
||||
takes the restore path, spawning a duplicate auto-approved session
|
||||
racing writes into the same conversation (CLI ``--resume`` and the
|
||||
``/resume`` command both hit this)."""
|
||||
|
||||
def _saved_ws(self, ws_id: str) -> None:
|
||||
from turnstone.core.memory import register_workstream, save_message
|
||||
|
||||
register_workstream(ws_id)
|
||||
save_message(ws_id, "user", "hi")
|
||||
|
||||
def test_nonfork_resume_moves_registration_to_adopted_id(self, tmp_db):
|
||||
self._saved_ws("resume-target")
|
||||
session = _make_session_for_dispatch()
|
||||
old_id = session._ws_id
|
||||
runner = _RecordingRunner()
|
||||
session.set_watch_runner(runner, wake_fn=None)
|
||||
|
||||
assert session.resume("resume-target") is True
|
||||
|
||||
# New key live BEFORE the old key is removed — a fire during the
|
||||
# transition can never observe an empty registry (which would
|
||||
# divert it to the restore path).
|
||||
assert runner.events == [
|
||||
("set", old_id),
|
||||
("set", "resume-target"),
|
||||
("remove", old_id),
|
||||
]
|
||||
assert set(runner.fns) == {"resume-target"}
|
||||
|
||||
def test_fork_resume_keeps_registration(self, tmp_db):
|
||||
self._saved_ws("fork-src")
|
||||
session = _make_session_for_dispatch()
|
||||
old_id = session._ws_id
|
||||
runner = _RecordingRunner()
|
||||
session.set_watch_runner(runner, wake_fn=None)
|
||||
|
||||
assert session.resume("fork-src", fork=True) is True
|
||||
|
||||
# Fork keeps its own identity — registration untouched.
|
||||
assert runner.events == [("set", old_id)]
|
||||
|
||||
def test_resume_without_runner_is_noop(self, tmp_db):
|
||||
# CLI --resume / restore-fn shape: resume() runs BEFORE any
|
||||
# set_watch_runner call — nothing to re-register, nothing raises.
|
||||
self._saved_ws("resume-bare")
|
||||
session = _make_session_for_dispatch()
|
||||
|
||||
assert session.resume("resume-bare") is True
|
||||
assert session._watch_runner is None
|
||||
|
||||
def test_reregistered_closure_keeps_wake_fn(self, tmp_db):
|
||||
# The stored wake_fn rides the re-registration: a watch firing on
|
||||
# the ADOPTED id must still wake the workstream.
|
||||
self._saved_ws("resume-wake")
|
||||
session = _make_session_for_dispatch()
|
||||
runner = _RecordingRunner()
|
||||
wake = MagicMock()
|
||||
session.set_watch_runner(runner, wake_fn=wake)
|
||||
|
||||
assert session.resume("resume-wake") is True
|
||||
|
||||
runner.fns["resume-wake"](_reminder("watch output"), "w1")
|
||||
assert len(session._nudge_queue) == 1
|
||||
wake.assert_called_once()
|
||||
|
||||
def test_resume_does_not_steal_another_live_registration(self, tmp_db):
|
||||
# In-session /resume of a workstream that is OPEN IN ANOTHER PANE
|
||||
# (a degenerate two-live-sessions state): the original owner keeps
|
||||
# its watch fires — the adopter neither clobbers the target's
|
||||
# registration nor (on a later resume-away or close) deletes it.
|
||||
self._saved_ws("shared-A")
|
||||
self._saved_ws("other-C")
|
||||
runner = _RecordingRunner()
|
||||
|
||||
pane_a = _make_session_for_dispatch()
|
||||
pane_a._ws_id = "shared-A" # pane A opened A and registered
|
||||
pane_a.set_watch_runner(runner, wake_fn=None)
|
||||
fn_a = runner.fns["shared-A"]
|
||||
|
||||
pane_b = _make_session_for_dispatch()
|
||||
pane_b.set_watch_runner(runner, wake_fn=None)
|
||||
|
||||
assert pane_b.resume("shared-A") is True
|
||||
# Pane A's registration survived the adoption…
|
||||
assert runner.fns["shared-A"] is fn_a
|
||||
|
||||
assert pane_b.resume("other-C") is True
|
||||
# …and the resume-away removed only pane B's own (absent) claim.
|
||||
assert runner.fns["shared-A"] is fn_a
|
||||
assert "other-C" in runner.fns
|
||||
|
||||
def test_new_command_moves_registration_to_fresh_id(self, tmp_db):
|
||||
# /new is the other identity rebind: watches created AFTER it stamp
|
||||
# the fresh id and must reach this session, while the old
|
||||
# workstream's fires must stop landing in a conversation that no
|
||||
# longer shows them (they divert to the restore path instead).
|
||||
session = _make_session_for_dispatch()
|
||||
old_id = session._ws_id
|
||||
runner = _RecordingRunner()
|
||||
session.set_watch_runner(runner, wake_fn=None)
|
||||
|
||||
# handle_command's return means "should exit" — /new never exits.
|
||||
assert session.handle_command("/new") is False
|
||||
|
||||
assert session._ws_id != old_id
|
||||
assert set(runner.fns) == {session._ws_id}
|
||||
assert ("remove", old_id) in runner.events
|
||||
|
||||
def test_close_removes_only_own_registration(self, tmp_db):
|
||||
# A watch-restore shell and a reopened pane can serve one ws_id in
|
||||
# sequence; the shell's later teardown must not unregister the pane.
|
||||
self._saved_ws("shared-W")
|
||||
runner = _RecordingRunner()
|
||||
|
||||
shell = _make_session_for_dispatch()
|
||||
shell._ws_id = "shared-W"
|
||||
shell.set_watch_runner(runner, wake_fn=None)
|
||||
|
||||
pane = _make_session_for_dispatch()
|
||||
pane._ws_id = "shared-W"
|
||||
pane.set_watch_runner(runner, wake_fn=None) # pane re-registers (last writer)
|
||||
pane_fn = runner.fns["shared-W"]
|
||||
|
||||
shell.close() # shell reaped (close_idle / eviction)
|
||||
|
||||
assert runner.fns.get("shared-W") is pane_fn # pane still registered
|
||||
|
||||
@@ -340,8 +340,8 @@ def test_poll_watch_terminal_fire_survives_drain(
|
||||
monkeypatch.setattr(session._nudge_queue, "enqueue", _spy_enqueue)
|
||||
|
||||
# For the max_polls=1 case the first poll has prev_output=None and
|
||||
# would not normally fire on output change; the max_polls branch
|
||||
# at watch.py:412-414 still marks is_final=True so dispatch runs.
|
||||
# would not normally fire on output change; _poll_watch's max_polls
|
||||
# branch still marks is_final=True so dispatch runs.
|
||||
due = storage.list_due_watches("2099-01-01T00:00:00")
|
||||
matching = [r for r in due if r["watch_id"] == f"w-regression-{label}"]
|
||||
assert len(matching) == 1, f"watch row not picked up by list_due_watches: {due!r}"
|
||||
|
||||
@@ -114,3 +114,85 @@ class TestVersionHtml:
|
||||
html = '<script src="/static/app.js?foo=bar"></script>'
|
||||
result = version_html(html)
|
||||
assert result == html # unchanged — already has query string
|
||||
|
||||
|
||||
class TestLatin1SafeFilename:
|
||||
"""Content-Disposition filename sanitizer — must yield a value that is
|
||||
both latin-1 encodable (Starlette) and control-char free (h11)."""
|
||||
|
||||
def _assert_wire_safe(self, out: str) -> None:
|
||||
# Independent oracle — deliberately does NOT reuse the impl's
|
||||
# isprintable() gate (that would pass by construction). Every char
|
||||
# must be printable ASCII (0x20..0x7e) and neither quoted-string
|
||||
# metacharacter, so the value is latin-1 clean, control-free, and
|
||||
# safely quotable.
|
||||
assert all(0x20 <= ord(c) <= 0x7E and c not in '"\\' for c in out)
|
||||
|
||||
def test_plain_ascii_unchanged(self):
|
||||
from turnstone.core.web_helpers import latin1_safe_filename
|
||||
|
||||
assert latin1_safe_filename("report_2026.md") == "report_2026.md"
|
||||
|
||||
def test_non_latin1_folds_to_question_marks(self):
|
||||
from turnstone.core.web_helpers import latin1_safe_filename
|
||||
|
||||
# CJK + em dash (U+2014) are printable but non-latin-1 → fold to '?'.
|
||||
out = latin1_safe_filename("文書 — v1.md")
|
||||
assert out == "?? ? v1.md"
|
||||
self._assert_wire_safe(out)
|
||||
|
||||
def test_latin1_but_control_chars_are_stripped(self):
|
||||
from turnstone.core.web_helpers import latin1_safe_filename
|
||||
|
||||
# All latin-1 encodable, so the old strip/fold left them in the header
|
||||
# and the HTTP server layer then 500'd (h11 rejects NUL/CR/LF/FF/VT;
|
||||
# httptools is stricter). NUL / form-feed / DEL / TAB / VT / C1-NEL
|
||||
# (0x85) must all be dropped, not merely folded.
|
||||
out = latin1_safe_filename("a\x00b\x0cc\x7fd\te\x0bf\x85g.md")
|
||||
assert out == "abcdefg.md"
|
||||
self._assert_wire_safe(out)
|
||||
|
||||
def test_crlf_and_quote_stripped(self):
|
||||
from turnstone.core.web_helpers import latin1_safe_filename
|
||||
|
||||
out = latin1_safe_filename('a"\r\nX-Evil: 1.md')
|
||||
assert "\r" not in out and "\n" not in out and '"' not in out
|
||||
self._assert_wire_safe(out)
|
||||
|
||||
def test_backslash_stripped(self):
|
||||
from turnstone.core.web_helpers import latin1_safe_filename
|
||||
|
||||
# Backslash is the RFC 6266 quoted-pair escape inside filename="..." —
|
||||
# a trailing '\' would escape the closing quote, and '\x' mid-name
|
||||
# becomes a spurious escape. Both must be dropped (Windows-origin
|
||||
# uploads legitimately carry '\').
|
||||
assert latin1_safe_filename("dir\\file.md") == "dirfile.md"
|
||||
assert latin1_safe_filename("trailing\\") == "trailing"
|
||||
self._assert_wire_safe(latin1_safe_filename("a\\b\\c"))
|
||||
|
||||
def test_empty_after_sanitizing_uses_fallback(self):
|
||||
from turnstone.core.web_helpers import latin1_safe_filename
|
||||
|
||||
# A name of only quotes / controls sanitizes to empty → fallback,
|
||||
# never ``filename=""``.
|
||||
assert latin1_safe_filename('"""') == "attachment"
|
||||
assert latin1_safe_filename("\x00\x0c\x7f") == "attachment"
|
||||
assert latin1_safe_filename("", fallback="preview") == "preview"
|
||||
|
||||
def test_all_non_latin1_stays_non_empty_no_fallback(self):
|
||||
from turnstone.core.web_helpers import latin1_safe_filename
|
||||
|
||||
# An all-CJK name folds to '???' (truthy) — must NOT hit the fallback.
|
||||
assert latin1_safe_filename("日本語", fallback="preview") == "???"
|
||||
|
||||
def test_fallback_is_also_sanitized(self):
|
||||
from turnstone.core.web_helpers import latin1_safe_filename
|
||||
|
||||
# The fallback fires only when the name sanitizes to empty, and it is
|
||||
# cleaned by the SAME rules — a caller can't reintroduce the crash /
|
||||
# corruption through an unsafe fallback.
|
||||
assert latin1_safe_filename("", fallback="—\x00.txt") == "?.txt"
|
||||
self._assert_wire_safe(latin1_safe_filename("", fallback="bad\\\x00name"))
|
||||
# If even the fallback sanitizes to empty, a safe constant backs it —
|
||||
# never filename="".
|
||||
assert latin1_safe_filename("", fallback='"\x00') == "download"
|
||||
|
||||
@@ -32,8 +32,14 @@ def _drain_global() -> list[dict]:
|
||||
class TestContentAccumulation:
|
||||
"""WebUI should accumulate content tokens and include in idle broadcast."""
|
||||
|
||||
def test_content_token_accumulates(self):
|
||||
"""on_content_token should append to _ws_turn_content."""
|
||||
def test_content_token_accumulates(self, monkeypatch):
|
||||
"""on_content_token should append to _ws_turn_content.
|
||||
|
||||
Batch window forced to 0 (per-token flush) — this pins the
|
||||
accumulator wiring, not the emit-time batching cadence (which
|
||||
coalesces fragments; see test_sse_token_batching.py)."""
|
||||
|
||||
monkeypatch.setattr("turnstone.core.session_ui_base._TOKEN_BATCH_WINDOW_SECS", 0.0)
|
||||
ui = _make_ui()
|
||||
ui.on_content_token("Hello ")
|
||||
ui.on_content_token("world")
|
||||
|
||||
@@ -23,6 +23,7 @@ the affected golden and inspecting the diff.
|
||||
from __future__ import annotations
|
||||
|
||||
import contextlib
|
||||
import dataclasses
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
@@ -308,3 +309,53 @@ def test_wire_payload_anthropic_compat(fixture_id: str) -> None:
|
||||
)
|
||||
assert "thinking" not in payload, "compat lane must never send the native thinking param"
|
||||
_assert_golden(f"anthropic_compat__{fixture_id}", payload)
|
||||
|
||||
|
||||
# GPT-5.6 is the first commercial OpenAI family to expose the "max"
|
||||
# reasoning effort across Sol, Terra, and Luna (see OPENAI_CAPABILITIES).
|
||||
# The base matrix above pins only the default-effort Responses shape
|
||||
# (gpt-5 → "medium"), so freeze a max-effort request to prove the new
|
||||
# level compiles onto the native ``reasoning={"effort": "max"}`` param.
|
||||
# Driving it through "gpt-5.6-sol" also exercises the longest-prefix
|
||||
# inheritance (that id resolves to the "gpt-5.6" row) on the real wire,
|
||||
# not just the capability lookup — over a bare turn and a tool round-trip.
|
||||
_OPENAI_MAX_FIXTURES = ("text", "toolcall_complete")
|
||||
|
||||
|
||||
@pytest.mark.parametrize("fixture_id", _OPENAI_MAX_FIXTURES)
|
||||
def test_wire_payload_openai_max(fixture_id: str) -> None:
|
||||
messages, opts = _FIXTURES[fixture_id]
|
||||
provider = OpenAIResponsesProvider()
|
||||
payload = _capture(
|
||||
provider,
|
||||
model="gpt-5.6-sol",
|
||||
messages=[dict(m) for m in messages],
|
||||
reasoning_effort="max",
|
||||
**opts,
|
||||
)
|
||||
assert payload["reasoning"] == {"effort": "max"}, "max must compile onto reasoning.effort"
|
||||
_assert_golden(f"openai_responses_max__{fixture_id}", payload)
|
||||
|
||||
|
||||
def test_wire_payload_openai_verbosity_pro() -> None:
|
||||
"""Operator-declared verbosity + pro mode compile onto ``text.verbosity``
|
||||
and ``reasoning.mode`` for GPT-5.6 Sol. Both are default-off; an operator
|
||||
turns them on via the model-definition capabilities JSON, which
|
||||
``ChatSession._resolve_capabilities`` merges into caps at request time —
|
||||
modeled here by the ``dataclasses.replace`` override _capture forwards."""
|
||||
provider = OpenAIResponsesProvider()
|
||||
caps = dataclasses.replace(
|
||||
provider.get_capabilities("gpt-5.6-sol"), verbosity="low", reasoning_mode="pro"
|
||||
)
|
||||
messages, opts = _FIXTURES["text"]
|
||||
payload = _capture(
|
||||
provider,
|
||||
model="gpt-5.6-sol",
|
||||
messages=[dict(m) for m in messages],
|
||||
caps=caps,
|
||||
reasoning_effort="high",
|
||||
**opts,
|
||||
)
|
||||
assert payload["text"] == {"verbosity": "low"}
|
||||
assert payload["reasoning"] == {"effort": "high", "mode": "pro"}
|
||||
_assert_golden("openai_responses_verbosity_pro__text", payload)
|
||||
|
||||
@@ -1,3 +1,3 @@
|
||||
"""turnstone - Multi-node AI orchestration platform with tool use, agent routing, and cluster simulation."""
|
||||
|
||||
__version__ = "1.7.0rc1"
|
||||
__version__ = "1.7.4"
|
||||
|
||||
@@ -101,6 +101,73 @@ def _cmd_create_user(args: argparse.Namespace) -> None:
|
||||
print(" (Save this token now — it cannot be retrieved again)")
|
||||
|
||||
|
||||
def _cmd_create_admin(args: argparse.Namespace) -> None:
|
||||
"""Create an admin user (or promote an existing one) with full access.
|
||||
|
||||
Unlike ``create-user`` — which creates a role-less user that logs into the
|
||||
web UI read-only — this assigns the built-in admin role, mirroring the web
|
||||
first-run setup wizard (``POST /api/auth/setup``). Use it for headless
|
||||
installs, or to unstick a ``create-user`` account that logs in only to hit
|
||||
"Forbidden: token lacks 'approve' scope".
|
||||
"""
|
||||
import getpass
|
||||
|
||||
from turnstone.core.auth import hash_password, is_valid_username
|
||||
|
||||
if not is_valid_username(args.username):
|
||||
print("Error: invalid username (1-64 chars: letters, digits, . _ -)", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
|
||||
storage = _get_storage(args)
|
||||
|
||||
# The admin role is seeded by DB migrations; without it we'd leave the
|
||||
# account read-only — the exact lockout this command exists to prevent.
|
||||
if storage.get_role("builtin-admin") is None:
|
||||
print(
|
||||
"Error: the built-in admin role is missing — run database migrations first.",
|
||||
file=sys.stderr,
|
||||
)
|
||||
sys.exit(1)
|
||||
|
||||
# Promote an existing user (recovery path: create-user assigns no role).
|
||||
existing = storage.get_user_by_username(args.username)
|
||||
if existing is not None:
|
||||
user_id = existing["user_id"]
|
||||
already_admin = any(
|
||||
r.get("role_id") == "builtin-admin" for r in storage.list_user_roles(user_id)
|
||||
)
|
||||
storage.assign_role(user_id, "builtin-admin", "")
|
||||
if already_admin:
|
||||
print(f"User '{args.username}' is already an admin (user {user_id}); no change.")
|
||||
else:
|
||||
print(f"Granted the admin role to existing user '{args.username}' (user {user_id}).")
|
||||
print(" Log out and back in for the new access to take effect.")
|
||||
return
|
||||
|
||||
# Create a fresh admin user.
|
||||
password = args.password
|
||||
if not password:
|
||||
password = getpass.getpass("Password: ")
|
||||
confirm = getpass.getpass("Confirm password: ")
|
||||
if password != confirm:
|
||||
print("Error: passwords do not match", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
# Match the web setup wizard's floor for the most privileged account.
|
||||
if len(password) < 8:
|
||||
print("Error: password must be at least 8 characters", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
|
||||
display_name = args.name or args.username
|
||||
user_id = uuid.uuid4().hex
|
||||
pw_hash = hash_password(password)
|
||||
storage.create_user(user_id, args.username, display_name, pw_hash)
|
||||
storage.assign_role(user_id, "builtin-admin", "")
|
||||
print(f"Created admin user: {user_id}")
|
||||
print(f" Username: {args.username}")
|
||||
print(f" Name: {display_name}")
|
||||
print(" Role: admin (full access)")
|
||||
|
||||
|
||||
def _cmd_create_token(args: argparse.Namespace) -> None:
|
||||
from turnstone.core.auth import (
|
||||
generate_token,
|
||||
@@ -596,6 +663,18 @@ def main() -> None:
|
||||
p_cu.add_argument("--token", action="store_true", help="Also create an initial API token")
|
||||
p_cu.add_argument("--scopes", default="read,write,approve", help="Scopes for initial token")
|
||||
|
||||
p_ca = sub.add_parser(
|
||||
"create-admin",
|
||||
help="Create an admin user (or promote an existing one) with full access",
|
||||
)
|
||||
p_ca.add_argument("--username", required=True, help="Login username")
|
||||
p_ca.add_argument("--name", default="", help="Display name (defaults to the username)")
|
||||
p_ca.add_argument(
|
||||
"--password",
|
||||
default="",
|
||||
help="Password (prompted if omitted; ignored when promoting an existing user)",
|
||||
)
|
||||
|
||||
p_ct = sub.add_parser("create-token", help="Create an API token for a user")
|
||||
p_ct.add_argument("--user", required=True, help="User ID")
|
||||
p_ct.add_argument("--name", default="", help="Human label for the token")
|
||||
@@ -690,6 +769,7 @@ def main() -> None:
|
||||
|
||||
dispatch = {
|
||||
"create-user": _cmd_create_user,
|
||||
"create-admin": _cmd_create_admin,
|
||||
"create-token": _cmd_create_token,
|
||||
"list-users": _cmd_list_users,
|
||||
"list-tokens": _cmd_list_tokens,
|
||||
|
||||
@@ -151,6 +151,10 @@ class ConsoleCreateWsRequest(BaseModel):
|
||||
default="",
|
||||
description="Persona slug; resolved and snapshotted at creation, empty = kind default",
|
||||
)
|
||||
project_id: str = Field(
|
||||
default="",
|
||||
description="Project to attach the workstream to (validated against membership, empty = none)",
|
||||
)
|
||||
resume_ws: str = Field(
|
||||
default="", description="Workstream ID to resume (loads previous conversation)"
|
||||
)
|
||||
|
||||
@@ -195,6 +195,8 @@ class CreateScheduleRequest(BaseModel):
|
||||
auto_approve: bool = Field(default=False)
|
||||
auto_approve_tools: list[str] = Field(default_factory=list)
|
||||
skill: str = Field(default="", description="Skill name (replaces default skills)")
|
||||
persona: str = Field(default="", description="Persona slug (empty = kind default)")
|
||||
project_id: str = Field(default="", description="Project to attach the workstream to")
|
||||
notify_targets: list[dict[str, str]] = Field(
|
||||
default_factory=list,
|
||||
description="Notification targets on completion (channel_type + channel_id/user_id)",
|
||||
@@ -216,6 +218,8 @@ class UpdateScheduleRequest(BaseModel):
|
||||
auto_approve: bool | None = None
|
||||
auto_approve_tools: list[str] | None = None
|
||||
skill: str | None = None
|
||||
persona: str | None = None
|
||||
project_id: str | None = None
|
||||
notify_targets: list[dict[str, str]] | None = None
|
||||
enabled: bool | None = None
|
||||
|
||||
@@ -235,6 +239,8 @@ class ScheduleInfo(BaseModel):
|
||||
auto_approve: bool = False
|
||||
auto_approve_tools: list[str] = Field(default_factory=list)
|
||||
skill: str = ""
|
||||
persona: str = ""
|
||||
project_id: str = ""
|
||||
notify_targets: list[dict[str, str]] = Field(default_factory=list)
|
||||
enabled: bool = True
|
||||
created_by: str = ""
|
||||
|
||||
@@ -224,6 +224,17 @@ class CreateWorkstreamResponse(BaseModel):
|
||||
"/v1/api/workstreams/{ws_id}/send."
|
||||
),
|
||||
)
|
||||
initial_message_status: Literal["queue_full", "refused_closed"] | None = Field(
|
||||
default=None,
|
||||
description=(
|
||||
"Present ONLY when the workstream was created but its "
|
||||
"initial_message could not be delivered: 'queue_full' (a raced "
|
||||
"live worker's interjection queue was at capacity — resend via "
|
||||
"/send; any uploads stay staged) or 'refused_closed' (the "
|
||||
"workstream was closed mid-create). Absent whenever the message "
|
||||
"was dispatched."
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
class CloseWorkstreamRequest(BaseModel):
|
||||
|
||||
+43
-4
@@ -7,6 +7,7 @@ model auto-detection, workstream management, and the main() REPL entry point.
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import contextlib
|
||||
import logging
|
||||
import os
|
||||
import readline
|
||||
@@ -277,7 +278,11 @@ class TerminalUI(SessionUI):
|
||||
output: str,
|
||||
*,
|
||||
is_error: bool = False,
|
||||
preview: dict[str, Any] | None = None,
|
||||
) -> None:
|
||||
# ``preview`` renders nowhere in a terminal -- the result line already
|
||||
# names what was shown, and the header carries the target for the
|
||||
# operator to open themselves.
|
||||
if is_error:
|
||||
with self._print_lock:
|
||||
sys.stderr.write(f"{RED}\u2717 {name}: {output}{RESET}\n")
|
||||
@@ -479,9 +484,10 @@ class WorkstreamTerminalUI(TerminalUI):
|
||||
output: str,
|
||||
*,
|
||||
is_error: bool = False,
|
||||
preview: dict[str, Any] | None = None,
|
||||
) -> None:
|
||||
if self.is_foreground:
|
||||
super().on_tool_result(call_id, name, output, is_error=is_error)
|
||||
super().on_tool_result(call_id, name, output, is_error=is_error, preview=preview)
|
||||
|
||||
def on_tool_output_chunk(self, call_id: str, chunk: str) -> None:
|
||||
if self.is_foreground:
|
||||
@@ -915,6 +921,41 @@ def resolve_cli_persona_kwargs(
|
||||
return {}
|
||||
|
||||
|
||||
def _close_all_sessions(manager: SessionManager) -> None:
|
||||
"""Close EVERY loaded session at CLI exit — not just the active one.
|
||||
|
||||
``ChatSession.close()`` removes MCP listeners AND reaps the workstream's
|
||||
background shells (#817). An active-only close would let a dev server
|
||||
started in workstream 1 survive ``/new`` + ``/exit`` forever: its
|
||||
detached process group outlives this process — the exact leaked-server
|
||||
class #816 removed.
|
||||
|
||||
Two phases so total exit latency doesn't stack per workstream: the kill
|
||||
signals land on EVERY session's shells first (microseconds each — after
|
||||
which nothing can outlive us), then the per-session closes pay their
|
||||
join budgets, which are near-zero once the kills have landed. A Ctrl-C
|
||||
during the close phase degrades gracefully instead of aborting the
|
||||
sweep: the signals are already delivered, the remaining joins are
|
||||
skipped, and the caller still runs MCP/registry shutdown. Best-effort
|
||||
per workstream either way — one bad teardown must not stop the rest.
|
||||
"""
|
||||
loaded = [(ws.id, ws.session) for ws in manager.list_all() if ws.session is not None]
|
||||
for _ws_id, session in loaded:
|
||||
with contextlib.suppress(Exception):
|
||||
session._background_shells.signal_all()
|
||||
try:
|
||||
for ws_id, session in loaded:
|
||||
try:
|
||||
session.close()
|
||||
except Exception:
|
||||
print(dim(f" (workstream {ws_id[:8]} teardown error, continuing)"))
|
||||
except KeyboardInterrupt:
|
||||
# The kills above already landed; skipping the remaining joins
|
||||
# leaks nothing — it only abandons wedged drain threads that die
|
||||
# with this process anyway.
|
||||
print(dim(" (interrupted — background shells already signalled)"))
|
||||
|
||||
|
||||
def main() -> None:
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Interactive CLI for OpenAI-compatible models with tool calling.",
|
||||
@@ -1384,9 +1425,7 @@ def main() -> None:
|
||||
except Exception as e:
|
||||
print(f"\n{red(f'Error: {e}')}")
|
||||
|
||||
# Close active session (removes MCP listener) before shutting down MCP
|
||||
if active and active.session:
|
||||
active.session.close()
|
||||
_close_all_sessions(manager)
|
||||
if mcp_client:
|
||||
mcp_client.shutdown()
|
||||
registry.shutdown()
|
||||
|
||||
@@ -318,6 +318,8 @@ class TaskScheduler:
|
||||
auto_approve_tools=",".join(self._parse_tools(task)),
|
||||
user_id=task.get("created_by", ""),
|
||||
skill=task.get("skill", ""),
|
||||
persona=task.get("persona", ""),
|
||||
project_id=task.get("project_id", ""),
|
||||
notify_targets=task.get("notify_targets", "[]"),
|
||||
# Mark the resulting ChatSession as non-interactive-for-
|
||||
# consent so OAuth-MCP errors get persisted to
|
||||
|
||||
+117
-1
@@ -3154,6 +3154,25 @@ async def proxy_non_api(request: Request) -> Response:
|
||||
return await _proxy_get(request, server_url, path)
|
||||
|
||||
|
||||
# Upstream response headers the generic proxy must carry through. These are
|
||||
# the node's hardening + disposition headers: dropping Content-Security-Policy
|
||||
# would serve previewed attacker HTML from the CONSOLE origin with no CSP
|
||||
# sandbox — opened top-level, its scripts would run with the operator's
|
||||
# console cookies, where the same bytes on the node origin are inert. The
|
||||
# rendezvous attachment proxy (route_attachment_proxy) already preserves
|
||||
# these; the /node/{id} lane must match.
|
||||
_PROXY_PASS_HEADERS = (
|
||||
"content-security-policy",
|
||||
"x-content-type-options",
|
||||
"content-disposition",
|
||||
"cache-control",
|
||||
)
|
||||
|
||||
|
||||
def _proxy_pass_headers(resp: httpx.Response) -> dict[str, str]:
|
||||
return {h: resp.headers[h] for h in _PROXY_PASS_HEADERS if h in resp.headers}
|
||||
|
||||
|
||||
async def _proxy_get(request: Request, server_url: str, path: str) -> Response:
|
||||
"""Forward a GET request to the target server."""
|
||||
client: httpx.AsyncClient = request.app.state.proxy_client
|
||||
@@ -3166,6 +3185,7 @@ async def _proxy_get(request: Request, server_url: str, path: str) -> Response:
|
||||
content=resp.content,
|
||||
status_code=resp.status_code,
|
||||
media_type=resp.headers.get("content-type", "application/json"),
|
||||
headers=_proxy_pass_headers(resp),
|
||||
)
|
||||
except httpx.HTTPError as exc:
|
||||
log.debug("Proxy GET error for %s: %s", target, exc)
|
||||
@@ -6075,6 +6095,43 @@ def _validate_schedule_fields(schedule_type: str, cron_expr: str, at_time: str)
|
||||
return None
|
||||
|
||||
|
||||
def _resolve_schedule_persona(storage: Any, persona: str) -> tuple[str, str | None]:
|
||||
"""Validate a schedule's persona slug against the interactive kind.
|
||||
|
||||
Schedules dispatch interactive workstreams, so the persona is resolved
|
||||
against that kind — the same eligibility rule the create handler applies,
|
||||
surfaced here so a bad slug fails at edit time rather than silently at the
|
||||
next firing. Returns ``(canonical_slug, None)`` on success (the resolved
|
||||
row's name, never the raw input, per the persona contract) or
|
||||
``("", error)`` on failure. Empty persona = kind default, always valid.
|
||||
"""
|
||||
if not persona:
|
||||
return "", None
|
||||
from turnstone.core.personas import resolve_persona_for_kind
|
||||
|
||||
row, err = resolve_persona_for_kind(storage, persona, "interactive")
|
||||
if err:
|
||||
return "", err
|
||||
return (str(row["name"]) if row else persona), None
|
||||
|
||||
|
||||
def _validate_schedule_project(
|
||||
storage: Any, user_id: str, project_id: str
|
||||
) -> tuple[int, str] | None:
|
||||
"""Gate attaching a schedule's dispatched workstream to *project_id*.
|
||||
|
||||
Checked against *user_id* — the schedule's ``created_by``, the identity the
|
||||
scheduler dispatches under — so the same owner/member rule the node enforces
|
||||
at dispatch is applied up front. Returns ``None`` when allowed, else the
|
||||
``(status, message)`` to surface. Empty project_id = no attach, allowed.
|
||||
"""
|
||||
if not project_id:
|
||||
return None
|
||||
from turnstone.core.auth import ensure_project_attachable
|
||||
|
||||
return ensure_project_attachable(user_id, project_id, storage=storage)
|
||||
|
||||
|
||||
async def admin_preview_schedule(request: Request) -> JSONResponse:
|
||||
"""POST /v1/api/admin/schedules/preview — validate timing, return next runs.
|
||||
|
||||
@@ -6168,7 +6225,17 @@ async def admin_create_schedule(request: Request) -> JSONResponse:
|
||||
raw_tools = body.get("auto_approve_tools", [])
|
||||
auto_approve_tools = raw_tools if isinstance(raw_tools, list) else []
|
||||
skill_name = str(body.get("skill", "")).strip()[:256]
|
||||
persona = str(body.get("persona", "")).strip()[:64]
|
||||
project_id = str(body.get("project_id", "")).strip()[:64]
|
||||
enabled = bool(body.get("enabled", True))
|
||||
# created_by is the authenticated admin — read via auth_result like every
|
||||
# other console endpoint (AuthMiddleware never sets request.state.user_id,
|
||||
# so the previous ``state.user_id`` read silently stored ""). It is now
|
||||
# load-bearing: the scheduler dispatches under this identity and the node
|
||||
# gates the project attach against it, so an empty value would make every
|
||||
# project-scoped schedule fail the attach.
|
||||
auth_result = getattr(getattr(request, "state", None), "auth_result", None)
|
||||
created_by = getattr(auth_result, "user_id", "") or ""
|
||||
|
||||
# Validate notify_targets
|
||||
from turnstone.server import _validate_notify_targets
|
||||
@@ -6188,6 +6255,13 @@ async def admin_create_schedule(request: Request) -> JSONResponse:
|
||||
return JSONResponse({"error": "initial_message is required"}, status_code=400)
|
||||
if skill_name and not storage.get_prompt_template_by_name(skill_name):
|
||||
return JSONResponse({"error": f"Skill not found: {skill_name}"}, status_code=400)
|
||||
persona, persona_err = _resolve_schedule_persona(storage, persona)
|
||||
if persona_err:
|
||||
return JSONResponse({"error": persona_err}, status_code=400)
|
||||
project_denied = _validate_schedule_project(storage, created_by, project_id)
|
||||
if project_denied is not None:
|
||||
status_code, message = project_denied
|
||||
return JSONResponse({"error": message}, status_code=status_code)
|
||||
|
||||
validation_err = _validate_schedule_fields(schedule_type, cron_expr, at_time)
|
||||
if validation_err:
|
||||
@@ -6206,7 +6280,6 @@ async def admin_create_schedule(request: Request) -> JSONResponse:
|
||||
|
||||
next_run = _compute_next_run(schedule_type, cron_expr, at_time)
|
||||
task_id = uuid.uuid4().hex
|
||||
created_by = getattr(getattr(request, "state", None), "user_id", "")
|
||||
|
||||
storage.create_scheduled_task(
|
||||
task_id=task_id,
|
||||
@@ -6224,6 +6297,8 @@ async def admin_create_schedule(request: Request) -> JSONResponse:
|
||||
next_run=next_run if enabled else "",
|
||||
skill=skill_name,
|
||||
notify_targets=notify_targets,
|
||||
persona=persona,
|
||||
project_id=project_id,
|
||||
)
|
||||
|
||||
if not enabled:
|
||||
@@ -6303,6 +6378,47 @@ async def admin_update_schedule(request: Request) -> JSONResponse:
|
||||
if skill_val and not storage.get_prompt_template_by_name(skill_val):
|
||||
return JSONResponse({"error": f"Skill not found: {skill_val}"}, status_code=400)
|
||||
updates["skill"] = skill_val
|
||||
if "persona" in body:
|
||||
persona_val = str(body["persona"]).strip()[:64]
|
||||
# Re-validate only when the persona actually changes: the edit shelf
|
||||
# always resends the current slug, and a schedule whose persona was
|
||||
# since disabled (or the picker couldn't show) must stay editable for
|
||||
# its other fields. A stale persona still fails loudly at dispatch,
|
||||
# where the node re-resolves it.
|
||||
if persona_val != (existing.get("persona") or ""):
|
||||
persona_val, persona_err = _resolve_schedule_persona(storage, persona_val)
|
||||
if persona_err:
|
||||
return JSONResponse({"error": persona_err}, status_code=400)
|
||||
updates["persona"] = persona_val
|
||||
if "project_id" in body:
|
||||
project_val = str(body["project_id"]).strip()[:64]
|
||||
# Re-gate only when the project actually changes: the edit shelf
|
||||
# resends the current value, and membership churn (or a project the
|
||||
# editing admin can't see) must not block unrelated edits — the node
|
||||
# re-gates against the owner at dispatch. On an actual change, the
|
||||
# schedule dispatches under created_by and the node gates the attach
|
||||
# against it, so a schedule created before the created_by fix ("")
|
||||
# could never attach. When a project is assigned to such an orphaned
|
||||
# schedule, adopt the editing admin as its owner (never overriding a
|
||||
# real created_by) and persist it so the create-time check and the
|
||||
# dispatch identity agree.
|
||||
if project_val != (existing.get("project_id") or ""):
|
||||
editing_admin = (
|
||||
getattr(
|
||||
getattr(getattr(request, "state", None), "auth_result", None),
|
||||
"user_id",
|
||||
"",
|
||||
)
|
||||
or ""
|
||||
)
|
||||
owner = existing.get("created_by", "") or editing_admin
|
||||
project_denied = _validate_schedule_project(storage, owner, project_val)
|
||||
if project_denied is not None:
|
||||
status_code, message = project_denied
|
||||
return JSONResponse({"error": message}, status_code=status_code)
|
||||
if project_val and not existing.get("created_by", ""):
|
||||
updates["created_by"] = owner
|
||||
updates["project_id"] = project_val
|
||||
if "enabled" in body:
|
||||
updates["enabled"] = bool(body["enabled"])
|
||||
if "notify_targets" in body:
|
||||
|
||||
@@ -1329,8 +1329,10 @@ function _populateScheduleSelect(selectId, url, labelKey, valueKey, opts) {
|
||||
.then(function (data) {
|
||||
const temp = sel.querySelector("[data-temporary]");
|
||||
if (temp) temp.remove();
|
||||
const items = opts && opts.listKey ? data[opts.listKey] : data;
|
||||
let items = opts && opts.listKey ? data[opts.listKey] : data;
|
||||
if (!Array.isArray(items)) return;
|
||||
if (opts && typeof opts.filter === "function")
|
||||
items = items.filter(opts.filter);
|
||||
items.forEach(function (item) {
|
||||
const opt = document.createElement("option");
|
||||
opt.value = item[valueKey];
|
||||
@@ -1338,7 +1340,22 @@ function _populateScheduleSelect(selectId, url, labelKey, valueKey, opts) {
|
||||
opts && opts.display ? opts.display(item) : item[labelKey];
|
||||
sel.appendChild(opt);
|
||||
});
|
||||
if (opts && opts.selected) sel.value = opts.selected;
|
||||
if (opts && opts.selected) {
|
||||
sel.value = opts.selected;
|
||||
if (sel.value !== opts.selected) {
|
||||
// The current value isn't in this list — it was filtered out
|
||||
// (a disabled/wrong-kind persona) or is outside the caller-scoped
|
||||
// feed (a private/archived project the editing admin can't see).
|
||||
// Re-add it as a "(current)" option so it round-trips; without this
|
||||
// the select falls back to the placeholder and saving an unrelated
|
||||
// field would silently CLEAR the setting.
|
||||
const keep = document.createElement("option");
|
||||
keep.value = opts.selected;
|
||||
keep.textContent = opts.selected + " (current)";
|
||||
sel.appendChild(keep);
|
||||
sel.value = opts.selected;
|
||||
}
|
||||
}
|
||||
// Caller hook for placeholder annotation / other post-load tweaks.
|
||||
// Used by the schedule modals to rewrite the bare "Default model"
|
||||
// placeholder with the resolved alias so the label matches the
|
||||
@@ -1923,7 +1940,12 @@ function _schResetForm() {
|
||||
_schSetMode("daily");
|
||||
}
|
||||
|
||||
function _schPopulateSelects(selectedModel, selectedSkill) {
|
||||
function _schPopulateSelects(
|
||||
selectedModel,
|
||||
selectedSkill,
|
||||
selectedPersona,
|
||||
selectedProject,
|
||||
) {
|
||||
_populateScheduleSelect("sch-model", "/v1/api/models", "alias", "alias", {
|
||||
listKey: "models",
|
||||
selected: selectedModel || "",
|
||||
@@ -1945,6 +1967,32 @@ function _schPopulateSelects(selectedModel, selectedSkill) {
|
||||
},
|
||||
},
|
||||
);
|
||||
// Schedules dispatch interactive workstreams, so only offer personas
|
||||
// eligible for that kind; the label matches the home/create picker
|
||||
// (display name, falling back to the slug).
|
||||
_populateScheduleSelect("sch-persona", "/v1/api/personas", "name", "name", {
|
||||
listKey: "personas",
|
||||
selected: selectedPersona || "",
|
||||
filter: function (p) {
|
||||
return (p.applies_to_kinds || []).indexOf("interactive") !== -1;
|
||||
},
|
||||
display: function (p) {
|
||||
return p.display_name || p.name;
|
||||
},
|
||||
});
|
||||
_populateScheduleSelect(
|
||||
"sch-project",
|
||||
"/v1/api/projects",
|
||||
"name",
|
||||
"project_id",
|
||||
{
|
||||
listKey: "projects",
|
||||
selected: selectedProject || "",
|
||||
display: function (p) {
|
||||
return p.name;
|
||||
},
|
||||
},
|
||||
);
|
||||
}
|
||||
|
||||
function _schOpen(title, tag, kind, submitLabel) {
|
||||
@@ -1962,7 +2010,7 @@ function showCreateScheduleModal() {
|
||||
_schWire();
|
||||
_schResetForm();
|
||||
document.getElementById("sch-enabled-row").hidden = true;
|
||||
_schPopulateSelects("", "");
|
||||
_schPopulateSelects("", "", "", "");
|
||||
_schOpen("New schedule", "SCH-NEW", "create", "Create");
|
||||
}
|
||||
|
||||
@@ -1991,7 +2039,12 @@ function showEditScheduleModal(taskId) {
|
||||
? s.target_mode
|
||||
: "";
|
||||
document.getElementById("sch-node-group").hidden = !isSpecificNode;
|
||||
_schPopulateSelects(s.model || "", s.skill || "");
|
||||
_schPopulateSelects(
|
||||
s.model || "",
|
||||
s.skill || "",
|
||||
s.persona || "",
|
||||
s.project_id || "",
|
||||
);
|
||||
document.getElementById("sch-message").value = s.initial_message || "";
|
||||
document.getElementById("sch-autoapprove").checked = !!s.auto_approve;
|
||||
document.getElementById("sch-enabled").checked = !!s.enabled;
|
||||
@@ -2036,6 +2089,8 @@ function _submitScheduleShelf() {
|
||||
target_mode: targetMode,
|
||||
model: (document.getElementById("sch-model").value || "").trim(),
|
||||
skill: (document.getElementById("sch-template").value || "").trim(),
|
||||
persona: (document.getElementById("sch-persona").value || "").trim(),
|
||||
project_id: (document.getElementById("sch-project").value || "").trim(),
|
||||
initial_message: message,
|
||||
auto_approve: document.getElementById("sch-autoapprove").checked,
|
||||
notify_targets: _collectNotifyTargets("sch"),
|
||||
@@ -6130,7 +6185,7 @@ let _modelDefaultAlias = "";
|
||||
// and are re-merged on save. Reset per modal open.
|
||||
let _rerankCalFields = {};
|
||||
|
||||
// Capability tile matrix — sparse-override semantics. The 9 tiles display
|
||||
// Capability tile matrix — sparse-override semantics. The tiles display
|
||||
// merge(dataclass defaults, known-model table baseline, explicit overrides);
|
||||
// only EXPLICIT keys persist (saved keys + tiles the user toggled), so a
|
||||
// known model keeps tracking future table updates instead of being pinned.
|
||||
@@ -6142,6 +6197,8 @@ const _MODEL_CAP_KEYS = [
|
||||
"supports_web_search",
|
||||
"supports_temperature",
|
||||
"supports_effort",
|
||||
"supports_verbosity",
|
||||
"supports_pro_mode",
|
||||
"supports_transcription",
|
||||
"supports_speech_synthesis",
|
||||
"supports_audio_input",
|
||||
@@ -6155,6 +6212,8 @@ const _MODEL_CAP_DEFAULTS = {
|
||||
supports_web_search: false,
|
||||
supports_temperature: true,
|
||||
supports_effort: false,
|
||||
supports_verbosity: false,
|
||||
supports_pro_mode: false,
|
||||
supports_transcription: false,
|
||||
supports_speech_synthesis: false,
|
||||
supports_audio_input: false,
|
||||
@@ -6178,6 +6237,152 @@ function _modelRenderTiles() {
|
||||
else if (k in _modelCapsBaseline) el.checked = !!_modelCapsBaseline[k];
|
||||
else el.checked = _MODEL_CAP_DEFAULTS[k];
|
||||
});
|
||||
_updateModelResponseControls();
|
||||
}
|
||||
|
||||
const _MODEL_RESPONSE_CONTROLS = [
|
||||
{
|
||||
key: "verbosity",
|
||||
supportKey: "supports_verbosity",
|
||||
elementId: "model-output-verbosity",
|
||||
fieldId: "model-output-verbosity-field",
|
||||
values: ["low", "medium", "high"],
|
||||
},
|
||||
{
|
||||
key: "reasoning_mode",
|
||||
supportKey: "supports_pro_mode",
|
||||
elementId: "model-reasoning-mode",
|
||||
fieldId: "model-reasoning-mode-field",
|
||||
values: ["standard", "pro"],
|
||||
},
|
||||
];
|
||||
let _modelResponseInitialIdentity = "";
|
||||
let _modelResponseCurrentIdentity = "";
|
||||
let _modelResponseCaptured = {};
|
||||
let _modelResponseDirty = {};
|
||||
|
||||
function _modelIdentity() {
|
||||
const provider = document.getElementById("model-provider").value;
|
||||
const model = document.getElementById("model-name").value.trim();
|
||||
const surface =
|
||||
provider === "openai-compatible"
|
||||
? document.getElementById("model-api-surface").value
|
||||
: "";
|
||||
return provider + "\n" + model + "\n" + surface;
|
||||
}
|
||||
|
||||
function _modelUsesResponsesSurface() {
|
||||
const provider = document.getElementById("model-provider").value;
|
||||
if (provider === "openai") return true;
|
||||
return (
|
||||
provider === "openai-compatible" &&
|
||||
document.getElementById("model-api-surface").value === "responses"
|
||||
);
|
||||
}
|
||||
|
||||
function _modelResponseValueValid(spec, value) {
|
||||
return typeof value === "string" && spec.values.indexOf(value) !== -1;
|
||||
}
|
||||
|
||||
function _updateModelResponseControls() {
|
||||
const group = document.getElementById("model-response-controls");
|
||||
if (!group) return;
|
||||
const responseSurface = _modelUsesResponsesSurface();
|
||||
const sameIdentity =
|
||||
_modelResponseInitialIdentity &&
|
||||
_modelIdentity() === _modelResponseInitialIdentity;
|
||||
let anyVisible = false;
|
||||
_MODEL_RESPONSE_CONTROLS.forEach(function (spec) {
|
||||
const field = document.getElementById(spec.fieldId);
|
||||
const select = document.getElementById(spec.elementId);
|
||||
if (!field || !select) return;
|
||||
// A value _captureModelResponseControls lifted out of the row's JSON
|
||||
// stays visible (and re-saveable) while the identity still matches the
|
||||
// row being edited, deliberately NOT consulting the capability
|
||||
// baseline: the baseline arrives async (or never, on the compat lane),
|
||||
// and yielding to it would hide the pinned value and silently drop it
|
||||
// on save — the same lift-then-restore contract as server_compat,
|
||||
// rerank calibration, and thinking_param. Wire safety is server-side:
|
||||
// emission gates on the merged supports_* flag, so a pinned value on
|
||||
// an unsupported model is inert; "Provider default" explicitly clears
|
||||
// it. An explicit tile override (either polarity) supersedes the
|
||||
// fallback — unchecking the tile is the operator's way to retire it.
|
||||
const capturedFallback =
|
||||
sameIdentity &&
|
||||
!(spec.supportKey in _modelCapsExplicit) &&
|
||||
_modelResponseValueValid(spec, select.value);
|
||||
const visible =
|
||||
responseSurface && (_modelGetTile(spec.supportKey) || capturedFallback);
|
||||
field.hidden = !visible;
|
||||
anyVisible = anyVisible || visible;
|
||||
});
|
||||
group.hidden = !anyVisible;
|
||||
}
|
||||
|
||||
function _resetModelResponseControls() {
|
||||
_MODEL_RESPONSE_CONTROLS.forEach(function (spec) {
|
||||
const select = document.getElementById(spec.elementId);
|
||||
if (select) select.value = "";
|
||||
});
|
||||
_modelResponseInitialIdentity = "";
|
||||
_modelResponseCurrentIdentity = _modelIdentity();
|
||||
_modelResponseCaptured = {};
|
||||
_modelResponseDirty = {};
|
||||
_updateModelResponseControls();
|
||||
}
|
||||
|
||||
function _captureModelResponseControls(capsObj) {
|
||||
if (!_modelUsesResponsesSurface()) return;
|
||||
_MODEL_RESPONSE_CONTROLS.forEach(function (spec) {
|
||||
const select = document.getElementById(spec.elementId);
|
||||
if (!select) return;
|
||||
const explicitlyUnsupported =
|
||||
spec.supportKey in _modelCapsExplicit &&
|
||||
!_modelCapsExplicit[spec.supportKey];
|
||||
const value = capsObj[spec.key];
|
||||
if (!explicitlyUnsupported && _modelResponseValueValid(spec, value)) {
|
||||
select.value = value;
|
||||
_modelResponseCaptured[spec.key] = value;
|
||||
delete capsObj[spec.key];
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
function _mergeModelResponseControls(caps) {
|
||||
if (!_modelUsesResponsesSurface()) return;
|
||||
const sameIdentity =
|
||||
_modelResponseInitialIdentity &&
|
||||
_modelIdentity() === _modelResponseInitialIdentity;
|
||||
_MODEL_RESPONSE_CONTROLS.forEach(function (spec) {
|
||||
// Dirty (select touched this session) lets the select override a
|
||||
// stale JSON key, but only for the identity that made it dirty —
|
||||
// after a model/provider/surface change the flag describes the OLD
|
||||
// row, and honoring it would delete a key hand-typed into the
|
||||
// Advanced JSON for the new one.
|
||||
if (_modelResponseDirty[spec.key] && sameIdentity) delete caps[spec.key];
|
||||
else if (spec.key in caps) return; // Advanced JSON wins.
|
||||
const select = document.getElementById(spec.elementId);
|
||||
if (!select || !_modelResponseValueValid(spec, select.value)) return;
|
||||
// Same capturedFallback contract as _updateModelResponseControls
|
||||
// (rationale there): a lifted same-identity value must re-save, or an
|
||||
// unrelated edit silently drops it from the row.
|
||||
const capturedFallback =
|
||||
sameIdentity && !(spec.supportKey in _modelCapsExplicit);
|
||||
if (_modelGetTile(spec.supportKey) || capturedFallback) {
|
||||
caps[spec.key] = select.value;
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
function _rememberModelResponseControl(spec) {
|
||||
_modelResponseDirty[spec.key] = true;
|
||||
if (_modelIdentity() !== _modelResponseInitialIdentity) return;
|
||||
const select = document.getElementById(spec.elementId);
|
||||
if (select && _modelResponseValueValid(spec, select.value)) {
|
||||
_modelResponseCaptured[spec.key] = select.value;
|
||||
} else {
|
||||
delete _modelResponseCaptured[spec.key];
|
||||
}
|
||||
}
|
||||
|
||||
// Roles surfaced in the Models → Roles sub-tab. Each entry maps a
|
||||
@@ -6713,6 +6918,25 @@ function _renderModels(items) {
|
||||
if (m.max_tokens != null) overrides.push("max_tok=" + m.max_tokens);
|
||||
if (m.reasoning_effort != null)
|
||||
overrides.push("effort=" + m.reasoning_effort);
|
||||
let displayCaps = m.capabilities;
|
||||
if (typeof displayCaps === "string") {
|
||||
try {
|
||||
displayCaps = JSON.parse(displayCaps || "{}");
|
||||
} catch (e) {
|
||||
displayCaps = {};
|
||||
}
|
||||
}
|
||||
if (!_isPlainObject(displayCaps)) displayCaps = {};
|
||||
if (
|
||||
displayCaps.supports_verbosity !== false &&
|
||||
["low", "medium", "high"].indexOf(displayCaps.verbosity) !== -1
|
||||
)
|
||||
overrides.push("verbosity=" + displayCaps.verbosity);
|
||||
if (
|
||||
displayCaps.supports_pro_mode !== false &&
|
||||
["standard", "pro"].indexOf(displayCaps.reasoning_mode) !== -1
|
||||
)
|
||||
overrides.push("mode=" + displayCaps.reasoning_mode);
|
||||
// Reasoning persistence flags surface only when non-default
|
||||
// (persist=False is the operator opt-out; replay=True is the
|
||||
// operator opt-in). Default values are silent.
|
||||
@@ -6930,8 +7154,10 @@ function showCreateModelModal() {
|
||||
if (_calChip) _calChip.style.display = "none";
|
||||
const _recalBtn = document.getElementById("model-recalibrate-btn");
|
||||
if (_recalBtn) _recalBtn.hidden = true;
|
||||
_modelCapsSeq++; // invalidate lookups from a prior shelf lifecycle
|
||||
_modelCapsBaseline = {};
|
||||
_modelCapsExplicit = {};
|
||||
_resetModelResponseControls();
|
||||
_modelRenderTiles();
|
||||
document.getElementById("model-autofill").hidden = true;
|
||||
_refreshModelSuggestions();
|
||||
@@ -7027,7 +7253,7 @@ function showEditModelModal(definitionId) {
|
||||
}
|
||||
},
|
||||
);
|
||||
// Lift the 9 matrix keys out of the JSON into the tiles — they are
|
||||
// Lift the capability keys out of the JSON into the tiles — they are
|
||||
// the row's explicit overrides and the textarea holds the remainder.
|
||||
_modelCapsExplicit = {};
|
||||
_MODEL_CAP_KEYS.forEach(function (k) {
|
||||
@@ -7036,6 +7262,9 @@ function showEditModelModal(definitionId) {
|
||||
delete capsObj[k];
|
||||
}
|
||||
});
|
||||
_modelResponseInitialIdentity = _modelIdentity();
|
||||
_modelResponseCurrentIdentity = _modelResponseInitialIdentity;
|
||||
_captureModelResponseControls(capsObj);
|
||||
_modelRenderTiles();
|
||||
_modelCapsRefreshBaseline();
|
||||
_scheduleEffortLadder();
|
||||
@@ -7204,6 +7433,7 @@ function submitCreateModel() {
|
||||
Object.keys(_modelCapsExplicit).forEach(function (k) {
|
||||
if (!(k in caps)) caps[k] = _modelGetTile(k);
|
||||
});
|
||||
_mergeModelResponseControls(caps);
|
||||
|
||||
// Re-merge reranker calibration fields extracted on edit so an unrelated edit
|
||||
// doesn't silently drop the calibration. A field typed directly into the
|
||||
@@ -7633,12 +7863,34 @@ function recalibrateModel() {
|
||||
}
|
||||
|
||||
/* Capability auto-fill: when the user types a known model name or
|
||||
changes the provider, look up static capabilities and pre-fill
|
||||
context_window and the capabilities textarea. */
|
||||
changes the provider, look up static capabilities and refresh the
|
||||
context window, capability tiles, and conditional response controls. */
|
||||
let _capsTimer = null;
|
||||
let _modelCapsSeq = 0;
|
||||
function _onModelFieldChange() {
|
||||
clearTimeout(_capsTimer);
|
||||
const nextIdentity = _modelIdentity();
|
||||
if (
|
||||
_modelResponseCurrentIdentity &&
|
||||
nextIdentity !== _modelResponseCurrentIdentity
|
||||
) {
|
||||
_MODEL_RESPONSE_CONTROLS.forEach(function (spec) {
|
||||
const select = document.getElementById(spec.elementId);
|
||||
if (!select) return;
|
||||
const captured = _modelResponseCaptured[spec.key];
|
||||
select.value =
|
||||
nextIdentity === _modelResponseInitialIdentity &&
|
||||
_modelResponseValueValid(spec, captured)
|
||||
? captured
|
||||
: "";
|
||||
});
|
||||
}
|
||||
_modelResponseCurrentIdentity = nextIdentity;
|
||||
_modelCapsSeq++; // invalidate any capability lookup already in flight
|
||||
_modelCapsBaseline = {};
|
||||
const banner = document.getElementById("model-autofill");
|
||||
if (banner) banner.hidden = true;
|
||||
_modelRenderTiles();
|
||||
_capsTimer = setTimeout(_modelCapsRefreshBaseline, 500);
|
||||
_scheduleEffortLadder();
|
||||
}
|
||||
@@ -7767,6 +8019,7 @@ function _modelCapsRefreshBaseline() {
|
||||
const provider = document.getElementById("model-provider").value;
|
||||
const modelName = document.getElementById("model-name").value.trim();
|
||||
const banner = document.getElementById("model-autofill");
|
||||
const seq = ++_modelCapsSeq;
|
||||
if (
|
||||
!modelName ||
|
||||
provider === "openai-compatible" ||
|
||||
@@ -7779,7 +8032,6 @@ function _modelCapsRefreshBaseline() {
|
||||
}
|
||||
// Two type-then-pause cycles can have both fetches in flight; a reordered
|
||||
// older response must not clobber the tiles (the _schPreviewSeq pattern).
|
||||
const seq = ++_modelCapsSeq;
|
||||
authFetch(
|
||||
"/v1/api/admin/model-capabilities?provider=" +
|
||||
encodeURIComponent(provider) +
|
||||
@@ -7863,6 +8115,7 @@ function _applyProviderDefaults() {
|
||||
if (serverFieldsRow) {
|
||||
serverFieldsRow.hidden = provider === "anthropic-compatible";
|
||||
}
|
||||
_updateModelResponseControls();
|
||||
}
|
||||
|
||||
/* Populate the model name datalist with known model prefixes for the
|
||||
@@ -7902,14 +8155,26 @@ function _refreshModelSuggestions() {
|
||||
const tmEl = document.getElementById("model-thinking-mode");
|
||||
if (tmEl) tmEl.addEventListener("change", _toggleThinkingParam);
|
||||
if (tmEl) tmEl.addEventListener("change", _scheduleEffortLadder);
|
||||
_MODEL_RESPONSE_CONTROLS.forEach(function (spec) {
|
||||
const select = document.getElementById(spec.elementId);
|
||||
if (select)
|
||||
select.addEventListener("change", function () {
|
||||
_rememberModelResponseControl(spec);
|
||||
});
|
||||
});
|
||||
["model-thinking-param", "model-effort-param", "model-capabilities"].forEach(
|
||||
function (id) {
|
||||
const el = document.getElementById(id);
|
||||
if (el) el.addEventListener("input", _scheduleEffortLadder);
|
||||
},
|
||||
);
|
||||
const rawCapsEl = document.getElementById("model-capabilities");
|
||||
if (rawCapsEl)
|
||||
rawCapsEl.addEventListener("input", function () {
|
||||
_modelResponseDirty = {};
|
||||
});
|
||||
const apiSurfEl = document.getElementById("model-api-surface");
|
||||
if (apiSurfEl) apiSurfEl.addEventListener("change", _scheduleEffortLadder);
|
||||
if (apiSurfEl) apiSurfEl.addEventListener("change", _onModelFieldChange);
|
||||
const grid = document.getElementById("model-capgrid");
|
||||
if (grid) {
|
||||
grid.addEventListener("change", function (e) {
|
||||
@@ -7917,6 +8182,17 @@ function _refreshModelSuggestions() {
|
||||
if (!cap) return;
|
||||
// a toggle IS the override decision — the key persists from here on
|
||||
_modelCapsExplicit[cap] = e.target.checked;
|
||||
if (cap === "supports_verbosity" || cap === "supports_pro_mode") {
|
||||
const spec = _MODEL_RESPONSE_CONTROLS.find(function (item) {
|
||||
return item.supportKey === cap;
|
||||
});
|
||||
if (spec && !e.target.checked) {
|
||||
const select = document.getElementById(spec.elementId);
|
||||
if (select) select.value = "";
|
||||
delete _modelResponseCaptured[spec.key];
|
||||
}
|
||||
_updateModelResponseControls();
|
||||
}
|
||||
if (cap === "supports_rerank") {
|
||||
const recalBtn = document.getElementById("model-recalibrate-btn");
|
||||
if (recalBtn)
|
||||
|
||||
@@ -41,6 +41,15 @@ import {
|
||||
indexLabel,
|
||||
} from "/shared/conversation.js";
|
||||
import { redactCredentials } from "/shared/redact_credentials.js";
|
||||
import {
|
||||
OVERFLOW_TRIP_COUNT,
|
||||
OVERFLOW_TRIP_WINDOW_MS,
|
||||
DEGRADED_COOLDOWN_BASE_MS,
|
||||
DEGRADED_COOLDOWN_MAX_MS,
|
||||
DEGRADED_COOLDOWN_RESET_MS,
|
||||
overflowWindowTripped,
|
||||
degradedCooldownStep,
|
||||
} from "/shared/sse_overflow.js";
|
||||
|
||||
function buildCoordChrome(root, opts) {
|
||||
opts = opts || {};
|
||||
@@ -430,15 +439,40 @@ function createCoordinatorPane(root, wsId, opts) {
|
||||
|
||||
let evtSource = null;
|
||||
let reconnectAttempts = 0;
|
||||
// Flag set in onerror, cleared in onopen. Drives the "did we
|
||||
// just recover from a gap?" decision in onopen so the replace-
|
||||
// mode refresh of children/tasks/wait/badge caches fires on
|
||||
// every reconnect — including the common case where native
|
||||
// EventSource auto-reconnect handles the underlying SSE transition
|
||||
// without scheduleReconnect running (which used to be the only
|
||||
// place reconnectAttempts incremented; that path is rarely hit
|
||||
// now that native reconnect handles transient errors).
|
||||
let disconnectedSinceLastOpen = false;
|
||||
// Wall-clock start of the CURRENT gap (0 = no gap in progress). Stamped by
|
||||
// markStreamGap (from onerror and every deliberate suspend), cleared by
|
||||
// onopen. A non-zero value IS the "did we just recover from a gap?" flag
|
||||
// that drives onopen's replace-mode children/tasks/badge refresh — no
|
||||
// separate boolean is kept in lockstep with it. Kept at the EARLIEST mark
|
||||
// so repeated onerror fires during one outage don't shrink the measured gap.
|
||||
// (The scheduleReconnect-after-CLOSED path bumps reconnectAttempts instead;
|
||||
// wasReconnecting ORs the two — that path is rarely hit now that native
|
||||
// reconnect handles transient errors.)
|
||||
let disconnectedAt = 0;
|
||||
// Reconnect-replay trust window for the sidebar. child_ws_* / task events
|
||||
// are ordinary ring-buffer entries, so a cursor reconnect (replay_ok)
|
||||
// redelivers them and the sidebar heals without any REST refetch; gaps the
|
||||
// ring could NOT cover announce themselves via replay_truncated. The one
|
||||
// blind spot is a stale cursor the server no longer recognises (process
|
||||
// restart resets event ids; the empty/reset ring reports replay_ok and
|
||||
// silently skips the gap). Past this gap length we stop trusting the cursor
|
||||
// and pull authoritative /children + /tasks state; a FASTER restart (under
|
||||
// the threshold) is caught instead by the backwards-event-id check in
|
||||
// onmessage (a live id below our saved cursor == the counter reset).
|
||||
// Momentary blur/focus cycles stay well below the threshold — no rebuild
|
||||
// flicker on an alt-tab.
|
||||
const GAP_REFRESH_THRESHOLD_MS = 60000;
|
||||
// True when THIS connection's onopen already ran refreshSidebarAfterGap —
|
||||
// lets the replay_truncated handler (first frame after open) skip a
|
||||
// back-to-back duplicate of the refresh it would otherwise trigger.
|
||||
let gapRefreshedAtOpen = false;
|
||||
// Set when replay_truncated arrives while a turn is mid-stream (refetching
|
||||
// then would detach the live bubble); consumed on the next state_change=idle.
|
||||
// Mirrors interactive.js's _pendingTruncatedResync — the deferral keeps a
|
||||
// ring-evicted gap from going unrepaired for the rest of the session, and
|
||||
// also catches a turn stranded by close-on-hide (finished while hidden, its
|
||||
// stream_end evicted before the show-edge reconnect).
|
||||
let pendingTruncatedResync = false;
|
||||
// Saved high-water mark for the manual-reconnect path. The
|
||||
// EventSource constructor can't set custom headers, so when we
|
||||
// construct a fresh source we thread ``?last_event_id=N`` instead
|
||||
@@ -449,6 +483,28 @@ function createCoordinatorPane(root, wsId, opts) {
|
||||
// close).
|
||||
let lastEventId = null;
|
||||
let reconnectTimer = null;
|
||||
// --- SSE overflow-recovery state (client half — mirrors interactive.js) ---
|
||||
// Field instrumentation for the two distinct "output stops while the backend
|
||||
// is healthy" causes: server-signalled overflow closes (dropped-events
|
||||
// class) vs client dispatch/render throws (wedge class). The console line
|
||||
// at each increment carries the running count, so a field report shows which
|
||||
// class fired without a debugger attached.
|
||||
const streamHealth = { overflows: 0, renderThrows: 0, malformedFrames: 0 };
|
||||
// Rolling timestamps of stream_overflow closes feeding the degraded-catchup
|
||||
// limiter (overflowWindowTripped); plus the cooldown-ladder state, keyed off
|
||||
// the last-trip timestamp via degradedCooldownStep — never off overflowTimes
|
||||
// (enterDegradedCatchup clears it each trip). See enterDegradedCatchup.
|
||||
const overflowTimes = [];
|
||||
let degradedTimer = null;
|
||||
let degradedCooldownMs = DEGRADED_COOLDOWN_BASE_MS;
|
||||
let lastDegradedAt = 0;
|
||||
// Close-on-hide / replay-on-show bookkeeping. A hidden tab's throttled event
|
||||
// loop is the likeliest too-slow SSE consumer, so the visibilitychange
|
||||
// handler closes the stream on hide and reconnects with the saved lastEventId
|
||||
// on show (replay_ok covers the gap). hiddenDisconnect marks that WE closed
|
||||
// for hide, so show never resurrects a stream closed deliberately elsewhere.
|
||||
let visHandler = null;
|
||||
let hiddenDisconnect = false;
|
||||
// Ids of operator-context system turns already painted from /history. A
|
||||
// later SSE replay that redelivers one (resume-cursor overlap) is skipped
|
||||
// by the system_turn handler — reset per refetchHistory. Mirrors
|
||||
@@ -1681,7 +1737,7 @@ function createCoordinatorPane(root, wsId, opts) {
|
||||
try {
|
||||
streamingRender(body, currentAssistantBuf);
|
||||
} catch (e) {
|
||||
console.warn("coordinator streamingRender failed", e);
|
||||
noteRenderThrow("streamingRender", e);
|
||||
body.textContent = currentAssistantBuf;
|
||||
}
|
||||
} else if (body) {
|
||||
@@ -1719,7 +1775,7 @@ function createCoordinatorPane(root, wsId, opts) {
|
||||
try {
|
||||
streamingRenderFinalize(body, currentAssistantBuf);
|
||||
} catch (e) {
|
||||
console.warn("coordinator streamingRenderFinalize failed", e);
|
||||
noteRenderThrow("streamingRenderFinalize", e);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -2090,11 +2146,13 @@ function createCoordinatorPane(root, wsId, opts) {
|
||||
}
|
||||
};
|
||||
try {
|
||||
if (evtSource) evtSource.close();
|
||||
if (reconnectTimer) {
|
||||
clearTimeout(reconnectTimer);
|
||||
reconnectTimer = null;
|
||||
}
|
||||
suspendStream();
|
||||
// Session teardown owns the stream from here: a tab hide→show while
|
||||
// the /close POST is in flight must NOT reopen a stream against the
|
||||
// workstream the server is tearing down (404 / reconnect churn against
|
||||
// a dead session). connectSSE reinstalls the handler, so the failure
|
||||
// paths below get close-on-hide back for free via resumeSse.
|
||||
removeVisibilityHandler();
|
||||
} catch (_) {
|
||||
/* best-effort suspension */
|
||||
}
|
||||
@@ -2184,32 +2242,47 @@ function createCoordinatorPane(root, wsId, opts) {
|
||||
}
|
||||
}
|
||||
|
||||
function connectSSE() {
|
||||
if (reconnectTimer) {
|
||||
clearTimeout(reconnectTimer);
|
||||
reconnectTimer = null;
|
||||
}
|
||||
// The transport-teardown chokepoint: close + null the EventSource and
|
||||
// cancel BOTH pending retry timers (reconnect backoff + degraded catch-up).
|
||||
// Every teardown path routes through here — connectSSE's redial prologue,
|
||||
// suspendStream (overflow / hide / close-session), destroy — so a new
|
||||
// transport timer gets cancelled in one place instead of by hand at each
|
||||
// call site. Cancelling the degraded timer on redial also keeps a pending
|
||||
// catch-up retry from firing mid-stream and double-opening;
|
||||
// enterDegradedCatchup re-arms AFTER its own suspend, so this never cancels
|
||||
// the timer it is about to set. Gap accounting stays OUT of this helper —
|
||||
// a redial is not itself a gap (suspendStream layers markStreamGap on top).
|
||||
function closeStreamTransport() {
|
||||
if (evtSource) {
|
||||
try {
|
||||
evtSource.close();
|
||||
} catch (_) {
|
||||
/* noop */
|
||||
}
|
||||
evtSource = null;
|
||||
}
|
||||
setSseStatus("connecting…", "");
|
||||
// Snapshot whether this is a reconnect BEFORE resetting
|
||||
// reconnectAttempts in onopen — child_ws_* events dispatched while
|
||||
// we were disconnected aren't replayed by the events SSE handler,
|
||||
// so the client has to pull authoritative state after any gap.
|
||||
// Snapshot whether this connect attempt follows a prior
|
||||
// disconnect. Native EventSource auto-reconnect no longer
|
||||
// routes through scheduleReconnect on the transient-error path,
|
||||
// so the legacy ``reconnectAttempts > 0`` check is always false
|
||||
// after PR-D — use ``disconnectedSinceLastOpen`` (set by onerror,
|
||||
// cleared by onopen below) as the authoritative "was-gap" flag.
|
||||
// Falls back to the legacy semantic for the genuinely manual
|
||||
// case (scheduleReconnect-driven reconnect after CLOSED state).
|
||||
const wasReconnecting = disconnectedSinceLastOpen || reconnectAttempts > 0;
|
||||
if (reconnectTimer) {
|
||||
clearTimeout(reconnectTimer);
|
||||
reconnectTimer = null;
|
||||
}
|
||||
if (degradedTimer) {
|
||||
clearTimeout(degradedTimer);
|
||||
degradedTimer = null;
|
||||
}
|
||||
}
|
||||
|
||||
function connectSSE() {
|
||||
closeStreamTransport();
|
||||
// Snapshot whether this connect attempt follows a prior disconnect
|
||||
// BEFORE onopen resets the flags — onopen's post-gap sidebar recovery
|
||||
// keys off it. Native EventSource auto-reconnect no longer routes
|
||||
// through scheduleReconnect on the transient-error path, so the legacy
|
||||
// ``reconnectAttempts > 0`` check is always false after PR-D — use
|
||||
// ``disconnectedAt`` (stamped by markStreamGap from onerror and from every
|
||||
// deliberate suspend, cleared by onopen below) as the authoritative
|
||||
// "was-gap" flag. Falls back to the legacy semantic for the genuinely
|
||||
// manual case (scheduleReconnect-driven reconnect after CLOSED state).
|
||||
const wasReconnecting = disconnectedAt !== 0 || reconnectAttempts > 0;
|
||||
let url = "/v1/api/workstreams/" + encodeURIComponent(wsId) + "/events";
|
||||
// ``!= null`` (not truthiness): a resume cursor of 0 is valid (the
|
||||
// ring buffer's first emitted event is id 1), and a brand-new ws's
|
||||
@@ -2218,12 +2291,47 @@ function createCoordinatorPane(root, wsId, opts) {
|
||||
if (lastEventId != null) {
|
||||
url += "?last_event_id=" + encodeURIComponent(lastEventId);
|
||||
}
|
||||
// Close-on-hide / replay-on-show: install once per pane, removed by
|
||||
// destroy(). A hidden tab's throttled drain is the likeliest slow consumer
|
||||
// behind a server-side queue overflow, and an idle hidden tab holds a node
|
||||
// connection for nothing — closing on hide removes both, and the saved
|
||||
// lastEventId makes the show-edge reconnect lossless (replay_ok).
|
||||
if (!visHandler) {
|
||||
// onVisibilityChange is a plain closure function (no `this` to bind), so
|
||||
// it serves directly as the once-install sentinel AND the
|
||||
// add/removeEventListener handle — no wrapper needed.
|
||||
visHandler = onVisibilityChange;
|
||||
document.addEventListener("visibilitychange", visHandler);
|
||||
}
|
||||
// Never open an EventSource into a hidden (throttled) tab — including a
|
||||
// FIRST connect in a background tab, where the close-on-hide handler never
|
||||
// fires because there was no open stream to close. This single connect
|
||||
// chokepoint backstops every caller (init, scheduleReconnect, degraded
|
||||
// retry, show edge); the saved lastEventId + the handler installed just
|
||||
// above make the show-edge reconnect replay the gap. The deferral IS a
|
||||
// gap — everything until the show edge is missed exactly as if the
|
||||
// transport had dropped — so mark it like one: without the mark, a pane
|
||||
// first opened in a background tab skipped onopen's post-gap recovery and
|
||||
// the sidebar silently missed every child/task the backend created while
|
||||
// hidden. And say so honestly — "connecting…" (set below, only when an
|
||||
// attempt really starts) used to pin here forever with nothing in flight.
|
||||
if (document.hidden) {
|
||||
markStreamGap();
|
||||
hiddenDisconnect = true;
|
||||
setSseStatus("paused — tab hidden", "");
|
||||
return;
|
||||
}
|
||||
setSseStatus("connecting…", "");
|
||||
evtSource = new EventSource(url, { withCredentials: true });
|
||||
evtSource.onopen = function () {
|
||||
reconnectAttempts = 0;
|
||||
// Clear the "was disconnected" flag now that the gap is
|
||||
// closed. Future onerror fires will set it again.
|
||||
disconnectedSinceLastOpen = false;
|
||||
// Measure the gap this open just closed, then clear it (disconnectedAt is
|
||||
// the was-gap flag). A gap with no start stamp (legacy scheduleReconnect
|
||||
// path) reads as Infinity — unknown length means we can't argue the
|
||||
// replay covered it, so refresh below.
|
||||
const gapMs = disconnectedAt ? Date.now() - disconnectedAt : Infinity;
|
||||
disconnectedAt = 0;
|
||||
gapRefreshedAtOpen = false;
|
||||
setSseStatus("live", "ok");
|
||||
// Lift the disconnected dim treatment + restore the last known
|
||||
// counters; the replay phase will overwrite with authoritative
|
||||
@@ -2236,26 +2344,22 @@ function createCoordinatorPane(root, wsId, opts) {
|
||||
statusBarEl.classList.remove("ws-sb-disconnected");
|
||||
if (lastStatusEvt) updateStatusBar(lastStatusEvt);
|
||||
else StatusBar.resetTokensPlaceholder(sbTokensEl);
|
||||
if (wasReconnecting) {
|
||||
// Replace-mode refresh: the server is authoritative after a
|
||||
// gap; any SSE-only rows the client accumulated before
|
||||
// disconnect are stale.
|
||||
loadChildren({ replace: true });
|
||||
loadTasks();
|
||||
// Drop the live-badge cache too — entries within the 5s TTL
|
||||
// can carry stale pending_approval_details (the child may
|
||||
// have resolved its approval during the SSE gap). Without
|
||||
// this clear, inline approve/deny buttons could render on
|
||||
// a row whose approval was resolved elsewhere; the next
|
||||
// scheduleLiveFetch from loadChildren's finally branch
|
||||
// (which fires for every visible row) repopulates with
|
||||
// authoritative state. Preserve `permanent: true` entries
|
||||
// (set on 403/404 — denied by permission/identity, not by
|
||||
// state) so a user lacking admin.cluster.inspect doesn't
|
||||
// pay one 403 per denied id on every reconnect.
|
||||
for (const [id, c] of liveBadgeCache) {
|
||||
if (!c || !c.permanent) _liveBadgeCacheDelete(id);
|
||||
}
|
||||
// Post-gap sidebar recovery. child_ws_* / task-mutating events are
|
||||
// ordinary ring-buffer entries, so a cursor reconnect (replay_ok)
|
||||
// redelivers them and the normal handlers heal the sidebar — no REST
|
||||
// refetch, no replace-mode rebuild flicker on a momentary blur/focus.
|
||||
// Refresh eagerly only when the replay CANNOT vouch for the gap: no
|
||||
// cursor to resume from (the fresh path's synthetic replay carries no
|
||||
// child events), or a gap long enough that a stale cursor could be
|
||||
// lying (see GAP_REFRESH_THRESHOLD_MS). The ring-evicted case
|
||||
// announces itself — the replay_truncated handler runs the same
|
||||
// refresh on arrival (gapRefreshedAtOpen keeps the two from stacking).
|
||||
if (
|
||||
wasReconnecting &&
|
||||
(lastEventId == null || gapMs > GAP_REFRESH_THRESHOLD_MS)
|
||||
) {
|
||||
refreshSidebarAfterGap();
|
||||
gapRefreshedAtOpen = true;
|
||||
}
|
||||
};
|
||||
evtSource.onerror = function () {
|
||||
@@ -2267,7 +2371,7 @@ function createCoordinatorPane(root, wsId, opts) {
|
||||
// reconnect, which is exactly the reconnect-with-replay defect
|
||||
// PR-D ships to fix. See
|
||||
// tests/test_app_js.py::test_coord_connectsse_onerror_preserves_native_reconnect.
|
||||
disconnectedSinceLastOpen = true;
|
||||
markStreamGap();
|
||||
setSseStatus("disconnected", "err");
|
||||
// Dim the status bar so a stale reading doesn't read as live.
|
||||
statusBarEl.classList.add("ws-sb-disconnected");
|
||||
@@ -2362,18 +2466,59 @@ function createCoordinatorPane(root, wsId, opts) {
|
||||
// doesn't desync the manual-reconnect fallback from native
|
||||
// auto-reconnect.
|
||||
if (evtSource && evtSource.lastEventId) {
|
||||
// A live event id BELOW our saved cursor means the server's per-ws
|
||||
// event counter reset — a coordinator process restart with a fresh,
|
||||
// empty ring. The replay path can't flag that (a cursor at/above the
|
||||
// ring's earliest id reports replay_ok even when it's past the new
|
||||
// max), so the gap is silent and the sidebar's pre-restart rows go
|
||||
// stale with nothing to replay them. Catch it here and pull
|
||||
// authoritative state — deduped per open against onopen's /
|
||||
// replay_truncated's own refresh. (Gaps that DON'T reset the counter
|
||||
// are handled at onopen by the cursor-trust window.)
|
||||
if (
|
||||
lastEventId != null &&
|
||||
!gapRefreshedAtOpen &&
|
||||
Number(evtSource.lastEventId) < Number(lastEventId)
|
||||
) {
|
||||
refreshSidebarAfterGap();
|
||||
gapRefreshedAtOpen = true;
|
||||
}
|
||||
lastEventId = evtSource.lastEventId;
|
||||
}
|
||||
let data = null;
|
||||
try {
|
||||
data = JSON.parse(event.data);
|
||||
} catch (_) {
|
||||
} catch (err) {
|
||||
streamHealth.malformedFrames += 1;
|
||||
console.warn(
|
||||
"coordinator: dropping malformed SSE frame (total " +
|
||||
streamHealth.malformedFrames +
|
||||
")",
|
||||
err,
|
||||
);
|
||||
return;
|
||||
}
|
||||
// Tag the event with its own SSE id so the system_turn handler can dedup
|
||||
// a turn already painted from /history (mirrors ui/static/app.js).
|
||||
if (event.lastEventId) data._event_id = event.lastEventId;
|
||||
handleEvent(data);
|
||||
// Guard the dispatch: an exception escaping onmessage does NOT close the
|
||||
// EventSource, so an unhandled throw here leaves the streaming refs stale
|
||||
// and every later turn paints into the poisoned segment — the "output
|
||||
// stops while the backend is healthy" wedge. Count it (render-throw
|
||||
// class) so a field report tells it apart from a dropped-events gap.
|
||||
try {
|
||||
handleEvent(data);
|
||||
} catch (err) {
|
||||
streamHealth.renderThrows += 1;
|
||||
console.error(
|
||||
"coordinator: handleEvent failed for " +
|
||||
(data && data.type) +
|
||||
" (render-throw total " +
|
||||
streamHealth.renderThrows +
|
||||
")",
|
||||
err,
|
||||
);
|
||||
}
|
||||
};
|
||||
}
|
||||
|
||||
@@ -2384,6 +2529,168 @@ function createCoordinatorPane(root, wsId, opts) {
|
||||
reconnectTimer = setTimeout(connectSSE, base + jitter);
|
||||
}
|
||||
|
||||
// ------------------------------------------------------------------
|
||||
// SSE overflow recovery (client half) — mirrors interactive.js. The
|
||||
// trip threshold + cooldown-ladder math is shared via sse_overflow.js;
|
||||
// the stateful glue below is coupled to this closure's evtSource seam.
|
||||
// ------------------------------------------------------------------
|
||||
|
||||
// Transport-only stream suspension for the overflow / visibility /
|
||||
// close-session paths: the closeStreamTransport teardown WITHOUT the full
|
||||
// destroy() cleanup (observers, task timers), plus gap accounting.
|
||||
// connectSSE re-opens from the saved lastEventId, so this is lossless for
|
||||
// committed turns.
|
||||
function suspendStream() {
|
||||
closeStreamTransport();
|
||||
// A deliberate suspend is still a gap. The transient-error path marks
|
||||
// it via onerror; the overflow/hide/close-session paths self-close (no
|
||||
// onerror fires after .close()), so mark it here instead.
|
||||
markStreamGap();
|
||||
}
|
||||
|
||||
// Open a gap in the stream's coverage: stamp its wall-clock start, which
|
||||
// doubles as the was-reconnecting flag the next onopen snapshots (see
|
||||
// connectSSE). Keep the EARLIEST stamp when marks pile up (repeated onerror
|
||||
// fires, hide followed by a deferred connect) so onopen measures the whole
|
||||
// outage, not just its last slice. onopen clears it.
|
||||
function markStreamGap() {
|
||||
if (!disconnectedAt) disconnectedAt = Date.now();
|
||||
}
|
||||
|
||||
// Replace-mode sidebar re-sync after a gap the reconnect replay could not
|
||||
// (or might not) have covered — the server is authoritative; any SSE-only
|
||||
// child/task rows accumulated before the disconnect are stale. Two
|
||||
// callers: onopen (no-cursor / over-threshold gaps) and the
|
||||
// replay_truncated handler (ring-evicted gaps). Ordinary short gaps need
|
||||
// neither — the ring replay redelivers child_ws_* / task events itself.
|
||||
function refreshSidebarAfterGap() {
|
||||
loadChildren({ replace: true });
|
||||
loadTasks();
|
||||
// Drop the live-badge cache too — entries within the 5s TTL can carry
|
||||
// stale pending_approval_details (the child may have resolved its
|
||||
// approval during the SSE gap). Without this clear, inline approve/deny
|
||||
// buttons could render on a row whose approval was resolved elsewhere;
|
||||
// the next scheduleLiveFetch from loadChildren's finally branch (which
|
||||
// fires for every visible row) repopulates with authoritative state.
|
||||
// Preserve `permanent: true` entries (set on 403/404 — denied by
|
||||
// permission/identity, not by state) so a user lacking
|
||||
// admin.cluster.inspect doesn't pay one 403 per denied id on every
|
||||
// refresh.
|
||||
for (const [id, c] of liveBadgeCache) {
|
||||
if (!c || !c.permanent) _liveBadgeCacheDelete(id);
|
||||
}
|
||||
}
|
||||
|
||||
// Count + log a caught render throw (wedge-class instrumentation). The
|
||||
// running total rides in the log line so a field report shows which class
|
||||
// fired — dropped events vs render wedge — without a debugger attached.
|
||||
// console.warn, not error: every caller recovers (plain-text fallback or
|
||||
// keeping the already-streamed text). The onmessage dispatch catch keeps
|
||||
// its own inline increment — that one is console.error (the whole event is
|
||||
// dropped, nothing recovers it) and names the event type.
|
||||
function noteRenderThrow(where, err) {
|
||||
streamHealth.renderThrows += 1;
|
||||
console.warn(
|
||||
"coordinator " +
|
||||
where +
|
||||
" failed (render-throw total " +
|
||||
streamHealth.renderThrows +
|
||||
")",
|
||||
err,
|
||||
);
|
||||
}
|
||||
|
||||
function noteStreamOverflow() {
|
||||
streamHealth.overflows += 1;
|
||||
const now = Date.now();
|
||||
overflowTimes.push(now);
|
||||
console.warn(
|
||||
"coordinator: server closed the stream after a send-queue overflow " +
|
||||
"(total " +
|
||||
streamHealth.overflows +
|
||||
"); reconnect will replay the gap",
|
||||
);
|
||||
// Trip when OVERFLOW_TRIP_COUNT closes land inside the rolling window. The
|
||||
// cooldown-ladder reset lives in enterDegradedCatchup (keyed off
|
||||
// lastDegradedAt), NOT here — this only counts and trips.
|
||||
if (
|
||||
overflowWindowTripped(
|
||||
overflowTimes,
|
||||
now,
|
||||
OVERFLOW_TRIP_COUNT,
|
||||
OVERFLOW_TRIP_WINDOW_MS,
|
||||
)
|
||||
) {
|
||||
enterDegradedCatchup();
|
||||
}
|
||||
}
|
||||
|
||||
function enterDegradedCatchup() {
|
||||
// Repeated overflow closes inside one window: this consumer cannot keep up
|
||||
// with live streaming right now, and each reconnect round just stalls
|
||||
// rendering behind the retry before re-saturating. Stop the churn: close
|
||||
// the stream, say so in plain language, and come back after a (doubling)
|
||||
// cooldown — that reconnect replays the gap from the server's ring buffer,
|
||||
// or falls to the replay_truncated → /history resync floor once the gap
|
||||
// has outgrown it. Either path is lossless for committed turns.
|
||||
const now = Date.now();
|
||||
// Escalate the cooldown when trips recur; reset to base only after a
|
||||
// genuine quiet gap. Keyed off lastDegradedAt (a timestamp), NOT
|
||||
// overflowTimes — this clears that array below, so keying the reset off it
|
||||
// would restart the ladder on the next storm's first overflow and the
|
||||
// doubling (15→30→60→120s) would never take effect.
|
||||
const step = degradedCooldownStep(
|
||||
degradedCooldownMs,
|
||||
lastDegradedAt,
|
||||
now,
|
||||
DEGRADED_COOLDOWN_BASE_MS,
|
||||
DEGRADED_COOLDOWN_MAX_MS,
|
||||
DEGRADED_COOLDOWN_RESET_MS,
|
||||
);
|
||||
lastDegradedAt = now;
|
||||
degradedCooldownMs = step.nextCooldownMs;
|
||||
overflowTimes.length = 0;
|
||||
suspendStream(); // also cancels any earlier degraded timer
|
||||
setSseStatus("catching up…", "err");
|
||||
statusBarEl.classList.add("ws-sb-disconnected");
|
||||
sbTokensEl.textContent = "Connection is slow — catching up…";
|
||||
const cooldown = step.cooldown;
|
||||
degradedTimer = setTimeout(function () {
|
||||
degradedTimer = null;
|
||||
if (document.hidden) {
|
||||
// Reopening into a throttled hidden tab would overflow again — defer to
|
||||
// the visibilitychange show edge instead.
|
||||
hiddenDisconnect = true;
|
||||
return;
|
||||
}
|
||||
connectSSE();
|
||||
}, cooldown);
|
||||
}
|
||||
|
||||
function onVisibilityChange() {
|
||||
if (document.hidden) {
|
||||
// Closing beats letting the hidden tab's throttled event loop starve the
|
||||
// drain until the server-side queue overflows. The streaming buffers
|
||||
// (currentAssistantBuf / currentAssistantEl) survive — suspendStream is
|
||||
// transport-only — so the visible tail is intact when the tab returns.
|
||||
if (evtSource) {
|
||||
suspendStream();
|
||||
hiddenDisconnect = true;
|
||||
}
|
||||
} else if (hiddenDisconnect) {
|
||||
hiddenDisconnect = false;
|
||||
connectSSE();
|
||||
}
|
||||
}
|
||||
|
||||
function removeVisibilityHandler() {
|
||||
if (visHandler) {
|
||||
document.removeEventListener("visibilitychange", visHandler);
|
||||
visHandler = null;
|
||||
}
|
||||
hiddenDisconnect = false;
|
||||
}
|
||||
|
||||
// ------------------------------------------------------------------
|
||||
// SSE event router
|
||||
// ------------------------------------------------------------------
|
||||
@@ -2428,7 +2735,7 @@ function createCoordinatorPane(root, wsId, opts) {
|
||||
try {
|
||||
streamingRender(abody, currentAssistantBuf);
|
||||
} catch (e) {
|
||||
console.warn("coordinator streamingRender failed", e);
|
||||
noteRenderThrow("in_progress_snapshot render", e);
|
||||
abody.textContent = currentAssistantBuf;
|
||||
}
|
||||
} else if (abody) {
|
||||
@@ -2440,6 +2747,15 @@ function createCoordinatorPane(root, wsId, opts) {
|
||||
case "stream_end":
|
||||
finishAssistantStream();
|
||||
break;
|
||||
case "stream_overflow":
|
||||
// The server poisoned this listener at its first queue overflow and
|
||||
// closes the stream right after this id-less frame. lastEventId still
|
||||
// points below the gap, so native EventSource reconnect replays it
|
||||
// losslessly from the ring buffer. Count the close: a persistently
|
||||
// slow consumer trips the degraded catch-up instead of churning
|
||||
// reconnects.
|
||||
noteStreamOverflow();
|
||||
break;
|
||||
case "tool_result":
|
||||
appendToolResult(
|
||||
ev.name || "tool",
|
||||
@@ -2641,6 +2957,24 @@ function createCoordinatorPane(root, wsId, opts) {
|
||||
// reset, idle-after-error). Mirrors the interactive pane.
|
||||
if (ev.state === "idle" || ev.state === "error") {
|
||||
setBusy(false);
|
||||
// Deferred replay_truncated re-sync: the truncation arrived while a
|
||||
// turn was mid-stream (refetching then would have detached the live
|
||||
// bubble), so repair the ring-evicted gap now that the turn is
|
||||
// settled and /history is complete. Also repairs a turn stranded by
|
||||
// close-on-hide — hidden mid-turn, its stream_end evicted, so the
|
||||
// show-edge replay_truncated latched the flag and the live bubble
|
||||
// never finalized. Reset the streaming refs first: refetchHistory
|
||||
// replaceChildren()s the DOM but does NOT null them, and a dangling
|
||||
// ref would strand the NEXT turn's tokens into a detached node.
|
||||
// Mirrors interactive.js.
|
||||
if (pendingTruncatedResync) {
|
||||
pendingTruncatedResync = false;
|
||||
currentAssistantEl = null;
|
||||
currentAssistantBuf = "";
|
||||
currentReasoningEl = null;
|
||||
currentReasoningBuf = "";
|
||||
refetchHistory();
|
||||
}
|
||||
} else if (
|
||||
ev.state === "running" ||
|
||||
ev.state === "thinking" ||
|
||||
@@ -2739,9 +3073,24 @@ function createCoordinatorPane(root, wsId, opts) {
|
||||
}
|
||||
case "replay_truncated":
|
||||
// Reconnect buffer evicted past our last-seen id — re-sync from REST.
|
||||
// Skip mid-stream: in_progress_snapshot already paints the live turn
|
||||
// and a replaceChildren() would detach the streaming bubble.
|
||||
if (!currentAssistantEl) refetchHistory();
|
||||
// Skip while a turn is mid-stream (BOTH a content bubble and a
|
||||
// reasoning-only one are detachable): the recovery floor's
|
||||
// in_progress_snapshot repaints it, and an async refetch's
|
||||
// replaceChildren() would detach the live bubble so deltas render
|
||||
// nowhere. Mid-stream the resync is DEFERRED, not dropped — skipping
|
||||
// outright left the ring-evicted gap unrepaired for the rest of the
|
||||
// session (no clean reconnect may come for hours); the idle edge
|
||||
// consumes the flag. Mirrors interactive.js's _pendingTruncatedResync.
|
||||
if (!currentAssistantEl && !currentReasoningEl) {
|
||||
refetchHistory();
|
||||
} else {
|
||||
pendingTruncatedResync = true;
|
||||
}
|
||||
// The evicted slice may have carried child_ws_* / task events the
|
||||
// sidebar will never see replayed — this is the server saying the
|
||||
// gap was NOT covered, so pull authoritative state (unless onopen
|
||||
// already did, milliseconds ago, for this same reconnect).
|
||||
if (!gapRefreshedAtOpen) refreshSidebarAfterGap();
|
||||
break;
|
||||
case "tool_pending":
|
||||
// Early paint — render the batch the instant the model commits to
|
||||
@@ -4959,10 +5308,8 @@ function createCoordinatorPane(root, wsId, opts) {
|
||||
// login out to every open pane.)
|
||||
function onLogin() {
|
||||
reconnectAttempts = 0;
|
||||
if (reconnectTimer) {
|
||||
clearTimeout(reconnectTimer);
|
||||
reconnectTimer = null;
|
||||
}
|
||||
// connectSSE's closeStreamTransport prologue cancels any pending
|
||||
// reconnect / degraded timers before dialling.
|
||||
connectSSE();
|
||||
}
|
||||
|
||||
@@ -4970,20 +5317,21 @@ function createCoordinatorPane(root, wsId, opts) {
|
||||
// per-instance pane must release the stream + every timer/observer or a
|
||||
// backgrounded pane keeps an SSE open and fires renders into detached DOM.
|
||||
function destroy() {
|
||||
if (evtSource) {
|
||||
evtSource.close();
|
||||
evtSource = null;
|
||||
}
|
||||
// Stream + both retry timers (reconnect backoff, degraded catch-up).
|
||||
closeStreamTransport();
|
||||
[
|
||||
reconnectTimer,
|
||||
cancelTimeoutId,
|
||||
forceTimeoutId,
|
||||
tasksRefreshTimer,
|
||||
liveBadgeFlushTimer,
|
||||
].forEach((t) => t && clearTimeout(t));
|
||||
reconnectTimer = cancelTimeoutId = forceTimeoutId = null;
|
||||
cancelTimeoutId = forceTimeoutId = null;
|
||||
tasksRefreshTimer = liveBadgeFlushTimer = null;
|
||||
if (pruneTimer) clearInterval(pruneTimer);
|
||||
// The document-level visibilitychange listener holds a strong ref to this
|
||||
// closure — leaving it registered would both leak the pane and let a show
|
||||
// edge reopen a stream for a destroyed pane.
|
||||
removeVisibilityHandler();
|
||||
if (_childObserver && _childObserver.disconnect)
|
||||
_childObserver.disconnect();
|
||||
}
|
||||
|
||||
@@ -25,6 +25,7 @@
|
||||
<link rel="stylesheet" href="/static/coordinator/coordinator.css" />
|
||||
<link rel="stylesheet" href="/static/coordinator/coord-chrome.css" />
|
||||
<link rel="stylesheet" href="/shared/interactive.css" />
|
||||
<link rel="stylesheet" href="/shared/preview.css" />
|
||||
<link rel="stylesheet" href="/shared/hatch.css" />
|
||||
</head>
|
||||
<body>
|
||||
@@ -1532,6 +1533,18 @@
|
||||
<select id="sch-template">
|
||||
<option value="">None</option>
|
||||
</select>
|
||||
<label for="sch-persona"
|
||||
>Persona <span class="label-hint">optional</span></label
|
||||
>
|
||||
<select id="sch-persona">
|
||||
<option value="">Default persona</option>
|
||||
</select>
|
||||
<label for="sch-project"
|
||||
>Project <span class="label-hint">optional</span></label
|
||||
>
|
||||
<select id="sch-project">
|
||||
<option value="">No project</option>
|
||||
</select>
|
||||
<label for="sch-message">Initial message</label>
|
||||
<textarea
|
||||
id="sch-message"
|
||||
@@ -1731,6 +1744,46 @@
|
||||
<option value="max">Max</option>
|
||||
</select>
|
||||
|
||||
<div
|
||||
id="model-response-controls"
|
||||
role="group"
|
||||
aria-labelledby="model-response-controls-title"
|
||||
hidden
|
||||
>
|
||||
<div class="sh-section" id="model-response-controls-title">
|
||||
Response controls
|
||||
</div>
|
||||
<div class="field-pair">
|
||||
<div id="model-output-verbosity-field" hidden>
|
||||
<label for="model-output-verbosity"
|
||||
>Output verbosity
|
||||
<span class="label-hint"
|
||||
>answer length, independent of effort</span
|
||||
></label
|
||||
>
|
||||
<select id="model-output-verbosity">
|
||||
<option value="">Provider default</option>
|
||||
<option value="low">Low — concise</option>
|
||||
<option value="medium">Medium — balanced</option>
|
||||
<option value="high">High — detailed</option>
|
||||
</select>
|
||||
</div>
|
||||
<div id="model-reasoning-mode-field" hidden>
|
||||
<label for="model-reasoning-mode"
|
||||
>Reasoning mode
|
||||
<span class="label-hint"
|
||||
>Pro applies more work before answering</span
|
||||
></label
|
||||
>
|
||||
<select id="model-reasoning-mode">
|
||||
<option value="">Provider default</option>
|
||||
<option value="standard">Standard</option>
|
||||
<option value="pro">Pro</option>
|
||||
</select>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div id="model-server-compat-section" hidden>
|
||||
<div class="sh-section">Server compatibility</div>
|
||||
<div class="field-pair" id="model-server-fields-row">
|
||||
@@ -1869,6 +1922,20 @@
|
||||
></span
|
||||
><span class="cap-name">Reasoning-effort control</span></label
|
||||
>
|
||||
<label class="cap"
|
||||
><input
|
||||
type="checkbox"
|
||||
data-cap="supports_verbosity"
|
||||
/><span class="cap-led"></span
|
||||
><span class="cap-name">Output verbosity</span></label
|
||||
>
|
||||
<label class="cap"
|
||||
><input
|
||||
type="checkbox"
|
||||
data-cap="supports_pro_mode"
|
||||
/><span class="cap-led"></span
|
||||
><span class="cap-name">Standard / Pro mode</span></label
|
||||
>
|
||||
<label class="cap"
|
||||
><input
|
||||
type="checkbox"
|
||||
|
||||
@@ -23,12 +23,28 @@ if TYPE_CHECKING:
|
||||
def cleanup_session_ui(ws: Workstream) -> None:
|
||||
"""Shared SessionKindAdapter cleanup_ui implementation.
|
||||
|
||||
Unblocks pending approval / plan / foreground events on the
|
||||
workstream's UI, broadcasts ``ws_closed`` to per-UI listener
|
||||
queues, then cancels + closes the session. The ``hasattr`` checks
|
||||
guard stub UIs used in tests — the real ``WebUI`` /
|
||||
Marks the workstream object dead (``ws._closed``) FIRST, under
|
||||
``ws._lock``, then unblocks pending approval / plan / foreground
|
||||
events on the workstream's UI, broadcasts ``ws_closed`` to per-UI
|
||||
listener queues, and cancels + closes the session. The ``hasattr``
|
||||
checks guard stub UIs used in tests — the real ``WebUI`` /
|
||||
``ConsoleCoordinatorUI`` always have these attributes.
|
||||
|
||||
The flag write lives HERE because every teardown path funnels
|
||||
through this function — ``close``, ``close_idle``, EVICTION,
|
||||
``delete``, ``discard`` — and per-path writes are not enough: a
|
||||
flag set by only some paths, or set after this teardown body,
|
||||
leaves windows where the workstream is being torn down while
|
||||
still reading as live. The wake paths that hold OBJECT
|
||||
references (the watch ``wake_fn``, ``session_worker``'s exit
|
||||
backstop) gate on this flag, and ``session_worker.send``
|
||||
re-checks it under the same lock: once this write lands, no wake
|
||||
can spawn a worker on the torn-down session — including on
|
||||
evicted or deleted workstreams, and including during the
|
||||
remainder of this teardown.
|
||||
"""
|
||||
with ws._lock:
|
||||
ws._closed = True
|
||||
if ws.session is not None and hasattr(ws.session, "cancel"):
|
||||
ws.session.cancel()
|
||||
ui = ws.ui
|
||||
@@ -68,6 +84,15 @@ def _broadcast_ws_closed_to_listeners(ui: SessionUI) -> None:
|
||||
return
|
||||
with listeners_lock:
|
||||
for lq in listeners:
|
||||
# Mark the stream closing BEFORE attempting the sentinel: a
|
||||
# poisoned/full ``_ListenerQueue`` rejects every put (the
|
||||
# eviction-safe retry below can't beat the poison latch), so
|
||||
# the drain loop needs this out-of-band flag to unwind as a
|
||||
# clean close instead of emitting a spurious ``stream_overflow``
|
||||
# frame. Best-effort on plain ``queue.Queue`` (no such method).
|
||||
mark_closing = getattr(lq, "mark_closing", None)
|
||||
if mark_closing is not None:
|
||||
mark_closing()
|
||||
try:
|
||||
lq.put_nowait({"type": "ws_closed"})
|
||||
except queue.Full:
|
||||
|
||||
@@ -0,0 +1,833 @@
|
||||
"""Per-session registry for explicitly backgrounded bash shells (#817).
|
||||
|
||||
#816 made the ``bash`` tool terminate its whole process group when the call
|
||||
returns — no leaked servers, no hangs, but also no way to keep a dev server
|
||||
alive across calls. This registry restores that as an explicit opt-in with
|
||||
the model-facing shape the frontier coding agents converged on: a boolean on
|
||||
the shell tool, a short ``bash_N`` handle, a delta-output reader that returns
|
||||
only lines produced since the previous read, and a kill tool.
|
||||
|
||||
Lifetime rules (the #816 rule, extended):
|
||||
|
||||
* The tracked command defines the shell's lifetime. When it exits —
|
||||
naturally, by ``kill``, or by registry teardown — its whole session group
|
||||
is SIGKILLed, so nothing the command backgrounded can outlive it.
|
||||
* Shells survive generation-cancel (they are deliberately detached) and die
|
||||
with the owning session: :meth:`BackgroundShellRegistry.close` runs from
|
||||
``ChatSession.close()``, which every workstream-teardown path funnels
|
||||
through.
|
||||
* Shells spawned inside a task_agent carry that agent's ``owner`` tag; the
|
||||
agent's ``finally`` reaps them, and owner-scoped lookup keeps parallel
|
||||
agents (and the parent) from touching each other's handles.
|
||||
|
||||
Output is buffered per shell as a rolling deque of lines (stderr tagged
|
||||
``[stderr] `` inline, arrival order) capped by total characters with
|
||||
drop-oldest semantics — a chatty server cannot grow a session's memory
|
||||
unbounded. Reads advance a cursor over the *logical* line stream, so a
|
||||
line dropped before it was ever read surfaces as an explicit gap count
|
||||
rather than silently vanishing.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import contextlib
|
||||
import itertools
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import signal
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import threading
|
||||
import time
|
||||
from collections import deque
|
||||
from dataclasses import dataclass
|
||||
from typing import TYPE_CHECKING, Any, Literal
|
||||
|
||||
from turnstone.core.log import get_logger
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from collections.abc import Callable
|
||||
|
||||
log = get_logger(__name__)
|
||||
|
||||
ShellStatus = Literal["running", "completed", "killed"]
|
||||
|
||||
# Live (status == "running") shells per session. A hard backstop against a
|
||||
# runaway loop of spawns on a multi-tenant node, not an operator knob.
|
||||
_DEFAULT_MAX_SHELLS = 8
|
||||
# Rolling per-shell buffer cap, in characters. Oldest whole lines drop
|
||||
# first; the newest line always survives even if it alone exceeds the cap.
|
||||
_DEFAULT_MAX_BUFFER_CHARS = 200_000
|
||||
# How long to wait for the drain threads after the group kill forces their
|
||||
# pipes to EOF. A grandchild that double-``setsid``-escaped the group can
|
||||
# hold a pipe open past this — the drain is a daemon thread and leaks
|
||||
# (logged) until that process dies, same acceptance as the foreground tool.
|
||||
_DRAIN_JOIN_TIMEOUT_S = 5
|
||||
# TOTAL join budget for ``kill``/``reap`` across all of a shell's threads
|
||||
# (not per-thread — a wedged drain must not stack timeouts).
|
||||
_WAITER_JOIN_TIMEOUT_S = 10
|
||||
# TOTAL join budget for ``close()`` across ALL shells. close() runs on the
|
||||
# workstream-teardown funnel, which the server can reach from an async
|
||||
# handler — an unbounded (or per-shell-stacking) wait here would freeze the
|
||||
# node's event loop, not just this workstream. Threads still alive past the
|
||||
# budget are daemons: logged and abandoned, they die with their pipes.
|
||||
_CLOSE_JOIN_BUDGET_S = 5
|
||||
# Exited records retained per registry (drop-oldest). Keeps a long-lived
|
||||
# workstream that backgrounds thousands of short jobs from accumulating
|
||||
# dead records (each can pin up to ``max_buffer_chars`` of buffer) while
|
||||
# still letting the model read recently-exited shells' output.
|
||||
_MAX_EXITED_RECORDS = 32
|
||||
# Bounds on the model-supplied ``filter`` regex: pattern length, how much of
|
||||
# each line the pattern sees, and wall-clock for the whole filter pass. The
|
||||
# pass runs in a SUBPROCESS, not a thread: CPython's sre engine holds the
|
||||
# GIL for the entire duration of one ``search`` call, so a catastrophic-
|
||||
# backtracking pattern freezes every thread in the interpreter — no
|
||||
# in-process timeout (thread join, signal, anything) can fire. A child
|
||||
# process is killable from outside the GIL; on timeout the read errors
|
||||
# WITHOUT consuming the delta (the cursor only commits on a completed pass).
|
||||
_MAX_FILTER_PATTERN_CHARS = 512
|
||||
_FILTER_MAX_LINE_CHARS = 4096
|
||||
_FILTER_TIMEOUT_S = 2.0
|
||||
|
||||
# Runs inside ``sys.executable -c``: reads {pattern, lines} as JSON on
|
||||
# stdin (lines already truncated parent-side), writes the MATCHING INDEXES
|
||||
# as JSON on stdout (indexes, not lines — no need to echo a 200K buffer
|
||||
# back through a pipe).
|
||||
_FILTER_HELPER_SRC = (
|
||||
"import json, re, sys\n"
|
||||
"d = json.load(sys.stdin)\n"
|
||||
"p = re.compile(d['pattern'])\n"
|
||||
"sys.stdout.write(json.dumps([i for i, ln in enumerate(d['lines']) if p.search(ln)]))\n"
|
||||
)
|
||||
|
||||
|
||||
class UnknownShellError(LookupError):
|
||||
"""No shell with that id is visible in the caller's owner scope."""
|
||||
|
||||
|
||||
class TooManyShellsError(RuntimeError):
|
||||
"""The per-session live-shell cap would be exceeded."""
|
||||
|
||||
|
||||
class FilterTimeoutError(ValueError):
|
||||
"""The ``filter`` regex did not finish within the time bound."""
|
||||
|
||||
|
||||
class FilterExecError(RuntimeError):
|
||||
"""The filter helper process failed for a non-pattern reason."""
|
||||
|
||||
|
||||
def _filter_lines_bounded(pattern: re.Pattern[str], lines: list[str], shell_id: str) -> list[str]:
|
||||
"""Apply ``pattern`` per line with a wall-clock bound.
|
||||
|
||||
A catastrophic-backtracking pattern would wedge the (auto-approved)
|
||||
tool call — the exact never-returns class #816 removed — and it cannot
|
||||
be bounded IN-PROCESS: sre holds the GIL for the whole ``search`` call,
|
||||
freezing every interpreter thread including any watchdog. So the pass
|
||||
runs in a small child process (killable from the OS): each line
|
||||
truncated PARENT-side to :data:`_FILTER_MAX_LINE_CHARS` before
|
||||
serialization (a filter targets log lines; shipping a retained multi-MB
|
||||
line through the pipe would spend the time budget on I/O and misreport
|
||||
a fine pattern as slow), the whole pass bounded by
|
||||
:data:`_FILTER_TIMEOUT_S`, SIGKILL on the child's group past that.
|
||||
|
||||
Raises :class:`FilterTimeoutError` on timeout and
|
||||
:class:`FilterExecError` on a helper failure that is NOT the pattern's
|
||||
fault (fork/OOM/env) — distinct messages, so the model doesn't
|
||||
"simplify" an innocent regex. Either way the caller consumes nothing.
|
||||
The ~tens-of-ms interpreter startup is paid only on filtered reads.
|
||||
|
||||
Threat model for the auto-approved path (``bash_output`` runs without
|
||||
operator approval): the only model-controlled inputs are the PATTERN
|
||||
and, transitively, the buffered text. The pattern is compiled
|
||||
parent-side before the fork (a non-regex payload fails there), the
|
||||
child executes only the fixed ``_FILTER_HELPER_SRC`` — the pattern is
|
||||
DATA on stdin, never code —, the child gets a scrubbed environment, no
|
||||
shell, read-only work, and a SIGKILL at the time bound. Worst case a
|
||||
hostile pattern buys ~2s of one core.
|
||||
"""
|
||||
from turnstone.core.env import scrubbed_env
|
||||
|
||||
payload = json.dumps(
|
||||
{
|
||||
"pattern": pattern.pattern,
|
||||
"lines": [ln[:_FILTER_MAX_LINE_CHARS] for ln in lines],
|
||||
},
|
||||
ensure_ascii=False,
|
||||
)
|
||||
try:
|
||||
proc = subprocess.Popen(
|
||||
[sys.executable, "-c", _FILTER_HELPER_SRC],
|
||||
stdin=subprocess.PIPE,
|
||||
stdout=subprocess.PIPE,
|
||||
stderr=subprocess.DEVNULL,
|
||||
# Pin BOTH pipe directions to UTF-8: ``text=True`` alone uses
|
||||
# the locale encoding, and on a C/POSIX-locale node a single
|
||||
# U+FFFD (from the drain's ``errors="replace"``) would raise
|
||||
# UnicodeEncodeError out of communicate() — escaping the
|
||||
# Timeout/Exec error taxonomy as a generic crash. The child's
|
||||
# own stdio decode is pinned via PYTHONIOENCODING.
|
||||
encoding="utf-8",
|
||||
errors="replace",
|
||||
# scrubbed_env, not os.environ: the helper needs no secrets (it
|
||||
# runs only our trusted source over already-buffered text), and
|
||||
# every other fork in this codebase strips API keys/tokens —
|
||||
# this one must not be the exception.
|
||||
env={**scrubbed_env(), "PYTHONIOENCODING": "utf-8"},
|
||||
start_new_session=True,
|
||||
)
|
||||
except OSError as e:
|
||||
# Fork pressure (EAGAIN) / exec failure — same containment class as
|
||||
# spawn()'s thread-start guard, and by contract NOT the pattern's
|
||||
# fault.
|
||||
log.warning("bg_shell.filter_helper_spawn_failed", shell_id=shell_id, error=str(e))
|
||||
raise FilterExecError(
|
||||
"the filter could not be applied (helper failed to start); this "
|
||||
"is not a problem with your pattern — no output was consumed; "
|
||||
"retry, or read without a filter"
|
||||
) from e
|
||||
try:
|
||||
out, _ = proc.communicate(payload, timeout=_FILTER_TIMEOUT_S)
|
||||
except subprocess.TimeoutExpired:
|
||||
with contextlib.suppress(OSError, ProcessLookupError):
|
||||
os.killpg(proc.pid, signal.SIGKILL)
|
||||
with contextlib.suppress(subprocess.TimeoutExpired):
|
||||
proc.wait(timeout=5)
|
||||
log.warning("bg_shell.filter_timeout", shell_id=shell_id, pattern=pattern.pattern[:80])
|
||||
raise FilterTimeoutError(
|
||||
f"filter regex took longer than {_FILTER_TIMEOUT_S:g}s to run; no "
|
||||
"output was consumed — simplify the pattern or retry without a filter"
|
||||
) from None
|
||||
if proc.returncode != 0:
|
||||
# The parent validated the compile, so a child failure is exotic
|
||||
# (fork pressure, interpreter env) — NOT the pattern's fault.
|
||||
log.warning(
|
||||
"bg_shell.filter_helper_failed",
|
||||
shell_id=shell_id,
|
||||
returncode=proc.returncode,
|
||||
)
|
||||
raise FilterExecError(
|
||||
f"the filter could not be applied (helper exited {proc.returncode}); "
|
||||
"this is not a problem with your pattern — no output was consumed; "
|
||||
"retry, or read without a filter"
|
||||
)
|
||||
try:
|
||||
indexes = json.loads(out)
|
||||
except ValueError:
|
||||
log.warning("bg_shell.filter_helper_bad_output", shell_id=shell_id)
|
||||
raise FilterExecError(
|
||||
"the filter could not be applied (helper returned malformed data); "
|
||||
"no output was consumed — retry, or read without a filter"
|
||||
) from None
|
||||
return [lines[i] for i in indexes if isinstance(i, int) and 0 <= i < len(lines)]
|
||||
|
||||
|
||||
def drain_pipe_lines(pipe: Any, on_line: Callable[[str], None]) -> None:
|
||||
"""Read ``pipe`` line-by-line until EOF, forwarding each to ``on_line``.
|
||||
|
||||
The drain half of the shared bash recipe (see :func:`spawn_group_leader`
|
||||
for the spawn half): both variants of the tool tolerate the same two
|
||||
end-of-stream shapes. A pipe torn down by the session-group kill is the
|
||||
expected end; anything else must not kill the drain silently. (The
|
||||
``errors="replace"`` on the shared Popen pre-empts UnicodeDecodeError —
|
||||
a ValueError that would otherwise end the drain early and drop ALL
|
||||
remaining output while reporting a clean success.)
|
||||
"""
|
||||
try:
|
||||
for line in pipe:
|
||||
on_line(line)
|
||||
except (ValueError, OSError):
|
||||
log.debug("bash.drain_read_error", exc_info=True)
|
||||
|
||||
|
||||
def spawn_group_leader(
|
||||
command: str, *, stop_on_error: bool, env: dict[str, str] | None
|
||||
) -> tuple[subprocess.Popen[str], int, str]:
|
||||
"""Write the script, fork the detached group leader, snapshot its pgid.
|
||||
|
||||
THE shared prologue for both runs of the model-facing bash tool — the
|
||||
foreground executor (``ChatSession._exec_bash``) and this registry — so
|
||||
the two variants of one tool cannot drift: same ``pipefail``/``set -e``
|
||||
preamble, same decode policy (``errors="replace"``), same session-group
|
||||
discipline. The script file exists because bash reads scripts lazily
|
||||
(robust to quoting/length; unlinking early could truncate a long script
|
||||
mid-run) — the CALLER owns the unlink on its own exit path. On a
|
||||
failed fork the script is unlinked here and the error propagates. The
|
||||
pgid snapshot happens while the leader is alive (``start_new_session``
|
||||
makes ``pgid == pid``); the microseconds-wide pid-wraparound TOCTOU is
|
||||
the same accepted one as always.
|
||||
"""
|
||||
with tempfile.NamedTemporaryFile(mode="w", suffix=".sh", delete=False) as f:
|
||||
preamble = "set -o pipefail\n"
|
||||
if stop_on_error:
|
||||
preamble += "set -e\n"
|
||||
f.write(preamble + command)
|
||||
script_path = f.name
|
||||
try:
|
||||
proc = subprocess.Popen(
|
||||
["bash", script_path],
|
||||
stdout=subprocess.PIPE,
|
||||
stderr=subprocess.PIPE,
|
||||
text=True,
|
||||
errors="replace",
|
||||
start_new_session=True,
|
||||
env=env,
|
||||
)
|
||||
except BaseException:
|
||||
with contextlib.suppress(OSError):
|
||||
os.unlink(script_path)
|
||||
raise
|
||||
try:
|
||||
pgid = os.getpgid(proc.pid)
|
||||
except OSError:
|
||||
pgid = proc.pid
|
||||
return proc, pgid, script_path
|
||||
|
||||
|
||||
@dataclass
|
||||
class ShellRead:
|
||||
"""One delta read: lines since the previous read, plus shell state.
|
||||
|
||||
``lines`` is post-filter (what the caller shows); ``new_line_count`` is
|
||||
the pre-filter delta size — the cursor advanced past all of them, so a
|
||||
filtered-out line is consumed, never deferred to a later read.
|
||||
``dropped_lines`` counts lines lost to the buffer cap before they were
|
||||
ever read (an explicit gap, not silence).
|
||||
"""
|
||||
|
||||
shell_id: str
|
||||
status: ShellStatus
|
||||
exit_code: int | None
|
||||
lines: list[str]
|
||||
new_line_count: int
|
||||
dropped_lines: int
|
||||
# Lines in this delta longer than the per-line filter window — their
|
||||
# tails were invisible to the pattern. Only populated on filtered
|
||||
# reads; the caller surfaces it so a "none matching" answer over
|
||||
# clipped evidence is never silent.
|
||||
clipped_lines: int = 0
|
||||
|
||||
|
||||
class BackgroundShell:
|
||||
"""One detached shell: process handles, rolling buffer, read cursor.
|
||||
|
||||
Mutable state is guarded by ``self.lock`` — the drain threads append
|
||||
while reads snapshot; the waiter thread flips ``status`` exactly once.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
shell_id: str,
|
||||
command: str,
|
||||
proc: subprocess.Popen[str],
|
||||
pgid: int,
|
||||
owner: str | None,
|
||||
script_path: str,
|
||||
max_buffer_chars: int,
|
||||
) -> None:
|
||||
self.shell_id = shell_id
|
||||
self.command = command
|
||||
self.proc = proc
|
||||
self.pid = proc.pid
|
||||
self.pgid = pgid
|
||||
self.owner = owner
|
||||
self.status: ShellStatus = "running"
|
||||
self.exit_code: int | None = None
|
||||
self.lock = threading.Lock()
|
||||
self._script_path = script_path
|
||||
self._max_buffer_chars = max_buffer_chars
|
||||
# Rolling buffer over the logical line stream: ``_buffer`` holds the
|
||||
# retained tail; ``_dropped_total``/``_total_lines`` are absolute
|
||||
# line counts so the cursor survives drop-oldest evictions.
|
||||
self._buffer: deque[str] = deque()
|
||||
self._buffered_chars = 0
|
||||
self._dropped_total = 0
|
||||
self._total_lines = 0
|
||||
self._read_cursor = 0
|
||||
# Set (under ``lock``) before the group kill on every deliberate
|
||||
# termination path so the waiter can distinguish "killed" from
|
||||
# "completed" and suppress the exit callback.
|
||||
self._killed = False
|
||||
self._threads: list[threading.Thread] = []
|
||||
# Serializes whole read passes (snapshot → filter → commit). The
|
||||
# buffer lock alone leaves a window where two concurrent reads of
|
||||
# the same shell snapshot the same cursor and BOTH return the delta
|
||||
# as new — double-delivering every line. Held across the filter
|
||||
# subprocess too: correctness over parallel reads of one shell.
|
||||
self.read_serial = threading.Lock()
|
||||
# Monotonic EXIT order (registry-assigned by the waiter), None while
|
||||
# running. Dead-record eviction sorts on this, never on spawn
|
||||
# order: a long-lived first-spawned server must not be the first
|
||||
# record evicted — least of all by its own exit's prune, which
|
||||
# would drop its promised exit notice and crash output unread.
|
||||
self._exit_seq: int | None = None
|
||||
|
||||
@property
|
||||
def unread_lines(self) -> int:
|
||||
"""Lines still READABLE that the cursor hasn't consumed — excludes
|
||||
lines the buffer cap already evicted, so an exit notice never
|
||||
promises more output than ``bash_output`` can actually return."""
|
||||
with self.lock:
|
||||
return self._total_lines - max(self._read_cursor, self._dropped_total)
|
||||
|
||||
def _append(self, line: str) -> None:
|
||||
with self.lock:
|
||||
self._buffer.append(line)
|
||||
self._buffered_chars += len(line)
|
||||
self._total_lines += 1
|
||||
# Drop oldest whole lines past the cap, but always keep the
|
||||
# newest — a single oversized line must not empty the buffer.
|
||||
while self._buffered_chars > self._max_buffer_chars and len(self._buffer) > 1:
|
||||
dropped = self._buffer.popleft()
|
||||
self._buffered_chars -= len(dropped)
|
||||
self._dropped_total += 1
|
||||
|
||||
def _snapshot_delta(self) -> tuple[list[str], int, int, ShellStatus, int | None]:
|
||||
"""Snapshot unread lines WITHOUT consuming them.
|
||||
|
||||
Returns ``(delta, gap, new_cursor, status, exit_code)``. The caller
|
||||
commits ``new_cursor`` via :meth:`_commit_cursor` only after any
|
||||
filtering succeeded — a failed/timed-out filter must not eat output.
|
||||
"""
|
||||
with self.lock:
|
||||
start = max(self._read_cursor, self._dropped_total)
|
||||
gap = start - self._read_cursor
|
||||
delta = list(itertools.islice(self._buffer, start - self._dropped_total, None))
|
||||
return delta, gap, self._total_lines, self.status, self.exit_code
|
||||
|
||||
def _commit_cursor(self, new_cursor: int) -> None:
|
||||
with self.lock:
|
||||
# max(): monotonic under concurrent reads of the same scope.
|
||||
self._read_cursor = max(self._read_cursor, new_cursor)
|
||||
|
||||
|
||||
class BackgroundShellRegistry:
|
||||
"""Session-scoped table of background shells, ``bash_N``-keyed.
|
||||
|
||||
Thread-safe: tool calls (spawn/read/kill), waiter threads (exit
|
||||
transitions), and teardown (close/reap) may interleave freely.
|
||||
``on_exit`` fires from the waiter thread on NATURAL exit only — never
|
||||
for ``kill``/``reap``/``close`` — after the drains have flushed, so a
|
||||
read triggered by the callback sees the complete output.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
*,
|
||||
max_shells: int = _DEFAULT_MAX_SHELLS,
|
||||
max_buffer_chars: int = _DEFAULT_MAX_BUFFER_CHARS,
|
||||
max_exited_records: int = _MAX_EXITED_RECORDS,
|
||||
on_exit: Callable[[BackgroundShell], None] | None = None,
|
||||
) -> None:
|
||||
self._max_shells = max_shells
|
||||
self._max_buffer_chars = max_buffer_chars
|
||||
self._max_exited_records = max_exited_records
|
||||
self._on_exit = on_exit
|
||||
self._shells: dict[str, BackgroundShell] = {}
|
||||
self._lock = threading.Lock()
|
||||
self._counter = 0
|
||||
self._exit_counter = 0
|
||||
self._closed = False
|
||||
|
||||
# -- Spawning -----------------------------------------------------------
|
||||
|
||||
def spawn(
|
||||
self,
|
||||
command: str,
|
||||
*,
|
||||
env: dict[str, str] | None = None,
|
||||
owner: str | None = None,
|
||||
stop_on_error: bool = False,
|
||||
) -> BackgroundShell:
|
||||
"""Start ``command`` as a detached shell; return its record.
|
||||
|
||||
Raises ``RuntimeError`` after :meth:`close`, :class:`TooManyShellsError`
|
||||
at the live-shell cap, and propagates ``OSError`` from a failed spawn.
|
||||
"""
|
||||
if env is None:
|
||||
from turnstone.core.env import scrubbed_env
|
||||
|
||||
env = scrubbed_env()
|
||||
# Fast-fail before paying disk + fork; re-checked authoritatively
|
||||
# under the lock after the fork (spawn stays lock-free through the
|
||||
# slow syscalls so close()/reap() — which serialize on the registry
|
||||
# lock with a total time budget — can never be blocked behind a
|
||||
# stalled filesystem write or fork).
|
||||
with self._lock:
|
||||
self._check_capacity_locked(owner)
|
||||
# Shared prologue with the foreground bash tool — the waiter unlinks
|
||||
# the script after exit.
|
||||
proc, pgid, script_path = spawn_group_leader(command, stop_on_error=stop_on_error, env=env)
|
||||
try:
|
||||
with self._lock:
|
||||
# Authoritative re-check: a concurrent spawn/close may have
|
||||
# won the race while we were forking. Refusal lands in the
|
||||
# outer handler, which reaps the freshly-forked group —
|
||||
# nothing may outlive a failed call (#816 rule).
|
||||
self._check_capacity_locked(owner)
|
||||
self._counter += 1
|
||||
shell = BackgroundShell(
|
||||
shell_id=f"bash_{self._counter}",
|
||||
command=command,
|
||||
proc=proc,
|
||||
pgid=pgid,
|
||||
owner=owner,
|
||||
script_path=script_path,
|
||||
max_buffer_chars=self._max_buffer_chars,
|
||||
)
|
||||
# Publish, wire and START the threads under the registry
|
||||
# lock: close()/reap() take the same lock, so they can never
|
||||
# observe a registered shell whose threads aren't started
|
||||
# (they would "join" nothing and return while the drains /
|
||||
# waiter start up behind them). Registration is popped on a
|
||||
# start failure IN the same hold, so a thread-exhausted node
|
||||
# (RLIMIT_NPROC) can't strand an orphan record whose
|
||||
# never-started Thread objects would make every later
|
||||
# ``join`` — hence every teardown — raise. The thread
|
||||
# bodies only ever take ``shell.lock`` or re-take the
|
||||
# registry lock AFTER this hold is released (the waiter's
|
||||
# prune), so starting them here cannot deadlock.
|
||||
self._shells[shell.shell_id] = shell
|
||||
assert proc.stdout is not None and proc.stderr is not None
|
||||
out_thread = threading.Thread(
|
||||
target=self._drain,
|
||||
args=(proc.stdout, shell, False),
|
||||
name=f"bg-shell-out-{shell.shell_id}",
|
||||
daemon=True,
|
||||
)
|
||||
err_thread = threading.Thread(
|
||||
target=self._drain,
|
||||
args=(proc.stderr, shell, True),
|
||||
name=f"bg-shell-err-{shell.shell_id}",
|
||||
daemon=True,
|
||||
)
|
||||
waiter = threading.Thread(
|
||||
target=self._wait_for_exit,
|
||||
args=(shell, out_thread, err_thread),
|
||||
name=f"bg-shell-wait-{shell.shell_id}",
|
||||
daemon=True,
|
||||
)
|
||||
shell._threads = [out_thread, err_thread, waiter]
|
||||
try:
|
||||
out_thread.start()
|
||||
err_thread.start()
|
||||
waiter.start()
|
||||
except BaseException:
|
||||
self._shells.pop(shell.shell_id, None)
|
||||
raise
|
||||
except BaseException:
|
||||
# Refused post-fork or thread start failed: reap the fresh group
|
||||
# (any started drain then EOFs and exits on its own) and surface
|
||||
# the original error to the tool layer.
|
||||
with contextlib.suppress(OSError, ProcessLookupError):
|
||||
os.killpg(pgid, signal.SIGKILL)
|
||||
with contextlib.suppress(subprocess.TimeoutExpired):
|
||||
proc.wait(timeout=5)
|
||||
with contextlib.suppress(OSError):
|
||||
os.unlink(script_path)
|
||||
raise
|
||||
log.info(
|
||||
"bg_shell.spawned",
|
||||
shell_id=shell.shell_id,
|
||||
pid=shell.pid,
|
||||
owner=owner or "",
|
||||
)
|
||||
return shell
|
||||
|
||||
def _check_capacity_locked(self, owner: str | None) -> None:
|
||||
"""Raise if closed or at the live-shell cap. Caller holds the lock."""
|
||||
if self._closed:
|
||||
raise RuntimeError("background shells unavailable: session is closing")
|
||||
live = [s for s in self._shells.values() if s.status == "running"]
|
||||
if len(live) < self._max_shells:
|
||||
return
|
||||
# The cap is registry-wide (it protects the node), but the advice
|
||||
# must be scope-honest: kill_shell is owner-scoped, so naming
|
||||
# another scope's ids would send the caller in circles.
|
||||
mine = [s.shell_id for s in live if s.owner == owner]
|
||||
others = len(live) - len(mine)
|
||||
if mine:
|
||||
detail = f"In your scope: {', '.join(mine)} — stop one with kill_shell"
|
||||
if others:
|
||||
detail += f"; {others} more belong to other agents"
|
||||
detail += "."
|
||||
else:
|
||||
detail = (
|
||||
f"All {others} belong to other agents' scopes and end when "
|
||||
"those agents finish; wait and retry."
|
||||
)
|
||||
raise TooManyShellsError(
|
||||
f"Background shell limit reached ({self._max_shells} running). {detail}"
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
def _drain(pipe: Any, shell: BackgroundShell, is_stderr: bool) -> None:
|
||||
drain_pipe_lines(
|
||||
pipe, lambda line: shell._append(f"[stderr] {line}" if is_stderr else line)
|
||||
)
|
||||
|
||||
def _wait_for_exit(
|
||||
self,
|
||||
shell: BackgroundShell,
|
||||
out_thread: threading.Thread,
|
||||
err_thread: threading.Thread,
|
||||
) -> None:
|
||||
"""Waiter thread: block on the leader, then tear down the group.
|
||||
|
||||
The kill-on-exit is what keeps the #816 guarantee: a child the
|
||||
command backgrounded dies with the command, and the drains hit EOF
|
||||
promptly instead of hanging on an inherited pipe write-end.
|
||||
"""
|
||||
shell.proc.wait()
|
||||
with contextlib.suppress(OSError, ProcessLookupError):
|
||||
os.killpg(shell.pgid, signal.SIGKILL)
|
||||
out_thread.join(timeout=_DRAIN_JOIN_TIMEOUT_S)
|
||||
err_thread.join(timeout=_DRAIN_JOIN_TIMEOUT_S)
|
||||
if out_thread.is_alive() or err_thread.is_alive():
|
||||
log.warning("bg_shell.drain_leaked", shell_id=shell.shell_id, pid=shell.pid)
|
||||
with contextlib.suppress(OSError):
|
||||
os.unlink(shell._script_path)
|
||||
with shell.lock:
|
||||
shell.exit_code = shell.proc.returncode
|
||||
shell.status = "killed" if shell._killed else "completed"
|
||||
notify = not shell._killed
|
||||
with self._lock:
|
||||
self._exit_counter += 1
|
||||
shell._exit_seq = self._exit_counter
|
||||
log.info(
|
||||
"bg_shell.exited",
|
||||
shell_id=shell.shell_id,
|
||||
exit_code=shell.exit_code,
|
||||
status=shell.status,
|
||||
)
|
||||
self._prune_exited()
|
||||
if notify and self._on_exit is not None:
|
||||
try:
|
||||
self._on_exit(shell)
|
||||
except Exception:
|
||||
log.warning("bg_shell.on_exit_failed", shell_id=shell.shell_id, exc_info=True)
|
||||
|
||||
def _prune_exited(self) -> None:
|
||||
"""Drop the OLDEST-EXITED records past ``max_exited_records``.
|
||||
|
||||
Exited records are kept so the model can read a finished shell's
|
||||
output later, but a workstream that backgrounds thousands of short
|
||||
jobs must not accumulate them (each can pin ``max_buffer_chars`` of
|
||||
buffer). Eviction sorts on exit order, NOT spawn order — the shell
|
||||
whose exit triggered this prune is by definition the newest-exited
|
||||
and therefore never its own victim (its exit notice and unread
|
||||
output survive). An exited shell whose ``_exit_seq`` isn't
|
||||
assigned yet (waiter mid-transition) sorts as newest for the same
|
||||
reason.
|
||||
"""
|
||||
with self._lock:
|
||||
if self._closed:
|
||||
return
|
||||
exited = sorted(
|
||||
(s for s in self._shells.values() if s.status != "running"),
|
||||
key=lambda s: s._exit_seq if s._exit_seq is not None else float("inf"),
|
||||
)
|
||||
for stale in exited[: max(0, len(exited) - self._max_exited_records)]:
|
||||
self._shells.pop(stale.shell_id, None)
|
||||
|
||||
# -- Lookup / reads ------------------------------------------------------
|
||||
|
||||
def _get(self, shell_id: str, owner: str | None) -> BackgroundShell:
|
||||
with self._lock:
|
||||
shell = self._shells.get(shell_id)
|
||||
if shell is not None and shell.owner == owner:
|
||||
return shell
|
||||
visible = [
|
||||
f"{s.shell_id} ({s.status})" for s in self._shells.values() if s.owner == owner
|
||||
]
|
||||
known = (
|
||||
f" Known shells: {', '.join(visible)}." if visible else " No background shells exist."
|
||||
)
|
||||
raise UnknownShellError(f"No background shell with id '{shell_id}'.{known}")
|
||||
|
||||
def has(self, shell_id: str) -> bool:
|
||||
with self._lock:
|
||||
return shell_id in self._shells
|
||||
|
||||
def shells(self, owner: str | None = None) -> list[BackgroundShell]:
|
||||
"""Snapshot of the given scope's shells, in spawn order."""
|
||||
with self._lock:
|
||||
return [s for s in self._shells.values() if s.owner == owner]
|
||||
|
||||
def read(
|
||||
self, shell_id: str, *, owner: str | None = None, filter_pattern: str | None = None
|
||||
) -> ShellRead:
|
||||
"""Return output produced since the last read of ``shell_id``.
|
||||
|
||||
``filter_pattern`` (a regex, ``search`` semantics per line — the
|
||||
tool-facing ``filter`` arg) narrows what is RETURNED, not what is
|
||||
consumed: on a successful read the cursor advances past the whole
|
||||
delta. A failed or timed-out filter consumes NOTHING — the model
|
||||
can retry without the filter and still get its output. Raises
|
||||
:class:`UnknownShellError` outside the caller's scope, ``re.error``
|
||||
for a bad pattern, and :class:`FilterTimeoutError` for a pattern
|
||||
that blows the time bound (catastrophic backtracking).
|
||||
"""
|
||||
shell = self._get(shell_id, owner)
|
||||
pattern: re.Pattern[str] | None = None
|
||||
if filter_pattern:
|
||||
if len(filter_pattern) > _MAX_FILTER_PATTERN_CHARS:
|
||||
raise re.error( # noqa: TRY003 — mirrors re.compile's own error type
|
||||
f"filter pattern too long ({len(filter_pattern)} chars, "
|
||||
f"max {_MAX_FILTER_PATTERN_CHARS})"
|
||||
)
|
||||
pattern = re.compile(filter_pattern)
|
||||
# Serialize the whole pass: concurrent reads of one shell (a
|
||||
# parallel tool batch) would otherwise snapshot the same cursor and
|
||||
# each return the full delta as "new".
|
||||
with shell.read_serial:
|
||||
delta, gap, new_cursor, status, exit_code = shell._snapshot_delta()
|
||||
clipped = 0
|
||||
if pattern is None:
|
||||
shown = delta
|
||||
else:
|
||||
# Raises FilterTimeoutError / FilterExecError BEFORE the
|
||||
# commit below — a failed filter consumes nothing.
|
||||
shown = _filter_lines_bounded(pattern, delta, shell.shell_id)
|
||||
clipped = sum(1 for ln in delta if len(ln) > _FILTER_MAX_LINE_CHARS)
|
||||
shell._commit_cursor(new_cursor)
|
||||
return ShellRead(
|
||||
shell_id=shell.shell_id,
|
||||
status=status,
|
||||
exit_code=exit_code,
|
||||
lines=shown,
|
||||
new_line_count=len(delta),
|
||||
dropped_lines=gap,
|
||||
clipped_lines=clipped,
|
||||
)
|
||||
|
||||
# -- Termination ---------------------------------------------------------
|
||||
|
||||
@staticmethod
|
||||
def _signal_group(shell: BackgroundShell) -> None:
|
||||
"""SIGKILL the shell's group IFF its leader is still running.
|
||||
|
||||
The liveness guard is load-bearing: a completed shell's ``pgid`` is
|
||||
an hours-stale snapshot the OS may have recycled to an unrelated
|
||||
process group — signalling it unconditionally would let the
|
||||
auto-approved ``kill_shell`` (or a routine ``close()``) SIGKILL
|
||||
another tenant's processes. A leader that exits between the
|
||||
``poll()`` and the ``killpg`` leaves the same microseconds-wide
|
||||
pid-wraparound TOCTOU as the foreground tool — accepted there,
|
||||
accepted here. The guard also keeps a kill racing a natural exit
|
||||
honest: the waiter labels the shell ``completed`` (with its real
|
||||
exit code and notice) instead of ``killed``.
|
||||
"""
|
||||
with shell.lock:
|
||||
if shell.proc.poll() is not None:
|
||||
return # already exited — the waiter's own group kill ran/runs
|
||||
shell._killed = True
|
||||
# killpg INSIDE the lock: poll-and-signal is atomic wrt our own
|
||||
# bookkeeping (nothing can observe _killed without the signal
|
||||
# having been attempted). The lock is never held around other
|
||||
# locks, so this cannot deadlock; the OS-level microseconds
|
||||
# pid-wraparound TOCTOU is the same accepted one as always.
|
||||
with contextlib.suppress(OSError, ProcessLookupError):
|
||||
os.killpg(shell.pgid, signal.SIGKILL)
|
||||
|
||||
@staticmethod
|
||||
def _join_threads(shells: list[BackgroundShell], budget_s: float) -> bool:
|
||||
"""Join every shell thread under ONE shared deadline; True if all done.
|
||||
|
||||
The budget is total, not per-thread: teardown latency must not stack
|
||||
by shell count (``close()`` can run under the server's async close
|
||||
route — see :data:`_CLOSE_JOIN_BUDGET_S`). Stragglers are daemons;
|
||||
the caller logs and abandons them.
|
||||
"""
|
||||
deadline = time.monotonic() + budget_s
|
||||
done = True
|
||||
for shell in shells:
|
||||
for t in shell._threads:
|
||||
# suppress: joining a never-started Thread raises
|
||||
# RuntimeError. spawn() unregisters on a start failure, so
|
||||
# this is pure belt — teardown must never die on a join.
|
||||
with contextlib.suppress(RuntimeError):
|
||||
t.join(timeout=max(0.0, deadline - time.monotonic()))
|
||||
done = done and not t.is_alive()
|
||||
return done
|
||||
|
||||
def kill(self, shell_id: str, *, owner: str | None = None) -> BackgroundShell:
|
||||
"""SIGKILL ``shell_id``'s whole group; return its (updated) record.
|
||||
|
||||
Suppresses the exit callback — the caller asked for this exit, so
|
||||
there is nothing to announce. Killing an already-exited shell
|
||||
signals nothing (see :meth:`_signal_group`) and returns the record
|
||||
unchanged. On return the record is usually terminal; a leader in
|
||||
uninterruptible sleep can still read ``running`` after the join
|
||||
budget — callers report that honestly rather than assuming.
|
||||
"""
|
||||
shell = self._get(shell_id, owner)
|
||||
self._signal_group(shell)
|
||||
if not self._join_threads([shell], _WAITER_JOIN_TIMEOUT_S):
|
||||
log.warning("bg_shell.kill_join_timeout", shell_id=shell.shell_id, pid=shell.pid)
|
||||
return shell
|
||||
|
||||
def reap(self, *, owner: str | None) -> None:
|
||||
"""Kill every shell belonging to ``owner`` and drop their records.
|
||||
|
||||
Used by the task_agent teardown: a sub-agent's shells are bound to
|
||||
the sub-agent's lifetime (never handed to the parent), and dropping
|
||||
the records keeps dead ``bash_N`` handles from cluttering scope
|
||||
listings. Suppresses the exit callback for the shells it kills,
|
||||
same as :meth:`kill` — teardown is the caller's own act, there is
|
||||
nothing to announce.
|
||||
"""
|
||||
with self._lock:
|
||||
mine = [s for s in self._shells.values() if s.owner == owner]
|
||||
for shell in mine:
|
||||
self._signal_group(shell)
|
||||
if not self._join_threads(mine, _WAITER_JOIN_TIMEOUT_S):
|
||||
log.warning("bg_shell.reap_join_timeout", owner=owner or "")
|
||||
with self._lock:
|
||||
for shell in mine:
|
||||
self._shells.pop(shell.shell_id, None)
|
||||
|
||||
def signal_all(self) -> None:
|
||||
"""SIGKILL every live shell's group WITHOUT joining or unregistering.
|
||||
|
||||
The instant half of teardown, separated so multi-session frontends
|
||||
can bound their total exit latency: signal every session's groups
|
||||
first (microseconds each), then pay the join budgets — or, on an
|
||||
impatient Ctrl-C, signal alone still guarantees no process outlives
|
||||
the frontend even though the joins are skipped. Liveness-guarded
|
||||
per shell (:meth:`_signal_group`), so completed shells' stale pgids
|
||||
are never touched. Idempotent; :meth:`close` remains the complete
|
||||
teardown.
|
||||
"""
|
||||
with self._lock:
|
||||
shells = list(self._shells.values())
|
||||
for shell in shells:
|
||||
self._signal_group(shell)
|
||||
|
||||
def close(self) -> None:
|
||||
"""Kill everything, join threads under a total budget, refuse spawns.
|
||||
|
||||
Idempotent; called from ``ChatSession.close()`` (the funnel every
|
||||
workstream-teardown path runs through). Signals are issued to all
|
||||
groups first (instant), then ONE shared join budget covers every
|
||||
thread — a pathological shell (escaped-group grandchild holding the
|
||||
pipes, D-state leader) delays teardown by at most
|
||||
:data:`_CLOSE_JOIN_BUDGET_S`, with the stragglers logged and left
|
||||
to die as daemons. Records are dropped so a queued exit notice's
|
||||
``valid_until`` predicate (``has(shell_id)``) goes stale and the
|
||||
drain discards it — nobody is left to read it.
|
||||
"""
|
||||
with self._lock:
|
||||
if self._closed:
|
||||
return
|
||||
self._closed = True
|
||||
shells = list(self._shells.values())
|
||||
for shell in shells:
|
||||
self._signal_group(shell)
|
||||
if not self._join_threads(shells, _CLOSE_JOIN_BUDGET_S):
|
||||
leaked = [t.name for s in shells for t in s._threads if t.is_alive()]
|
||||
log.warning("bg_shell.close_join_timeout", leaked=",".join(leaked))
|
||||
with self._lock:
|
||||
self._shells.clear()
|
||||
@@ -630,6 +630,12 @@ def project_history_messages(
|
||||
result_call_id = msg.get("tool_call_id")
|
||||
if result_call_id:
|
||||
entry["tool_call_id"] = str(result_call_id)
|
||||
# Preview-pane descriptor → top-level ``preview``, mirroring the
|
||||
# live ``tool_result`` SSE event's field so replay renders the
|
||||
# same reopen chip the live path did.
|
||||
preview = msg.get("_preview")
|
||||
if isinstance(preview, dict) and preview:
|
||||
entry["preview"] = preview
|
||||
if isinstance(content, list):
|
||||
# Renderers require a string (``replayHistory`` calls
|
||||
# ``stripAnsi(content).trim()``; coord joins text parts), so
|
||||
|
||||
@@ -1,8 +1,18 @@
|
||||
"""Idle wake-trigger for the metacog NudgeQueue pipeline.
|
||||
"""Idle wake-triggers for the metacog NudgeQueue pipeline.
|
||||
|
||||
Hosts :class:`IdleNudgeWatcher` plus the
|
||||
:func:`install_idle_nudge_watcher` / :func:`shutdown_idle_nudge_watchers`
|
||||
lifespan helpers. Pulled out of :mod:`turnstone.core.metacognition`
|
||||
Hosts the two wake entry points plus their lifespan helpers:
|
||||
|
||||
* :class:`IdleNudgeWatcher` — event-driven: a workstream transitions
|
||||
to IDLE while nudges are ALREADY queued.
|
||||
* :func:`wake_workstream_if_pending` — the shared wake gate, also
|
||||
called directly by asynchronous producers that enqueue onto an
|
||||
ALREADY-idle workstream (the watch dispatch closure, via
|
||||
``ChatSession.set_watch_runner``'s ``wake_fn``). Such producers see
|
||||
no IDLE transition — the workstream has been idle all along — so
|
||||
the watcher alone would leave their entries queued until the next
|
||||
user message.
|
||||
|
||||
Pulled out of :mod:`turnstone.core.metacognition`
|
||||
because the watcher is subscriber-lifecycle / runtime-orchestration
|
||||
code with different concerns from the static nudge-text templates and
|
||||
detection heuristics that live in metacognition; mixing them grew the
|
||||
@@ -16,32 +26,132 @@ from typing import TYPE_CHECKING, Any
|
||||
|
||||
from turnstone.core import session_worker
|
||||
from turnstone.core.log import get_logger
|
||||
from turnstone.core.nudge_queue import USER_DRAIN
|
||||
from turnstone.core.nudge_queue import WAKE_PENDING, NudgeQueue
|
||||
from turnstone.core.workstream import WorkstreamState
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from collections.abc import Callable
|
||||
|
||||
from turnstone.core.session_manager import SessionManager
|
||||
from turnstone.core.workstream import Workstream
|
||||
|
||||
log = get_logger(__name__)
|
||||
|
||||
|
||||
def wake_workstream_if_pending(ws: Workstream, *, trigger: str = "unspecified") -> bool:
|
||||
"""Spawn a wake send for *ws* when it is idle with drainable nudges.
|
||||
|
||||
The shared gate behind both wake triggers:
|
||||
|
||||
* :class:`IdleNudgeWatcher` — the workstream just transitioned to
|
||||
IDLE with nudges already queued.
|
||||
* the watch dispatch closure (``ChatSession.set_watch_runner``'s
|
||||
``wake_fn``) — a watch fired on a workstream that is ALREADY
|
||||
idle, so no IDLE transition will ever re-check the queue.
|
||||
|
||||
*trigger* is a short label naming which path requested the wake
|
||||
(``"idle-transition"``, ``"watch-fire"``, ``"worker-exit"``); it is
|
||||
used only to tag the log lines below and never affects control flow.
|
||||
|
||||
Gates, in order:
|
||||
|
||||
* ``ws.session is None`` — workstream tracked but session not
|
||||
built — or a session whose ``_nudge_queue`` is not a real
|
||||
:class:`NudgeQueue` (bare stubs; mock sessions). The wake
|
||||
contract REQUIRES real drain semantics: the spawned worker's
|
||||
``deliver_wake_nudge_from_queue`` must actually CONSUME what
|
||||
``has_pending`` saw, or the worker-exit backstop respawns wake
|
||||
workers forever — a mock queue's truthy ``has_pending`` plus a
|
||||
no-op deliver is exactly that storm, so the gate refuses on
|
||||
TYPE, not just presence.
|
||||
* ``ws._closed`` — ``close()`` already ran (or is racing us); its
|
||||
storage row says ``closed`` and a wake send would drive a
|
||||
torn-down session. Lockless FAST-PATH only: a stale ``False``
|
||||
falls through to ``session_worker.send``, which re-checks
|
||||
``_closed`` under ``ws._lock`` — the same lock ``close()`` sets
|
||||
it under — and refuses, so a wake racing a close can never spawn
|
||||
a worker on the torn-down session.
|
||||
* ``ws.state is not IDLE`` — a busy workstream's worker drains the
|
||||
queue at its own seams (``ATTENTION``/``THINKING``/``RUNNING``
|
||||
all imply a live worker), and ``ERROR`` stays parked for the
|
||||
operator rather than burning inference unattended.
|
||||
* nothing gate-eligible under ``WAKE_PENDING`` — tool-only/quiet entries
|
||||
belong to the next tool-result seam, not a synthetic empty user
|
||||
turn (``deliver_wake_nudge_from_queue`` would no-op on them).
|
||||
|
||||
Past the gates, exactly one info line is emitted per call:
|
||||
|
||||
* ``nudge_wake.deferred_worker_busy`` — the reuse-path drop: a
|
||||
worker owned the workstream, so ``session_worker.send`` called
|
||||
the no-op ``enqueue`` instead of spawning. The entry stays
|
||||
queued; the owning worker's exit backstop (or its next drain
|
||||
seam) delivers it.
|
||||
* ``nudge_wake.dispatched`` — a fresh wake daemon was spawned.
|
||||
* ``nudge_wake.refused`` — ``session_worker.send`` declined the
|
||||
spawn: its authoritative under-lock ``_closed`` re-check caught a
|
||||
teardown this gate's lockless peek missed. The entry dies with
|
||||
the workstream; logged here so a dropped wake is traceable to its
|
||||
trigger during production troubleshooting.
|
||||
|
||||
Returns ``True`` iff the wake was handed to
|
||||
``session_worker.send`` — which may still downgrade it to a no-op
|
||||
enqueue when a worker owns the workstream (see the race-semantics
|
||||
section on :class:`IdleNudgeWatcher`).
|
||||
"""
|
||||
session = ws.session
|
||||
if session is None or ws._closed or ws.state is not WorkstreamState.IDLE:
|
||||
return False
|
||||
nudge_queue = getattr(session, "_nudge_queue", None)
|
||||
# Gate on WAKE_PENDING, not USER_DRAIN: ``"quiet"`` entries (external
|
||||
# events demoted by a user cancel) deliver at the next legitimate seam
|
||||
# but must never themselves wake the workstream the user just stopped.
|
||||
if not isinstance(nudge_queue, NudgeQueue) or not nudge_queue.has_pending(WAKE_PENDING):
|
||||
return False
|
||||
|
||||
deferred = False
|
||||
|
||||
def _noop_enqueue() -> None:
|
||||
nonlocal deferred
|
||||
deferred = True
|
||||
|
||||
ok = session_worker.send(
|
||||
ws,
|
||||
enqueue=_noop_enqueue,
|
||||
run=session.deliver_wake_nudge_from_queue,
|
||||
thread_name=f"wake-nudge-{ws.id[:8]}",
|
||||
)
|
||||
if deferred:
|
||||
log.info("nudge_wake.deferred_worker_busy ws=%s trigger=%s", ws.id[:8], trigger)
|
||||
elif ok:
|
||||
log.info("nudge_wake.dispatched ws=%s trigger=%s", ws.id[:8], trigger)
|
||||
else:
|
||||
log.info("nudge_wake.refused ws=%s trigger=%s", ws.id[:8], trigger)
|
||||
return ok
|
||||
|
||||
|
||||
class IdleNudgeWatcher:
|
||||
"""Convert a workstream IDLE transition into a wake send when the
|
||||
session has queued nudges.
|
||||
|
||||
Subscribes to :meth:`SessionManager.subscribe_to_state` and listens
|
||||
for ``WorkstreamState.IDLE``. If the workstream's
|
||||
for ``WorkstreamState.IDLE``, then defers to
|
||||
:func:`wake_workstream_if_pending` (the shared gate — see its
|
||||
docstring for the full gate order). If the workstream's
|
||||
:class:`NudgeQueue` has any drainable entry for the wake's drain
|
||||
filter (``USER_DRAIN`` — channels ``"user"`` or ``"any"``),
|
||||
dispatches via ``session_worker.send`` with a no-op ``enqueue``
|
||||
callback. Tool-only entries don't fire the wake — they belong to
|
||||
the next tool-result seam, not a synthetic empty user turn —
|
||||
otherwise every IDLE event with a queued tool advisory would spawn
|
||||
a wake daemon that immediately no-ops at
|
||||
gate (``WAKE_PENDING`` — channels ``"user"`` or ``"any"``), the
|
||||
gate dispatches via ``session_worker.send`` with a no-op
|
||||
``enqueue`` callback. Tool-only entries don't fire the wake —
|
||||
they belong to the next tool-result seam, not a synthetic empty
|
||||
user turn — otherwise every IDLE event with a queued tool advisory
|
||||
would spawn a wake daemon that immediately no-ops at
|
||||
``deliver_wake_nudge_from_queue``'s drain guard.
|
||||
|
||||
This watcher only covers nudges that are already queued when the
|
||||
IDLE transition fires. Producers that enqueue asynchronously onto
|
||||
an already-idle workstream (watch fires) call
|
||||
:func:`wake_workstream_if_pending` themselves — there is no state
|
||||
transition for this watcher to observe in that case.
|
||||
|
||||
**Race semantics.** ``session_worker.send`` decides atomically
|
||||
under ``ws._lock`` whether a worker thread already owns the
|
||||
workstream. Three outcomes:
|
||||
@@ -51,10 +161,17 @@ class IdleNudgeWatcher:
|
||||
drains its own queue and runs the synthetic empty-user turn).
|
||||
* Worker running → call our ``enqueue`` lambda, which is a no-op.
|
||||
The wake is silently dropped; the queued nudge stays in
|
||||
``NudgeQueue`` and the in-flight worker picks it up at its next
|
||||
user-message-attach or tool-result seam (whichever fires first
|
||||
for the entry's channel). This is the load-bearing fallback —
|
||||
we never spawn a competing worker.
|
||||
``NudgeQueue``. We never spawn a competing worker. This branch
|
||||
is the COMMON case for IDLE-transition wakes, not the exception:
|
||||
``set_state`` subscribers fire on the calling thread, and IDLE
|
||||
is emitted from inside ``run()`` at the end of a send — so the
|
||||
transitioning worker still owns the flag while this watcher
|
||||
dispatches. Delivery is then owed to one of two follow-ups:
|
||||
the in-flight worker's next drain seam (when IDLE fired
|
||||
mid-turn), or — for the end-of-send case, where no later seam
|
||||
exists — ``session_worker``'s ownership-clear backstop
|
||||
(``_retry_pending_wake``), which re-runs
|
||||
:func:`wake_workstream_if_pending` the moment the worker exits.
|
||||
* Workstream gone (``ws is None``) or session not built
|
||||
(``ws.session is None``) → bail.
|
||||
|
||||
@@ -82,17 +199,9 @@ class IdleNudgeWatcher:
|
||||
if state is not WorkstreamState.IDLE:
|
||||
return
|
||||
ws = self._manager.get(ws_id)
|
||||
if ws is None or ws.session is None:
|
||||
if ws is None:
|
||||
return
|
||||
session = ws.session
|
||||
if not session._nudge_queue.has_pending(USER_DRAIN):
|
||||
return
|
||||
session_worker.send(
|
||||
ws,
|
||||
enqueue=lambda: None,
|
||||
run=session.deliver_wake_nudge_from_queue,
|
||||
thread_name=f"wake-nudge-{ws.id[:8]}",
|
||||
)
|
||||
wake_workstream_if_pending(ws, trigger="idle-transition")
|
||||
|
||||
self._callback = _on_state
|
||||
self._manager.subscribe_to_state(_on_state)
|
||||
|
||||
+59
-6
@@ -15,7 +15,7 @@ import re
|
||||
import threading
|
||||
import time
|
||||
import uuid
|
||||
from dataclasses import dataclass, field
|
||||
from dataclasses import dataclass, field, fields, replace
|
||||
from functools import partial
|
||||
from pathlib import Path
|
||||
from typing import TYPE_CHECKING, Any
|
||||
@@ -30,7 +30,7 @@ from turnstone.core.log import get_logger
|
||||
if TYPE_CHECKING:
|
||||
from collections.abc import Callable
|
||||
|
||||
from turnstone.core.providers._protocol import LLMProvider
|
||||
from turnstone.core.providers._protocol import LLMProvider, ModelCapabilities
|
||||
|
||||
log = get_logger(__name__)
|
||||
|
||||
@@ -844,6 +844,33 @@ def _positive_window(*candidates: Any, floor: int = _DEFAULT_JUDGE_CONTEXT_WINDO
|
||||
return floor
|
||||
|
||||
|
||||
def _resolve_model_capabilities(provider: LLMProvider, model: str, cfg: Any) -> ModelCapabilities:
|
||||
"""Provider base capabilities with a model definition's ``capabilities``
|
||||
overrides applied — the same lowering ``ChatSession._resolve_capabilities``
|
||||
performs for the session, utility, and sub-agent completion lanes.
|
||||
|
||||
The judges are the only completion callers that live outside ``ChatSession``,
|
||||
so they cannot reach ``self._resolve_capabilities``; this mirrors it so a
|
||||
judge alias honors operator-declared capabilities (effort passthrough, tool
|
||||
support, temperature, verbosity) exactly like the main loop. ``cfg`` is the
|
||||
alias's ``ModelConfig``; a missing or non-dict ``capabilities`` is ignored
|
||||
rather than raised — capability resolution must never crash a judge turn.
|
||||
|
||||
It does NOT fold in ``ModelConfig.context_window``: that is a separate field,
|
||||
not part of the capabilities JSON, and the caller sizes the judge's window
|
||||
budget off it directly (the static caps table reports 200000 for local
|
||||
models, which would silently over-budget them).
|
||||
"""
|
||||
caps = provider.get_capabilities(model)
|
||||
overrides = getattr(cfg, "capabilities", None)
|
||||
if isinstance(overrides, dict) and overrides:
|
||||
names = {f.name for f in fields(type(caps))}
|
||||
applied = {k: v for k, v in overrides.items() if k in names}
|
||||
if applied:
|
||||
caps = replace(caps, **applied)
|
||||
return caps
|
||||
|
||||
|
||||
def honest_truncate(text: str, budget: int) -> str:
|
||||
"""Return *text* untouched when it fits *budget* characters, otherwise the
|
||||
leading ``budget`` characters followed by an explicit note of exactly how
|
||||
@@ -971,13 +998,21 @@ class IntentJudge:
|
||||
session_provider: LLMProvider,
|
||||
session_client: Any,
|
||||
session_model: str,
|
||||
context_window: int = 200_000,
|
||||
session_capabilities: ModelCapabilities | None = None,
|
||||
rule_registry: Any | None = None,
|
||||
model_registry: Any | None = None,
|
||||
) -> None:
|
||||
self._config = config
|
||||
self._context_window = context_window
|
||||
self._rule_registry = rule_registry
|
||||
# The caller (ChatSession) resolves the session model's real caps from
|
||||
# _get_capabilities (config/registry-aware) and passes them in; they are
|
||||
# this judge's wire capabilities and window when it inherits the session
|
||||
# model. The window is taken ONLY from these resolved caps (else a
|
||||
# floor) — NEVER provider.get_capabilities(), whose static 200000 for a
|
||||
# local model would blind the budget to overflow.
|
||||
session_window = (
|
||||
session_capabilities.context_window if session_capabilities is not None else None
|
||||
)
|
||||
|
||||
# Resolve judge model via ModelRegistry alias, otherwise self-
|
||||
# consistency on the session model. ``judge.model`` is alias-only
|
||||
@@ -1001,6 +1036,9 @@ class IntentJudge:
|
||||
self._provider.provider_name,
|
||||
)
|
||||
self._model = model_name
|
||||
self._capabilities = _resolve_model_capabilities(
|
||||
self._provider, self._model, model_cfg
|
||||
)
|
||||
# Use the registry's per-model context window, NOT
|
||||
# ``provider.get_capabilities().context_window``: the static
|
||||
# capability table returns 200000 for every model absent
|
||||
@@ -1013,7 +1051,8 @@ class IntentJudge:
|
||||
# the session ``context_window`` then a floor, so it neither
|
||||
# aborts resolution nor zeroes the budgets.
|
||||
self._judge_context_window = _positive_window(
|
||||
getattr(model_cfg, "context_window", None), context_window
|
||||
getattr(model_cfg, "context_window", None),
|
||||
session_window,
|
||||
)
|
||||
resolved = True
|
||||
except Exception:
|
||||
@@ -1034,8 +1073,15 @@ class IntentJudge:
|
||||
session_provider.provider_name,
|
||||
)
|
||||
self._model = session_model
|
||||
# Wire caps: the caller's resolved session caps, or the provider's
|
||||
# static table as a last resort for degraded / legacy callers.
|
||||
self._capabilities = (
|
||||
session_capabilities
|
||||
if session_capabilities is not None
|
||||
else session_provider.get_capabilities(session_model)
|
||||
)
|
||||
# Coerce here too, defensively against a non-positive session window.
|
||||
self._judge_context_window = _positive_window(context_window)
|
||||
self._judge_context_window = _positive_window(session_window)
|
||||
|
||||
# -- Client lifecycle helpers -------------------------------------------
|
||||
|
||||
@@ -1336,6 +1382,13 @@ class IntentJudge:
|
||||
max_tokens=2048,
|
||||
temperature=0.0,
|
||||
reasoning_effort="medium",
|
||||
# Thread the judge model's operator-declared capabilities
|
||||
# onto the wire like every other lane — resolved from the
|
||||
# judge alias's model definition, or the session model on
|
||||
# fallback. Without this the provider would fall back to
|
||||
# its static capability table and silently ignore the
|
||||
# definition's overrides on judge calls alone.
|
||||
capabilities=self._capabilities,
|
||||
),
|
||||
timeout=per_call_timeout,
|
||||
cancel_event=cancel_event,
|
||||
|
||||
+101
-14
@@ -18,7 +18,15 @@ This module owns the three provider-neutral lowering passes:
|
||||
``deepseek_v4``, which ``json.loads`` the arguments at request-render time)
|
||||
can't reject the whole request. Mutates the transient wire copy only — the
|
||||
canonical trajectory keeps the raw output. See
|
||||
:func:`sanitize_tool_call_arguments`.
|
||||
:func:`sanitize_tool_call_arguments`. The id sibling,
|
||||
:func:`restore_provider_tool_ids`, maps session-minted sub-agent tool ids
|
||||
(``{parent}::r{run}s{step}::{provider_id}``) back to the provider's OWN ids
|
||||
on the wire copy, so the provider-native block lane — whose ``tool_use``
|
||||
blocks carry the provider id verbatim, under a reasoning signature that must
|
||||
never be touched — stays id-consistent with the top-level mirror and the
|
||||
tool results. It runs at the AGENT wire seam (``ChatSession._run_agent``'s
|
||||
``_api_call``) only — main-loop ids are provider-issued or uuid-filled and
|
||||
already consistent with their native lane.
|
||||
* **repair** (validity) — synthesizing cancellation results for orphaned client
|
||||
tool calls. See :func:`repair_wire_messages`.
|
||||
|
||||
@@ -224,7 +232,7 @@ def tool_args_preview(arguments: Any) -> str:
|
||||
return _ARGS_PREVIEW_CONTROL_RE.sub(" ", redact_credentials(text))[:120]
|
||||
|
||||
|
||||
def _legalized_arguments(arguments: Any) -> str | None:
|
||||
def legalized_arguments(arguments: Any) -> str | None:
|
||||
"""A wire-valid replacement for *arguments*, or ``None`` if already valid.
|
||||
|
||||
A raw ``dict`` (an internal shape that reached the wire seat) is serialized;
|
||||
@@ -243,6 +251,33 @@ def _legalized_arguments(arguments: Any) -> str | None:
|
||||
return "{}"
|
||||
|
||||
|
||||
def legalize_tool_call_entry(tc: dict[str, Any]) -> dict[str, Any] | None:
|
||||
"""A legalized copy of one wire tool-call entry, or ``None`` when its
|
||||
``arguments`` are already wire-valid — or its ``function`` is not a dict
|
||||
(someone else's malformation to surface, not silently rename).
|
||||
|
||||
Emits the standard ``wire.tool_args_legalized`` breadcrumb when it fixes
|
||||
an entry. The ONE per-entry legalizer: shared by
|
||||
:func:`sanitize_tool_call_arguments` and the Google fidelity swap
|
||||
(``providers/_google.py``), which re-applies the same floor to the raw
|
||||
provider dicts it swaps over the sanitized mirror — so the two seats
|
||||
cannot drift on what "wire-valid" means or on the diagnosis trail.
|
||||
"""
|
||||
fn = tc.get("function")
|
||||
if not isinstance(fn, dict):
|
||||
return None
|
||||
replacement = legalized_arguments(fn.get("arguments"))
|
||||
if replacement is None:
|
||||
return None # already wire-valid — leave byte-for-byte untouched
|
||||
log.debug(
|
||||
"wire.tool_args_legalized",
|
||||
tool=fn.get("name", "?"),
|
||||
call_id=tc.get("id", ""),
|
||||
raw_preview=tool_args_preview(fn.get("arguments")),
|
||||
)
|
||||
return {**tc, "function": {**fn, "arguments": replacement}}
|
||||
|
||||
|
||||
def sanitize_tool_call_arguments(messages: list[dict[str, Any]]) -> list[dict[str, Any]]:
|
||||
"""Return *messages* with every assistant tool call's ``arguments`` made
|
||||
wire-valid — the legalize pass (see the module docstring).
|
||||
@@ -265,21 +300,12 @@ def sanitize_tool_call_arguments(messages: list[dict[str, Any]]) -> list[dict[st
|
||||
continue
|
||||
repaired: list[dict[str, Any]] | None = None
|
||||
for ci, tc in enumerate(msg["tool_calls"]):
|
||||
fn = tc.get("function")
|
||||
if not isinstance(fn, dict):
|
||||
fixed = legalize_tool_call_entry(tc)
|
||||
if fixed is None:
|
||||
continue
|
||||
replacement = _legalized_arguments(fn.get("arguments"))
|
||||
if replacement is None:
|
||||
continue # already wire-valid — leave byte-for-byte untouched
|
||||
if repaired is None:
|
||||
repaired = list(msg["tool_calls"])
|
||||
log.debug(
|
||||
"wire.tool_args_legalized",
|
||||
tool=fn.get("name", "?"),
|
||||
call_id=tc.get("id", ""),
|
||||
raw_preview=tool_args_preview(fn.get("arguments")),
|
||||
)
|
||||
repaired[ci] = {**tc, "function": {**fn, "arguments": replacement}}
|
||||
repaired[ci] = fixed
|
||||
if repaired is not None:
|
||||
if out is None:
|
||||
out = list(messages)
|
||||
@@ -287,6 +313,67 @@ def sanitize_tool_call_arguments(messages: list[dict[str, Any]]) -> list[dict[st
|
||||
return messages if out is None else out
|
||||
|
||||
|
||||
def restore_provider_tool_ids(
|
||||
messages: list[dict[str, Any]], id_map: dict[str, str]
|
||||
) -> list[dict[str, Any]]:
|
||||
"""Return *messages* with session-minted sub-agent tool ids mapped back to
|
||||
the provider's own ids — the id half of the legalize pass, applied at the
|
||||
AGENT wire seam.
|
||||
|
||||
*id_map* is the per-``_run_agent`` ``{minted_id: provider_original_id}``
|
||||
record built at the mint site. Rewriting assistant ``tool_calls[*].id``
|
||||
and tool ``tool_call_id`` back to the provider originals makes every wire
|
||||
representation agree: the provider-native ``tool_use`` block (which holds
|
||||
the provider id verbatim and must never be rewritten — its bytes sit under
|
||||
the turn's reasoning signature), the top-level ``tool_calls`` mirror, and
|
||||
the ``tool_result``. A translator that replays the native lane and one
|
||||
that rebuilds from ``tool_calls`` therefore emit the same ids, so the
|
||||
pairing holds on both paths. The minted id stays the internal key
|
||||
(registry / DOM / recall / cancel ledger) untouched — only the transient
|
||||
wire copy is mapped.
|
||||
|
||||
Replaying the provider's own ids to the producing provider is the proven
|
||||
prior behaviour, including the duplicate-ish ids a local server that
|
||||
reissues per-response ids ("call_0") produces — an agent run is pinned to
|
||||
one provider, so the ids always return to the backend that issued them.
|
||||
Ids not in the map (uuid back-fills, an unparented run that never minted)
|
||||
pass through untouched; recovery is by MAP ONLY, never by string-splitting
|
||||
the mint suffix (provider ids can contain surprising characters,
|
||||
including the mint's own delimiter).
|
||||
|
||||
Copy-on-write + identity-preserving, exactly like
|
||||
:func:`sanitize_tool_call_arguments`: an empty map or a conversation with
|
||||
no minted id returns the same object.
|
||||
"""
|
||||
if not id_map:
|
||||
return messages
|
||||
out: list[dict[str, Any]] | None = None # copy-on-write: None until first fix
|
||||
for idx, msg in enumerate(messages):
|
||||
role = msg.get("role")
|
||||
if role == "assistant" and msg.get("tool_calls"):
|
||||
repaired: list[dict[str, Any]] | None = None
|
||||
for ci, tc in enumerate(msg["tool_calls"]):
|
||||
tc_id = tc.get("id")
|
||||
original = id_map.get(tc_id) if isinstance(tc_id, str) else None
|
||||
if original is None or original == tc_id:
|
||||
continue
|
||||
if repaired is None:
|
||||
repaired = list(msg["tool_calls"])
|
||||
repaired[ci] = {**tc, "id": original}
|
||||
if repaired is not None:
|
||||
if out is None:
|
||||
out = list(messages)
|
||||
out[idx] = {**msg, "tool_calls": repaired}
|
||||
elif role == "tool":
|
||||
tc_id = msg.get("tool_call_id")
|
||||
original = id_map.get(tc_id) if isinstance(tc_id, str) else None
|
||||
if original is not None and original != tc_id:
|
||||
if out is None:
|
||||
out = list(messages)
|
||||
out[idx] = {**msg, "tool_call_id": original}
|
||||
return messages if out is None else out
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# Fold — operator-context representation (A); runs BEFORE repair on the wire.
|
||||
# --------------------------------------------------------------------------- #
|
||||
|
||||
@@ -158,6 +158,10 @@ _NUDGE_MAP: dict[str, str] = {
|
||||
# consumers recognise the type.
|
||||
"idle_children": "",
|
||||
"watch_triggered": "",
|
||||
# background_shell_exit (#817) likewise: per-fire text is composed by
|
||||
# ``ChatSession._on_background_shell_exit`` and rides the shared
|
||||
# external-event rail, never :func:`format_nudge`.
|
||||
"background_shell_exit": "",
|
||||
# participant_joined likewise carries no static body — the per-fire text
|
||||
# ("<name> has joined this shared workstream…") is composed by its producer
|
||||
# (``ChatSession._maybe_note_new_participant``) and emitted via
|
||||
|
||||
@@ -145,9 +145,35 @@ class ModelRegistry:
|
||||
raise ValueError(f"Unknown model alias: {alias}")
|
||||
if alias not in self._clients:
|
||||
cfg = self._models[alias]
|
||||
self._clients[alias] = create_client(
|
||||
cfg.provider, base_url=cfg.base_url, api_key=cfg.api_key
|
||||
)
|
||||
try:
|
||||
self._clients[alias] = create_client(
|
||||
cfg.provider, base_url=cfg.base_url, api_key=cfg.api_key
|
||||
)
|
||||
except ValueError:
|
||||
# create_client's own misconfig errors already carry
|
||||
# remediation text — pass through untouched.
|
||||
raise
|
||||
except Exception as exc:
|
||||
# SDK construction can fail on environment problems the
|
||||
# config never sees — e.g. httpx resolving a CA-bundle
|
||||
# path that a venv rebuild deleted (FileNotFoundError).
|
||||
# Routes map ValueError to a 503 with the message;
|
||||
# anything else surfaces as an opaque 500, so re-type
|
||||
# here where the alias is known. The ValueError text is
|
||||
# echoed to HTTP callers, so it carries only the
|
||||
# exception TYPE — arbitrary SDK exception text can
|
||||
# embed filesystem paths; the full detail goes to the
|
||||
# server log instead.
|
||||
log.warning(
|
||||
"Client construction failed for model alias %r (provider %s)",
|
||||
alias,
|
||||
cfg.provider,
|
||||
exc_info=True,
|
||||
)
|
||||
raise ValueError(
|
||||
f"failed to construct {cfg.provider} client for model "
|
||||
f"alias {alias!r}: {type(exc).__name__} (details in server log)"
|
||||
) from exc
|
||||
return self._clients[alias]
|
||||
|
||||
def get_provider(self, alias: str) -> LLMProvider:
|
||||
|
||||
+112
-14
@@ -15,7 +15,13 @@ Channels:
|
||||
* ``"tool"`` — only drains at tool-result seams.
|
||||
* ``"any"`` — drains at whichever seam fires first (used for
|
||||
wake-trigger-driven nudges that should not be pinned to a
|
||||
specific drain seam).
|
||||
specific drain seam) AND counts toward the idle-wake gate
|
||||
(:data:`WAKE_PENDING`).
|
||||
* ``"quiet"`` — drains at whichever seam fires first, but does NOT
|
||||
count toward the idle-wake gate. A user cancel demotes pending
|
||||
``"any"`` entries here: the external event (watch fire,
|
||||
background-shell exit) is still delivered at the next seam, but it
|
||||
must not wake the workstream the user just stopped.
|
||||
|
||||
Drain preserves FIFO order; non-matching entries stay queued. Each
|
||||
entry can carry an optional ``valid_until`` predicate that drain
|
||||
@@ -40,19 +46,38 @@ if TYPE_CHECKING:
|
||||
|
||||
log = get_logger(__name__)
|
||||
|
||||
Channel = Literal["user", "tool", "any"]
|
||||
_VALID_CHANNELS: frozenset[str] = frozenset({"user", "tool", "any"})
|
||||
Channel = Literal["user", "tool", "any", "quiet"]
|
||||
_VALID_CHANNELS: frozenset[str] = frozenset({"user", "tool", "any", "quiet"})
|
||||
|
||||
# Module-level filter constants — most callers want one of these and
|
||||
# pre-allocating spares us a frozenset construction at every drain seam.
|
||||
USER_DRAIN: frozenset[str] = frozenset({"user", "any"})
|
||||
TOOL_DRAIN: frozenset[str] = frozenset({"tool", "any"})
|
||||
USER_DRAIN: frozenset[str] = frozenset({"user", "any", "quiet"})
|
||||
TOOL_DRAIN: frozenset[str] = frozenset({"tool", "any", "quiet"})
|
||||
# The idle-wake GATE (``IdleNudgeWatcher``): which pending channels justify
|
||||
# waking an idle workstream. Deliberately excludes ``"quiet"`` — entries a
|
||||
# user cancel demoted must ride the next legitimate seam/wake, never cause
|
||||
# one, or Stop is followed seconds later by an autonomous resume.
|
||||
WAKE_PENDING: frozenset[str] = frozenset({"user", "any"})
|
||||
# The quiet channel, named once: the demotion target for external events a
|
||||
# user cancel must not let re-wake the workstream, and the ride-along drain
|
||||
# the wake path uses after its WAKE_PENDING pass.
|
||||
QUIET_CHANNEL: Channel = "quiet"
|
||||
QUIET_DRAIN: frozenset[str] = frozenset({QUIET_CHANNEL})
|
||||
|
||||
|
||||
class _Entry(NamedTuple):
|
||||
class Entry(NamedTuple):
|
||||
"""One queued nudge. Public so consumers of
|
||||
:meth:`NudgeQueue.drain_entries` can give entries back via
|
||||
:meth:`NudgeQueue.requeue` — which preserves ``valid_until`` AND the
|
||||
original ``seq`` (a plain :meth:`NudgeQueue.enqueue` would assign a
|
||||
fresh seq and re-order a recovered older notice after newer events).
|
||||
``seq`` is the queue-global insertion number multi-channel drains sort
|
||||
on to restore chronology."""
|
||||
|
||||
nudge_type: str
|
||||
text: str
|
||||
channel: Channel
|
||||
seq: int = 0
|
||||
valid_until: Callable[[], bool] | None = None
|
||||
# Producer-supplied optional fields that ride alongside ``text`` when
|
||||
# drained — used by ``watch_triggered`` to carry ``watch_name`` /
|
||||
@@ -70,7 +95,8 @@ class NudgeQueue:
|
||||
"""Single-session FIFO queue with channel-tagged entries."""
|
||||
|
||||
def __init__(self) -> None:
|
||||
self._items: deque[_Entry] = deque()
|
||||
self._items: deque[Entry] = deque()
|
||||
self._seq = 0
|
||||
self._lock = threading.Lock()
|
||||
|
||||
def enqueue(
|
||||
@@ -105,7 +131,8 @@ class NudgeQueue:
|
||||
if channel not in _VALID_CHANNELS:
|
||||
raise ValueError(f"channel={channel!r}; expected one of {sorted(_VALID_CHANNELS)}")
|
||||
with self._lock:
|
||||
self._items.append(_Entry(nudge_type, text, channel, valid_until, metadata))
|
||||
self._seq += 1
|
||||
self._items.append(Entry(nudge_type, text, channel, self._seq, valid_until, metadata))
|
||||
|
||||
def drain(
|
||||
self, channels: frozenset[str] | set[str]
|
||||
@@ -122,6 +149,15 @@ class NudgeQueue:
|
||||
without delivering it. Already-removed-from-queue either way —
|
||||
dropped entries don't ride a future drain.
|
||||
"""
|
||||
return [(e.nudge_type, e.text, e.metadata) for e in self.drain_entries(channels)]
|
||||
|
||||
def drain_entries(self, channels: frozenset[str] | set[str]) -> list[Entry]:
|
||||
"""Like :meth:`drain` but returns the surviving :class:`Entry`
|
||||
records whole — ``seq`` for cross-channel chronology merges and
|
||||
``valid_until`` so a consumer that must give an entry back (the
|
||||
wake path's failed-send re-enqueue) can do so without stripping
|
||||
its staleness predicate.
|
||||
"""
|
||||
with self._lock:
|
||||
if not self._items:
|
||||
return []
|
||||
@@ -131,10 +167,10 @@ class NudgeQueue:
|
||||
# ``USER_DRAIN`` / ``TOOL_DRAIN`` (channel + "any") and
|
||||
# most queues hold only one channel's entries at a time.
|
||||
if all(entry.channel in channels for entry in self._items):
|
||||
candidates: list[_Entry] = list(self._items)
|
||||
candidates: list[Entry] = list(self._items)
|
||||
self._items = deque()
|
||||
else:
|
||||
kept: deque[_Entry] = deque()
|
||||
kept: deque[Entry] = deque()
|
||||
candidates = []
|
||||
for entry in self._items:
|
||||
if entry.channel in channels:
|
||||
@@ -150,14 +186,14 @@ class NudgeQueue:
|
||||
# every child closed) and logs at ``info``; a raised exception
|
||||
# is a wiring bug (predicate is misbehaving) and stays at
|
||||
# ``warning`` with ``exc_info`` so the traceback surfaces.
|
||||
out: list[tuple[str, str, dict[str, Any] | None]] = []
|
||||
out: list[Entry] = []
|
||||
for entry in candidates:
|
||||
if entry.valid_until is None:
|
||||
out.append((entry.nudge_type, entry.text, entry.metadata))
|
||||
out.append(entry)
|
||||
continue
|
||||
try:
|
||||
if entry.valid_until():
|
||||
out.append((entry.nudge_type, entry.text, entry.metadata))
|
||||
out.append(entry)
|
||||
continue
|
||||
log.info(
|
||||
"nudge_queue.predicate_dropped",
|
||||
@@ -187,12 +223,74 @@ class NudgeQueue:
|
||||
return len(self._items)
|
||||
|
||||
def clear(self) -> int:
|
||||
"""Drop every entry; return the count cleared. Used in cancel paths."""
|
||||
"""Drop every entry regardless of channel; return the count cleared.
|
||||
|
||||
No longer on the cancel path — abandoned generations use
|
||||
:meth:`clear_channels` + :meth:`demote_channel` so external events
|
||||
survive. Kept for tests and for full-reset callers that truly mean
|
||||
"everything".
|
||||
"""
|
||||
with self._lock:
|
||||
n = len(self._items)
|
||||
self._items.clear()
|
||||
return n
|
||||
|
||||
def requeue(self, entry: Entry, *, channel: Channel | None = None) -> None:
|
||||
"""Give a drained :class:`Entry` back to the queue, KEEPING its seq.
|
||||
|
||||
A plain :meth:`enqueue` would assign a fresh (higher) seq, so a
|
||||
failed delivery's re-queued OLDER notice would sort after events
|
||||
that arrived during the failed attempt — running poll counters
|
||||
backwards at the next seq-merged wake. Insertion is positioned by
|
||||
seq so plain FIFO drains stay chronological too. ``channel``
|
||||
overrides the entry's channel (the wake path demotes ``"any"`` →
|
||||
``"quiet"``); ``valid_until`` and ``metadata`` ride unchanged.
|
||||
"""
|
||||
dst = channel if channel is not None else entry.channel
|
||||
if dst not in _VALID_CHANNELS:
|
||||
raise ValueError(f"channel={dst!r}; expected one of {sorted(_VALID_CHANNELS)}")
|
||||
restored = entry._replace(channel=dst)
|
||||
with self._lock:
|
||||
for i, existing in enumerate(self._items):
|
||||
if existing.seq > restored.seq:
|
||||
self._items.insert(i, restored)
|
||||
return
|
||||
self._items.append(restored)
|
||||
|
||||
def demote_channel(self, src: Channel, dst: Channel) -> int:
|
||||
"""Atomically re-tag every ``src``-channel entry as ``dst``; return
|
||||
the count. Order, text, metadata and ``valid_until`` are preserved
|
||||
— only drain/wake eligibility changes. The cancel path uses this to
|
||||
take ``"any"`` entries out of the idle-wake gate (→ ``"quiet"``)
|
||||
without dropping the external events they announce.
|
||||
"""
|
||||
if dst not in _VALID_CHANNELS:
|
||||
raise ValueError(f"channel={dst!r}; expected one of {sorted(_VALID_CHANNELS)}")
|
||||
with self._lock:
|
||||
demoted = 0
|
||||
for i, entry in enumerate(self._items):
|
||||
if entry.channel == src:
|
||||
self._items[i] = entry._replace(channel=dst)
|
||||
demoted += 1
|
||||
return demoted
|
||||
|
||||
def clear_channels(self, channels: frozenset[str] | set[str]) -> int:
|
||||
"""Drop entries whose channel is in ``channels``; return the count.
|
||||
|
||||
The abandoned-generation paths use this instead of :meth:`clear`:
|
||||
``"tool"``/``"user"`` advisories are generation-scoped commentary
|
||||
(a stale ``repeat`` nudge must not bleed into the next send), but
|
||||
``"any"``-channel entries are EXTERNAL events — a watch fire or a
|
||||
background-shell exit that happened during the doomed generation
|
||||
still happened, and dropping it would silently break the "you will
|
||||
be notified" contract those producers promised the model.
|
||||
"""
|
||||
with self._lock:
|
||||
kept = deque(e for e in self._items if e.channel not in channels)
|
||||
n = len(self._items) - len(kept)
|
||||
self._items = kept
|
||||
return n
|
||||
|
||||
def count_by_type(self, nudge_type: str, channel: Channel | None = None) -> int:
|
||||
"""Return the number of queued entries matching ``nudge_type``.
|
||||
|
||||
|
||||
@@ -58,6 +58,17 @@ class OAuthSSRFError(Exception):
|
||||
"""
|
||||
|
||||
|
||||
class OAuthSSRFPrivateAddressError(OAuthSSRFError):
|
||||
"""A hostname resolved to a non-public address, specifically.
|
||||
|
||||
A distinct subclass so callers with an operator-facing opt-in
|
||||
(``[oidc] allow_private_network``) can catch this case and append the
|
||||
remediation hint, while callers with no such opt-in (``mcp_oauth``,
|
||||
where endpoint URLs come from untrusted remote-server metadata) keep
|
||||
catching :class:`OAuthSSRFError` and stay strict.
|
||||
"""
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Helpers
|
||||
# ---------------------------------------------------------------------------
|
||||
@@ -92,13 +103,23 @@ def effective_port(parsed: urllib.parse.ParseResult) -> int | None:
|
||||
return {"http": 80, "https": 443}.get(parsed.scheme)
|
||||
|
||||
|
||||
def validate_url_no_ssrf(url: str, *, allow_http: bool) -> urllib.parse.ParseResult:
|
||||
def validate_url_no_ssrf(
|
||||
url: str, *, allow_http: bool, allow_private: bool = False
|
||||
) -> urllib.parse.ParseResult:
|
||||
"""Run the scheme/userinfo/SSRF checks shared by issuer and discovered URLs.
|
||||
|
||||
Returns the parsed URL on success. Raises :class:`OAuthSSRFError` on
|
||||
failure. The ``allow_http`` flag is the only knob: when ``True``,
|
||||
``http://`` is accepted *if* the hostname is also a localhost form;
|
||||
when ``False``, only ``https://`` is accepted.
|
||||
failure. Two knobs: ``allow_http=True`` accepts ``http://`` *if* the
|
||||
hostname is also a localhost form (when ``False``, only ``https://``
|
||||
is accepted); ``allow_private=True`` accepts hostnames resolving to
|
||||
private-range addresses (RFC 1918, ULA, CGNAT, loopback) for
|
||||
operator-trusted URLs — a self-hosted IdP on an internal network.
|
||||
Even with ``allow_private``, link-local, multicast, unspecified, and
|
||||
reserved addresses stay refused: cloud metadata services
|
||||
(169.254.169.254) are the canonical SSRF target, and no legitimate
|
||||
IdP lives in those ranges. Non-public rejections raise the
|
||||
:class:`OAuthSSRFPrivateAddressError` subclass so callers that *have*
|
||||
an opt-in can point the operator at it.
|
||||
"""
|
||||
parsed = urllib.parse.urlparse(url)
|
||||
|
||||
@@ -127,8 +148,19 @@ def validate_url_no_ssrf(url: str, *, allow_http: bool) -> urllib.parse.ParseRes
|
||||
raise OAuthSSRFError(
|
||||
f"endpoint hostname resolved to invalid IP {sockaddr[0]!r}: {hostname}"
|
||||
) from exc
|
||||
if not addr.is_global and not is_localhost(hostname):
|
||||
raise OAuthSSRFError(f"endpoint URL resolves to non-public address ({addr}): {url}")
|
||||
if addr.is_global or is_localhost(hostname):
|
||||
continue
|
||||
if allow_private:
|
||||
if addr.is_link_local or addr.is_multicast or addr.is_unspecified or addr.is_reserved:
|
||||
raise OAuthSSRFError(
|
||||
f"endpoint URL resolves to a link-local/multicast/"
|
||||
f"unspecified/reserved address ({addr}), refused even "
|
||||
f"with private addresses allowed: {url}"
|
||||
)
|
||||
continue
|
||||
raise OAuthSSRFPrivateAddressError(
|
||||
f"endpoint URL resolves to non-public address ({addr}): {url}"
|
||||
)
|
||||
|
||||
return parsed
|
||||
|
||||
@@ -139,6 +171,7 @@ def validate_discovered_endpoint(
|
||||
*,
|
||||
allow_http: bool,
|
||||
trusted_endpoint_hosts: frozenset[str],
|
||||
allow_private: bool = False,
|
||||
) -> None:
|
||||
"""Validate an endpoint URL pulled from an OIDC/OAuth discovery document.
|
||||
|
||||
@@ -146,11 +179,12 @@ def validate_discovered_endpoint(
|
||||
constraint: the endpoint host must equal the issuer host, be in the
|
||||
well-known trust map, or be in the operator-supplied
|
||||
``trusted_endpoint_hosts``. Effective port (with scheme defaults
|
||||
applied) and scheme must match the issuer.
|
||||
applied) and scheme must match the issuer. ``allow_private`` forwards
|
||||
to :func:`validate_url_no_ssrf`.
|
||||
|
||||
Raises :class:`OAuthSSRFError` on validation failure.
|
||||
"""
|
||||
parsed = validate_url_no_ssrf(url, allow_http=allow_http)
|
||||
parsed = validate_url_no_ssrf(url, allow_http=allow_http, allow_private=allow_private)
|
||||
|
||||
issuer_hostname = (issuer_parsed.hostname or "").lower()
|
||||
endpoint_hostname = (parsed.hostname or "").lower()
|
||||
@@ -182,7 +216,9 @@ def validate_discovered_endpoint(
|
||||
)
|
||||
|
||||
|
||||
async def validate_url_no_ssrf_async(url: str, *, allow_http: bool) -> urllib.parse.ParseResult:
|
||||
async def validate_url_no_ssrf_async(
|
||||
url: str, *, allow_http: bool, allow_private: bool = False
|
||||
) -> urllib.parse.ParseResult:
|
||||
"""Async variant of :func:`validate_url_no_ssrf` for hot-path callers.
|
||||
|
||||
The synchronous variant calls ``socket.getaddrinfo``, which blocks
|
||||
@@ -191,7 +227,9 @@ async def validate_url_no_ssrf_async(url: str, *, allow_http: bool) -> urllib.pa
|
||||
:func:`asyncio.to_thread` to keep the loop responsive. This wrapper
|
||||
centralises that wrapping so callers don't repeat the idiom.
|
||||
"""
|
||||
return await asyncio.to_thread(validate_url_no_ssrf, url, allow_http=allow_http)
|
||||
return await asyncio.to_thread(
|
||||
validate_url_no_ssrf, url, allow_http=allow_http, allow_private=allow_private
|
||||
)
|
||||
|
||||
|
||||
async def validate_discovered_endpoint_async(
|
||||
@@ -200,6 +238,7 @@ async def validate_discovered_endpoint_async(
|
||||
*,
|
||||
allow_http: bool,
|
||||
trusted_endpoint_hosts: frozenset[str],
|
||||
allow_private: bool = False,
|
||||
) -> None:
|
||||
"""Async variant of :func:`validate_discovered_endpoint`."""
|
||||
await asyncio.to_thread(
|
||||
@@ -208,12 +247,14 @@ async def validate_discovered_endpoint_async(
|
||||
issuer_parsed,
|
||||
allow_http=allow_http,
|
||||
trusted_endpoint_hosts=trusted_endpoint_hosts,
|
||||
allow_private=allow_private,
|
||||
)
|
||||
|
||||
|
||||
__all__ = [
|
||||
"KNOWN_TRUSTED_OAUTH_ENDPOINT_HOSTS",
|
||||
"OAuthSSRFError",
|
||||
"OAuthSSRFPrivateAddressError",
|
||||
"effective_port",
|
||||
"is_localhost",
|
||||
"sanitize_log_text",
|
||||
|
||||
+50
-8
@@ -27,6 +27,7 @@ if TYPE_CHECKING:
|
||||
from turnstone.core.log import get_logger
|
||||
from turnstone.core.oauth_ssrf import (
|
||||
OAuthSSRFError,
|
||||
OAuthSSRFPrivateAddressError,
|
||||
is_localhost,
|
||||
)
|
||||
from turnstone.core.oauth_ssrf import (
|
||||
@@ -105,7 +106,7 @@ class OIDCConfig:
|
||||
Startup-config fields (set by :func:`load_oidc_config`):
|
||||
``enabled``, ``issuer``, ``client_id``, ``client_secret``, ``scopes``,
|
||||
``provider_name``, ``role_claim``, ``role_map``, ``password_enabled``,
|
||||
``redirect_base``, ``trusted_endpoint_hosts``.
|
||||
``redirect_base``, ``trusted_endpoint_hosts``, ``allow_private_network``.
|
||||
|
||||
Discovery-derived fields (set by :func:`discover_oidc`; empty before
|
||||
discovery completes):
|
||||
@@ -124,6 +125,10 @@ class OIDCConfig:
|
||||
password_enabled: bool = True
|
||||
redirect_base: str = ""
|
||||
trusted_endpoint_hosts: tuple[str, ...] = ()
|
||||
# Opt-in for self-hosted IdPs on internal networks: permit the issuer
|
||||
# (and its same-origin discovered endpoints) to resolve to private
|
||||
# addresses. Link-local/multicast/reserved stay refused regardless.
|
||||
allow_private_network: bool = False
|
||||
# Discovered from .well-known/openid-configuration
|
||||
authorization_endpoint: str = ""
|
||||
token_endpoint: str = ""
|
||||
@@ -189,6 +194,9 @@ def load_oidc_config() -> OIDCConfig:
|
||||
password_enabled = _env_or_cfg_bool(
|
||||
"TURNSTONE_OIDC_PASSWORD_ENABLED", cfg, "password_enabled", True
|
||||
)
|
||||
allow_private_network = _env_or_cfg_bool(
|
||||
"TURNSTONE_OIDC_ALLOW_PRIVATE_NETWORK", cfg, "allow_private_network", False
|
||||
)
|
||||
|
||||
# Role map: env var is "admin:builtin-admin,eng:builtin-operator"
|
||||
role_map_raw = os.environ.get("TURNSTONE_OIDC_ROLE_MAP", "").strip()
|
||||
@@ -259,7 +267,12 @@ def load_oidc_config() -> OIDCConfig:
|
||||
enabled = bool(issuer and client_id and client_secret)
|
||||
|
||||
if enabled:
|
||||
log.info("OIDC enabled: issuer=%s provider=%s", issuer, provider_name)
|
||||
log.info(
|
||||
"OIDC enabled: issuer=%s provider=%s%s",
|
||||
issuer,
|
||||
provider_name,
|
||||
" (private-network IdP allowed)" if allow_private_network else "",
|
||||
)
|
||||
else:
|
||||
log.debug("OIDC not configured (issuer/client_id/client_secret incomplete)")
|
||||
|
||||
@@ -275,6 +288,7 @@ def load_oidc_config() -> OIDCConfig:
|
||||
password_enabled=password_enabled,
|
||||
redirect_base=redirect_base,
|
||||
trusted_endpoint_hosts=trusted_endpoint_hosts,
|
||||
allow_private_network=allow_private_network,
|
||||
)
|
||||
|
||||
|
||||
@@ -286,26 +300,44 @@ def load_oidc_config() -> OIDCConfig:
|
||||
# converting :class:`OAuthSSRFError` to :class:`OIDCError`.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
# Appended to every private-address rejection in this module (issuer and
|
||||
# discovered endpoints alike): the login-flow URLs are operator-configured,
|
||||
# so pointing the operator at the opt-in is safe here — unlike ``mcp_oauth``,
|
||||
# where the URLs come from untrusted remote-server metadata and no such
|
||||
# opt-in exists.
|
||||
_PRIVATE_NETWORK_HINT = (
|
||||
" — to allow a self-hosted IdP on a private network, set "
|
||||
"allow_private_network = true in the [oidc] section of config.toml "
|
||||
"(or TURNSTONE_OIDC_ALLOW_PRIVATE_NETWORK=true)"
|
||||
)
|
||||
|
||||
def _validate_url_no_ssrf(url: str, *, allow_http: bool) -> urllib.parse.ParseResult:
|
||||
|
||||
def _validate_url_no_ssrf(
|
||||
url: str, *, allow_http: bool, allow_private: bool = False
|
||||
) -> urllib.parse.ParseResult:
|
||||
"""OIDC-flavoured wrapper around :func:`oauth_ssrf.validate_url_no_ssrf`."""
|
||||
try:
|
||||
return _ssrf_validate_url_no_ssrf(url, allow_http=allow_http)
|
||||
return _ssrf_validate_url_no_ssrf(url, allow_http=allow_http, allow_private=allow_private)
|
||||
except OAuthSSRFPrivateAddressError as exc:
|
||||
raise OIDCError(f"{exc}{_PRIVATE_NETWORK_HINT}") from exc
|
||||
except OAuthSSRFError as exc:
|
||||
raise OIDCError(str(exc)) from exc
|
||||
|
||||
|
||||
def validate_issuer_url(url: str) -> None:
|
||||
def validate_issuer_url(url: str, *, allow_private: bool = False) -> None:
|
||||
"""Validate an OIDC issuer URL to prevent SSRF.
|
||||
|
||||
Rejects:
|
||||
- Non-HTTPS URLs (except localhost for development)
|
||||
- URLs with embedded credentials (userinfo)
|
||||
- Hostnames that resolve to private/internal/loopback IP addresses
|
||||
- Hostnames that resolve to private/internal/loopback IP addresses,
|
||||
unless ``allow_private`` is set (the ``allow_private_network``
|
||||
opt-in for self-hosted IdPs; link-local/multicast/reserved
|
||||
addresses stay refused regardless)
|
||||
|
||||
Raises :class:`OIDCError` on validation failure.
|
||||
"""
|
||||
_validate_url_no_ssrf(url, allow_http=True)
|
||||
_validate_url_no_ssrf(url, allow_http=True, allow_private=allow_private)
|
||||
|
||||
|
||||
def validate_discovered_endpoint(
|
||||
@@ -314,6 +346,7 @@ def validate_discovered_endpoint(
|
||||
*,
|
||||
allow_http: bool,
|
||||
trusted_endpoint_hosts: frozenset[str],
|
||||
allow_private: bool = False,
|
||||
) -> None:
|
||||
"""Validate an endpoint pulled from an IdP discovery document.
|
||||
|
||||
@@ -340,7 +373,12 @@ def validate_discovered_endpoint(
|
||||
issuer_parsed,
|
||||
allow_http=allow_http,
|
||||
trusted_endpoint_hosts=trusted_endpoint_hosts,
|
||||
allow_private=allow_private,
|
||||
)
|
||||
except OAuthSSRFPrivateAddressError as exc:
|
||||
# A discovered endpoint (or trusted host) resolving private is fixed
|
||||
# by the same opt-in as the issuer — carry the hint here too.
|
||||
raise OIDCError(f"{exc}{_PRIVATE_NETWORK_HINT}") from exc
|
||||
except OAuthSSRFError as exc:
|
||||
raise OIDCError(str(exc)) from exc
|
||||
|
||||
@@ -367,7 +405,9 @@ async def discover_oidc(
|
||||
return dataclasses.replace(config, enabled=False)
|
||||
|
||||
try:
|
||||
issuer_parsed = _validate_url_no_ssrf(config.issuer, allow_http=True)
|
||||
issuer_parsed = _validate_url_no_ssrf(
|
||||
config.issuer, allow_http=True, allow_private=config.allow_private_network
|
||||
)
|
||||
except OIDCError as exc:
|
||||
log.warning("OIDC issuer URL rejected: %s", exc)
|
||||
return dataclasses.replace(config, enabled=False)
|
||||
@@ -422,6 +462,7 @@ async def discover_oidc(
|
||||
issuer_parsed,
|
||||
allow_http=allow_http,
|
||||
trusted_endpoint_hosts=trusted_hosts,
|
||||
allow_private=config.allow_private_network,
|
||||
)
|
||||
except OIDCError as exc:
|
||||
log.warning("OIDC discovered %s rejected (url=%s): %s", name, endpoint_url, exc)
|
||||
@@ -434,6 +475,7 @@ async def discover_oidc(
|
||||
issuer_parsed,
|
||||
allow_http=allow_http,
|
||||
trusted_endpoint_hosts=trusted_hosts,
|
||||
allow_private=config.allow_private_network,
|
||||
)
|
||||
except OIDCError as exc:
|
||||
log.warning(
|
||||
|
||||
@@ -94,10 +94,16 @@ _RE_PRIVATE_KEY_BLOCK = re.compile(
|
||||
# ``redact_credentials`` — error persistence, audit details,
|
||||
# coordinator inspect/wait surfaces. The structural form ``[^:@\s]+:
|
||||
# [^@\s]+@`` is specific enough that ``https://example.com:8080/path``
|
||||
# (host:port without ``@``) doesn't match.
|
||||
# (host:port without ``@``) doesn't match. The optional ``+suffix``
|
||||
# covers SQLAlchemy dialect+driver URLs (``postgresql+psycopg2``,
|
||||
# ``postgresql+asyncpg``, ``mysql+pymysql``) and ``mongodb+srv`` —
|
||||
# enumerating drivers is a losing game, the suffix shape isn't.
|
||||
# Schemes are case-insensitive per RFC 3986, hence IGNORECASE:
|
||||
# ``POSTGRESQL://`` leaks the same password ``postgresql://`` does.
|
||||
_RE_CONNECTION_STRING = re.compile(
|
||||
r"(?:postgresql\+?(?:psycopg)?|mysql|mongodb(?:\+srv)?|rediss?|amqps?|sqlite|https?)"
|
||||
r"(?:postgresql|mysql|mongodb|rediss?|amqps?|sqlite|https?)(?:\+[a-z0-9]*)?"
|
||||
r"://[^:@\s]+:[^@\s]+@",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
_RE_ENV_SECRET_LINE = re.compile(r"[A-Z][A-Z_0-9]+=\S+")
|
||||
_RE_ENV_SECRET_KEY = re.compile(
|
||||
|
||||
@@ -50,8 +50,8 @@ from turnstone.core.deadline import (
|
||||
)
|
||||
from turnstone.core.judge import (
|
||||
_CHARS_PER_TOKEN,
|
||||
_DEFAULT_JUDGE_CONTEXT_WINDOW,
|
||||
_positive_window,
|
||||
_resolve_model_capabilities,
|
||||
)
|
||||
from turnstone.core.log import get_logger
|
||||
|
||||
@@ -59,7 +59,7 @@ if TYPE_CHECKING:
|
||||
import threading
|
||||
|
||||
from turnstone.core.judge import JudgeConfig
|
||||
from turnstone.core.providers._protocol import LLMProvider
|
||||
from turnstone.core.providers._protocol import LLMProvider, ModelCapabilities
|
||||
|
||||
log = get_logger(__name__)
|
||||
|
||||
@@ -247,7 +247,7 @@ class OutputGuardJudge:
|
||||
Construction resolves the configured ``judge.output_guard_model``
|
||||
alias inline; on resolution failure (alias unset or unknown) the
|
||||
session model is used as a fallback. Mirrors :class:`IntentJudge`'s
|
||||
own resolution at ``judge.py:917-960``.
|
||||
own alias resolution.
|
||||
|
||||
The HTTP client is lazy-initialised on the first ``evaluate()`` call
|
||||
and reused for the lifetime of the judge instance — see
|
||||
@@ -267,16 +267,23 @@ class OutputGuardJudge:
|
||||
session_client: Any,
|
||||
session_model: str,
|
||||
model_registry: Any | None = None,
|
||||
context_window: int = _DEFAULT_JUDGE_CONTEXT_WINDOW,
|
||||
session_capabilities: ModelCapabilities | None = None,
|
||||
) -> None:
|
||||
self._config = config
|
||||
# Alias resolution mirrors IntentJudge.__init__ at judge.py:917-960.
|
||||
# Caller's resolved session-model caps (config/registry-aware): the wire
|
||||
# capabilities + window when this judge inherits the session model, and
|
||||
# the alias path's window fallback. The window comes ONLY from these
|
||||
# (else a floor), never provider.get_capabilities() — see below.
|
||||
session_window = (
|
||||
session_capabilities.context_window if session_capabilities is not None else None
|
||||
)
|
||||
# Alias resolution mirrors IntentJudge.__init__.
|
||||
# An empty / unset alias falls through to the session model silently;
|
||||
# a set-but-unknown alias logs a warning and also falls through.
|
||||
# Judge model's context window drives the oversize-output guard in
|
||||
# ``evaluate``. It comes from the registry's ModelConfig on the alias
|
||||
# path and the session's real window (``context_window``, resolved by
|
||||
# the caller from _get_capabilities) on the fallback path — NEVER
|
||||
# path and the session's real window (``session_capabilities``, resolved
|
||||
# by the caller from _get_capabilities) on the fallback path — NEVER
|
||||
# ``provider.get_capabilities()``, which returns a static 200000 for
|
||||
# every model absent from its table (i.e. every local / self-hosted
|
||||
# judge), so a guard keyed off it would never trip for the small-window
|
||||
@@ -296,8 +303,12 @@ class OutputGuardJudge:
|
||||
)
|
||||
self._model = model_name
|
||||
self._judge_model_alias = config.output_guard_model
|
||||
self._capabilities = _resolve_model_capabilities(
|
||||
self._provider, self._model, model_cfg
|
||||
)
|
||||
self._judge_context_window = _positive_window(
|
||||
getattr(model_cfg, "context_window", None), context_window
|
||||
getattr(model_cfg, "context_window", None),
|
||||
session_window,
|
||||
)
|
||||
resolved = True
|
||||
except Exception:
|
||||
@@ -321,12 +332,19 @@ class OutputGuardJudge:
|
||||
)
|
||||
self._model = session_model
|
||||
self._judge_model_alias = ""
|
||||
# Wire caps: the caller's resolved session caps, or the provider's
|
||||
# static table as a last resort for degraded / legacy callers.
|
||||
self._capabilities = (
|
||||
session_capabilities
|
||||
if session_capabilities is not None
|
||||
else session_provider.get_capabilities(session_model)
|
||||
)
|
||||
# Session-model fallback: use the session's real context window
|
||||
# (the caller resolved it from _get_capabilities, config/registry-
|
||||
# aware) — NOT provider.get_capabilities(), which reports 200000 for
|
||||
# a local session model and would leave the guard blind to overflow,
|
||||
# the very failure this fixes. Mirrors IntentJudge's fallback.
|
||||
self._judge_context_window = _positive_window(context_window)
|
||||
self._judge_context_window = _positive_window(session_window)
|
||||
|
||||
# Lazy-init in _create_client(); reused across evaluate() calls.
|
||||
# Session swaps the entire OutputGuardJudge on credential / model
|
||||
@@ -341,7 +359,7 @@ class OutputGuardJudge:
|
||||
|
||||
Reads ``base_url`` and ``api_key`` from the client and returns
|
||||
the dict ``turnstone.core.providers.create_client`` accepts.
|
||||
Inlined from IntentJudge's helper at ``judge.py:965-969``.
|
||||
Inlined from IntentJudge's ``_extract_client_config``.
|
||||
"""
|
||||
base_url = str(getattr(client, "base_url", getattr(client, "_base_url", "")))
|
||||
api_key = getattr(client, "api_key", "") or ""
|
||||
@@ -491,6 +509,10 @@ class OutputGuardJudge:
|
||||
max_tokens=512,
|
||||
temperature=0.0,
|
||||
reasoning_effort="low",
|
||||
# Operator-declared capabilities reach the wire like every
|
||||
# other lane — from the output_guard alias's definition, or
|
||||
# the session model on fallback. See IntentJudge for why.
|
||||
capabilities=self._capabilities,
|
||||
),
|
||||
timeout=timeout,
|
||||
cancel_event=cancel_event,
|
||||
|
||||
@@ -0,0 +1,362 @@
|
||||
"""Preview-content policy for the ``open_preview`` tool.
|
||||
|
||||
The preview pane renders tool-selected content — a fetched web page, a PDF, an
|
||||
image, a data table, a text/markdown document — in a dedicated frontend pane
|
||||
beside the conversation. This module owns the pure policy so the session
|
||||
executor and the serving route share one definition: the content-kind
|
||||
vocabulary, how bytes + hints resolve to a kind, the per-kind size caps, the
|
||||
serving MIME allowlist + response headers, and the small HTML mutations
|
||||
(base-href injection, title extraction) applied to fetched pages at store
|
||||
time.
|
||||
|
||||
Preview blobs are persisted content-addressed with attachment kind
|
||||
``PREVIEW_BLOB_KIND``. That kind is deliberately outside the model-visible
|
||||
attachment vocabulary (image / text / pdf / audio): trajectory reconstruction
|
||||
skips it, so a preview blob can never be lifted into a turn's content and
|
||||
materialized onto the wire — the tool turn's ``meta.extra["preview"]``
|
||||
descriptor is the only carrier, and it is frontend-facing only.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import html
|
||||
import re
|
||||
from typing import Any
|
||||
|
||||
from turnstone.core.attachments import (
|
||||
ALLOWED_IMAGE_MIMES,
|
||||
IMAGE_SIZE_CAP,
|
||||
PDF_SIZE_CAP,
|
||||
TEXT_DOC_SIZE_CAP,
|
||||
sniff_image_mime,
|
||||
sniff_pdf_mime,
|
||||
)
|
||||
from turnstone.core.web_helpers import latin1_safe_filename
|
||||
|
||||
# Rendered-content kinds the pane knows how to display. ``web`` is a fetched
|
||||
# HTML document (sandboxed iframe); ``table`` is CSV/TSV/JSON parsed and
|
||||
# rendered client-side; the rest map 1:1 onto native browser rendering.
|
||||
PREVIEW_KINDS: frozenset[str] = frozenset({"web", "pdf", "image", "table", "text", "markdown"})
|
||||
|
||||
# Storage ``kind`` for preview blobs — see the module docstring for why this
|
||||
# is not one of the model-visible attachment kinds.
|
||||
PREVIEW_BLOB_KIND = "preview"
|
||||
|
||||
# Per-kind byte caps on the STORED preview content. image/pdf/text reuse the
|
||||
# attachment-subsystem caps so a previewable file and an uploadable file agree
|
||||
# on "too big". Fetched pages get their own cap (real-world pages fit well
|
||||
# under it; over-cap pages error rather than truncate — a mid-tag cut renders
|
||||
# garbage). Tables get headroom over plain text: a few-MB CSV is a normal
|
||||
# artifact of "bash produced data", and the client-side renderer row-caps.
|
||||
PREVIEW_SIZE_CAPS: dict[str, int] = {
|
||||
"web": 4 * 1024 * 1024,
|
||||
"pdf": PDF_SIZE_CAP,
|
||||
"image": IMAGE_SIZE_CAP,
|
||||
"table": 2 * 1024 * 1024,
|
||||
"text": TEXT_DOC_SIZE_CAP,
|
||||
"markdown": TEXT_DOC_SIZE_CAP,
|
||||
}
|
||||
|
||||
# MIME types the preview route will serve with a renderable Content-Type.
|
||||
# Everything stored by ``open_preview`` lands in this set; the route still
|
||||
# allowlists defensively so a non-preview blob addressed by id serves nothing
|
||||
# renderable. Parameterized types (``text/html; charset=utf-8``) match on the
|
||||
# bare type.
|
||||
PREVIEW_SERVE_MIMES: frozenset[str] = frozenset(
|
||||
{
|
||||
"text/html",
|
||||
"application/pdf",
|
||||
"text/plain",
|
||||
"text/csv",
|
||||
"text/tab-separated-values",
|
||||
"application/json",
|
||||
"text/markdown",
|
||||
}
|
||||
| set(ALLOWED_IMAGE_MIMES)
|
||||
)
|
||||
|
||||
# Extension → (kind, stored mime). Consulted after magic bytes and the
|
||||
# transport MIME hint; keys are lowercase with the dot.
|
||||
_EXT_KINDS: dict[str, tuple[str, str]] = {
|
||||
".html": ("web", "text/html; charset=utf-8"),
|
||||
".htm": ("web", "text/html; charset=utf-8"),
|
||||
".pdf": ("pdf", "application/pdf"),
|
||||
".csv": ("table", "text/csv; charset=utf-8"),
|
||||
".tsv": ("table", "text/tab-separated-values; charset=utf-8"),
|
||||
".json": ("table", "application/json"),
|
||||
".md": ("markdown", "text/markdown; charset=utf-8"),
|
||||
".markdown": ("markdown", "text/markdown; charset=utf-8"),
|
||||
}
|
||||
|
||||
# Stored mime per kind when the kind is chosen first (explicit ``kind`` arg or
|
||||
# a MIME-hint match): the inverse of ``_EXT_KINDS`` plus the text fallback.
|
||||
_KIND_MIMES: dict[str, str] = {
|
||||
"web": "text/html; charset=utf-8",
|
||||
"pdf": "application/pdf",
|
||||
"table": "text/csv; charset=utf-8",
|
||||
"text": "text/plain; charset=utf-8",
|
||||
"markdown": "text/markdown; charset=utf-8",
|
||||
}
|
||||
|
||||
|
||||
def _is_utf8_text(data: bytes) -> bool:
|
||||
"""True when *data* decodes as UTF-8 and carries no NUL (binary tell)."""
|
||||
if b"\x00" in data:
|
||||
return False
|
||||
try:
|
||||
data.decode("utf-8")
|
||||
except UnicodeDecodeError:
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
def _is_decodable_text(data: bytes) -> bool:
|
||||
"""True when *data* carries no NUL byte — the gate for DECLARED text.
|
||||
|
||||
A text-family MIME hint / extension / ``kind`` override says "this is
|
||||
text"; the store-time transcode ladder (:func:`transcode_text`) then
|
||||
decodes it whatever the charset, so the only hard reject left is the NUL
|
||||
byte that marks genuinely-binary content. The *undeclared* fallback lane
|
||||
keeps the stricter :func:`_is_utf8_text`: cp1252-with-replacement never
|
||||
fails, so unknown bytes must prove UTF-8 rather than be waved through as
|
||||
text.
|
||||
"""
|
||||
return b"\x00" not in data
|
||||
|
||||
|
||||
def _charset_param(mime: str) -> str | None:
|
||||
"""The ``charset=`` value from a MIME string, lowercased, or ``None``."""
|
||||
for part in mime.split(";")[1:]:
|
||||
key, sep, value = part.partition("=")
|
||||
if sep and key.strip().lower() == "charset":
|
||||
return value.strip().strip('"').lower() or None
|
||||
return None
|
||||
|
||||
|
||||
def transcode_text(body: bytes, mime_hint: str) -> str:
|
||||
"""Decode text-family *body* to ``str`` via a charset ladder.
|
||||
|
||||
Rungs: (a) the ``charset=`` parameter from *mime_hint* when it names a
|
||||
codec Python knows, (b) UTF-8, (c) cp1252 with ``errors="replace"``. The
|
||||
last rung never fails, so the return is always a usable string — this is
|
||||
the store-time transcode that lets a legacy-charset page / CSV / log render
|
||||
as UTF-8. Binary rejection stays upstream in :func:`resolve_preview_kind`
|
||||
(the NUL check); by the time bytes reach here they are already classified
|
||||
text.
|
||||
"""
|
||||
charset = _charset_param(mime_hint)
|
||||
if charset:
|
||||
try:
|
||||
return body.decode(charset)
|
||||
except (LookupError, UnicodeDecodeError):
|
||||
pass
|
||||
try:
|
||||
return body.decode("utf-8")
|
||||
except UnicodeDecodeError:
|
||||
return body.decode("cp1252", errors="replace")
|
||||
|
||||
|
||||
def _kind_from_mime(mime: str) -> tuple[str, str] | None:
|
||||
"""Map a transport MIME hint to ``(kind, stored_mime)``, or ``None``."""
|
||||
bare = mime.split(";", 1)[0].strip().lower()
|
||||
if not bare:
|
||||
return None
|
||||
if "html" in bare:
|
||||
return "web", _KIND_MIMES["web"]
|
||||
if bare == "application/pdf":
|
||||
return "pdf", _KIND_MIMES["pdf"]
|
||||
if bare in ALLOWED_IMAGE_MIMES:
|
||||
return "image", bare
|
||||
if bare == "text/csv":
|
||||
return "table", "text/csv; charset=utf-8"
|
||||
if bare == "text/tab-separated-values":
|
||||
return "table", "text/tab-separated-values; charset=utf-8"
|
||||
if bare in ("application/json", "text/json"):
|
||||
return "table", "application/json"
|
||||
if bare == "text/markdown":
|
||||
return "markdown", _KIND_MIMES["markdown"]
|
||||
if bare.startswith("text/"):
|
||||
return "text", _KIND_MIMES["text"]
|
||||
return None
|
||||
|
||||
|
||||
def resolve_preview_kind(
|
||||
mime_hint: str,
|
||||
name_hint: str,
|
||||
body: bytes,
|
||||
kind_override: str | None = None,
|
||||
) -> tuple[str, str] | None:
|
||||
"""Resolve ``(kind, stored_mime)`` for *body*, or ``None`` if unpreviewable.
|
||||
|
||||
Precedence: explicit *kind_override* (the model's ``kind`` argument) →
|
||||
magic bytes (image / pdf — never extension-trusted, mirroring the upload
|
||||
classifier) → transport MIME hint → filename/URL extension → UTF-8 text
|
||||
fallback. A binary body that matches nothing is not previewable.
|
||||
"""
|
||||
if kind_override:
|
||||
if kind_override not in PREVIEW_KINDS:
|
||||
return None
|
||||
if kind_override == "image":
|
||||
sniffed_image = sniff_image_mime(body)
|
||||
return ("image", sniffed_image) if sniffed_image else None
|
||||
if kind_override == "pdf":
|
||||
return ("pdf", "application/pdf") if sniff_pdf_mime(body) else None
|
||||
# Text-family overrides (table / text / markdown) reject only genuine
|
||||
# binary here — the NUL check. The executor transcodes the bytes to
|
||||
# UTF-8 at store time, so a legacy-charset body forced to a text kind
|
||||
# still renders (web always took this path; the others now join it).
|
||||
if kind_override != "web" and not _is_decodable_text(body):
|
||||
return None
|
||||
if kind_override == "table":
|
||||
# Preserve a JSON payload's real type so the client parser branches.
|
||||
bare = mime_hint.split(";", 1)[0].strip().lower()
|
||||
ext = _name_ext(name_hint)
|
||||
if bare in ("application/json", "text/json") or ext == ".json":
|
||||
return "table", "application/json"
|
||||
if bare == "text/tab-separated-values" or ext == ".tsv":
|
||||
return "table", "text/tab-separated-values; charset=utf-8"
|
||||
return "table", _KIND_MIMES["table"]
|
||||
return kind_override, _KIND_MIMES[kind_override]
|
||||
|
||||
sniffed = sniff_image_mime(body)
|
||||
if sniffed:
|
||||
return "image", sniffed
|
||||
if sniff_pdf_mime(body):
|
||||
return "pdf", "application/pdf"
|
||||
from_mime = _kind_from_mime(mime_hint)
|
||||
if from_mime:
|
||||
# A text-family MIME hint declares text: reject only genuine binary
|
||||
# (the NUL check). Legacy charsets (windows-1252 / Shift-JIS pages,
|
||||
# iso-8859-1 CSVs / logs) are not UTF-8 on the raw bytes, and the
|
||||
# executor transcodes every text-family kind to UTF-8 at store time
|
||||
# (charset-aware for fetches, ladder-decoded otherwise).
|
||||
if from_mime[0] in ("table", "text", "markdown") and not _is_decodable_text(body):
|
||||
return None
|
||||
return from_mime
|
||||
ext_match = _EXT_KINDS.get(_name_ext(name_hint))
|
||||
if ext_match:
|
||||
# A text-family extension declares text too — same NUL-only gate; the
|
||||
# store-time ladder handles whatever charset the bytes are in.
|
||||
if ext_match[0] != "web" and not _is_decodable_text(body):
|
||||
return None
|
||||
return ext_match
|
||||
if _is_utf8_text(body):
|
||||
return "text", _KIND_MIMES["text"]
|
||||
return None
|
||||
|
||||
|
||||
def _name_ext(name: str) -> str:
|
||||
"""Lowercase extension of a path / URL tail (query and fragment stripped)."""
|
||||
tail = name.rsplit("/", 1)[-1].split("?", 1)[0].split("#", 1)[0]
|
||||
dot = tail.rfind(".")
|
||||
return tail[dot:].lower() if dot >= 0 else ""
|
||||
|
||||
|
||||
# ``<base>`` / ``<head>`` / ``<html>`` / doctype openers in the first slice of
|
||||
# the document — enough for any real page; scanning megabytes for a head that
|
||||
# must appear early is wasted work.
|
||||
_HEAD_SCAN_LIMIT = 65536
|
||||
_BASE_TAG_RE = re.compile(r"<base[\s>/]", re.IGNORECASE)
|
||||
_HEAD_OPEN_RE = re.compile(r"<head(?:\s[^>]*)?>", re.IGNORECASE)
|
||||
_HTML_OPEN_RE = re.compile(r"<html(?:\s[^>]*)?>", re.IGNORECASE)
|
||||
_DOCTYPE_RE = re.compile(r"<!doctype[^>]*>", re.IGNORECASE)
|
||||
|
||||
|
||||
def inject_base_href(html_text: str, base_url: str) -> str:
|
||||
"""Give a fetched page a ``<base href>`` so relative assets resolve.
|
||||
|
||||
The stored bytes are what the fetch saw; without a base, every relative
|
||||
``src``/``href`` inside the sandboxed iframe would resolve against the
|
||||
turnstone origin and 404. A page that declares its own ``<base>`` is left
|
||||
alone. Insertion goes right after the ``<head>`` opener when present,
|
||||
else after ``<html>`` / the doctype — the parser hoists the tag into the
|
||||
implied head from there. Never ahead of the doctype: markup before
|
||||
``<!doctype`` voids it and drops the whole preview into quirks mode.
|
||||
"""
|
||||
head_slice = html_text[:_HEAD_SCAN_LIMIT]
|
||||
if _BASE_TAG_RE.search(head_slice):
|
||||
return html_text
|
||||
tag = f'<base href="{html.escape(base_url, quote=True)}">'
|
||||
m = _HEAD_OPEN_RE.search(head_slice) or _HTML_OPEN_RE.search(head_slice)
|
||||
if not m:
|
||||
m = _DOCTYPE_RE.search(head_slice)
|
||||
if m:
|
||||
return html_text[: m.end()] + tag + html_text[m.end() :]
|
||||
return tag + html_text
|
||||
|
||||
|
||||
_TITLE_RE = re.compile(r"<title[^>]*>(.*?)</title>", re.IGNORECASE | re.DOTALL)
|
||||
|
||||
|
||||
def page_title(html_text: str) -> str | None:
|
||||
"""The document's ``<title>`` text (unescaped, whitespace-collapsed), or None."""
|
||||
m = _TITLE_RE.search(html_text[:_HEAD_SCAN_LIMIT])
|
||||
if not m:
|
||||
return None
|
||||
title = " ".join(html.unescape(m.group(1)).split())
|
||||
return title[:200] or None
|
||||
|
||||
|
||||
def build_preview_descriptor(
|
||||
*,
|
||||
kind: str,
|
||||
title: str,
|
||||
source: str,
|
||||
attachment_id: str,
|
||||
content_type: str,
|
||||
size: int,
|
||||
) -> dict[str, Any]:
|
||||
"""The structured descriptor that rides the tool turn's meta to the frontend.
|
||||
|
||||
One shape on every boundary — the live ``tool_result`` SSE event, the
|
||||
persisted ``conversations.meta`` column, and the ``/history`` projection —
|
||||
so the pane renders identically live and on replay.
|
||||
"""
|
||||
return {
|
||||
"kind": kind,
|
||||
"title": title,
|
||||
"source": source,
|
||||
"attachment_id": attachment_id,
|
||||
"content_type": content_type,
|
||||
"size": size,
|
||||
}
|
||||
|
||||
|
||||
def preview_response_headers(
|
||||
bare_mime: str, filename: str, *, allow_remote_assets: bool = False
|
||||
) -> dict[str, str]:
|
||||
"""Response headers for the preview serving route, per rendered MIME.
|
||||
|
||||
``text/html`` is served sandboxed either way — scripts never run and its
|
||||
origin is opaque, so it can't touch the app origin's cookies or DOM, and
|
||||
the embedding iframe carries the ``sandbox`` attribute too. The default
|
||||
(``allow_remote_assets=False``) additionally locks the document out of the
|
||||
network: it renders with its inline styling and data-URI images but cannot
|
||||
fetch anything, so previewing a page never discloses the viewer's IP or
|
||||
traffic to the origin site. ``allow_remote_assets=True`` (a per-pane
|
||||
opt-in) drops back to the bare ``sandbox`` so the page's own images / CSS
|
||||
load. ``application/pdf`` gets no CSP: Chromium's PDF viewer refuses to
|
||||
paint inside a sandboxed context, and the response is inert media rendered
|
||||
by browser chrome, not an active document. Everything else keeps the
|
||||
attachment endpoints' full ``default-src 'none'; sandbox`` posture.
|
||||
"""
|
||||
# Page-title-derived filenames routinely carry em dashes / CJK (non-latin-1)
|
||||
# and can carry control bytes — either would 500 the serving route, so run
|
||||
# the shared header sanitizer rather than emit them verbatim.
|
||||
safe_name = latin1_safe_filename(filename, fallback="preview")
|
||||
headers = {
|
||||
"X-Content-Type-Options": "nosniff",
|
||||
"Content-Disposition": f'inline; filename="{safe_name}"',
|
||||
"Cache-Control": "private, no-store",
|
||||
}
|
||||
if bare_mime == "text/html":
|
||||
if allow_remote_assets:
|
||||
headers["Content-Security-Policy"] = "sandbox"
|
||||
else:
|
||||
headers["Content-Security-Policy"] = (
|
||||
"sandbox; default-src 'none'; style-src 'unsafe-inline'; "
|
||||
"img-src data:; font-src data:"
|
||||
)
|
||||
elif bare_mime != "application/pdf":
|
||||
headers["Content-Security-Policy"] = "default-src 'none'; sandbox"
|
||||
return headers
|
||||
@@ -23,6 +23,7 @@ from typing import TYPE_CHECKING, Any
|
||||
if TYPE_CHECKING:
|
||||
from collections.abc import Iterator
|
||||
|
||||
from turnstone.core.lowering import legalize_tool_call_entry
|
||||
from turnstone.core.providers._openai_chat import OpenAIChatCompletionsProvider
|
||||
from turnstone.core.providers._openai_common import sanitize_messages
|
||||
from turnstone.core.providers._protocol import ModelCapabilities, StreamChunk
|
||||
@@ -98,9 +99,31 @@ class GoogleProvider(OpenAIChatCompletionsProvider):
|
||||
# Only type=="function" is expected today; if Gemini adds
|
||||
# other tool types (e.g. code_execution) they will need
|
||||
# their own round-trip handling here.
|
||||
raw_tcs = [b for b in pc if b.get("type") == "function"]
|
||||
if raw_tcs:
|
||||
msg["tool_calls"] = raw_tcs
|
||||
raw_tcs = [b for b in pc if isinstance(b, dict) and b.get("type") == "function"]
|
||||
# Swap ONLY when the raw lane is a faithful counterpart of
|
||||
# the mirror: every raw dict carries an id (a blank id
|
||||
# predates the capture-time blank-id gate — swapping it in
|
||||
# would resurrect the blank id on every replay of that
|
||||
# historical row), and the raw list is the same length as
|
||||
# the mirror (a shorter list — a corrupted lane whose
|
||||
# non-dict elements the extraction filtered — would DROP
|
||||
# mirrored calls whose tool results remain in history and
|
||||
# orphan them). A turn failing either check keeps the
|
||||
# sanitized mirror — losing the raw lane, exactly what the
|
||||
# capture-time gate now produces for new degenerate turns.
|
||||
if (
|
||||
raw_tcs
|
||||
and len(raw_tcs) == len(msg.get("tool_calls") or [])
|
||||
and all(b.get("id") for b in raw_tcs)
|
||||
):
|
||||
# The raw dicts carry the model's ORIGINAL arguments;
|
||||
# the mirror this swap replaces may have been legalized
|
||||
# upstream (lowering.sanitize_tool_call_arguments), so
|
||||
# re-apply the SAME per-entry legalizer — otherwise the
|
||||
# fidelity swap resurrects a malformed arguments value
|
||||
# on every replay. Copy-on-write per offending entry;
|
||||
# ids and ``thought_signature`` stay untouched.
|
||||
msg["tool_calls"] = [legalize_tool_call_entry(b) or b for b in raw_tcs]
|
||||
cleaned.append(msg)
|
||||
return sanitize_messages(cleaned)
|
||||
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user