8 Commits

Author	SHA1	Message	Date
Patrick Buckley	480a1426b3	Fail-closed history-commit handoff (#1005) ... * fix(session): fail-closed history-commit handoff (#981) The deleted-workstream discovery is now a terminal, ws_id-keyed latch: keyed conversation commits refuse admission once the durable parent is gone (convergence finalizers and force-abandon are exempt), history handoff refuses to mint a proof token so /history fails closed with a 503 instead of silently wiping the pane, and the SSE stream carries a workstream_gone resync reason. Discarded commits leave a forensic log of commit keys and roles, never content. Conversation rows gain a commit_key (migration 071): keyed saves are idempotent under retry, validated against the full commit identity, and refused when they would cross a workstream deletion. The prune orphan category now requires a NULL alias plus a two-hour updated grace, with cutoffs computed at discovery time and carried into both dialects' rechecks. The mid-turn interjection queue is owner-partitioned with no per-site mode flags: pops take the acting principal's and unowned rows, other participants' rows are structurally retained, and enforcement lives at queue admission plus the shared before_spawn gates. The retraction ledger is bounded by open pop windows: pops open a window atomically with the queue delete, restores close their ids atomically with the ledger consume, every other exit closes through one helper, and misses for unheld ids record nothing. The workstream-gone latch refuses unattended wakes at all three gates (watcher spawn, claim, delivery pre-pop), and the retry dispatcher regained its pre-envelope cancel/error convergence net. Persistence-state reporting derives through the session bound to each UI instead of a registry lookup by id that failed open to healthy during tombstone retention. The dashboard roster no longer re-inserts ghost entries from trailing activity events, the history tool-outcome scan tolerates interleaved non-turn rows, and the shared handoff-deadline handle owns its own retirement. Single-sourced across call sites: keyed-commit row values, attachment save wrappers, tail-truncation and conflict-resolution bodies for both storage dialects; worker-slot lifecycle field sets; the direct-commit admission frame; queued-row layout accessors; the string-aware comment stripper shared by every JS harness suite. Refs #981 #964 * fix(session): sweep handoff fixes to their sibling surfaces The interactive replay loop treated a system row as a tool-batch boundary, so every tool result after an interleaved row vanished from that pane while the coordinator rendered the same history correctly. Only a conversational turn ends the batch window now, matching the shared outcome index. Accepted user turns clear the composer's attachment chips on the same viewer policy that settles optimistic bubbles rather than on having matched a local bubble, so a workstream created with an upload no longer keeps a chip for an attachment the create dispatch already consumed. The coordinator's raced-Stop arm emits the stream-end hook it inherits alongside the idle state, leaving no unfinalized bubble or unflushed tool output. Ending a session surfaces a failure toast when the request never lands or answers with a non-JSON body. The per-second persistence reconcile now probes each session without blocking: a workstream whose generation and handoff locks are held is skipped until the next pass instead of contending the locks every commit needs. The one-shot repair that gates workstream creation at capacity keeps a definite probe — it has no next pass, and the sessions likeliest to be contended are the ones whose unresolved journals emptied its candidate list. Single-sourced: the attachment lane builds its conversation row through the shared commit-identity builder; the ordinary worker exit releases its slot through the lifecycle owner; both operator surfaces snapshot their counters through one non-consuming helper; the replay preamble loses its per-kind wrappers and its config hook; the browser harness suites share one brace walker; and each in-flight history attempt is one record carrying both its abort controller and its deadline. Refs #981 #964	2026-08-11 04:18:36 -07:00
Patrick Buckley	f4fd7e1f67	fix(security): classify outbound addresses by what they reach (GHSA-wm4f-79pw-pfr9) (#1003) ... * fix(security): classify outbound addresses by what they reach (GHSA-wm4f-79pw-pfr9) Five guards screened outbound URLs and each hand-rolled its own address normalization and policy tests, so each had a different hole. An IPv6 transition address carries an IPv4 destination in its low bits and `ipaddress` classifies the wrapper, not the destination: 64:ff9b::a9fe:a9fe reports is_global because 64:ff9b::/96 is global unicast, while a NAT64 gateway routes it to the cloud metadata endpoint. CGNAT (100.64.0.0/10) is neither is_private nor is_global, so a denylist built on is_private missed it with no gateway involved at all. Add turnstone/core/ip_classify.py as the single classifier. One function returns exactly one policy lane — PUBLIC, PRIVATE (operator-approvable) or NEVER — and every guard branches on the lane rather than re-deriving it. Two overlapping booleans would make a verdict depend on which one a caller tested first; several addresses are simultaneously globally routable and metadata-reaching. - Decode transition addresses per RFC 6052 §2.2 (NAT64 well-known and local-use prefixes, 6to4, Teredo, IPv4-mapped, IPv4-compatible) and judge them by the IPv4 they reach. The local-use prefix does not say which layout its gateway uses, so every length it can carry is decoded and the worst result classified. - Share hostname resolution too. The five copies had already drifted on which failures they caught, and getaddrinfo raises UnicodeError — not an OSError — from the IDNA encoder. - Resolution failure is a refusal, not a pass: the fetch resolves again, so an authority answering the guard with SERVFAIL and the fetch with an internal address would otherwise switch the guard off for that hop. - Screen every redirect hop in every mode. allow_private_origin widens which lanes are acceptable rather than turning screening off, and the permission is revoked after any hop that is not wholly private. - Cleartext http is allowed only for a hostname that RESOLVES to loopback. *.localhost is ordinary DNS, and trusting the name put an OIDC token exchange on the wire in the clear. - Screen doctor and console-probe URLs through the classifier. Both used a host.startswith("169.254.") string test that never resolved, so any DNS name pointing at the metadata service passed and its body was returned to the model. - Add known vendor metadata prefixes the stdlib does not flag, and place deprecated IPv6 site-local outside the public lane. The operator's private-network opt-in still admits the whole home lab, including IPv6 loopback, CGNAT and split-horizon hosts. Metadata, link-local, multicast, unspecified and reserved addresses stay refused regardless of the opt-in, including as a redirect target from an approved private origin — the settings help and docs now say so. Reported by @tonghuaroot. * fix(security): close Azure/Oracle metadata gap and restore dual-stack origins Review follow-ups on the address-classification rework. Azure's host-agent endpoint (168.63.129.16) and Oracle Cloud's metadata endpoint (192.0.0.192) sit in ordinary unicast space, so the stdlib reported them as globally routable and both classified PUBLIC — reachable with no opt-in at all, a worse position than the RFC 1918 host beside them, and directly contradicting the "metadata stays refused even with the opt-in" guarantee the settings help and docs now advertise. Both join the shared vendor list. Revoking the private-hop permission on the ORIGIN hop broke the case `_screen_tool_url` deliberately admits: a dual-stack or split-horizon home-lab host answering with both a LAN and a public record was approved, then refused on its own `302 /login` — one hop was all it ever got. Track the approved HOST instead, so redirects that stay on it remain covered while a redirect to any other private host is still refused once the chain is no longer wholly private. Also: - Try several registry candidates for the collector-scope probe instead of abandoning it when the first is unresolvable, which also stopped a healthy registry from logging as malformed. - Bound the probe's name resolution with an explicit timeout matching the 2s the httpx connect deadline used to provide; it runs before the console lifespan yields and getaddrinfo has no timeout of its own. - Route doctor and the console probe through `web.screen_url` rather than keeping a third and fourth copy of parse/resolve/classify/fold, which had already diverged on default port and empty-hostname wording. An empty hostname no longer reports as a cloud-metadata refusal. - Give `screen_url` a scheme-aware default port. - Stop doubling the word "hostname" in the OAuth resolution refusal. - Correct the `_screen_tool_url` docstring: it described `private_origin` as requiring every record to be private, which the mixed-record decision reversed, and `private_block` as a property of a refusal when it reports the lane on the success path too. - Make the preview tests' screening stub opt-in rather than autouse — as a module-wide fixture it also stubbed the tests whose subject IS the screen, so one of them would have passed even if screening refused everything. Verified the module now passes with all name resolution blocked. * fix(security): refuse mixed-record private origins instead of exempting them The previous commit let an approved private origin redirect to itself by exempting its hostname from the chain-wide revocation. That exemption was wrong three ways: it was captured once and never cleared, so a public hop could steer the fetcher back into the approved host at a path of its choosing — reopening the private -> public -> private bypass; it was re-entrant across same-host redirects with fresh DNS each time, so a self-redirecting host could walk arbitrary internal addresses; and it matched on bare hostname, so it spanned every port on the approved box. All three were reproduced against the parent commit, which refuses them. Delete the exemption rather than repair it. The case it existed for — a dual-stack host answering with both a LAN and a public record — is now refused where it is actually decidable, in `_screen_tool_url`, with the remedy in the message: point the tool at the LAN address directly. A granted chain therefore always starts wholly private, so the fetch guard needs no notion of an approved host and stays one unconditional rule. That the accommodation could not be expressed safely in the guard is the signal: the connection may land on either record, so approving such a host never described where the fetch would go. Also from the same review: - Walk the whole service registry for a collector-scope probe candidate instead of the first three, and split the outcome into three log lines, so entries that are merely unreachable stop raising the malformed-registry alarm and skipping the boot check cluster-wide. - Stop the candidate walk on a resolver timeout. `asyncio.timeout` bounds the await, not the work, so continuing left one parked thread per timed-out candidate on the shared executor. - Move the metadata-hostname denylist into `ip_classify` and enforce it in `screen_url`, so doctor and the console probe inherit it instead of each keeping a copy. - Drop the scheme-aware default port: a numeric service does not change which addresses resolution returns, and classification reads only those. `parsed.port` is still touched so an out-of-range value refuses. - Correct the vendor-metadata comment, which generalized a claim true of Azure's and Oracle's addresses to Alibaba's CGNAT one. - Rename a test class that was still named for the rule it no longer tests.	2026-08-11 02:18:03 -07:00
Patrick Buckley	7a06f5e8bc	refactor(session): make ModelLane the provider boundary (#979) (#989) ... * refactor(session): make ModelLane the provider boundary (#979) ## Summary This closes the model-lane ownership gap left by #832: `ChatSession` no longer stores raw provider/client handles. `ResolvedModelBinding` now carries the provider, client, model, capabilities, registry generation, and backend-auth configuration as one coherent snapshot. - Atomically rebind existing sessions after model-registry changes while pinning each in-flight send, fallback, judge, output guard, task agent, title, compaction, perception, and voice operation to its initiating principal and binding. - Fence UI publication, canonical trajectory folds, durable writes, streams, retries, child scopes, and judge work by generation. Stop can hand off to a successor without accepting late state; cancelled tools retain typed effect receipts, and concurrent approval batches resolve by exact cycle or call. - Make create, fork, open, close, and delete race-safe with hidden `creating` reservations, incarnation-aware state tails, and an ACL-rechecked transaction that clones checkpoint-bounded history, configuration, project/persona state, and attachment references. - Extend REST/OpenAPI and Python/TypeScript SDK contracts for create/fork inputs, routed-create metadata, live-workstream probes, targeted approvals, and structured cancellation results. - Update architecture, storage, authentication, judge, channel, console, API, and SDK documentation, including regenerated architecture diagrams and OpenAPI artifacts. ## Validation - SQLite suite: 11,188 passed, 9 skipped, 10 deselected - PostgreSQL suite: 11,195 passed, 2 skipped, 10 deselected - Live backend: 3 passed - SSE recovery: 6 passed; browser recovery harness passed all scenarios - Ruff: clean; 595 files correctly formatted - mypy: 243 source files clean - TypeScript: typecheck/build and 35 tests passed - OpenAPI artifacts fresh; all 14 changed diagrams reproduce byte-for-byte - `git diff --check` and Git LFS integrity clean Closes #979. * fix(deps): update nanoid for GHSA-2v37-7h3g-55p8 Refresh the transitive lock entry admitted by PostCSS so the TypeScript security gate no longer resolves the vulnerable custom-generator implementation. Validation: - npm ci - npm audit --audit-level=moderate: 0 vulnerabilities - TypeScript typecheck and build - TypeScript tests: 35 passed * fix(test): assert canonical model registry URLs Replace prefix checks with exact canonical base URL assertions so the tests do not model incomplete URL validation. Validation: tests/test_model_registry.py (185 passed); Ruff check/format; mypy.	2026-08-08 16:13:35 -07:00
Patrick Buckley	bbe92faca1	fix(preview): fetch ceiling tracks the widest kind cap, not a flat 10 MB ... Review feedback (PR #800): the URL lane hard-capped fetched bodies at 10 MB before kind resolution, making the 32 MiB pdf cap unreachable for URL targets while path targets honored it. The flat pre-check is gone; the guarded fetch's max_bytes now tracks max(PREVIEW_SIZE_CAPS.values()) - mirroring the path lane's stat pre-check - and the per-kind caps after resolution stay authoritative. Also drops a redundant function-local asyncio import in test_console.py.	2026-07-07 08:20:57 -07:00
Patrick Buckley	29a4bbf876	fix(preview,web): stream guarded fetches under a byte budget; salt preview blob ids ... fetch_with_ssrf_guard now streams the response under a max_bytes budget (default 32 MiB, counted on decoded bytes so gzip cannot expand past it) instead of buffering blind - an unbounded body previously filled memory before any caller-side size cap could run. Redirect-hop bodies are no longer read at all, and the realized response drops stale wire-framing headers (content-encoding/content-length/transfer-encoding) that no longer describe the decoded content it carries. Preview blob ids are salted out of the model-visible attachment namespace (sha256("preview:" + body)): uploads use bare sha256(body) and save_attachment freezes kind at first insert, so a byte-identical preview/upload pair would otherwise share a row - whichever landed second inherited the other's kind, silently hiding an upload from model context or materializing preview bytes into a tool turn.	2026-07-07 08:20:57 -07:00
Patrick Buckley	09abc9d199	feat(tools): allow_private_network opt-in for private-address fetch/preview ... turnstone's primary audience self-hosts it beside other lab services — a web_fetch or open_preview aimed at Grafana, Home Assistant, or a dev node on the local network is the operator using their own network, not an attack. The hard SSRF refusal made those targets unreachable. New runtime setting tools.allow_private_network (settings registry, default off, rendered in console Settings → Tools; hot — read per tool call, no restart). When enabled, a call NAMING a private address becomes approvable: the approval prompt tags it "(private network)" so the operator approves it as what it is, and the human gate stays. The redirect side-door stays closed either way: a PUBLIC target that 302s into private address space is refused regardless of the opt-in — that address never appeared on the approval card, so it is never fetched. Only a chain whose approved origin was itself private skips hop screening (its redirects are the operator's own network). Refusals now teach the knob (mirrors the oidc opt-in hint): the error names tools.allow_private_network and where to enable it. Surfaces without a ConfigStore (bare CLI, eval) stay strict — there is no admin surface to have opted in on.	2026-07-07 08:20:57 -07:00
Patrick Buckley	1e2ab91ec2	feat(preview): probe preflight, legacy charsets, remote-assets opt-in, md vendor parity ... Four follow-ups to the preview pane: - Probe-mode preflight: the pane preflights src-loaded kinds with GET ?probe=1 (204, real hardening headers, no body) instead of HEAD — the console reverse proxy forwards HEAD as a full GET, so the old preflight dragged the whole blob across the node→console hop twice. Ownership gate + renderable-type check still run on probes. - Legacy-charset text: table/text/markdown now transcode to UTF-8 at store time (declared charset → UTF-8 → cp1252-replace ladder), same model the web kind already used. The ladder applies only when the text kind was DECLARED (MIME/extension/override); the bare no-hint fallback stays strict UTF-8 and NUL bytes still hard-reject, so binary rejection is unchanged. - Remote assets default OFF: previewed pages are now served under "sandbox; default-src 'none'; style-src 'unsafe-inline'; img-src data:; font-src data:" — they render with inline styling but cannot contact their origin site (no viewer IP/traffic disclosure). A per-pane "Load remote images & styles" checkbox (web previews only, sticky, not persisted) reloads with ?assets=1 for the permissive bare-sandbox mode. - Markdown vendor parity: preview markdown now runs renderer.js's postRenderMarkdown (hljs token coloring + lazy mermaid diagrams) like the conversation pane, with preview-scoped code-block/KaTeX chrome (the conversation theme is .msg.assistant-scoped). Tests: probe/assets HTTP + policy coverage, charset ladder units + stored-bytes round-trip, JS static guards for the probe form, the default-off toggle, and the post-pass; headless-chrome harness grew to 41 assertions (probe-not-HEAD, toggle visibility/default, fenced-code render). Full suite green.	2026-07-07 08:20:57 -07:00
Patrick Buckley	e010124008	feat(preview): rich preview pane + open_preview tool ... Tool results only ever rendered as plain text in the transcript. This adds the model-driven rich-preview lane every comparable surface has, in turnstone's developer-tool idiom: a preview pane that opens BESIDE the conversation, keyboard-operable, sandboxed, never replacing the transcript that spawned it. Backend - New built-in open_preview(target, kind?, title?): resolves an http(s) URL, a file path, or attachment:<id> to bytes; classifies into web/pdf/image/table/text/markdown (magic bytes > MIME hint > extension > UTF-8 fallback, legacy-charset pages transcoded); caps size per kind; persists content-addressed with kind="preview" — refcounted and GC'd with the workstream, skipped by trajectory reconstruction so preview bytes can never materialize onto the wire. URL targets gate like web_fetch (network egress); paths/attachments run unprompted like read_file. - New core.web.fetch_with_ssrf_guard: manual redirect walk that SSRF-screens every hop BEFORE requesting it (follow_redirects=True checked nothing between hops); adopted by both open_preview and web_fetch. URL userinfo is stripped before the descriptor or the stored bytes see it; <base href> is injected doctype-safely so relative assets resolve without quirks mode. - The preview descriptor rides the tool turn's meta side channel with ONE shape on every boundary: the live tool_result SSE event, the conversations.meta column, and the /history projection. Cancelled batches commit an already-announced preview (blob + meta) instead of stranding the open pane on a permanent 404. - New GET {ws}/attachments/{id}/preview (read scope, same ownership gate as /content) serves the STORED type with per-MIME hardening: bare CSP sandbox for text/html (renderable, scriptless, opaque origin), no CSP for application/pdf (Chromium's viewer refuses sandboxed contexts), full default-src 'none' otherwise; filenames fold to latin-1-safe ASCII. The console /node proxy now forwards CSP/nosniff/disposition/cache-control instead of dropping them. - History loads exclude preview blobs from the bulk content fetch at the query (they were read and discarded on every load). Frontend - New "preview" pane type registered in the shared shell (server + console): openPaneBeside placement, per-kind renderers — fully sandboxed iframe for pages, browser PDF viewer, sortable tables (CSV/TSV/JSON, ragged-file safe, 5k-row cap), rendered markdown, text — plus back/forward history with arrow keys, reload persistence via pane meta, and backoff auto-retry (0.9s..7.2s) bridging the gap between the live descriptor and the batch fold that commits its blob. - Tool results carrying a descriptor render a credential-redacted preview chip (the reopen + replay affordance); live results auto-open the pane only while the originating pane holds focus. Docs: docs/tools.md + prompts/tools.md. Tests: policy unit tests, tool prepare/exec (mocked fetch), serving route + proxy header pass-through, storage exclusion on both backends, cancel-path commit, JS static guards; a headless-Chrome harness drives the real module graph (32 DOM assertions).	2026-07-07 08:20:57 -07:00