Compare commits

..

12 Commits

Author SHA1 Message Date
Patrick Buckley 0397640567 feat(tls): support cross-host mTLS with lacme 1.2 2026-08-13 02:19:45 -07:00
Patrick Buckley 28ef63a10c fix: revalidate frontend assets across builds 2026-08-12 19:00:31 -07:00
Patrick Buckley 998271b016 feat: convert large pastes to attachments 2026-08-12 00:27:38 -07:00
Patrick Buckley 6eae1c3954 fix(providers): support OpenAI v3 HTTPX2 transport 2026-08-11 23:30:44 -07:00
Patrick Buckley d961af5c45 Remove unreachable MCP owner branch 2026-08-11 22:03:10 -07:00
Patrick Buckley 0599d72625 Apply MCP formatting 2026-08-11 22:03:10 -07:00
Patrick Buckley 2b51b2f8fa Resolve FastMCP settings before lifecycle tests 2026-08-11 22:03:10 -07:00
Patrick Buckley 8517b42ec1 Handle partial MCP resource discovery 2026-08-11 22:03:10 -07:00
Patrick Buckley ed6286ab63 fix(deps): constrain OpenAI SDK below v3 2026-08-11 21:58:21 -07:00
Patrick Buckley 3252f3fd95 fix(memory): preserve replay identity and validation 2026-08-11 21:58:21 -07:00
Patrick Buckley cc84f9d176 fix(memory): harden project scope authorization and consistency 2026-08-11 21:58:21 -07:00
Copilot d2a6c2852e Stabilize context-overflow compaction test in Python 3.11 CI (#1006)
* Stabilize overflow compaction test expectations

Co-authored-by: eous <13773563+eous@users.noreply.github.com>

* Fix ObservedRLock compatibility with Python 3.14 Condition

Co-authored-by: eous <13773563+eous@users.noreply.github.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: eous <13773563+eous@users.noreply.github.com>
2026-08-11 05:46:02 -07:00
109 changed files with 9450 additions and 2014 deletions
+3
View File
@@ -42,6 +42,9 @@
# CONSOLE_HTTPS_PORT=8443 # Caddy (dashboard HTTPS)
# POSTGRES_PORT=5432 # exposed for bare-metal host joins
# POSTGRES_BIND=127.0.0.1 # set 0.0.0.0 to let another machine join
# TURNSTONE_HOST_IP=127.0.0.1 # dev-stack bind address for cross-host joins
# TURNSTONE_CONSOLE_HTTP_BIND=127.0.0.1 # production TLS-overlay ACME/API bind
# TURNSTONE_ACME_EXTERNAL_URL=http://192.0.2.1:8090/acme # routable ACME base; bracket IPv6; include /acme
# -- Workspace ----------------------------------------------------------------
# Bind-mount a host directory the model can read/write at /workspace:
+1 -1
View File
@@ -55,7 +55,7 @@
{
"description": "LLM SDKs — always review manually",
"groupName": "LLM SDKs",
"matchPackageNames": ["openai", "anthropic", "mcp"],
"matchPackageNames": ["openai", "httpx2", "anthropic", "mcp"],
"schedule": ["before 9am on Monday"],
"automerge": false
},
+45
View File
@@ -18,6 +18,12 @@ Earlier stable lines (`stable/1.6`, `stable/1.5`) are frozen.
### Added
- **Large plain-text pastes become attachments.** Pasting text longer than the
fixed 2,000-character threshold stages `pasted-text.txt` across all five
attachment-capable create/send composers. Clipboard files retain priority,
text above the 512 KiB upload ceiling stays inline, identical synthesized
pastes collapse to one chip, and rejected attachment sends keep the staged
message and files so they can be corrected or retried.
- **`server_parses_reasoning` model capability.** Declare it on a model
definition whose backend segregates reasoning into its own channel (a
vLLM launched with a reasoning parser, a commercial provider): the
@@ -152,6 +158,35 @@ Earlier stable lines (`stable/1.6`, `stable/1.5`) are frozen.
### Changed
- **lacme 1.2.0 and HTTPX2 now back the core mTLS path.** TLS/ACME is a typed
core dependency rather than optional-import-era code; renewal and admin
clients use lacme's public close/stop lifecycle. Consoles can set
`TURNSTONE_ACME_EXTERNAL_URL` to a routable responder base ending in `/acme`,
so nodes on another host receive usable directory and follow-up URLs during
enrollment. Signing routes now require a dedicated, purpose-confined rotating
service JWT, and HTTPX2 pins that credential to canonical resources on
configured responder origins. Internal and external console identities use
separate persistence namespaces; cluster identities are reused only after
key, SAN, validity, EKU, and active-root verification. Responder-side keyless
CSR results cannot overwrite managed identities, failed live reloads restore
the last usable bundle, certificate lifetime stays at 48 hours with a
12-hour renewal cadence, and cancellation drains renewal clients before
propagating. Operator-supplied IPv4 and IPv6 literals are passed to lacme as
typed IP identifiers and issued as IP SANs; unexpired legacy certificates
containing `DNS:<ip>` are reissued instead of being reused as an invalid IP
identity
([#1011](https://github.com/turnstonelabs/turnstone/issues/1011)).
- **OpenAI SDK v3 and its HTTPX2 default transport are now supported (#1009).**
Chat Completions and Responses streams normalize native HTTPX2 connection
deaths through the same retry boundary as legacy HTTPX-backed providers,
including failures observed after safe cross-thread client closure during a
model-registry reload. The OpenAI v3 runtime escape hatch for explicitly
injected legacy HTTPX clients remains supported. OpenAI connections now
follow HTTPX2's operating-system trust store by default; deployments that
relied on a modified `certifi` bundle must install that CA in the system
store or set `SSL_CERT_FILE` / `SSL_CERT_DIR`.
- **Log event rename: `drain_stream.post_finish_blip` is now
`stream.post_finish_blip`; its `usage_captured` field is retained.** The
single-shot drain normalizes mid-body transport deaths through the same
@@ -217,6 +252,16 @@ Earlier stable lines (`stable/1.6`, `stable/1.5`) are frozen.
### Fixed
- **MCP servers may expose resources without resource templates, or templates
without concrete resources (#993).** The MCP handshake has one aggregate
`resources` capability, but implementations are not required to support both
list methods. An exact JSON-RPC `Method not found` response from either
method is now treated as an empty half-catalog during static and per-user
discovery and refresh, while every other error still fails closed. A failed
static registration also tears down its transport and publishes none of its
staged tools/resources/prompts, eliminating the live callable “ghost” tools
that could remain after the UI reported registration failure.
- **A cancelled judge, guard, or compaction call can now stop before its
request goes out (#972).** Previously it could not: `model_turn` refused
to *re-issue* an abandoned call after a mid-stream death, but nothing
+14 -5
View File
@@ -53,8 +53,9 @@
# The node registers in Postgres, auto-enrolls its mTLS cert from the console's
# ACME endpoint (when the cluster runs mTLS), and the console collector reaches
# it back via host.docker.internal. To join from ANOTHER machine, set
# TURNSTONE_HOST_IP to this host's LAN IP and use it in the URLs above (and the
# node's TURNSTONE_ADVERTISE_URL = the NODE host's IP) — see docs/docker.md.
# TURNSTONE_HOST_IP to this host's LAN IP and set TURNSTONE_ACME_EXTERNAL_URL to
# http://<this-host-ip>:8090/acme. Use the same host IP in the node's URLs above
# (and the NODE host's IP in TURNSTONE_ADVERTISE_URL) — see docs/docker.md.
# =============================================================================
name: turnstone
@@ -154,9 +155,11 @@ services:
# Publishes the console's plain-HTTP listener so a bare-metal node can reach
# the ACME endpoint, fetch the CA, and enroll its cert (the console serves
# HTTP here even under mTLS). Bound to 127.0.0.1 by default; setting
# TURNSTONE_HOST_IP exposes the WHOLE console HTTP API — including the
# cert-issuing ACME endpoint — on that interface, so the JWT secret's
# strength is the only gate. Browsers use Caddy :8443, never this port.
# TURNSTONE_HOST_IP exposes the WHOLE console HTTP API on that interface.
# ACME signing routes require a dedicated short-lived service JWT, but the
# listener and bearer token are still plain HTTP: bind only a trusted LAN
# or VPN interface and restrict it to enrolling nodes. Browsers use Caddy
# :8443, never this port.
ports:
- "${TURNSTONE_HOST_IP:-127.0.0.1}:8090:8090"
environment:
@@ -164,6 +167,9 @@ services:
TURNSTONE_DB_BACKEND: *db-backend
TURNSTONE_DB_URL: *db-url
TURNSTONE_CONSOLE_URL: http://console:8090
# Separate from TURNSTONE_CONSOLE_URL: this is the canonical responder
# base embedded in ACME directory/order URLs for cross-host enrollment.
TURNSTONE_ACME_EXTERNAL_URL: "${TURNSTONE_ACME_EXTERNAL_URL:-}"
extra_hosts:
- "host.docker.internal:host-gateway"
networks:
@@ -305,6 +311,9 @@ services:
SKIP_PERMISSIONS: ${SKIP_PERMISSIONS:-}
TURNSTONE_NODE_ID: node-1
TURNSTONE_ADVERTISE_URL: http://node-1:8080
# Lets the authenticated ACME client follow the console's canonical LAN
# URLs without trusting destinations learned from the public directory.
TURNSTONE_ACME_EXTERNAL_URL: "${TURNSTONE_ACME_EXTERNAL_URL:-}"
extra_hosts:
- "host.docker.internal:host-gateway"
networks:
+9 -5
View File
@@ -38,10 +38,18 @@ services:
condition: service_completed_successfully
volumes:
- tls-certs:/certs:ro
# The production base keeps :8090 private. The TLS overlay publishes it on
# localhost for same-host enrollment; use a trusted LAN/VPN address for a
# remote node and firewall it to that node.
ports:
- "${TURNSTONE_CONSOLE_HTTP_BIND:-127.0.0.1}:8090:8090"
environment:
TURNSTONE_TLS_ENABLED: "true"
TURNSTONE_TLS_SANS: "console"
TURNSTONE_CONSOLE_URL: "http://console:8090"
# Canonical ACME responder base advertised to enrolling nodes. Set this
# when they reach the console through a different host/address.
TURNSTONE_ACME_EXTERNAL_URL: "${TURNSTONE_ACME_EXTERNAL_URL:-}"
command:
- turnstone-console
- --host=0.0.0.0
@@ -58,11 +66,7 @@ services:
environment:
TURNSTONE_TLS_ENABLED: "true"
TURNSTONE_TLS_SANS: "server"
# Disable healthcheck — server serves HTTPS with mTLS which the
# stdlib healthcheck script can't satisfy. The base compose
# healthcheck uses plain HTTP which won't work on an HTTPS listener.
healthcheck:
disable: true
TURNSTONE_ACME_EXTERNAL_URL: "${TURNSTONE_ACME_EXTERNAL_URL:-}"
# Channel: TLS
channel:
+23 -8
View File
@@ -15,11 +15,14 @@ counterpart to the quick `turnstone-server …` invocation in
## Cluster-side prerequisite
The compose stack must publish Postgres, the console's ACME endpoint, and SearxNG
on an address the bare-metal host can reach. Start it with `TURNSTONE_HOST_IP`
set to the compose host's LAN IP (default `127.0.0.1` keeps everything host-local):
on an address the bare-metal host can reach. Use a trusted LAN or VPN interface,
firewall it to the joining node, and advertise the same reachable ACME endpoint
(default `127.0.0.1` keeps everything host-local):
```bash
TURNSTONE_HOST_IP=<compose-host-ip> docker compose up -d
TURNSTONE_HOST_IP=<compose-host-ip> \
TURNSTONE_ACME_EXTERNAL_URL=http://<compose-host-ip>:8090/acme \
docker compose up -d
```
## Install (run as root on the bare-metal host)
@@ -65,8 +68,20 @@ journalctl -u turnstone-server -f # watch it register + (if the cluster
shared settings (the database). If the cluster runs mTLS, the node auto-enrolls a
cert from the console's ACME endpoint and re-advertises itself over `https://`.
> **mTLS + cross-host caveat:** a node on a *different* host than the console
> currently can't complete ACME enrollment — the console advertises an
> unroutable in-container address in its ACME directory
> ([turnstonelabs/lacme#22](https://github.com/turnstonelabs/lacme/issues/22)).
> Same-host bare-metal nodes, and any node in a non-mTLS cluster, are unaffected.
For a node on a different host, `TURNSTONE_ACME_EXTERNAL_URL` is required on the
console and should also be set in the node drop-in. It is the full, externally
reachable responder base
(including `/acme`) that the console embeds in the ACME protocol's follow-up
URLs and that the node trusts as an enrollment-credential destination. The
node's `TURNSTONE_CONSOLE_URL` should point at the same host and port, without
the `/acme` suffix.
For mTLS, `TURNSTONE_ADVERTISE_URL` may use a resolvable DNS hostname or a
literal IP address. Turnstone enrolls literals as IP SANs. Bracket IPv6 literals
inside URLs, for example `http://[2001:db8::10]:8080`; do not use wildcard,
unspecified, or scoped addresses as certificate identities. Restart the node
after changing its advertised identity so it enrolls a matching certificate.
The dedicated service JWT authenticates enrollment but the direct `:8090`
bootstrap is still plain HTTP/TOFU. Use HTTPS through an independently trusted
proxy when the network itself is not trusted.
@@ -4,16 +4,20 @@
# they live here; the JWT secret + DB URL live in /etc/turnstone/config.toml.
#
# Addresses below use RFC 5737 documentation IPs — replace them:
# <this-host> = the bare-metal host's own LAN IP (what the console dials back)
# <this-host> = the bare-metal host's own reachable DNS name or IP address
# (its mTLS identity and the address peers dial)
# <compose-host> = the host running the cluster / docker-compose stack, started
# with TURNSTONE_HOST_IP=<compose-host> so :8090 and :8081 are
# published on its LAN interface (see docs/docker.md).
# with TURNSTONE_HOST_IP=<compose-host> and
# TURNSTONE_ACME_EXTERNAL_URL=http://<compose-host>:8090/acme
# so enrollment links and published ports are reachable
# (see docs/docker.md).
[Service]
# Unique node id (defaults to the hostname if unset).
Environment=TURNSTONE_NODE_ID=host-1
# The address peers + the console collector dial back. Auto-upgrades to https://
# once the node enrolls its mTLS cert.
# once the node enrolls its mTLS cert. IPv6 literals require URL brackets, for
# example http://[2001:db8::10]:8080.
Environment=TURNSTONE_ADVERTISE_URL=http://192.0.2.10:8080
# The cluster console's reachable plain-HTTP ACME/API endpoint. A bare-metal node
@@ -21,5 +25,9 @@ Environment=TURNSTONE_ADVERTISE_URL=http://192.0.2.10:8080
# port; turnstone-server honors this for cert enrollment.
Environment=TURNSTONE_CONSOLE_URL=http://192.0.2.1:8090
# Trusted canonical responder base. This pins where the node may send its
# enrollment JWT; it must match the console-side value (a literal IP is fine).
Environment=TURNSTONE_ACME_EXTERNAL_URL=http://192.0.2.1:8090/acme
# The cluster's published SearxNG, for the web_search tool.
Environment=TURNSTONE_SEARXNG_URL=http://192.0.2.1:8081
+26 -16
View File
@@ -1758,16 +1758,19 @@ Status code: `403`
### `GET /v1/api/memories`
List structured memories with optional filters. Requires `read` scope.
List structured memories with optional filters. Requires `read` scope. Without
`scope`, returns only `global` plus the authenticated caller's `user`
namespace. The public endpoint accepts `global`, `workstream`, and `user`;
explicit workstream access is owner-bound.
**Query parameters:**
| Parameter | Type | Required | Default | Description |
|------------|--------|----------|---------|------------------------------|
| `type` | string | no | `""` | Filter by memory type (user, project, feedback, reference) |
| `type` | string | no | `""` | Filter by memory type (user, general, feedback, reference) |
| `scope` | string | no | `""` | Filter by scope (global, workstream, user) |
| `scope_id` | string | no | `""` | Scope qualifier. Auto-resolved for `scope=user` when auth is active. |
| `limit` | int | no | `100` | Max results (capped at 200) |
| `limit` | int | no | `100` | Max results (1-200) |
**Response:**
@@ -1778,7 +1781,7 @@ List structured memories with optional filters. Requires `read` scope.
"memory_id": "a1b2c3d4-e5f6-...",
"name": "project_architecture",
"description": "Core architecture patterns",
"type": "project",
"type": "general",
"scope": "global",
"scope_id": "",
"content": "The project uses a hexagonal architecture...",
@@ -1795,7 +1798,8 @@ List structured memories with optional filters. Requires `read` scope.
### `POST /v1/api/memories`
Save or upsert a structured memory. Requires `write` scope. Returns `201` on
create, `200` on update.
create, `200` on update. Every write must include a non-empty, non-whitespace
`description`; content-only updates are rejected.
**Request body:**
@@ -1804,7 +1808,7 @@ create, `200` on update.
"name": "deployment_process",
"content": "Deploy via GitHub Actions. Staging auto-deploys on push to main.",
"description": "CI/CD deployment workflow",
"type": "project",
"type": "general",
"scope": "global",
"scope_id": ""
}
@@ -1814,8 +1818,8 @@ create, `200` on update.
|--------------|--------|----------|-------------|--------------------------------------|
| `name` | string | yes | -- | Memory name (max 256 chars) |
| `content` | string | yes | -- | Memory content (max 65536 chars) |
| `description`| string | no | `""` | Short description for search ranking |
| `type` | string | no | `"project"` | One of: user, project, feedback, reference |
| `description`| string | yes | -- | Non-empty relevance summary, required on create and update |
| `type` | string | no | unset | user, general, feedback, or reference |
| `scope` | string | no | `"global"` | One of: global, workstream, user |
| `scope_id` | string | no | `""` | Scope qualifier (auto-resolved for user scope) |
@@ -1826,7 +1830,7 @@ create, `200` on update.
"memory_id": "a1b2c3d4-e5f6-...",
"name": "deployment_process",
"description": "CI/CD deployment workflow",
"type": "project",
"type": "general",
"scope": "global",
"scope_id": "",
"content": "Deploy via GitHub Actions...",
@@ -1839,21 +1843,25 @@ create, `200` on update.
| Status | Condition |
|--------|--------------------------------------------------------|
| 400 | Missing name, empty content, invalid type/scope, name too long, content too long |
| 400 | Invalid input, public scope, scope ID, or limit |
| 403 | Cross-user or non-owner workstream access |
| 404 | Explicit workstream does not exist |
| 500 | Storage mutation failed |
---
### `POST /v1/api/memories/search`
Search memories by query. Uses POST for the request body but is non-mutating
(requires only `read` scope).
(requires only `read` scope). An omitted scope searches only `global` plus the
authenticated caller's `user` namespace.
**Request body:**
```json
{
"query": "authentication",
"type": "project",
"type": "general",
"scope": "",
"limit": 20
}
@@ -1865,7 +1873,7 @@ Search memories by query. Uses POST for the request body but is non-mutating
| `type` | string | no | `""` | Filter by type |
| `scope` | string | no | `""` | Filter by scope |
| `scope_id` | string | no | `""` | Filter by scope ID |
| `limit` | int | no | `20` | Max results (capped at 50) |
| `limit` | int | no | `20` | Max results (1-50) |
**Response:**
@@ -1876,7 +1884,7 @@ Search memories by query. Uses POST for the request body but is non-mutating
"memory_id": "a1b2c3d4-e5f6-...",
"name": "auth_patterns",
"description": "Authentication architecture",
"type": "project",
"type": "general",
"scope": "global",
"scope_id": "",
"content": "JWT tokens with HS256...",
@@ -1894,7 +1902,9 @@ Search memories by query. Uses POST for the request body but is non-mutating
### `DELETE /v1/api/memories/{name}`
Delete a memory by name and scope. Requires `write` scope.
Delete a memory by name and scope. Requires `write` scope. The delete returns
success only for the row atomically removed and records the authenticated
actor in the audit log.
**Path parameters:**
@@ -1978,7 +1988,7 @@ Get a single memory by ID. Requires `admin.memories` permission.
"memory_id": "a1b2c3d4-e5f6-...",
"name": "project_architecture",
"description": "Core architecture patterns",
"type": "project",
"type": "general",
"scope": "global",
"scope_id": "",
"content": "The project uses...",
+14
View File
@@ -429,6 +429,20 @@ Files require a non-empty initial task so the first turn consumes the staged
attachments. The console shell does not currently expose a fork action; use the
node's standalone workstream UI or the create API's `resume_ws` field.
### Large pasted text
Browser composers turn plain text longer than 2,000 Unicode code points into a
`text/plain` attachment named `pasted-text.txt`. A paste exactly at the
threshold stays inline. This applies to the interactive and coordinator send
boxes, the console home launcher, and the node dashboard and new-workstream
composers.
Clipboard files take priority over clipboard text. Text larger than the 512 KiB
attachment ceiling also stays inline, so the browser does not discard it before
a rejected upload. Attachments require a companion message and cannot be sent
as live-turn interjections; a busy composer preserves its message and chips for
an idle retry.
### Saved and filtered sessions
Saved coordinator and interactive sessions share one list with kind and persona
+1 -1
View File
@@ -77,7 +77,7 @@ or MCP config can do adds to it. Current members:
| `delete_workstream` | wind-down | Hard-delete one child. Requires approval. |
| `list_nodes` | discover | Enumerate live cluster nodes + capabilities. |
| `skills` (action=find) | discover | Browse the skill catalog; opt-in `kind` filter narrows by audience. |
| `memory` | persist | Durable orchestration memory (`coordinator` scope, per-user — survives across coordinator sessions). |
| `memory` | persist | Durable acting-user orchestration memory (`coordinator`), plus shared memory when attached to a project. |
| `notify` | broadcast | Post a status update to a human channel at a narrative beat. |
| `tasks` | plan | Orchestrator-only scratchpad. Children don't see it. |
+44 -38
View File
@@ -20,58 +20,61 @@ participant "SDK Client\n(sdk/)" as SDK <<sdk>>
== Phase 1: Tool Path (session.send) ==
Session -> Session : _prepare_tool_calls()\nparse memory(action=...)
Session -> Session : pin acting principal\nparse memory(action=...)
note right
Tool schema: 4 actions
save, search, delete, list
Tool schema: 5 actions
save, get, search, delete, list
Auto-approved (no approval needed)
end note
Session -> Session : resolve live project access\nselect exact/inherited scope
Session -> Session : _exec_memory(item)
alt action = save
Session -> Facade : save_structured_memory(\nname, content, description,\nmem_type, scope, scope_id)
Session -> Session : require non-empty description
Session -> Facade : save_structured_memory_strict(\n..., require_active_project)
Facade -> Facade : normalize_key(name)
Facade -> Storage : create_structured_memory()
alt unique constraint violation
Storage --> Facade : IntegrityError
Facade -> Storage : get_structured_memory_by_name()
Storage --> Facade : existing row
Facade -> Storage : update_structured_memory()
end
Storage --> Facade : memory_id
Facade --> Session : (memory_id, old_content)
Session -> Session : _init_system_messages()\nrefresh BM25 context
Facade -> Storage : guarded atomic upsert\nON CONFLICT ... RETURNING
Storage --> Facade : (saved row, was_update)
Facade --> Session : saved row
Session -> Session : invalidate prefix/cache\naudit acting principal
end
alt action = get
Session -> Facade : get_structured_memory_by_name_strict()
Facade -> Storage : exact scoped-name lookup
Storage --> Session : full row / not found
end
alt action = search
Session -> Facade : search_structured_memories(\nquery, mem_type, scope,\nscope_id, limit)
Facade -> Storage : search_structured_memories()
Session -> Storage : search exact scope or\nactor-visible scope union
Storage --> Session : matched rows
end
alt action = delete
Session -> Facade : delete_structured_memory(\nname, scope, scope_id)
Facade -> Storage : delete_structured_memory()
Storage --> Session : bool (existed)
Session -> Session : _init_system_messages()\nrefresh BM25 context
Session -> Facade : delete_structured_memory_returning_strict()
Facade -> Storage : DELETE ... RETURNING
Storage --> Session : deleted row / not found
Session -> Session : invalidate + audit\nmark prefix dirty
end
== Phase 2: BM25 Relevance Injection ==
Session -> Session : _init_system_messages()\nevery conversation turn
Session -> Session : resolve acting principal\nand live project ACL
Session -> Session : _list_visible_memories(\nlimit=fetch_limit)
note right
**Scope resolution:**
1. global scope (always)
2. workstream scope (ws_id)
3. user scope (user_id, if auth)
Combined and deduplicated.
Interactive: global + workstream
+ acting user + readable project
Coordinator: acting user's coordinator
+ readable project
end note
Session -> Facade : list_structured_memories()\nper scope
Facade -> Storage : list_structured_memories()
Session -> Facade : list_visible_structured_memories()
Facade -> Storage : one visibility-union query
Storage --> Session : up to fetch_limit rows
Session -> Relevance : extract_recent_context(\nmessages, max_messages=3)
@@ -103,28 +106,31 @@ Session -> Session : inject into\nsystem message
== Phase 3: Server API Path ==
SDK -> API : GET /v1/api/memories\n?type=project&limit=20
API -> Facade : list_structured_memories()
Facade -> Storage : list_structured_memories()
SDK -> API : GET /v1/api/memories\n?type=general&limit=20
API -> API : bind scope to caller\ndefault global + caller user
API -> Storage : list visible rows
Storage --> API : rows
API --> SDK : {"memories": [...], "total": N}
SDK -> API : POST /v1/api/memories\n{name, content, ...}
API -> API : validate type, scope,\nname length, content length
API -> Facade : save_structured_memory()
Facade -> Storage : create / update
SDK -> API : POST /v1/api/memories\n{name, content, description, ...}
API -> API : validate type, scope,\nname/content/description
API -> API : reject internal scopes\nowner-bind workstream scope
API -> Facade : save_structured_memory_strict()
Facade -> Storage : atomic upsert
Storage --> API : memory row
API -> API : record_audit(actor)
API --> SDK : 201 (created) / 200 (updated)
SDK -> API : POST /v1/api/memories/search\n{query, type, ...}
API -> Facade : search_structured_memories()
Facade -> Storage : search_structured_memories()
API -> API : bind scope to caller
API -> Storage : search visible rows
Storage --> API : matched rows
API --> SDK : {"memories": [...], "total": N}
SDK -> API : DELETE /v1/api/memories/{name}\n?scope=global
API -> Facade : delete_structured_memory()
Facade -> Storage : delete row
API -> Facade : delete_structured_memory_returning_strict()
Facade -> Storage : DELETE ... RETURNING
API -> API : record_audit(actor)
API --> SDK : {"status": "ok"}
== Phase 4: Console Admin Path ==
@@ -141,7 +147,7 @@ Storage --> Admin : memory row
Admin --> SDK : memory JSON
SDK -> Admin : DELETE /v1/api/admin/memories/{id}
Admin -> Storage : delete_structured_memory_by_id()
Admin -> Storage : delete_structured_memory_by_id_returning()
Admin -> Admin : record_audit(\n"memory.delete")
Admin --> SDK : {"status": "ok"}
+2 -2
View File
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:bd7fe8bf5c2b56b075453a316e54d61214b0ae912517d9cad6c1e88785aac722
size 300010
oid sha256:137d6c91a34695c820d8b0a33fd753e79165604aa92bf2ac8480d3744b2ef844
size 305199
+27 -7
View File
@@ -102,13 +102,26 @@ the cluster runs mTLS; harmless otherwise), and `TURNSTONE_SEARXNG_URL` points
are the dev-stack defaults — match whatever you set in `.env` if you changed them.
To let a server on a **different** machine join, start the stack with
`TURNSTONE_HOST_IP=<this host's LAN IP>` — that binds PostgreSQL, the console
ACME endpoint, and SearxNG to that interface. Then on the remote box set the
three URLs above to that IP, and set `TURNSTONE_ADVERTISE_URL` to the **remote**
box's own IP (the address the console dials back). **Set a strong
`POSTGRES_PASSWORD` first** — `TURNSTONE_HOST_IP` exposes the database (and every
user account + API-token hash in it), the console API, and the unauthenticated
SearxNG to your network.
both `TURNSTONE_HOST_IP=<this host's LAN IP>` and
`TURNSTONE_ACME_EXTERNAL_URL=http://<this host's LAN IP>:8090/acme`. The first
binds PostgreSQL, the console ACME endpoint, and SearxNG to that interface; the
second makes every URL in the ACME directory routable from the remote node (the
full value must include the `/acme` mount). Set the same
`TURNSTONE_ACME_EXTERNAL_URL` on the remote node so its authenticated ACME
client can pin that credential destination. Set `TURNSTONE_CONSOLE_URL` and
`TURNSTONE_SEARXNG_URL` to the compose host's IP, but set
`TURNSTONE_ADVERTISE_URL=http://192.0.2.10:8080` to the **remote** box's own
address. A resolvable DNS name works too. IPv6 literals must be bracketed in
URLs, for example `http://[2001:db8::10]:8080`; Turnstone enrolls literal
addresses as IP SANs rather than numeric DNS SANs.
Use a trusted LAN or VPN address and firewall `:8090` to enrolling nodes. ACME
signing routes require a dedicated short-lived service JWT, but direct bootstrap
is still plain HTTP/TOFU: a bearer token provides authentication, not transport
confidentiality or protection from an active on-path attacker. **Set a strong
`POSTGRES_PASSWORD` first** — `TURNSTONE_HOST_IP` also exposes the database (and
every user account + API-token hash in it), the console API, and the
unauthenticated SearxNG to your network.
To run the bare-metal node as a hardened, persistent service instead of by hand,
use the systemd units in [`deploy/systemd/`](../deploy/systemd/).
@@ -148,6 +161,11 @@ certs via the console's ACME endpoint:
docker compose -f turnstone/deploy/compose.yaml -f deploy/docker-compose.tls.yml up
```
The overlay publishes the console's plain-HTTP bootstrap/API port on
`TURNSTONE_CONSOLE_HTTP_BIND` (default `127.0.0.1`). For a cross-host node, set
that to a trusted LAN/VPN address, set `TURNSTONE_ACME_EXTERNAL_URL` to the same
address plus `/acme`, and firewall the port to enrolling nodes.
See [tls.md](tls.md) for details.
## Configuration
@@ -206,6 +224,8 @@ Caddy or proxied by the console:
| `POSTGRES_PORT` | `5432` | Host port for PostgreSQL (for bare-metal joins) |
| `SEARXNG_API_PORT` | `8081` | Host port for the SearxNG API a bare-metal node's `web_search` dials (dev stack) |
| `TURNSTONE_HOST_IP` | `127.0.0.1` | Interface PostgreSQL, the console ACME endpoint, and SearxNG bind on (dev stack). Set to this host's LAN IP so a bare-metal node on **another machine** can reach them — set a strong `POSTGRES_PASSWORD` first (it also exposes the DB and the unauthenticated SearxNG to your network). |
| `TURNSTONE_CONSOLE_HTTP_BIND` | `127.0.0.1` | Production TLS-overlay interface for the console's plain-HTTP bootstrap/API listener. Use only a trusted LAN/VPN address and firewall it to enrolling nodes. |
| `TURNSTONE_ACME_EXTERNAL_URL` | request-derived | Canonical externally reachable ACME responder base, including the final `/acme` mount (for example `http://192.0.2.1:8090/acme`). Set it on the console and clients for cross-host mTLS: the console advertises it, while clients pin it as an allowed enrollment-JWT destination. A reverse-proxy prefix is supported only when the proxy maps it to Turnstone's internal `/acme` mount. |
| `POSTGRES_BIND` | `127.0.0.1` | Production stack (`turnstone/deploy/compose.yaml`) only: interface PostgreSQL binds on; set to the host's LAN IP for remote joins. |
### Channel gateway
+11 -8
View File
@@ -177,7 +177,8 @@ Runs a complete multi-turn conversation:
1. Appends the user message.
2. Checks `_cancelled` event — stops if set (timeout cleanup).
3. Calls the model API (non-streaming).
3. Calls the model through the production streaming provider path and drains
the result.
4. If tool calls are returned, executes them (with stdout suppressed) and
logs each call to `self.tool_call_log`.
5. Repeats up to `max_turns` or until the model responds without tool calls.
@@ -190,14 +191,15 @@ Parallel tool calls are capped at 10 per turn to prevent degenerate repetition.
Each test runs in a `ThreadPoolExecutor(max_workers=1)` with a per-test
timeout (`--test-timeout`). Each attempt gets its own `OpenAI` client with
a matching httpx read timeout. On timeout, three layers of defense prevent
zombie connections:
a matching per-read HTTP transport timeout. Because a trickling stream can
continually reset that read timeout, three layers bound the harness and stop
follow-on work:
1. **httpx timeout**: Per-request read timeout aborts the HTTP call and
releases the server slot.
1. **Executor wall clock**: The harness stops waiting after `--test-timeout`.
2. **`_cancelled` event**: Prevents the orphan thread from starting new turns.
3. **`run_client.close()`**: Closes the connection pool to abort any
in-flight request.
3. **`run_client.close()`**: Retires the connection pool and prevents reuse.
HTTPX2 does not promise that cross-thread client closure immediately aborts
an active body read; that read unwinds on its next wire event or read timeout.
### Retry Logic
@@ -214,7 +216,8 @@ Each test case runs in isolation:
1. A fresh temp directory is created.
2. Setup files are written to the temp directory.
3. The working directory is changed to the temp directory.
4. A per-attempt `OpenAI` client is created with httpx timeout matching `--test-timeout`.
4. A per-attempt `OpenAI` client is created with an HTTP transport timeout
matching `--test-timeout`.
5. A new `HeadlessSession` is created with the current developer prompt.
6. `send_headless()` runs the user prompt through the conversation loop.
7. The tool log is scored against expected actions.
+94 -42
View File
@@ -20,7 +20,7 @@ Each memory has three dimensions:
| Type | Purpose |
|-------------|------------------------------------------------------------|
| `user` | User preferences, conventions, working style |
| `project` | Project-specific knowledge, architecture, patterns |
| `general` | General knowledge, architecture, patterns |
| `feedback` | Corrections, lessons learned, things to avoid |
| `reference` | Reference material, documentation, specifications |
@@ -31,25 +31,37 @@ Each memory has three dimensions:
| `global` | Visible to all workstreams and users |
| `workstream` | Visible only within the originating workstream |
| `user` | Follows the authenticated user across workstreams |
| `coordinator` | Coordinator sessions only; follows the user across coordinators |
| `coordinator` | Coordinator sessions only; follows the acting user |
| `project` | Shared by workstreams attached to one active project |
A memory's identity is the tuple `(name, scope, scope_id)`. Saving a memory
with the same identity upserts -- updating content while preserving the ID.
### Coordinator scope
### Inherited target and coordinator scope
Coordinator sessions are isolated to a single scope: `coordinator`, keyed by
the coordinator's creator `user_id`. It is durable -- every coordinator
session the same user runs (including concurrent ones) shares one
orchestration namespace, so procedures and lessons survive close/reopen.
Name-based operations use one inherited target when `scope` is omitted:
- An attached active project selects `project` for `save`, `get`, and
`delete`.
- Read-only project access permits `get`, but `save` and `delete` fail. They do
not fall back to a broader namespace.
- Without a project, interactive sessions select `global`; coordinator
sessions select `coordinator`.
A valid explicit scope selects exactly that scope. `search` and `list` are the
only actions that span every visible scope when `scope` is omitted.
Each coordinator's private `coordinator` namespace is keyed by the acting
user's `user_id`. It is durable -- every coordinator session that user runs
(including concurrent ones) shares one orchestration namespace, so procedures
and lessons survive close/reopen.
Isolation is bidirectional and enforced by session kind, not by secrecy of
the scope id:
- A coordinator session can read and write **only** `coordinator`-scope rows.
It never sees `global`/`workstream`/`user` memories, so content written by
interactive sessions (which routinely ingest untrusted MCP/attachment
output) cannot reach a coordinator's system message.
- A coordinator session sees its acting user's `coordinator` scope and, when
attached, the shared `project` scope. It never sees
`global`/`workstream`/`user` memories.
- Interactive sessions -- including a coordinator's own children, which share
its `user_id` -- are rejected from the `coordinator` scope on every memory
action. Children cannot plant rows the parent coordinator would read.
@@ -64,12 +76,13 @@ coordinator cannot be constructed, so the scope id is always a real user.
On every conversation turn, the system:
1. Fetches up to `fetch_limit` memories visible in the current scope
2. Extracts context from the last 3 user messages
3. Scores memories against that context using a BM25 index
4. Injects the top `relevance_k` memories into the system message as
1. Resolves the acting principal and their live project access
2. Fetches up to `fetch_limit` memories across that visibility envelope
3. Extracts context from the last 3 user messages
4. Scores memories against that context using a BM25 index
5. Injects the top `relevance_k` memories into the system message as
`<memories>` XML tags
5. Appends a hint telling the model how many memories are in scope
6. Appends a hint telling the model how many memories are in scope
This means the model always has its most relevant memories available without
explicit recall -- but can still use `memory(action='search')` for deeper
@@ -107,19 +120,23 @@ All fields are optional. Defaults are shown above.
## Tool Usage
The `memory` tool supports four actions:
The `memory` tool supports five actions:
### save
Store or update a memory.
Every save is a complete write for the relevance summary: `description` must
be supplied and contain non-whitespace text on both creation and update.
Content-only updates are rejected.
```json
{
"action": "save",
"name": "project_architecture",
"content": "The project uses a hexagonal architecture with...",
"description": "Core architecture patterns",
"type": "project",
"type": "general",
"scope": "global"
}
```
@@ -128,9 +145,26 @@ Store or update a memory.
|---------------|----------|-------------|------------------------------------------|
| `name` | yes | -- | Snake_case identifier (max 256 chars) |
| `content` | yes | -- | Memory content (max `max_content` chars) |
| `description` | no | `""` | Short description for relevance matching |
| `type` | no | `"project"` | One of: user, project, feedback, reference |
| `scope` | no | `"global"` | One of: global, workstream, user |
| `description` | yes | -- | Non-empty relevance summary, required on create and update |
| `type` | no | `"general"` | One of: user, general, feedback, reference |
| `scope` | no | inherited | Kind-valid scope; see inherited target above |
### get
Retrieve the full content of one memory by name.
```json
{
"action": "get",
"name": "project_architecture",
"scope": "project"
}
```
| Parameter | Required | Default | Description |
|-----------|----------|-----------|----------------------------|
| `name` | yes | -- | Memory name to retrieve |
| `scope` | no | inherited | Exact scope to query |
### search
@@ -140,7 +174,7 @@ Find memories by query (BM25 full-text search).
{
"action": "search",
"query": "authentication patterns",
"type": "project",
"type": "general",
"limit": 10
}
```
@@ -167,7 +201,7 @@ Remove a memory by name.
| Parameter | Required | Default | Description |
|------------|----------|------------|--------------------------|
| `name` | yes | -- | Memory name to delete |
| `scope` | no | `"global"` | Scope of the memory |
| `scope` | no | inherited | Exact scope to delete |
### list
@@ -197,6 +231,12 @@ Four endpoints on the server for programmatic memory access.
List memories with optional filters.
Without `scope`, the response is restricted to `global` plus the authenticated
caller's `user` namespace. The public API accepts only `global`, `user`, and
`workstream`; internal `project` and `coordinator` namespaces remain available
through the session tool and admin API. Explicit `workstream` access requires
its persisted owner (or a service token).
**Query parameters:**
| Parameter | Type | Required | Default | Description |
@@ -204,10 +244,10 @@ List memories with optional filters.
| `type` | string | no | `""` | Filter by memory type |
| `scope` | string | no | `""` | Filter by scope |
| `scope_id` | string | no | `""` | Filter by scope ID |
| `limit` | int | no | `100` | Max results (capped at 200) |
| `limit` | int | no | `100` | Max results (1-200) |
When `scope=user` and `scope_id` is omitted, the authenticated user's ID is
used automatically.
When `scope=user`, the authenticated user's ID is used automatically and a
different supplied ID is rejected. `scope=workstream` requires `scope_id`.
**Response:** `200`
@@ -218,7 +258,7 @@ used automatically.
"memory_id": "a1b2c3d4-e5f6-...",
"name": "project_architecture",
"description": "Core architecture patterns",
"type": "project",
"type": "general",
"scope": "global",
"scope_id": "",
"content": "The project uses a hexagonal architecture...",
@@ -236,6 +276,9 @@ used automatically.
Save or upsert a structured memory.
`description` is mandatory for both creates and updates and must contain
non-whitespace text. The API rejects content-only updates.
**Request body:**
```json
@@ -243,7 +286,7 @@ Save or upsert a structured memory.
"name": "deployment_process",
"content": "Deploy via GitHub Actions. Staging auto-deploys on push to main.",
"description": "CI/CD deployment workflow",
"type": "project",
"type": "general",
"scope": "global",
"scope_id": ""
}
@@ -253,8 +296,8 @@ Save or upsert a structured memory.
|--------------|--------|----------|-------------|--------------------------------------|
| `name` | string | yes | -- | Memory name (max 256 chars) |
| `content` | string | yes | -- | Memory content (max 65536 chars) |
| `description`| string | no | `""` | Short description for search ranking |
| `type` | string | no | `"project"` | One of: user, project, feedback, reference |
| `description`| string | yes | -- | Non-empty relevance summary, required on create and update |
| `type` | string | no | unset | user, general, feedback, or reference |
| `scope` | string | no | `"global"` | One of: global, workstream, user |
| `scope_id` | string | no | `""` | Scope qualifier (auto-resolved for user scope) |
@@ -265,7 +308,7 @@ Save or upsert a structured memory.
"memory_id": "a1b2c3d4-e5f6-...",
"name": "deployment_process",
"description": "CI/CD deployment workflow",
"type": "project",
"type": "general",
"scope": "global",
"scope_id": "",
"content": "Deploy via GitHub Actions...",
@@ -281,7 +324,10 @@ same `(name, scope, scope_id)` already existed.
| Status | Condition |
|--------|------------------------------------|
| 400 | Missing name, empty content, invalid type/scope, content too long |
| 400 | Invalid input, scope, scope ID, or limit |
| 403 | Cross-user or non-owner workstream access |
| 404 | Explicit workstream does not exist |
| 500 | Storage mutation failed |
---
@@ -290,12 +336,15 @@ same `(name, scope, scope_id)` already existed.
Search memories by query. Uses POST for the request body but is non-mutating
(requires only `read` scope).
An omitted scope searches the same caller-bound `global` + `user` envelope as
the list endpoint. It never means every row in the table.
**Request body:**
```json
{
"query": "authentication",
"type": "project",
"type": "general",
"scope": "",
"scope_id": "",
"limit": 20
@@ -308,7 +357,7 @@ Search memories by query. Uses POST for the request body but is non-mutating
| `type` | string | no | `""` | Filter by type |
| `scope` | string | no | `""` | Filter by scope |
| `scope_id` | string | no | `""` | Filter by scope ID |
| `limit` | int | no | `20` | Max results (capped at 50) |
| `limit` | int | no | `20` | Max results (1-50) |
**Response:** `200`
@@ -319,7 +368,7 @@ Search memories by query. Uses POST for the request body but is non-mutating
"memory_id": "a1b2c3d4-e5f6-...",
"name": "auth_patterns",
"description": "Authentication architecture",
"type": "project",
"type": "general",
"scope": "global",
"scope_id": "",
"content": "JWT tokens with HS256...",
@@ -337,6 +386,9 @@ Search memories by query. Uses POST for the request body but is non-mutating
Delete a memory by name and scope.
Deletes are atomic: the row used for the success result and audit event is the
row actually removed. A storage failure returns `500`, not a false `404`.
**Path parameters:**
| Parameter | Type | Description |
@@ -391,7 +443,7 @@ List memories across all scopes (no automatic scope resolution).
"memory_id": "a1b2c3d4-e5f6-...",
"name": "project_architecture",
"description": "Core architecture patterns",
"type": "project",
"type": "general",
"scope": "global",
"scope_id": "",
"content": "The project uses...",
@@ -440,7 +492,7 @@ Get a single memory by ID.
"memory_id": "a1b2c3d4-e5f6-...",
"name": "project_architecture",
"description": "Core architecture patterns",
"type": "project",
"type": "general",
"scope": "global",
"scope_id": "",
"content": "The project uses...",
@@ -497,13 +549,13 @@ with TurnstoneServer("http://localhost:8080", token="tok_xxx") as client:
"api_conventions",
"All endpoints use /v1/ prefix. JSON responses.",
description="API design patterns",
mem_type="project",
mem_type="general",
scope="global",
)
print(mem.memory_id)
# Search memories
results = client.search_memories("authentication", mem_type="project", limit=10)
results = client.search_memories("authentication", mem_type="general", limit=10)
for m in results.memories:
print(f"{m['name']}: {m['description']}")
@@ -524,7 +576,7 @@ with TurnstoneConsole("http://localhost:9090", token="tok_xxx") as admin:
result = admin.list_memories(scope="global", limit=100)
# Search
result = admin.search_memories("architecture", mem_type="project")
result = admin.search_memories("architecture", mem_type="general")
# Get by ID
mem = admin.get_memory("a1b2c3d4-e5f6-...")
@@ -548,14 +600,14 @@ const mem = await client.saveMemory({
name: "api_conventions",
content: "All endpoints use /v1/ prefix. JSON responses.",
description: "API design patterns",
type: "project",
type: "general",
scope: "global",
});
// Search memories
const results = await client.searchMemories({
query: "authentication",
type: "project",
type: "general",
limit: 10,
});
+65 -17
View File
@@ -54,15 +54,11 @@ docker compose exec caddy \
cat /data/caddy/pki/authorities/local/root.crt # import into your OS/browser
```
**Can Caddy get its cert from the console's internal CA instead?** Technically
yes — the console exposes a real ACME directory (`/acme/directory`) with
auto-approval, so Caddy's `tls { ca http://console:8090/acme/directory }` would
mint a cert for any name. It's not recommended as the default: lacme's ACME
responder is built for turnstone's own client (interop with Caddy's client is
unverified), it couples Caddy startup to the console, and the browser must trust
a private CA either way — so it buys nothing over `tls internal`. For a publicly
trusted cert (no warning), point Caddy at Let's Encrypt with a real domain
instead.
**Can Caddy get its cert from the console's internal CA instead?** Not directly.
The console's ACME signing routes require Turnstone's rotating enrollment JWT,
which a standard Caddy ACME issuer does not attach. Keep `tls internal`, or use a
public ACME CA for a publicly trusted certificate. An authenticated gateway or
Caddy plugin would be required to use Turnstone's responder.
---
@@ -106,13 +102,15 @@ An mTLS listener rejects plain-HTTP probes at the socket, so
it presents the node's own cert as the client cert and pins the cluster
CA, using the PEM files the server writes at boot under
`$TURNSTONE_TLS_PEM_DIR` (default `<tmpdir>/turnstone-tls`). The probe
dials `localhost` for the TLS attempt — the internal CA issues DNS SANs
only, so a literal-IP URL would fail verification. Cert renewal rewrites
dials `localhost` for the TLS attempt, which every service certificate carries
as a DNS SAN. Cert renewal rewrites
the PEM dir alongside the live listener swap, so the probe's client cert
never outlives the served cert. With TLS disabled the plain probe succeeds
and the PEM directory is never consulted. On bare metal with multiple
nodes per host, set `TURNSTONE_TLS_PEM_DIR` per node (each boot clears
stale `lacme-pem-*` dirs under its root).
The production TLS Compose overlay inherits this healthcheck from the base
service; it remains enabled under mTLS.
---
@@ -125,6 +123,13 @@ stale `lacme-pem-*` dirs under its root).
| `tls.enabled` | `false` | Master switch for internal mTLS |
| `tls.acme_directory` | `""` | External ACME CA URL for console frontend cert |
### ACME topology environment
| Variable | Default | Description |
|----------|---------|-------------|
| `TURNSTONE_ACME_EXTERNAL_URL` | request-derived | Canonical externally reachable responder base, including `/acme` (for example `http://192.0.2.1:8090/acme`). Set it on the console so advertised URLs are routable and on in-cluster clients so their enrollment JWT is allowed only at that configured destination. A public path prefix is valid only when a reverse proxy maps it to Turnstone's internal `/acme` mount. |
| `TURNSTONE_CONSOLE_HTTP_BIND` | `127.0.0.1` | Production TLS-overlay bind for the console's plain-HTTP bootstrap/API port. For cross-host enrollment, use a trusted LAN/VPN interface and firewall it to enrolling nodes. |
### Bootstrap Config (config.toml)
These are needed before storage is available:
@@ -144,7 +149,7 @@ sslkey = "" # path to client key
| CA common name | "Turnstone CA" | |
| CA validity | 10 years | |
| Cert validity | 48 hours | Short-lived, auto-renewed |
| Renewal interval | 24 hours | Half of validity |
| Renewal interval | 12 hours | Leaves retry headroom before expiry |
| ACME auto-approve | true | Internal network, no challenge validation |
---
@@ -177,10 +182,14 @@ turnstone-admin tls-ca-cert --out ca.pem --console-url http://console:8080
# Request a cert for a domain
turnstone-admin tls-issue worker-1.internal --out /certs --console-url http://console:8080
# List issued certs
# List managed cluster certs
turnstone-admin tls-list --console-url http://console:8080
```
`tls-ca-cert` preserves the supplied scheme. An `https://` console URL is
verified with the system trust store; an explicitly supplied `http://` URL is
TOFU and prints a fingerprint that must be checked out of band.
### Console URL Discovery
If `--console-url` is not provided, the CLI discovers it from the `services`
@@ -193,7 +202,9 @@ table in the shared database. The console registers itself on startup.
The **TLS** tab in the console admin panel (System group) shows:
- CA status (common name, certificate count)
- Certificate table (domain, SANs, issued, expires)
- Force-renew and delete actions per certificate
- Force-renew for the console-owned internal identity; remote nodes renew and
hot-reload their own keys
- Delete for expired, remotely managed certificate rows
---
@@ -248,8 +259,15 @@ const client = new TurnstoneServer({
`TURNSTONE_CONSOLE_URL` (a bare-metal node outside the compose network can't
resolve the in-cluster `console` name, so it points this at the console's
published ACME endpoint)
3. Fetches CA root cert from `http://console/acme/ca.pem` (plain HTTP, TOFU)
4. Requests a service cert via ACME (plain HTTP, JWS-signed). The cert's
3. Fetches the CA root from the configured console scheme. Direct deployments
use `http://console/acme/ca.pem` (plain HTTP, TOFU); an explicitly configured
HTTPS proxy is preserved and verified with the system trust store.
4. Requests a service cert via ACME with a dedicated, short-lived Turnstone
service JWT pinned to configured responder origins. lacme emits ACME JWS
messages, but its lightweight responder deliberately does not validate their
signatures or nonces; the service JWT is the enrollment authorization gate.
Direct HTTP bootstrap therefore still requires a trusted LAN/VPN (or an
independently trusted HTTPS proxy). The cert's
primary domain / SAN is the node's **advertised host** (the host of
`TURNSTONE_ADVERTISE_URL`, e.g. `node-1`) — the name peers actually dial,
not the container hostname. This makes mTLS hostname verification succeed
@@ -264,7 +282,12 @@ const client = new TurnstoneServer({
1. Read `tls.enabled` from ConfigStore
2. Initialize CA (load from DB or generate new root key)
3. Mount ACME responder at `/acme` (serves `/ca.pem` natively)
3. Mount ACME responder at `/acme` (serves `/ca.pem` natively). When
`TURNSTONE_ACME_EXTERNAL_URL` is set, use it for every advertised directory,
order, authorization, and certificate URL; otherwise derive URLs from each
request as before. Directory, nonce, and CA bootstrap resources stay public;
account/order/challenge/finalization/certificate routes require the dedicated
enrollment service JWT.
4. Issue console certs (internal + optional frontend)
5. Start CA-direct auto-renewal (no network, signs directly), scoped to the
console's own cert, plus a periodic GC that reclaims cert rows for
@@ -290,6 +313,15 @@ fronted under a second hostname). Symptom if this is wrong: the console
dashboard shows nodes as unreachable and `openssl s_client` reports the served
cert's SANs don't include the dialed name.
The advertised host and extra SANs may be DNS names or literal IPv4/IPv6
addresses. Turnstone converts IP literals to typed ACME identifiers so the
certificate contains `IPAddress` SANs that normal IP hostname verification can
use; DNS spelling is preserved. Bracket an IPv6 address when it appears in a URL
(for example `TURNSTONE_ADVERTISE_URL=http://[2001:db8::10]:8080`), but use the
bare address in `TURNSTONE_TLS_SANS`. Unspecified bind addresses (`0.0.0.0` and
`::`) and scoped IPv6 addresses such as `fe80::1%eth0` are not certificate
identities. Restart after changing the advertised identity or extra SANs.
### "No console service found"
The console registers itself in the `services` table on startup. If the console
@@ -297,6 +329,22 @@ hasn't started or the registration expired (1 hour TTL), nodes can't discover
it. Set `TURNSTONE_CONSOLE_URL` to a reachable console address (this is also how
a bare-metal node that can't resolve the in-cluster `console` name enrolls).
### Cross-host ACME links point at the container
For a node on another host, publish port 8090 on a reachable interface and set
`TURNSTONE_ACME_EXTERNAL_URL` on the console **and in-cluster nodes** to that full
responder base, including `/acme` (for example
`http://192.0.2.1:8090/acme`). The console advertises it; clients use it as a
trusted enrollment-token destination. Keep
`TURNSTONE_CONSOLE_URL=http://console:8090` for in-cluster service discovery.
A remote node whose `TURNSTONE_CONSOLE_URL` already names the public origin can
derive the same `/acme` base, but setting both values explicitly avoids drift.
Bind only a trusted LAN/VPN interface and firewall it to enrolling nodes. The
JWT authenticates the client, but a direct plain-HTTP bootstrap remains TOFU and
does not resist an active on-path attacker. If the network is untrusted, expose
the responder through an independently trusted HTTPS proxy instead.
### Browser HTTPS to the console
The console serves plain HTTP (it's the ACME bootstrap endpoint — see
+25 -10
View File
@@ -152,7 +152,7 @@ into its successor's trajectory.
**Auto-approved** (no user confirmation needed at runtime):
- `read_file` -- reads files, no side effects
- `search` -- grep-style search, no side effects
- `memory` -- structured persistent memory (save/search/delete/list)
- `memory` -- structured persistent memory (save/get/search/delete/list)
- `recall` -- searches conversation history
- `notify` -- sends notifications to linked channels (time-sensitive, auto-approved for urgency)
@@ -433,7 +433,7 @@ Delegate a general-purpose task to an autonomous sub-agent.
|-----------|--------|----------|-------------|
| `prompt` | string | yes | Complete task description for the sub-agent. |
- **What it does**: Spawns a sub-agent that inherits the `TASK_AGENT_TOOLS` set (read, write, edit, search, bash, web tools, memory tools). The sub-agent runs autonomously to completion. Use for work that requires file modifications or command execution.
- **What it does**: Spawns a sub-agent that inherits the `TASK_AGENT_TOOLS` set (read, write, edit, search, bash, and web tools). The sub-agent runs autonomously to completion. Use for work that requires file modifications or command execution.
- **Auto-approve**: No -- requires user confirmation.
- **Agent availability**: Top-level only.
@@ -447,18 +447,26 @@ Structured persistent memory across sessions with typed, scoped entries.
| Parameter | Type | Required | Description |
|---------------|---------|----------|-------------|
| `action` | string | yes | `save`, `search`, `delete`, or `list`. |
| `name` | string | save/delete | Short snake_case identifier for the memory. |
| `action` | string | yes | `save`, `get`, `search`, `delete`, or `list`. |
| `name` | string | save/get/delete | Short snake_case identifier for the memory. |
| `content` | string | save | Memory content to store. |
| `description` | string | no | Short description for relevance matching (recommended for `save`). |
| `type` | string | no | Memory type: `user`, `project`, `feedback`, or `reference`. Default: `project`. |
| `scope` | string | no | Memory scope: `global`, `workstream`, or `user`. Default: `global`. |
| `description` | string | save | Non-empty description for relevance matching; required on create and update. |
| `type` | string | no | Memory type: `user`, `general`, `feedback`, or `reference`. Default: `general`. |
| `scope` | string | no | Memory scope: `global`, `workstream`, `user`, `coordinator`, or `project`. See defaults below. |
| `query` | string | search | Search query for finding memories. |
| `limit` | integer | no | Max results for `search` or `list`. Default: 20. |
- **What it does**: Manages structured persistent memories in the database. Memories persist across sessions, have a type classification (user preferences, project knowledge, feedback, reference material) and a scope (global across all workstreams, private to a workstream, or following a user). Relevant memories are included in the system prompt on startup.
- **What it does**: Manages structured persistent memories in the database.
Memories persist across sessions, have a type classification, and live in a
role-specific visible scope. Unscoped `save`/`get`/`delete` resolve to one
target: the attached active project, otherwise `global` for an interactive
session or `coordinator` for a coordinator. Read-only project access permits
`get` but makes `save`/`delete` fail without falling back. A valid explicit
scope selects exactly that scope. Unscoped `search`/`list` cover all visible
scopes; use the displayed scope when following a result with `get` or
`delete`.
- **Auto-approve**: Yes.
- **Agent availability**: Not available to sub-agents (top-level only).
- **Agent availability**: Not available to task agents.
---
@@ -473,7 +481,7 @@ Search conversation history for past messages and tool results.
- **What it does**: Searches conversation history across sessions using FTS5 full-text search. Returns matching messages, tool calls, and tool results with timestamps and workstream context.
- **Auto-approve**: Yes.
- **Agent availability**: Not available to sub-agents (top-level only).
- **Agent availability**: Not available to task agents.
---
@@ -871,6 +879,13 @@ capabilities for the `resources` capability. For servers that declare it:
2. `list_resource_templates` fetches URI templates (parameterized patterns like
`db://tables/{table}/rows/{id}`).
The protocol advertises both lists through one aggregate `resources`
capability, so a server may implement only one of them. If either request
returns the JSON-RPC `Method not found` code (`-32601`), turnstone treats that
half of the catalog as empty and keeps the other half; authentication,
validation, transport, and all other discovery errors still fail the
connection or refresh.
Both are stored as `{uri, name, description, mimeType, server}` dicts and
merged into a unified catalog.
+14 -6
View File
@@ -23,9 +23,14 @@ classifiers = [
"Topic :: Scientific/Engineering :: Artificial Intelligence",
]
dependencies = [
"openai>=2.45", # GPT-5.6: typed reasoning.mode, prompt_cache_options, and cache_write_tokens
# Version 3 moves the default transport to HTTPX2. Keep major upgrades
# deliberate because the stream retry boundary depends on that contract.
"openai>=3,<4",
"anthropic>=0.117", # tracks the release current at claude-opus-5 onboarding; hard runtime floor is still 0.105 (mid-conversation system blocks) — Opus 5 itself needs no new SDK surface (model ids are opaque strings; "refusal" has been in the StopReason literal since ~0.95). Raise this when adopting fast mode / server-side fallbacks / advisor / mid-conversation tool changes, which DO need newer typed params.
"httpx>=0.28",
# Direct because the provider boundary catches this exception family;
# OpenAI v3's transitive dependency alone is not an import contract.
"httpx2>=2.7,<3",
"mcp>=1.27,<2", # v2 is a breaking rewrite (2.0.0a1 live 2026-06-11; stable ~2026-07-27) — streamablehttp_client removed, 2-tuple transport, snake_case types; migrate deliberately
"starlette>=1.3.1", # CVE-2026-54282 (path->authority host spoof) + CVE-2026-54283 (url-encoded form DoS); supersedes the PYSEC-2026-161 host-header path-injection floor
"uvicorn>=0.34",
@@ -42,7 +47,9 @@ dependencies = [
"PyJWT>=2.8",
"bcrypt>=4.0",
"cryptography>=48.0.1", # GHSA-537c-gmf6-5ccf: PyPI wheels <48.0.1 bundle a vulnerable statically-linked OpenSSL (2026-06-09 secadv)
"lacme>=1.0.5",
# Core mTLS/ACME contract. 1.1 moves lacme onto HTTPX2; 1.2 adds typed IP
# identifiers and retains the external responder URL for cross-host enrollment.
"lacme>=1.2,<2",
"python-frontmatter>=1.0",
"pypdfium2>=4", # PDF text-extract + rasterize for models without native PDF input (core/pdf.py)
"pillow>=10", # PNG encoding for the PDF->images rasterize fallback (vision models, core/pdf.py)
@@ -106,6 +113,11 @@ markers = [
"e2e_recovery: opt-in end-to-end SSE recovery harness (real server + real SSE consumers, scripted provider — NOT live, no LLM backend needed); tens of seconds each. CI lanes run ``-m 'not live and not e2e_recovery'``; select with ``-m e2e_recovery``.",
]
filterwarnings = [
# The MCP v1 FastMCP integration fixture rebuilds its incomplete generic
# Settings model before construction. Keep the new pydantic-settings 2.15
# diagnostic fatal so removing that compatibility step cannot regress
# into a warning hidden in the full-suite summary.
"error:Field 'lifespan' has an incomplete definition",
# mcp v1 deprecates streamablehttp_client for an entry point whose call
# shape only settles in v2 — adoption rides the deliberate v2 migration
# (pin capped <2); silence exactly this message until then.
@@ -187,10 +199,6 @@ ignore_missing_imports = true
module = ["frontmatter", "frontmatter.*"]
ignore_missing_imports = true
[[tool.mypy.overrides]]
module = ["lacme", "lacme.*"]
ignore_missing_imports = true
[[tool.mypy.overrides]]
module = ["pypdfium2", "pypdfium2.*"]
ignore_missing_imports = true
+458
View File
@@ -0,0 +1,458 @@
#!/usr/bin/env python3
"""Browser regression for the two-layer frontend cache contract.
This harness serves a versioned entry module which imports the real,
unversioned ``shared/interactive.js`` module. It loads an old pane build in a
real Chrome profile, switches the server to a same-version replacement whose
pane has the same byte length and mtime, then performs a normal browser reload
without clearing or disabling the HTTP cache.
The old fixture advertises ``user_turn=0&tool_turn=0`` in its EventSource URL;
the current source advertises both capabilities as ``1``. A passing run
therefore proves both layers of the contract:
* the package-versioned entry URL revalidates and may return 304; and
* its unversioned transitive pane import revalidates by content and returns the
current bytes rather than surviving from the prior build.
The page also loads representative KaTeX, Highlight.js, and HLS.js assets.
Their installed version directories are discovered from ``shared_static`` at
runtime, so a normal vendor-version bump requires no harness edit; reload must
reuse them under the immutable policy.
Usage::
uv run python scripts/asset_cache_e2e.py
"""
from __future__ import annotations
import contextlib
import json
import os
import shutil
import socket
import sys
import tempfile
import threading
import time
from dataclasses import dataclass, field
from pathlib import Path
from typing import Any
ROOT = Path(__file__).resolve().parents[1]
if str(ROOT) not in sys.path:
sys.path.insert(0, str(ROOT))
from recovery_e2e import CDP, _find_chrome, _launch_chrome, _page_ws_url # noqa: E402
from turnstone import __version__ # noqa: E402
from turnstone.core.web_helpers import ( # noqa: E402
RevalidatingStaticFiles,
version_html,
)
_PAGE_HTML = """<!doctype html>
<html>
<head>
<meta charset="utf-8">
<title>ASSET-CACHE-PENDING</title>
<!-- VENDORED_ASSET_TAGS -->
<script>
window.__assetCacheUrls = [];
class RecordingEventSource {
static CONNECTING = 0;
static OPEN = 1;
static CLOSED = 2;
constructor(url) {
this.url = String(url);
this.readyState = RecordingEventSource.CONNECTING;
window.__assetCacheUrls.push(this.url);
}
close() {
this.readyState = RecordingEventSource.CLOSED;
}
}
window.EventSource = RecordingEventSource;
window.addEventListener("error", (event) => {
document.title = "ASSET-CACHE-FAILED-" + String(event.message || "script").slice(0, 80);
});
window.addEventListener("unhandledrejection", (event) => {
document.title = "ASSET-CACHE-FAILED-" + String(event.reason || "promise").slice(0, 80);
});
</script>
</head>
<body>
<main id="pane"></main>
<script type="module" src="/static/asset_cache_boot.js"
onerror="document.title='ASSET-CACHE-FAILED-module-load'"></script>
</body>
</html>
"""
_BOOT_JS = """import { InteractivePane } from "/shared/interactive.js";
const fixtureWorkstreamId = "00000000-0000-0000-0000-000000000001";
const pane = new InteractivePane(fixtureWorkstreamId, { base: "" });
document.getElementById("pane").appendChild(pane.el);
pane.connectSSE(fixtureWorkstreamId);
const url = window.__assetCacheUrls.at(-1) || "";
const generation = url.includes("user_turn=1") && url.includes("tool_turn=1")
? "CURRENT"
: url.includes("user_turn=0") && url.includes("tool_turn=0")
? "OLD"
: "FAILED-CAPABILITIES";
window.__assetCacheResult = { generation, url };
document.title = "ASSET-CACHE-" + generation;
"""
@dataclass
class CacheState:
phase: str = "old"
requests: list[dict[str, Any]] = field(default_factory=list)
lock: threading.Lock = field(default_factory=threading.Lock)
def record(self, item: dict[str, Any]) -> None:
with self.lock:
self.requests.append(item)
def matching(self, phase: str, path: str) -> list[dict[str, Any]]:
with self.lock:
return [
item for item in self.requests if item["phase"] == phase and item["path"] == path
]
class SwitchingStaticFiles:
"""Select one immutable build snapshot at request dispatch time."""
def __init__(self, state: CacheState, old_dir: Path, current_dir: Path) -> None:
self._state = state
self._apps = {
"old": RevalidatingStaticFiles(directory=str(old_dir)),
"current": RevalidatingStaticFiles(directory=str(current_dir)),
}
async def __call__(self, scope: Any, receive: Any, send: Any) -> None:
await self._apps[self._state.phase](scope, receive, send)
class RecordingApp:
"""Record static HTTP validators and status without perturbing streaming."""
def __init__(self, app: Any, state: CacheState) -> None:
self._app = app
self._state = state
async def __call__(self, scope: Any, receive: Any, send: Any) -> None:
path = str(scope.get("path", ""))
if scope.get("type") != "http" or not path.startswith(("/static/", "/shared/")):
await self._app(scope, receive, send)
return
phase = self._state.phase
request_headers = {
key.decode("latin-1").lower(): value.decode("latin-1")
for key, value in scope.get("headers", [])
}
async def record_send(message: dict[str, Any]) -> None:
if message["type"] == "http.response.start":
response_headers = {
key.decode("latin-1").lower(): value.decode("latin-1")
for key, value in message.get("headers", [])
}
self._state.record(
{
"phase": phase,
"path": path,
"query": scope.get("query_string", b"").decode("latin-1"),
"if_none_match": request_headers.get("if-none-match"),
"status": message["status"],
"etag": response_headers.get("etag"),
"cache_control": response_headers.get("cache-control"),
}
)
await send(message)
await self._app(scope, receive, record_send)
def _old_interactive(current: bytes) -> bytes:
old = current
for needle, replacement in (
(b'"user_turn=1"', b'"user_turn=0"'),
(b'"&tool_turn=1"', b'"&tool_turn=0"'),
):
if old.count(needle) != 1:
raise RuntimeError(f"expected exactly one {needle.decode()} capability literal")
old = old.replace(needle, replacement, 1)
if len(old) != len(current):
raise AssertionError("old and current pane fixtures must have equal byte length")
return old
def _discover_vendor_assets(source_shared: Path) -> tuple[str, ...]:
selected = (
("katex", "katex.min.css"),
("katex", "katex.min.js"),
("hljs", "highlight.min.js"),
("hls", "hls.min.js"),
)
paths = []
for library, filename in selected:
matches = sorted(source_shared.glob(f"{library}-*/{filename}"))
if not matches:
raise RuntimeError(f"no vendored {library} asset named {filename} was found")
paths.extend(f"/shared/{match.relative_to(source_shared).as_posix()}" for match in matches)
return tuple(paths)
def _vendor_tags(vendor_paths: tuple[str, ...]) -> str:
tags = []
for path in vendor_paths:
if path.endswith(".css"):
tags.append(f'<link rel="stylesheet" href="{path}">')
else:
tags.append(f'<script src="{path}"></script>')
return "\n ".join(tags)
def _prepare_builds(scratch: Path) -> tuple[Path, Path, Path, tuple[str, ...]]:
import turnstone
package_dir = Path(turnstone.__file__).resolve().parent
source_shared = package_dir / "shared_static"
vendor_paths = _discover_vendor_assets(source_shared)
old_shared = scratch / "old" / "shared"
current_shared = scratch / "current" / "shared"
static_dir = scratch / "static"
shutil.copytree(source_shared, old_shared)
shutil.copytree(source_shared, current_shared)
static_dir.mkdir()
(static_dir / "asset_cache_boot.js").write_text(_BOOT_JS, encoding="utf-8")
current_asset = current_shared / "interactive.js"
old_asset = old_shared / "interactive.js"
current = current_asset.read_bytes()
old_asset.write_bytes(_old_interactive(current))
# Reproduce the metadata collision which defeated Starlette's default ETag.
fixed_mtime_ns = 1_700_000_000_123_456_789
for asset in (old_asset, current_asset):
os.utime(asset, ns=(fixed_mtime_ns, fixed_mtime_ns))
old_stat = old_asset.stat()
current_stat = current_asset.stat()
if (old_stat.st_size, old_stat.st_mtime_ns) != (
current_stat.st_size,
current_stat.st_mtime_ns,
):
raise AssertionError("pane fixture size/mtime collision was not preserved")
return old_shared, current_shared, static_dir, vendor_paths
def _make_app(
state: CacheState,
old_dir: Path,
current_dir: Path,
static_dir: Path,
vendor_paths: tuple[str, ...],
) -> Any:
from starlette.applications import Starlette
from starlette.responses import HTMLResponse
from starlette.routing import Mount, Route
async def page(_request: Any) -> HTMLResponse:
page_html = _PAGE_HTML.replace("<!-- VENDORED_ASSET_TAGS -->", _vendor_tags(vendor_paths))
return HTMLResponse(
version_html(page_html),
headers={"Cache-Control": "no-store"},
)
app = Starlette(
routes=[
Route("/asset-cache-e2e", page),
Mount(
"/static",
app=RevalidatingStaticFiles(directory=str(static_dir)),
),
Mount(
"/shared",
app=SwitchingStaticFiles(state, old_dir, current_dir),
),
]
)
return RecordingApp(app, state)
def _start_server(app: Any) -> tuple[Any, threading.Thread, socket.socket, str]:
import uvicorn
sock = socket.socket()
sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
sock.bind(("127.0.0.1", 0))
sock.listen(128)
port = int(sock.getsockname()[1])
server = uvicorn.Server(
uvicorn.Config(app, host="127.0.0.1", port=port, log_level="warning", lifespan="off")
)
thread = threading.Thread(
target=server.run,
kwargs={"sockets": [sock]},
name="asset-cache-e2e-server",
daemon=True,
)
thread.start()
deadline = time.monotonic() + 10
while not server.started and thread.is_alive() and time.monotonic() < deadline:
time.sleep(0.05)
if not server.started:
server.should_exit = True
thread.join(timeout=2)
sock.close()
raise RuntimeError("asset cache test server did not start")
return server, thread, sock, f"http://127.0.0.1:{port}"
def _wait_for_generation(cdp: CDP, expected: str, timeout: float = 20) -> dict[str, str]:
deadline = time.monotonic() + timeout
last_title = ""
while time.monotonic() < deadline:
last_title = cdp.title()
result = cdp.evaluate("window.__assetCacheResult || null")
if isinstance(result, dict) and result.get("generation") == expected:
return {"generation": str(result["generation"]), "url": str(result["url"])}
if last_title.startswith("ASSET-CACHE-FAILED"):
raise RuntimeError(last_title)
time.sleep(0.1)
raise TimeoutError(f"expected {expected}, last title was {last_title!r}")
def _one(state: CacheState, phase: str, path: str) -> dict[str, Any]:
requests = state.matching(phase, path)
if len(requests) != 1:
raise AssertionError(f"expected one {phase} request for {path}, got {requests!r}")
return requests[0]
def _verify_trace(state: CacheState, vendor_paths: tuple[str, ...]) -> tuple[str, list[str]]:
entry_path = "/static/asset_cache_boot.js"
pane_path = "/shared/interactive.js"
old_entry = _one(state, "old", entry_path)
current_entry = _one(state, "current", entry_path)
old_pane = _one(state, "old", pane_path)
current_pane = _one(state, "current", pane_path)
expected_query = f"v={__version__}"
if old_entry["query"] != expected_query or current_entry["query"] != expected_query:
raise AssertionError("entry URL did not retain the same package version across builds")
if old_entry["status"] != 200 or old_pane["status"] != 200:
raise AssertionError("old build did not populate the browser cache")
if current_entry["status"] != 304 or not current_entry["if_none_match"]:
raise AssertionError(f"versioned entry did not revalidate to 304: {current_entry!r}")
if current_pane["status"] != 200 or not current_pane["if_none_match"]:
raise AssertionError(
f"transitive pane did not revalidate to current bytes: {current_pane!r}"
)
if old_pane["etag"] == current_pane["etag"]:
raise AssertionError("content-derived pane validators did not change")
if current_pane["cache_control"] != "no-cache":
raise AssertionError("transitive pane lost its revalidation policy")
vendor_trace = []
immutable = "public, max-age=31536000, immutable"
for path in vendor_paths:
old_vendor = _one(state, "old", path)
if old_vendor["query"] or old_vendor["status"] != 200:
raise AssertionError(f"versioned vendor URL was rewritten or failed: {old_vendor!r}")
if old_vendor["cache_control"] != immutable:
raise AssertionError(f"versioned vendor asset was not immutable: {old_vendor!r}")
revisits = state.matching("current", path)
if revisits:
if len(revisits) != 1 or revisits[0]["status"] not in (200, 304):
raise AssertionError(f"unexpected vendor reload trace: {revisits!r}")
if revisits[0]["cache_control"] != immutable:
raise AssertionError(f"vendor reload lost immutable policy: {revisits[0]!r}")
vendor_trace.append(f"{path}: revisited-{revisits[0]['status']}")
else:
vendor_trace.append(f"{path}: cache-hit")
verdict = f"ASSET-CACHE-READY-entry304-pane200-user1-tool1-vendor{len(vendor_paths)}"
return verdict, vendor_trace
def _stop_process(proc: Any) -> None:
if proc.poll() is not None:
return
proc.terminate()
with contextlib.suppress(Exception):
proc.wait(timeout=5)
if proc.poll() is None:
proc.kill()
with contextlib.suppress(Exception):
proc.wait(timeout=2)
def main() -> int:
chrome = _find_chrome()
if not chrome:
print("ASSET-CACHE-FAILED-no-chrome")
return 2
with tempfile.TemporaryDirectory(prefix="turnstone-asset-cache-e2e-") as raw_scratch:
scratch = Path(raw_scratch)
old_dir, current_dir, static_dir, vendor_paths = _prepare_builds(scratch)
state = CacheState()
server, server_thread, sock, base_url = _start_server(
_make_app(state, old_dir, current_dir, static_dir, vendor_paths)
)
chrome_proc = None
cdp = None
try:
chrome_proc, cdp_port = _launch_chrome(chrome, scratch / "chrome-profile")
cdp = CDP(_page_ws_url(cdp_port))
cdp.cmd("Page.enable")
cdp.cmd("Runtime.enable")
cdp.cmd("Network.enable")
cdp.cmd("Page.navigate", {"url": f"{base_url}/asset-cache-e2e"})
old_result = _wait_for_generation(cdp, "OLD")
state.phase = "current"
cdp.cmd("Page.reload", {"ignoreCache": False})
current_result = _wait_for_generation(cdp, "CURRENT")
if "user_turn=0" not in old_result["url"] or "tool_turn=0" not in old_result["url"]:
raise AssertionError(f"old pane did not expose old capabilities: {old_result!r}")
if (
"user_turn=1" not in current_result["url"]
or "tool_turn=1" not in current_result["url"]
):
raise AssertionError(
f"reloaded pane did not expose current capabilities: {current_result!r}"
)
verdict, vendor_trace = _verify_trace(state, vendor_paths)
cdp.evaluate(f"document.title = {json.dumps(verdict)}")
print(verdict)
print(f" old EventSource: {old_result['url']}")
print(f" current EventSource: {current_result['url']}")
for item in vendor_trace:
print(f" vendor: {item}")
return 0
except Exception as exc:
print(f"ASSET-CACHE-FAILED-{type(exc).__name__}: {exc}")
return 1
finally:
if cdp is not None:
cdp.close()
if chrome_proc is not None:
_stop_process(chrome_proc)
server.should_exit = True
server_thread.join(timeout=10)
with contextlib.suppress(OSError):
sock.close()
if __name__ == "__main__":
raise SystemExit(main())
+189 -2
View File
@@ -74,6 +74,21 @@ Attachments harness (/attachments/livepass.html): the composer attachment
thumbnail crop/size, the native audio-control fit at the constrained
height, the snippet contrast, and how a long filename behaves at the
340px chip cap.
Paste-over-HTTP harness (/paste/livepass.html): the REAL Composer's
large-text paste path, exercised with a browser-generated, trusted paste
event on an explicitly non-secure HTTP origin. No
``navigator.clipboard`` stub, synthetic ``ClipboardEvent``, or
secure-context override is involved. The page requires
``window.isSecureContext === false``, ``event.isTrusted``, a ``text/plain``
clipboard item, canceled inline insertion, and an exact
``pasted-text.txt`` File round-trip before stamping ``PASTE-HTTP-READY``.
It fails closed as ``PASTE-HTTP-FAILED-<reason>``. Serve beyond loopback,
open the page by a LAN address (localhost and 127.0.0.1 are treated as
trustworthy origins by browsers), then use the browser's normal Copy and
Paste commands:
python3 scripts/livepass.py --serve 8950 --bind 0.0.0.0
Task-agent harness (/taskagent/livepass.html): the task_agent card a task
agent's sub-tool steps nested under its conversation row, driven through the
REAL InteractivePane.handleEvent (parent tool_pending/tool_info -> child
@@ -1031,6 +1046,165 @@ ATTACH_TEMPLATE = """<!doctype html>
"""
# --------------------------------------------------------------------------
# Native paste-over-HTTP harness. Unlike the copy-affordance harness, this
# must never stub clipboard access or dispatch a script-created paste event:
# its purpose is to prove that the production ClipboardEvent path still sees
# user-agent clipboard data on a non-secure origin.
# --------------------------------------------------------------------------
PASTE_TEMPLATE = """<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<title>PASTE-HTTP-BOOTING</title>
<link rel="stylesheet" href="shared/base.css" />
<link rel="stylesheet" href="shared/ui-base.css" />
<link rel="stylesheet" href="shared/chat.css" />
<style>
body {
margin: 0; padding: 32px;
background: var(--bg); color: var(--fg);
font-family: var(--font-sans, system-ui, sans-serif);
}
main { width: min(760px, 100%); margin: 0 auto; }
#paste-source {
box-sizing: border-box; width: 100%; height: 80px;
}
#composer-mount, #probe-status { margin-top: 16px; }
#probe-status { white-space: pre-wrap; }
</style>
</head>
<body>
<main>
<h1>Native paste on plain HTTP</h1>
<p>
This passes only when a trusted paste exposes clipboard text to the
real Composer on a non-secure HTTP origin.
</p>
<p>1. Select this 2001-character fixture, then copy it normally.</p>
<textarea id="paste-source" aria-label="Paste test source"></textarea>
<button id="select-source" type="button">Select source text</button>
<p>2. Focus the composer and paste normally.</p>
<div id="composer-mount"></div>
<pre id="probe-status" role="status" aria-live="polite">Booting</pre>
</main>
<script type="module">
import { Composer } from "./shared/composer.js";
import { PASTE_ATTACHMENT_CHARS } from "./shared/composer_paste_text.js";
const prefix = "TURNSTONE-PASTE-HTTP:";
const seedText =
prefix + "x".repeat(PASTE_ATTACHMENT_CHARS + 1 - prefix.length);
const source = document.getElementById("paste-source");
const status = document.getElementById("probe-status");
source.value = seedText;
let copyTrusted = false;
let paste = null;
let attachment = null;
let fileSettled = false;
let failed = false;
function paint() {
status.textContent = document.title + "\\n" + JSON.stringify({
origin: location.origin,
secureContext: window.isSecureContext,
copyTrusted: copyTrusted,
paste: paste,
attachment: attachment,
}, null, 2);
}
function fail(reason) {
if (failed) return;
failed = true;
document.title = "PASTE-HTTP-FAILED-" + reason;
paint();
}
function finish() {
if (failed || !paste || !fileSettled) return;
const checks = [
["copy-untrusted", copyTrusted],
["paste-untrusted", paste.trusted],
["no-clipboard-data", paste.hasClipboardData],
["no-text-plain", paste.hasPlainText],
["paste-not-canceled", paste.defaultPrevented],
["attach-count", attachment.calls === 1],
["filename", attachment.name === "pasted-text.txt"],
["mime", attachment.type === "text/plain"],
["size", attachment.size === new Blob([seedText]).size],
["content", attachment.contentMatches],
["inline-insert", paste.inputValue === ""],
];
const firstFailure = checks.find(function (entry) { return !entry[1]; });
if (firstFailure) return fail(firstFailure[0]);
document.title = "PASTE-HTTP-READY";
paint();
}
document.addEventListener("copy", function (event) {
copyTrusted = event.isTrusted;
paint();
}, true);
document.addEventListener("paste", function (event) {
const observed = {
trusted: event.isTrusted,
hasClipboardData: !!event.clipboardData,
hasPlainText: !!event.clipboardData &&
Array.from(event.clipboardData.types || []).includes("text/plain"),
};
setTimeout(function () {
paste = Object.assign(observed, {
defaultPrevented: event.defaultPrevented,
inputValue: composer.inputEl.value,
});
if (!attachment) fail("no-attachment");
else finish();
}, 0);
});
const composer = new Composer(document.getElementById("composer-mount"), {
onSend: function () {},
placeholder: "Paste the copied fixture here…",
attachments: {
onAttach: function (file) {
attachment = {
calls: attachment ? attachment.calls + 1 : 1,
name: file.name,
type: file.type,
size: file.size,
contentMatches: false,
};
file.text().then(function (text) {
attachment.contentMatches = text === seedText;
fileSettled = true;
finish();
}, function () { fail("file-read"); });
return true;
},
},
});
document.getElementById("select-source").addEventListener("click", function () {
source.focus();
source.select();
});
if (location.protocol !== "http:") fail("not-http");
else if (window.isSecureContext) fail("secure-context");
else {
document.title = "PASTE-HTTP-WAITING";
paint();
}
</script>
</body>
</html>
"""
# --------------------------------------------------------------------------
# Task-agent harness — the task_agent card: a task agent's sub-tool steps
# nested under its conversation row. Driven through the REAL
@@ -1958,6 +2132,12 @@ def build(out: Path) -> None:
(att / "livepass.html").write_text(ATTACH_TEMPLATE, encoding="utf-8")
print(f"{att}/livepass.html — composer chips + message attachment pills")
paste = out / "paste"
paste.mkdir(parents=True, exist_ok=True)
symlink(paste / "shared", ROOT / "turnstone/shared_static")
(paste / "livepass.html").write_text(PASTE_TEMPLATE, encoding="utf-8")
print(f"{paste}/livepass.html — trusted native paste on insecure HTTP")
ta = out / "taskagent"
ta.mkdir(parents=True, exist_ok=True)
symlink(ta / "shared", ROOT / "turnstone/shared_static")
@@ -2218,6 +2398,12 @@ def main() -> None:
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
ap.add_argument("--out", type=Path, default=Path("/tmp/livepass"))
ap.add_argument("--serve", type=int, metavar="PORT")
ap.add_argument(
"--bind",
default="127.0.0.1",
metavar="HOST",
help="listen address for --serve (use 0.0.0.0 for a manual insecure-origin pass)",
)
ap.add_argument("--perf", action="store_true", help="run the perf baseline and exit")
ap.add_argument(
"--perf-n",
@@ -2235,8 +2421,9 @@ def main() -> None:
import functools
handler = functools.partial(_HarnessHandler, directory=str(args.out))
print(f"serving {args.out} on http://localhost:{args.serve}/ — Ctrl+C stops")
http.server.ThreadingHTTPServer(("127.0.0.1", args.serve), handler).serve_forever()
display_host = "localhost" if args.bind == "127.0.0.1" else args.bind
print(f"serving {args.out} on http://{display_host}:{args.serve}/ — Ctrl+C stops")
http.server.ThreadingHTTPServer((args.bind, args.serve), handler).serve_forever()
if __name__ == "__main__":
+2 -2
View File
@@ -185,8 +185,8 @@ esac
echo ""
echo "NOTE: If you added a NEW library (not just updating a version), also update"
echo " the _ASSET_RE regex in turnstone/core/web_helpers.py — its negative lookahead"
echo " skips vendored directories to avoid double-versioning static asset URLs."
echo " _VERSIONED_VENDOR_DIR in turnstone/core/web_helpers.py — it controls both"
echo " HTML version rewriting and immutable static-response caching."
echo ""
echo "Verify the update:"
echo " git diff --stat"
+2 -2
View File
@@ -2,7 +2,7 @@
"openapi": "3.1.0",
"info": {
"title": "turnstone Console API",
"version": "1.8.0a6",
"version": "1.8.0a7",
"description": "Cluster-wide visibility and control across all turnstone nodes."
},
"paths": {
@@ -4700,7 +4700,7 @@
},
"/v1/api/admin/tls/certs": {
"get": {
"summary": "List all issued TLS certificates",
"summary": "List all managed TLS certificates",
"operationId": "v1_api_admin_tls_certs_get",
"tags": [
"Admin"
+168 -16
View File
@@ -2,7 +2,7 @@
"openapi": "3.1.0",
"info": {
"title": "turnstone Server API",
"version": "1.8.0a6",
"version": "1.8.0a7",
"description": "Single-node workstream management, chat interaction, and real-time streaming."
},
"paths": {
@@ -1869,7 +1869,7 @@
},
"/v1/api/memories": {
"get": {
"summary": "List structured memories",
"summary": "List structured memories. Without a scope, returns global plus the authenticated user's memories; workstream scope is owner-bound.",
"operationId": "v1_api_memories_get",
"tags": [
"Memories"
@@ -1891,7 +1891,7 @@
"schema": {
"type": "string"
},
"description": "Filter by scope"
"description": "Filter by public scope: global, workstream, or user"
},
{
"name": "scope_id",
@@ -1923,6 +1923,46 @@
}
}
}
},
"400": {
"description": "Error 400",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"403": {
"description": "Error 403",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"404": {
"description": "Error 404",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"500": {
"description": "Error 500",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
}
}
},
@@ -1962,13 +2002,43 @@
}
}
}
},
"403": {
"description": "Error 403",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"404": {
"description": "Error 404",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"500": {
"description": "Error 500",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
}
}
}
},
"/v1/api/memories/search": {
"post": {
"summary": "Search structured memories by query",
"summary": "Search structured memories by query. Without a scope, searches global plus the authenticated user's memories.",
"operationId": "v1_api_memories_search_post",
"tags": [
"Memories"
@@ -1993,6 +2063,46 @@
}
}
}
},
"400": {
"description": "Error 400",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"403": {
"description": "Error 403",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"404": {
"description": "Error 404",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"500": {
"description": "Error 500",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
}
}
}
@@ -2043,6 +2153,26 @@
}
}
},
"400": {
"description": "Error 400",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"403": {
"description": "Error 403",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"404": {
"description": "Error 404",
"content": {
@@ -2052,6 +2182,16 @@
}
}
}
},
"500": {
"description": "Error 500",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
}
}
}
@@ -3854,32 +3994,42 @@
"properties": {
"name": {
"description": "Memory identifier (normalized to snake_case)",
"maxLength": 256,
"minLength": 1,
"title": "Name",
"type": "string"
},
"content": {
"description": "Memory content",
"maxLength": 65536,
"minLength": 1,
"title": "Content",
"type": "string"
},
"description": {
"default": "",
"description": "Short description for relevance matching",
"description": "Required non-empty description used for relevance matching",
"minLength": 1,
"title": "Description",
"type": "string"
},
"type": {
"default": "general",
"description": "Memory type",
"enum": [
"user",
"general",
"feedback",
"reference"
"anyOf": [
{
"enum": [
"user",
"general",
"feedback",
"reference"
],
"type": "string"
},
{
"type": "null"
}
],
"title": "Type",
"type": "string"
"default": null,
"description": "Memory type; omission preserves it on update and defaults on insert",
"title": "Type"
},
"scope": {
"default": "global",
@@ -3901,7 +4051,8 @@
},
"required": [
"name",
"content"
"content",
"description"
],
"title": "SaveMemoryRequest",
"type": "object"
@@ -3995,6 +4146,7 @@
"properties": {
"query": {
"description": "Search query text",
"minLength": 1,
"title": "Query",
"type": "string"
},
+8 -1
View File
@@ -394,7 +394,14 @@ export class TurnstoneServer extends BaseClient {
}
async saveMemory(opts: SaveMemoryRequest): Promise<MemoryInfo> {
return this.request("POST", "/v1/api/memories", { json: opts });
if (typeof opts.description !== "string" || !opts.description.trim()) {
throw new TypeError(
"memory description is required and must be non-empty",
);
}
return this.request("POST", "/v1/api/memories", {
json: { ...opts, description: opts.description.trim() },
});
}
async searchMemories(
+1 -1
View File
@@ -894,7 +894,7 @@ export interface WorkstreamsOptions {
export interface SaveMemoryRequest {
name: string;
content: string;
description?: string;
description: string;
type?: "user" | "general" | "feedback" | "reference";
scope?: "global" | "workstream" | "user";
scope_id?: string;
+37
View File
@@ -75,6 +75,43 @@ describe("TurnstoneServer", () => {
});
});
it("saveMemory requires and normalizes the description", async () => {
const fetchFn = mockFetch({
memory_id: "m1",
name: "deployment_process",
description: "Production deployment workflow",
type: "general",
scope: "global",
scope_id: "",
content: "Deploy from main",
created: "2026-08-11T00:00:00",
updated: "2026-08-11T00:00:00",
});
const client = new TurnstoneServer({
baseUrl: "http://test",
fetch: fetchFn,
});
await client.saveMemory({
name: "deployment_process",
content: "Deploy from main",
description: " Production deployment workflow ",
});
const [, init] = (fetchFn as ReturnType<typeof vi.fn>).mock.calls[0];
expect(JSON.parse(init.body)).toMatchObject({
description: "Production deployment workflow",
});
await expect(
client.saveMemory({
name: "deployment_process",
content: "Deploy from main",
description: " ",
}),
).rejects.toThrow("description is required");
expect(fetchFn).toHaveBeenCalledTimes(1);
});
it("send posts correct payload", async () => {
const fetchFn = mockFetch({ status: "ok" });
const client = new TurnstoneServer({
+3 -5
View File
@@ -1,10 +1,8 @@
"""Shared mock factory for ``events_replay`` tests.
Both interactive (:func:`turnstone.server._interactive_events_replay`)
and coord (:func:`turnstone.console.server._coord_events_replay`) drive
the same shared preamble at
:func:`turnstone.core.session_replay.session_replay_preamble`. Their
test suites share the underlying mock surface (session.model,
Interactive and coordinator replay tests exercise the same shared preamble at
:func:`turnstone.core.session_replay.session_replay_preamble` plus their
kind-specific tails. Their test suites share the underlying mock surface (session.model,
session.model_alias, session._last_usage, ui._pending_*, ui._ws_lock,
counters); this module is the single home for that shape so a future
field add lands once.
+14
View File
@@ -15,6 +15,20 @@ from unittest.mock import MagicMock
import pytest
def pytest_sessionstart(session: pytest.Session) -> None:
"""Resolve MCP v1's generic FastMCP settings model for test servers.
MCP 1.29.0 defines ``Settings`` before ``FastMCP``, so its ``lifespan``
annotation remains an unresolved forward reference after import. Rebuild
once, after the module is fully loaded, before any integration fixture
constructs a FastMCP server. Pydantic's public hook is a no-op once the SDK
ships a complete model.
"""
from mcp.server.fastmcp.server import Settings as FastMCPSettings
FastMCPSettings.model_rebuild()
def stop_loop_thread(loop: asyncio.AbstractEventLoop, thread: threading.Thread) -> None:
"""Fully tear down a ``loop.run_forever``-in-a-thread test loop.
+4
View File
@@ -91,6 +91,8 @@ class TestServerVersioning:
def test_shared_static_unversioned(self, client):
resp = client.get("/shared/base.css")
assert resp.status_code == 200
assert resp.headers["cache-control"] == "no-cache"
assert resp.headers["etag"]
class TestConsoleVersioning:
@@ -147,3 +149,5 @@ class TestConsoleVersioning:
resp = client.get("/static/app.js")
body = resp.text
assert "/v1/api/cluster" in body
assert resp.headers["cache-control"] == "no-cache"
assert resp.headers["etag"]
+178
View File
@@ -11,6 +11,8 @@ from turnstone.core.auth import (
AUTH_COOKIE,
AUTH_COOKIE_CONSOLE,
AUTH_COOKIE_SERVER,
JWT_AUD_CONSOLE,
TLS_ACME_TOKEN_SOURCE,
WRITE_PATHS,
_extract_bearer,
_extract_cookie,
@@ -53,6 +55,17 @@ class TestIsPublicPath:
def test_shared_js_public(self):
assert is_public_path("/shared/utils.js") is True
@pytest.mark.parametrize("path", ["/acme/directory", "/acme/new-nonce", "/acme/ca.pem"])
def test_acme_bootstrap_resources_public(self, path):
assert is_public_path(path) is True
@pytest.mark.parametrize(
"path",
["/acme/new-account", "/acme/new-order", "/acme/authz/order-1", "/acme/cert/1"],
)
def test_acme_signing_resources_not_public(self, path):
assert is_public_path(path) is False
def test_api_workstreams_not_public(self):
assert is_public_path("/api/workstreams") is False
@@ -129,6 +142,12 @@ class TestRequiredScope:
def test_get_api_needs_read(self):
assert required_scope("GET", "/api/workstreams") == "read"
@pytest.mark.parametrize(
"path", ["/acme/new-account", "/acme/new-order", "/acme/finalize/order-1"]
)
def test_acme_signing_requires_service(self, path):
assert required_scope("POST", path) == "service"
def test_get_events_needs_read(self):
assert required_scope("GET", "/api/events") == "read"
@@ -479,6 +498,146 @@ class TestCheckRequest:
)
assert allowed is True
def test_acme_signing_no_token_401(self):
allowed, status, _msg, _result = check_request(
"POST", "/acme/new-order", None, cookie_name=AUTH_COOKIE_CONSOLE
)
assert allowed is False
assert status == 401
def test_acme_rejects_generic_service_token(self):
token = create_jwt(
"node-1",
frozenset({"service"}),
"console",
self._SECRET,
audience=JWT_AUD_CONSOLE,
)
allowed, status, msg, _result = check_request(
"POST",
"/acme/new-order",
f"Bearer {token}",
jwt_secret=self._SECRET,
jwt_audience=JWT_AUD_CONSOLE,
cookie_name=AUTH_COOKIE_CONSOLE,
)
assert allowed is False
assert status == 403
assert "ACME enrollment" in msg
def test_acme_rejects_enrollment_token_without_service_scope(self):
token = create_jwt(
"node-1",
frozenset({"read"}),
TLS_ACME_TOKEN_SOURCE,
self._SECRET,
audience=JWT_AUD_CONSOLE,
)
allowed, status, _msg, _result = check_request(
"POST",
"/acme/new-order",
f"Bearer {token}",
jwt_secret=self._SECRET,
jwt_audience=JWT_AUD_CONSOLE,
cookie_name=AUTH_COOKIE_CONSOLE,
)
assert allowed is False
assert status == 403
def test_acme_accepts_purpose_specific_service_token(self):
token = create_jwt(
"node-1",
frozenset({"service"}),
TLS_ACME_TOKEN_SOURCE,
self._SECRET,
audience=JWT_AUD_CONSOLE,
)
allowed, status, _msg, result = check_request(
"POST",
"/acme/new-order",
f"Bearer {token}",
jwt_secret=self._SECRET,
jwt_audience=JWT_AUD_CONSOLE,
cookie_name=AUTH_COOKIE_CONSOLE,
)
assert allowed is True
assert status == 200
assert result is not None and result.token_source == TLS_ACME_TOKEN_SOURCE
@pytest.mark.parametrize(
("method", "path"),
[
("GET", "/v1/api/workstreams"),
("POST", "/v1/api/workstreams/new"),
("GET", "/v1/api/admin/users"),
("POST", "/v1/api/admin/tls/certs/node-1/renew"),
("GET", "/v1/node/node-1/api/workstreams"),
],
)
def test_enrollment_token_rejected_outside_acme(self, method, path):
token = create_jwt(
"node-1",
frozenset({"service"}),
TLS_ACME_TOKEN_SOURCE,
self._SECRET,
audience=JWT_AUD_CONSOLE,
)
allowed, status, msg, result = check_request(
method,
path,
f"Bearer {token}",
jwt_secret=self._SECRET,
jwt_audience=JWT_AUD_CONSOLE,
cookie_name=AUTH_COOKIE_CONSOLE,
)
assert allowed is False
assert status == 403
assert "enrollment token" in msg
assert result is None
def test_versioned_acme_accepts_purpose_specific_service_token(self):
token = create_jwt(
"node-1",
frozenset({"service"}),
TLS_ACME_TOKEN_SOURCE,
self._SECRET,
audience=JWT_AUD_CONSOLE,
)
allowed, status, _msg, result = check_request(
"POST",
"/v1/acme/new-order",
f"Bearer {token}",
jwt_secret=self._SECRET,
jwt_audience=JWT_AUD_CONSOLE,
cookie_name=AUTH_COOKIE_CONSOLE,
)
assert allowed is True
assert status == 200
assert result is not None and result.token_source == TLS_ACME_TOKEN_SOURCE
def test_acme_rejects_wrong_audience(self):
token = create_jwt(
"node-1",
frozenset({"service"}),
TLS_ACME_TOKEN_SOURCE,
self._SECRET,
audience="another-service",
)
allowed, status, _msg, _result = check_request(
"POST",
"/acme/new-order",
f"Bearer {token}",
jwt_secret=self._SECRET,
jwt_audience=JWT_AUD_CONSOLE,
cookie_name=AUTH_COOKIE_CONSOLE,
)
assert allowed is False
assert status == 401
def test_api_no_token_401(self):
allowed, status, msg, _result = check_request(
"GET", "/api/workstreams", None, cookie_name=AUTH_COOKIE_SERVER
@@ -1406,6 +1565,25 @@ class TestConsoleLogin:
)
assert resp.status_code == 401
def test_login_enrollment_token_rejected(self):
token = create_jwt(
"node-1",
frozenset({"service"}),
TLS_ACME_TOKEN_SOURCE,
self._jwt_secret,
audience=JWT_AUD_CONSOLE,
)
resp = self.test_client.post(
"/v1/api/auth/login",
json={"token": token},
)
assert resp.status_code == 401
assert resp.json() == {"error": "Invalid credentials"}
assert "jwt" not in resp.json()
assert AUTH_COOKIE_CONSOLE not in resp.headers.get("set-cookie", "")
def test_login_password_ok(self):
resp = self.test_client.post(
"/v1/api/auth/login",
+3
View File
@@ -225,6 +225,9 @@ class _ObservedRLock:
def release(self) -> None:
self._lock.release()
def locked(self) -> bool:
return self._lock.locked()
def __enter__(self):
self.acquire()
return self
+253
View File
@@ -0,0 +1,253 @@
"""Behavior and wiring tests for large-paste attachment conversion."""
from __future__ import annotations
import json
import shutil
import subprocess
from pathlib import Path
import pytest
_ROOT = Path(__file__).resolve().parent.parent
_HELPER = _ROOT / "turnstone/shared_static/composer_paste_text.js"
_COMPOSER = _ROOT / "turnstone/shared_static/composer.js"
_ATTACHMENTS = _ROOT / "turnstone/shared_static/composer_attachments.js"
_INTERACTIVE = _ROOT / "turnstone/shared_static/interactive.js"
_UI_APP = _ROOT / "turnstone/ui/static/app.js"
_CONSOLE_APP = _ROOT / "turnstone/console/static/app.js"
_COORDINATOR = _ROOT / "turnstone/console/static/coordinator/coordinator.js"
def test_paste_text_helper_behavior(tmp_path: Path) -> None:
"""Execute the real ESM and pin its fixed threshold, byte cap, and dedup."""
if shutil.which("node") is None:
pytest.skip("node binary not available on PATH")
module = tmp_path / "composer_paste_text.mjs"
module.write_text(_HELPER.read_text(encoding="utf-8"), encoding="utf-8")
script = tmp_path / "paste_harness.mjs"
module_url = json.dumps(module.as_uri())
script.write_text(
f"const moduleUrl = {module_url};\n"
+ r"""
import { Blob, File } from "node:buffer";
globalThis.Blob = Blob;
globalThis.File = File;
globalThis.window = globalThis;
function check(condition, message) {
if (!condition) throw new Error(message);
}
const paste = await import(moduleUrl);
check(paste.PASTE_ATTACHMENT_CHARS === 2000, "fixed threshold drifted");
check(
paste.pasteTextToFile("x".repeat(1999)) === null,
"below-threshold text converted",
);
const exactText = "x".repeat(2000);
const exact = paste.pasteTextToFile(exactText);
check(exact === null, "exact-threshold text converted");
const convertedText = "x".repeat(2001);
const converted = paste.pasteTextToFile(convertedText);
check(converted instanceof File, "above-threshold text did not convert");
check(converted.name === "pasted-text.txt", "filename drifted");
check(converted.type === "text/plain", "MIME drifted");
check(converted.size === 2001, "byte size drifted");
check(
(await converted.text()) === convertedText,
"file content did not round-trip",
);
check(
paste.pasteTextToFile("") === null,
"empty text converted",
);
check(
paste.pasteTextToFile("😀".repeat(2000)) === null,
"UTF-16 code units were counted as characters",
);
check(
paste.pasteTextToFile("😀".repeat(2001)) instanceof File,
"Unicode code points above the threshold did not convert",
);
const cjkAtCap = "".repeat(Math.floor(paste.TEXT_ATTACHMENT_MAX_BYTES / 3));
check(
paste.pasteTextToFile(cjkAtCap) instanceof File,
"text within the byte ceiling did not convert",
);
check(
paste.pasteTextToFile(cjkAtCap + "") === null,
"multibyte text escaped the byte ceiling",
);
const sameText = "same".repeat(501);
const sameA = paste.pasteTextToFile(sameText);
const sameB = paste.pasteTextToFile(sameText);
const different = paste.pasteTextToFile("size".repeat(501));
check(
paste.isDuplicatePastedTextFile(sameB, [sameA]),
"identical synthesized pastes did not deduplicate",
);
check(
!paste.isDuplicatePastedTextFile(different, [sameA]),
"different same-size pastes were deduplicated",
);
check(
!paste.isDuplicatePastedTextFile(
new File([sameText], "ordinary.txt", { type: "text/plain" }),
[sameA],
),
"ordinary user files entered synthesized-paste dedup",
);
""",
encoding="utf-8",
)
proc = subprocess.run(
["node", str(script)],
capture_output=True,
text=True,
timeout=15,
)
assert proc.returncode == 0, f"paste harness failed:\n{proc.stderr}\n{proc.stdout}"
def test_attachment_snapshot_reports_in_flight_upload(tmp_path: Path) -> None:
"""A synthesized paste cannot disappear from a send-time snapshot while
its immediate upload is still resolving."""
if shutil.which("node") is None:
pytest.skip("node binary not available on PATH")
script = tmp_path / "attachment_snapshot_harness.mjs"
module_url = json.dumps(_ATTACHMENTS.as_uri())
script.write_text(
f"const moduleUrl = {module_url};\n"
+ r"""
import { File } from "node:buffer";
globalThis.File = File;
globalThis.window = globalThis;
function fakeElement() {
return {
children: [],
dataset: {},
appendChild(child) {
this.children.push(child);
return child;
},
addEventListener() {},
querySelector() { return null; },
setAttribute() {},
remove() {},
};
}
globalThis.document = { createElement: () => fakeElement() };
let resolveUpload;
const uploadResponse = new Promise((resolve) => { resolveUpload = resolve; });
const attachmentsModule = await import(moduleUrl);
const controller = attachmentsModule.createAttachmentController({
chipsEl: fakeElement(),
getWsId: () => "ws-1",
authFetch: () => uploadResponse,
});
let snap = controller.snapshot();
if (snap.uploading || snap.attachment_ids.length)
throw new Error("empty controller reported an upload");
controller.upload(new File(["large paste"], "pasted-text.txt", {
type: "text/plain",
}));
snap = controller.snapshot();
if (!snap.uploading)
throw new Error("in-flight placeholder was omitted from snapshot state");
if (snap.attachments.length || snap.attachment_ids.length)
throw new Error("placeholder escaped into stable attachment arrays");
resolveUpload({
ok: true,
status: 200,
json: () => Promise.resolve({
attachment_id: "attachment-1",
filename: "pasted-text.txt",
size_bytes: 11,
mime_type: "text/plain",
kind: "text",
}),
});
await new Promise((resolve) => setTimeout(resolve, 0));
snap = controller.snapshot();
if (snap.uploading)
throw new Error("settled upload remained marked in flight");
if (snap.attachment_ids.join(",") !== "attachment-1")
throw new Error("settled upload was not sendable: " + snap.attachment_ids);
""",
encoding="utf-8",
)
proc = subprocess.run(
["node", str(script)],
capture_output=True,
text=True,
timeout=15,
)
assert proc.returncode == 0, (
f"attachment snapshot harness failed:\n{proc.stderr}\n{proc.stdout}"
)
def test_paste_text_wiring_guard_rails() -> None:
"""Guard every surface around the behavior-tested shared helper."""
helper = _HELPER.read_text(encoding="utf-8")
composer = _COMPOSER.read_text(encoding="utf-8")
attachments = _ATTACHMENTS.read_text(encoding="utf-8")
interactive = _INTERACTIVE.read_text(encoding="utf-8")
ui_app = _UI_APP.read_text(encoding="utf-8")
console_app = _CONSOLE_APP.read_text(encoding="utf-8")
coordinator = _COORDINATOR.read_text(encoding="utf-8")
assert "window.TurnstonePasteText" in helper
assert "PASTE_ATTACHMENT_CHARS = 2000" in helper
assert "paste_attachment_chars" not in helper
assert "loadPasteThresholdChars" not in helper
assert 'from "./composer_paste_text.js"' in composer
assert "pasteTextToFile(text" in composer
assert "2000" not in composer, "the threshold belongs in the shared helper"
assert composer.index("if (uploaded > 0)") < composer.index("pasteTextToFile(text")
assert "if (accepted !== false) e.preventDefault();" in composer
assert "if (pending.has(info.attachment_id))" in attachments
assert "pending.delete(placeholderId);" in attachments
assert "uploading: uploading" in attachments
assert "function _handleComposerPaste(" in ui_app
assert "if (files.length > 0)" in ui_app
assert "if (!textFile || addFiles([textFile]) === false) return false;" in ui_app
assert "_handleComposerPaste(event, _newWsAddFiles)" in ui_app
assert "_handleComposerPaste(e, _addDashboardFiles)" in ui_app
assert "initEl.onpaste" in ui_app
assert ui_app.count("Add a message to send with this attachment.") == 2
assert "_loadPasteAttachmentSetting" not in ui_app
assert "return _homeStageFile(file);" in console_app
assert "isDuplicatePastedTextFile(file, _homeStagedFiles)" in console_app
assert "Add a message to send with this attachment." in console_app
assert "_loadPasteAttachmentSetting" not in console_app
for pane in (interactive, coordinator):
assert "Add a message to send with this attachment." in pane
assert "Attachments can't be sent while the assistant is working." in pane
assert "if (snap.uploading)" in pane
assert "Wait for attachments to finish uploading before sending." in pane
assert "!attachments.isEmpty()" in pane or "!this.attachments.isEmpty()" in pane
assert "loadPasteThresholdChars" not in coordinator
for page in (
_ROOT / "turnstone/ui/static/index.html",
_ROOT / "turnstone/console/static/index.html",
_ROOT / "turnstone/console/static/coordinator/index.html",
):
body = page.read_text(encoding="utf-8")
assert "/shared/composer_paste_text.js" in body, page
assert body.index("/shared/composer_paste_text.js") < body.index("/shared/composer.js"), (
page
)
+163
View File
@@ -1587,6 +1587,168 @@ class TestConsoleProxy:
mock_collector.get_node_detail.return_value = None
resp = client.get("/node/unknown/static/app.js")
assert resp.status_code == 404
assert resp.headers["cache-control"] == "no-store"
@pytest.mark.parametrize("mount", ["static", "shared"])
@pytest.mark.parametrize(
"suffix",
[
"%2e%2e/%2e%2e/v1/api/workstreams/private/history",
"%2E%2E/%2E%2E/v1/api/workstreams/private/history",
"%2e%2e%2f%2e%2e%2fv1%2fapi%2fworkstreams",
"%5c..%5c..%5cv1%5capi%5cworkstreams",
],
)
def test_proxy_static_rejects_encoded_traversal_before_upstream(self, mount, suffix):
from starlette.applications import Starlette
from starlette.routing import Route
from starlette.testclient import TestClient
from turnstone.console import server as csrv
handler = csrv.proxy_static if mount == "static" else csrv.proxy_shared_static
app = Starlette(
routes=[Route(f"/node/{{node_id}}/{mount}/{{path:path}}", endpoint=handler)]
)
app.state.proxy_client = MagicMock()
with TestClient(app) as client:
resp = client.get(f"/node/node-a/{mount}/katex-0.18.4/{suffix}")
assert resp.status_code == 400
assert resp.headers["cache-control"] == "no-store"
app.state.proxy_client.get.assert_not_called()
def test_proxy_static_percent_encodes_each_valid_path_segment(self, monkeypatch):
from types import SimpleNamespace
import httpx
from turnstone.console import server as csrv
upstream = httpx.Response(
200,
content=b"asset",
request=httpx.Request("GET", "http://n:1/static/nested/asset"),
)
calls = []
async def _mock_get(url, *, headers):
calls.append((url, headers))
return upstream
proxy_client = MagicMock(spec=httpx.AsyncClient)
proxy_client.get = MagicMock(side_effect=_mock_get)
request = SimpleNamespace(
app=SimpleNamespace(state=SimpleNamespace(proxy_client=proxy_client)),
path_params={"node_id": "node-a", "path": "nested/asset name?#.js"},
headers={},
)
monkeypatch.setattr(csrv, "_proxy_auth_headers", lambda request: {})
monkeypatch.setattr(csrv, "_get_server_url", lambda request, node_id: "http://n:1")
resp = asyncio.run(csrv.proxy_static(request))
assert calls == [("http://n:1/static/nested/asset%20name%3F%23.js", {})]
assert resp.status_code == 200
assert resp.headers["cache-control"] == "no-cache"
@pytest.mark.parametrize(
("handler_name", "mount", "path"),
[
("proxy_static", "static", "app.js"),
("proxy_shared_static", "shared", "interactive.js"),
],
)
def test_proxy_static_forwards_conditional_request_and_validators(
self, monkeypatch, handler_name, mount, path
):
from types import SimpleNamespace
import httpx
from turnstone.console import server as csrv
upstream = httpx.Response(
304,
headers={
"etag": '"current-build"',
"last-modified": "Wed, 12 Aug 2026 12:00:00 GMT",
},
request=httpx.Request("GET", f"http://n:1/{mount}/{path}"),
)
calls = []
async def _mock_get(url, *, headers):
calls.append((url, headers))
return upstream
proxy_client = MagicMock(spec=httpx.AsyncClient)
proxy_client.get = MagicMock(side_effect=_mock_get)
request = SimpleNamespace(
app=SimpleNamespace(state=SimpleNamespace(proxy_client=proxy_client)),
path_params={"node_id": "node-a", "path": path},
headers={"if-none-match": '"current-build"'},
)
monkeypatch.setattr(csrv, "_proxy_auth_headers", lambda request: {"X-Auth": "test"})
monkeypatch.setattr(csrv, "_get_server_url", lambda request, node_id: "http://n:1")
resp = asyncio.run(getattr(csrv, handler_name)(request))
assert calls == [
(
f"http://n:1/{mount}/{path}",
{"X-Auth": "test", "if-none-match": '"current-build"'},
)
]
assert resp.status_code == 304
assert resp.headers["cache-control"] == "no-cache"
assert resp.headers["etag"] == '"current-build"'
assert resp.headers["last-modified"] == "Wed, 12 Aug 2026 12:00:00 GMT"
@pytest.mark.parametrize("status_code", [404, 500])
def test_proxy_vendor_static_error_is_never_immutable(self, status_code):
import httpx
from turnstone.console.server import _proxy_static_response
upstream = httpx.Response(
status_code,
content=b"transient failure",
request=httpx.Request("GET", "http://n:1/shared/katex-0.18.4/missing.css"),
)
resp = _proxy_static_response(upstream, "katex-0.18.4/missing.css")
assert resp.status_code == status_code
assert resp.headers["cache-control"] == "no-store"
@pytest.mark.parametrize(
("upstream_policy", "expected"),
[
("no-store", "no-store"),
("private, max-age=600", "private, max-age=600"),
("public, max-age=0, must-revalidate", "public, max-age=0, must-revalidate"),
("public, max-age=60", "public, max-age=60"),
],
)
def test_proxy_vendor_static_preserves_stricter_upstream_policy(
self, upstream_policy, expected
):
import httpx
from turnstone.console.server import _proxy_static_response
upstream = httpx.Response(
200,
content=b"asset",
headers={"cache-control": upstream_policy},
request=httpx.Request("GET", "http://n:1/shared/katex-0.18.4/katex.js"),
)
resp = _proxy_static_response(upstream, "katex-0.18.4/katex.js")
assert resp.headers["cache-control"] == expected
def test_proxy_api_unknown_node_returns_404(self, client, mock_collector):
mock_collector.get_node_detail.return_value = None
@@ -2447,6 +2609,7 @@ class TestProxySharedStatic:
client = TestClient(app, raise_server_exceptions=False, headers=_TEST_AUTH_HEADERS)
resp = client.get("/node/unknown/shared/base.css")
assert resp.status_code == 404
assert resp.headers["cache-control"] == "no-store"
client.close()
+14 -27
View File
@@ -2102,39 +2102,35 @@ def test_coord_cancel_cascade_failure_does_not_fail_owner_cancel(storage):
from tests._replay_helpers import make_replay_mocks as _make_coord_replay_mocks # noqa: E402
def test_coord_events_replay_yields_connected_first():
"""Pre-status-bar coord replay only re-injected pending_approval +
pending_plan_review. Post-status-bar parity with interactive
yields ``connected`` first so the dashboard's status bar populates
the model cell before any history arrives mirrors the
interactive replay (turnstone/server.py:_interactive_events_replay)."""
from turnstone.console.server import _coord_events_replay
def test_coord_shared_preamble_yields_connected_first():
"""The shared preamble gives coordinator streams the same bootstrap."""
from turnstone.core.session_replay import session_replay_preamble
ws, ui, request = _make_coord_replay_mocks()
out = list(_coord_events_replay(ws, ui, request))
ws, ui, _request = _make_coord_replay_mocks()
out = list(session_replay_preamble(ws.session, ui))
assert out[0]["type"] == "connected"
assert out[0]["model"] == "gpt-5"
assert out[0]["model_alias"] == "default"
assert out[0]["skip_permissions"] is False
def test_coord_events_replay_includes_status_only_when_last_usage_present():
def test_coord_shared_preamble_includes_status_only_when_last_usage_present():
"""The ``status`` event populates the per-tab token-usage bar on
resume. Skipped when ``session._last_usage`` is None (a freshly-
created coordinator that hasn't completed a turn) — matches
interactive behaviour."""
from turnstone.console.server import _coord_events_replay
from turnstone.core.session_replay import session_replay_preamble
ws, ui, request = _make_coord_replay_mocks()
out = list(_coord_events_replay(ws, ui, request))
ws, ui, _request = _make_coord_replay_mocks()
out = list(session_replay_preamble(ws.session, ui))
assert "status" not in {ev["type"] for ev in out}
def test_coord_events_replay_status_payload_shape():
def test_coord_shared_preamble_status_payload_shape():
"""When ``last_usage`` exists, the replayed ``status`` event carries
every field the dashboard's updateStatusBar() reads — same shape
SessionUI.on_status emits live."""
from turnstone.console.server import _coord_events_replay
from turnstone.core.session_replay import session_replay_preamble
ws, ui, request = _make_coord_replay_mocks(
last_usage={
@@ -2146,7 +2142,7 @@ def test_coord_events_replay_status_payload_shape():
_ws_turn_tool_calls=3,
_ws_messages=7,
)
out = list(_coord_events_replay(ws, ui, request))
out = list(session_replay_preamble(ws.session, ui))
status = next(ev for ev in out if ev["type"] == "status")
assert status["prompt_tokens"] == 40000
assert status["completion_tokens"] == 6310
@@ -2174,12 +2170,7 @@ def test_coord_events_replay_skips_session_block_when_no_session():
def test_coord_events_replay_yields_pending_approval():
"""The lifted coord ``events_replay`` callback yields, after the
connected preamble, the pending approval (if any). Pre-lift coord
pushed it onto the listener queue via ``put_nowait``; the lift
restructures as a generator the lifted body iterates and yields as
``data:`` lines, but the payload identity is preserved. Pure-read
never mutates ``ui``."""
"""The coord replay tail yields pending approval without mutation."""
from turnstone.console.server import _coord_events_replay
ws, ui, request = _make_coord_replay_mocks(
@@ -2188,8 +2179,6 @@ def test_coord_events_replay_yields_pending_approval():
out = list(_coord_events_replay(ws, ui, request))
types = [ev["type"] for ev in out]
# Status preamble is yielded first (no last_usage → no status); the
# pending-approval re-injection then matches the pre-lift body.
assert types[0] == "connected"
assert "approve_request" in types
@@ -2246,9 +2235,7 @@ def test_coord_events_replay_skips_verdict_replay_without_pending_approval():
def test_coord_events_replay_yields_only_connected_when_no_pending():
"""A workstream with a session but no pending approval / plan
review and no last_usage yields just the ``connected`` preamble.
The lifted body falls through to the live loop immediately after."""
"""Without controls or usage, replay contains only the preamble."""
from turnstone.console.server import _coord_events_replay
ws, ui, request = _make_coord_replay_mocks()
+23 -3
View File
@@ -26,6 +26,8 @@ def client():
def test_valid_ws_id_injects_data_attr(client):
from turnstone import __version__
ws_id = "a" * 32
resp = client.get(f"/coordinator/{ws_id}")
assert resp.status_code == 200
@@ -35,9 +37,27 @@ def test_valid_ws_id_injects_data_attr(client):
assert f'data-ws-id="{ws_id}"' in body
# Template placeholder is fully substituted.
assert "{{WS_ID}}" not in body
# Sanity: the shared static imports are wired.
assert "/shared/base.css" in body
assert "/static/coordinator/coordinator.js" in body
# First-party tags are versioned; version-named vendor assets stay stable.
assert f"/shared/base.css?v={__version__}" in body
assert f"/static/coordinator/coordinator.css?v={__version__}" in body
assert "/shared/katex-0.18.4/katex.min.css?v=" not in body
# Inline module imports are outside version_html's src/href boundary. The
# static route's no-cache policy makes this URL revalidate on every reload.
assert 'from "/static/coordinator/coordinator.js"' in body
def test_coordinator_page_revalidates_with_etag(client):
ws_id = "b" * 32
first = client.get(f"/coordinator/{ws_id}")
assert first.headers["cache-control"] == "no-cache"
assert first.headers["etag"]
unchanged = client.get(
f"/coordinator/{ws_id}", headers={"If-None-Match": first.headers["etag"]}
)
assert unchanged.status_code == 304
assert unchanged.headers["cache-control"] == "no-cache"
assert unchanged.headers["etag"] == first.headers["etag"]
def test_non_hex_ws_id_returns_400(client):
+1
View File
@@ -2261,6 +2261,7 @@ def test_prepare_and_write_path_refuse_in_the_same_words(coord_session):
item = sess._prepare_tool(_tc("tasks", args))
assert "error" in item, args
expected = sess._coord_tool_error("call-1", "tasks", f"{action}: {authoritative['error']}")
expected["_principal_id"] = sess._tool_prepare_principal_id()
assert item == expected, (args, item["error"])
+9 -2
View File
@@ -18,9 +18,8 @@ from pathlib import Path
import pytest
lacme = pytest.importorskip("lacme")
SCRIPT = Path(__file__).parent.parent / "docker" / "healthcheck.py"
TLS_COMPOSE_OVERLAY = Path(__file__).parent.parent / "deploy" / "docker-compose.tls.yml"
def run_healthcheck(url: str, pem_root: Path | None = None) -> subprocess.CompletedProcess:
@@ -220,6 +219,14 @@ def test_default_pem_root_matches_server(monkeypatch):
assert _load_script_module()._pem_root() == tls_pem_runtime_dir()
def test_tls_compose_overlay_keeps_server_healthcheck_enabled():
"""The mTLS-aware base healthcheck must survive the production overlay."""
import yaml
overlay = yaml.safe_load(TLS_COMPOSE_OVERLAY.read_text())
assert "healthcheck" not in overlay["services"]["server"]
def test_find_pem_dir_accepts_real_pem_layout(monkeypatch, mtls_setup):
"""Drift guard: lacme's on-disk layout is accepted by _find_pem_dir.
+6 -1
View File
@@ -654,10 +654,15 @@ class TestPreflightHelpers:
def test_relevant_env_redacts_secrets(self) -> None:
out = _relevant_env(
{"TURNSTONE_JWT_SECRET": "supersecret", "TURNSTONE_HOST_IP": "10.0.0.1"}
{
"TURNSTONE_JWT_SECRET": "supersecret",
"TURNSTONE_HOST_IP": "10.0.0.1",
"TURNSTONE_ACME_EXTERNAL_URL": "http://ca.internal:8090/acme",
}
)
assert out["TURNSTONE_JWT_SECRET"] == "set (hidden)"
assert out["TURNSTONE_HOST_IP"] == "10.0.0.1"
assert out["TURNSTONE_ACME_EXTERNAL_URL"] == "http://ca.internal:8090/acme"
def test_relevant_env_redacts_db_url_creds(self) -> None:
out = _relevant_env({"TURNSTONE_DB_URL": "postgresql://u:pw@h/db"})
+39 -23
View File
@@ -9,6 +9,8 @@ provider_blocks, trailing-citation fold).
from __future__ import annotations
import httpx
import httpx2
import pytest
from turnstone.core.providers import (
@@ -418,11 +420,18 @@ class TestTransportGuarded:
"""``transport_guarded`` — drain's conversion rule for consumers that
keep streaming semantics (the interactive loop)."""
@pytest.mark.parametrize("exc_name", ["ReadError", "RemoteProtocolError"])
def test_pre_finish_transport_error_becomes_retryable_incomplete(self, exc_name):
import httpx
exc_cls = getattr(httpx, exc_name)
@pytest.mark.parametrize(
"exc_cls",
[
httpx.ReadError,
httpx.RemoteProtocolError,
httpx2.ReadError,
httpx2.RemoteProtocolError,
],
ids=["httpx-read", "httpx-protocol", "httpx2-read", "httpx2-protocol"],
)
def test_pre_finish_transport_error_becomes_retryable_incomplete(self, exc_cls):
exc_name = exc_cls.__name__
def chunks():
yield StreamChunk(content_delta="partial")
@@ -434,14 +443,17 @@ class TestTransportGuarded:
next(it)
assert isinstance(excinfo.value.__cause__, exc_cls)
def test_message_byte_matches_drains_shape(self):
@pytest.mark.parametrize(
"exc_cls",
[httpx.ReadError, httpx2.ReadError],
ids=["httpx", "httpx2"],
)
def test_message_byte_matches_drains_shape(self, exc_cls):
# The wrapper and the drain are the SAME conversion rule; any test
# or log filter pinned to drain's message must match the wrapper's.
import httpx
def chunks():
yield StreamChunk(content_delta="partial")
raise httpx.ReadError("[SSL] record layer failure (_ssl.c:2590)")
raise exc_cls("[SSL] record layer failure (_ssl.c:2590)")
with pytest.raises(IncompleteStreamError) as guarded:
list(transport_guarded(chunks()))
@@ -449,17 +461,20 @@ class TestTransportGuarded:
drain_stream(chunks())
assert str(guarded.value) == str(drained.value)
def test_post_finish_blip_ends_stream_cleanly(self, caplog):
@pytest.mark.parametrize(
"exc_cls",
[httpx.ReadError, httpx2.ReadError],
ids=["httpx", "httpx2"],
)
def test_post_finish_blip_ends_stream_cleanly(self, caplog, exc_cls):
# The generation already completed — the blip only cost trailing
# metadata, so the stream ends instead of raising.
import logging
import httpx
def chunks():
yield StreamChunk(content_delta="done")
yield StreamChunk(finish_reason="stop")
raise httpx.ReadError("late blip")
raise exc_cls("late blip")
with caplog.at_level(logging.WARNING, logger="turnstone.core.providers._protocol"):
out = list(transport_guarded(chunks()))
@@ -502,32 +517,33 @@ class TestErrorPropagation:
# The generation completed (finish reason in hand) — a trailing
# transport blip forfeits only trailing metadata (here: the usage
# chunk), never the completed result.
import httpx
def chunks():
yield StreamChunk(content_delta="whole answer")
yield StreamChunk(finish_reason="stop")
raise httpx.ReadError("late blip")
raise httpx2.ReadError("late blip")
result = drain_stream(chunks())
assert result.content == "whole answer"
assert result.finish_reason == "stop"
assert result.usage is None
def test_httpx_transport_error_becomes_retryable_incomplete(self):
@pytest.mark.parametrize(
"exc_cls",
[httpx.RemoteProtocolError, httpx2.RemoteProtocolError],
ids=["httpx", "httpx2"],
)
def test_transport_error_becomes_retryable_incomplete(self, exc_cls):
# Streaming moves the body read out of the SDK's wrapped request:
# a mid-body wire failure surfaces as a raw httpx.TransportError
# no retry predicate recognizes. The drain re-raises it (chained,
# a mid-body wire failure surfaces as a raw HTTPX-family TransportError
# no retry predicate recognizes. The drain re-raises it (chained,
# message preserved) as the retryable IncompleteStreamError.
import httpx
def chunks():
yield StreamChunk(content_delta="partial")
raise httpx.RemoteProtocolError("peer closed connection")
raise exc_cls("peer closed connection")
with pytest.raises(IncompleteStreamError, match="RemoteProtocolError") as excinfo:
drain_stream(chunks())
assert isinstance(excinfo.value.__cause__, httpx.RemoteProtocolError)
assert isinstance(excinfo.value.__cause__, exc_cls)
def test_mid_stream_exception_propagates_verbatim(self):
# Retry/deadline/fallback policy is the caller's — the drain adds
+2 -1
View File
@@ -256,6 +256,7 @@ class TestWorldSeeding:
"memory": [
{
"name": "proj-context",
"description": "Project deployment context",
"content": "acme-api: staging tracks main.",
"type": "reference",
}
@@ -2752,7 +2753,7 @@ class TestRunResourceLifecycle:
assert not is_storage_initialized()
def test_a_hung_generation_is_bounded_by_the_wall_clock(self, monkeypatch):
"""The per-request httpx timeout cannot bound a STREAM — a
"""The per-request HTTP transport timeout cannot bound a STREAM — a
trickling response resets the read timeout indefinitely so
without the executor wall clock a hung generation occupies a run
slot forever and is scored as a body regression when the sweep
+89 -2
View File
@@ -1209,8 +1209,8 @@ def test_deferred_send_settle_protocol_pins() -> None:
assert "deferred: !!data.deferred" in composer_queue
assert "attachedCount: (data.attached_ids || []).length" in composer_queue
assert "ctx.busyIsOptimistic()" in composer_queue
assert composer_queue.count("ctx.optimisticEl.remove()") >= 2, (
"both the retro-convert and queue_full arms must clear the optimistic bubble"
assert composer_queue.count("ctx.optimisticEl.remove()") >= 3, (
"retro-convert, queue_full, and attachments_busy must clear false optimistic bubbles"
)
# The missed-edge settle: a non-deferred chip binding onto an
# already-idle pane missed its only sweep — the post-bind promote
@@ -1233,6 +1233,10 @@ def test_deferred_send_settle_protocol_pins() -> None:
assert "settleSendResponse(" not in src, f"{name}: must not bypass the fetch stage"
assert "busyIsOptimistic" in src, name
assert "paneIsBusy" in src, f"{name}: the missed-edge settle needs the live flag"
assert "mergeRejectedComposerText" in src, f"{name}: refused text must be restored"
assert src.count("restoreInput:") == 2, (
f"{name}: composer send and edit-resend both need refusal restoration"
)
assert 'setBusy(true, "optimistic")' in src, f"{name}: optimistic flip must stamp"
assert "parsePriority(" in src, f"{name}: shared !!! parse"
assert 'case "message_dispatched"' in src, f"{name}: settle event not consumed"
@@ -1332,6 +1336,89 @@ console.log("settle matrix OK");
assert proc.returncode == 0, f"settle harness failed:\n{proc.stderr}\n{proc.stdout}"
def test_stale_idle_refusals_restore_input(tmp_path) -> None:
"""A stale local idle state must not render either busy refusal as
delivered or discard its companion text. Text entered during the POST is
retained after the rejected text, and an SSE idle that already arrived
prevents the old optimistic busy state from being reasserted."""
import shutil
import subprocess
if shutil.which("node") is None:
pytest.skip("node binary not available on PATH")
helper = _ROOT / "turnstone/shared_static/composer_queue.js"
script = tmp_path / "attachments_busy_harness.mjs"
script.write_text(
rf"""const {{ mergeRejectedComposerText, settleSendResponse }} =
await import("file://{helper}");
function run(status, optimisticBusy) {{
const calls = [];
let composerValue = "typed during request";
const optimisticEl = {{
isConnected: true,
dataset: {{}},
remove: () => calls.push("remove-optimistic"),
}};
settleSendResponse(
{{ remove: () => calls.push("remove-queued") }},
{{ status }},
{{
queuedEl: null,
optimisticEl,
isBusy: false,
setBusy: (value) => calls.push("busy:" + value),
busyIsOptimistic: () => optimisticBusy,
paneIsBusy: () => optimisticBusy,
restoreInput: () => {{
composerValue = mergeRejectedComposerText("rejected", composerValue);
calls.push("restore");
}},
renderError: () => calls.push("error"),
consumeAttachments: () => calls.push("consume"),
}},
);
return {{ calls, composerValue }};
}}
for (const [status, expectedBusy] of [
["attachments_busy", "busy:true"],
["cross_user_interjection", "busy:false"],
["queue_full", "busy:false"],
]) {{
let result = run(status, true);
if (result.composerValue !== "rejected\ntyped during request")
throw new Error(status + " companion/current text merge drifted: " + result.composerValue);
for (const call of ["remove-optimistic", "restore", expectedBusy, "error"]) {{
if (!result.calls.includes(call))
throw new Error(status + " missing stale-idle settlement " + call + ": " + result.calls);
}}
if (result.calls.includes("consume") || result.calls.includes("remove-queued"))
throw new Error(status + " attachments/chip state was consumed: " + result.calls);
result = run(status, false);
if (result.calls.some((call) => call.startsWith("busy:")))
throw new Error(status + " overwrote a raced SSE state: " + result.calls);
}}
if (
mergeRejectedComposerText("rejected", "rejected\nlater") !==
"rejected\nrejected\nlater"
)
throw new Error("independently typed matching text was discarded");
if (mergeRejectedComposerText("rejected", "") !== "rejected")
throw new Error("empty composer did not restore rejected text");
""",
encoding="utf-8",
)
proc = subprocess.run(
["node", str(script)],
capture_output=True,
text=True,
timeout=15,
)
assert proc.returncode == 0, f"attachments_busy harness failed:\n{proc.stderr}\n{proc.stdout}"
def test_accepted_tool_event_recorded_only_when_painted() -> None:
"""An unpainted accepted tool_result must stay replayable.
+69
View File
@@ -0,0 +1,69 @@
"""Regression guards for the native paste-over-HTTP livepass."""
from __future__ import annotations
import importlib.util
import re
import shutil
import subprocess
from pathlib import Path
from typing import Any
import pytest
_ROOT = Path(__file__).resolve().parent.parent
_SCRIPT = _ROOT / "scripts/livepass.py"
def _load_livepass() -> Any:
spec = importlib.util.spec_from_file_location("livepass_paste_script", _SCRIPT)
assert spec is not None and spec.loader is not None
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
return module
def test_paste_livepass_builds_real_clipboard_event_path(tmp_path: Path) -> None:
livepass = _load_livepass()
livepass.build(tmp_path)
page_path = tmp_path / "paste/livepass.html"
page = page_path.read_text(encoding="utf-8")
assert (tmp_path / "paste/shared").resolve() == _ROOT / "turnstone/shared_static"
assert 'import { Composer } from "./shared/composer.js";' in page
assert "PASTE_ATTACHMENT_CHARS" in page
assert "new Composer(" in page
assert "event.isTrusted" in page
assert "event.clipboardData" in page
assert "window.isSecureContext" in page
assert 'attachment.name === "pasted-text.txt"' in page
assert 'attachment.type === "text/plain"' in page
assert 'document.title = "PASTE-HTTP-READY"' in page
assert 'document.title = "PASTE-HTTP-FAILED-" + reason' in page
# These shortcuts would make the harness green without exercising the
# browser's native clipboard event and therefore invalidate its purpose.
assert "navigator.clipboard" not in page
assert "new ClipboardEvent" not in page
assert ".dispatchEvent(" not in page
assert "__pasteProbe" not in page
def test_generated_paste_module_is_valid_javascript(tmp_path: Path) -> None:
if shutil.which("node") is None:
pytest.skip("node binary not available on PATH")
livepass = _load_livepass()
livepass.build(tmp_path)
page = (tmp_path / "paste/livepass.html").read_text(encoding="utf-8")
match = re.search(r'<script type="module">\s*(.*?)\s*</script>', page, re.S)
assert match is not None
module = tmp_path / "paste_livepass.mjs"
module.write_text(match.group(1), encoding="utf-8")
result = subprocess.run(
["node", "--check", str(module)],
capture_output=True,
text=True,
timeout=15,
)
assert result.returncode == 0, result.stderr
+69
View File
@@ -4069,6 +4069,75 @@ class TestStaticNotificationRefresh:
"the hung sibling must be cancelled and reaped inside the scope"
)
def test_resources_refresh_accepts_missing_template_method(self) -> None:
"""Push/manual refresh shares the per-method compatibility behavior."""
from mcp import McpError
mgr = MCPClientManager({})
session = MagicMock()
session.list_resources = AsyncMock(
return_value=mcp_types.ListResourcesResult(
resources=[
mcp_types.Resource(
uri="res://fresh",
name="fresh",
mimeType="text/plain",
)
]
)
)
session.list_resource_templates = AsyncMock(
side_effect=McpError(
mcp_types.ErrorData(
code=mcp_types.METHOD_NOT_FOUND,
message="templates disabled",
)
)
)
state = _seed_static_state(
mgr,
"srv",
session=session,
supports_resources=True,
resources=[{"uri": "res://old", "server": "srv"}],
)
mgr._rebuild_resources()
asyncio.run(mgr._refresh_server_resources("srv"))
assert [r["uri"] for r in state.resources] == ["res://fresh"]
assert [r["uri"] for r in mgr.get_resources()] == ["res://fresh"]
def test_resources_refresh_rejects_message_only_method_match(self) -> None:
"""Only -32601 is normalized; a lookalike error leaves catalog intact."""
from mcp import McpError
mgr = MCPClientManager({})
session = MagicMock()
session.list_resources = AsyncMock(return_value=mcp_types.ListResourcesResult(resources=[]))
session.list_resource_templates = AsyncMock(
side_effect=McpError(
mcp_types.ErrorData(
code=mcp_types.INVALID_PARAMS,
message="Method not found",
)
)
)
old_resources = [{"uri": "res://old", "server": "srv"}]
state = _seed_static_state(
mgr,
"srv",
session=session,
supports_resources=True,
resources=old_resources,
)
with pytest.raises(McpError) as raised:
asyncio.run(mgr._refresh_server_resources("srv"))
assert raised.value.error.code == mcp_types.INVALID_PARAMS
assert state.resources is old_resources
def test_reap_bounded_reraises_external_cancel(self) -> None:
"""An EXTERNAL cancel delivered during the reap window must be
HONOURED (re-raised), not swallowed else a shutdown/cancel of
+392
View File
@@ -26,7 +26,9 @@ from contextlib import asynccontextmanager
from typing import Any
from unittest.mock import AsyncMock, MagicMock, patch
import mcp.types as mcp_types
import pytest
from mcp import McpError
from turnstone.core.mcp_client import MCPClientManager
@@ -155,6 +157,348 @@ class TestTransportOwnerLifecycle:
assert state.owner_task is None
assert state.close_requested is None
def test_tools_only_connect_skips_unchanged_catalog_listeners(self, running_loop_mgr) -> None:
"""A tools-only registration must not rebuild every live chat twice."""
mgr, loop, _ = running_loop_mgr
patches: dict[str, Any] = {}
_fake_transport_and_session(patches)
notifications: list[str] = []
mgr.add_listener(lambda: notifications.append("tools"))
mgr.add_resource_listener(lambda: notifications.append("resources"))
mgr.add_prompt_listener(lambda: notifications.append("prompts"))
with (
patch("turnstone.core.mcp_client.stdio_client", patches["stdio_client"]),
patch("turnstone.core.mcp_client.ClientSession", patches["ClientSession"]),
):
_run(loop, mgr._connect_one_locked("srv", mgr._server_configs["srv"]))
assert notifications == ["tools"]
_run(loop, mgr._teardown_static_session("srv"))
def test_connect_notifies_when_dropped_capability_clears_stale_catalog(
self, running_loop_mgr
) -> None:
"""Capability loss clears and broadcasts genuinely stale catalogs."""
mgr, loop, _ = running_loop_mgr
state = mgr._ensure_static_state("srv")
state.resources = [
{
"uri": "res://stale",
"name": "stale",
"description": "",
"mimeType": "text/plain",
"server": "srv",
}
]
state.prompts = [
{
"name": "mcp__srv__stale",
"original_name": "stale",
"server": "srv",
"description": "",
"arguments": [],
}
]
mgr._rebuild_resources()
mgr._rebuild_prompts()
patches: dict[str, Any] = {}
_fake_transport_and_session(patches)
notifications: list[str] = []
mgr.add_listener(lambda: notifications.append("tools"))
mgr.add_resource_listener(lambda: notifications.append("resources"))
mgr.add_prompt_listener(lambda: notifications.append("prompts"))
with (
patch("turnstone.core.mcp_client.stdio_client", patches["stdio_client"]),
patch("turnstone.core.mcp_client.ClientSession", patches["ClientSession"]),
):
_run(loop, mgr._connect_one_locked("srv", mgr._server_configs["srv"]))
assert state.resources == []
assert state.prompts == []
assert mgr.get_resources() == []
assert mgr.get_prompts() == []
assert notifications == ["tools", "resources", "prompts"]
_run(loop, mgr._teardown_static_session("srv"))
def test_static_connect_accepts_resources_without_template_method(
self, running_loop_mgr
) -> None:
"""The aggregate resources capability does not require both list methods.
A server with concrete resources but no templates still publishes its
tools/resources and retains a usable live session when the unsupported
half returns the protocol's exact METHOD_NOT_FOUND code.
"""
mgr, loop, _ = running_loop_mgr
patches: dict[str, Any] = {}
fake = _fake_transport_and_session(patches)
session = fake["session"]
session.get_server_capabilities = MagicMock(
return_value=mcp_types.ServerCapabilities(
tools=mcp_types.ToolsCapability(listChanged=False),
resources=mcp_types.ResourcesCapability(listChanged=False),
)
)
session.list_tools = AsyncMock(
return_value=mcp_types.ListToolsResult(
tools=[
mcp_types.Tool(
name="echo",
description="Echo input",
inputSchema={"type": "object", "properties": {}},
)
]
)
)
session.list_resources = AsyncMock(
return_value=mcp_types.ListResourcesResult(
resources=[
mcp_types.Resource(
uri="res://one",
name="one",
mimeType="text/plain",
)
]
)
)
session.list_resource_templates = AsyncMock(
side_effect=McpError(
mcp_types.ErrorData(
code=mcp_types.METHOD_NOT_FOUND,
message="templates disabled",
)
)
)
with (
patch("turnstone.core.mcp_client.stdio_client", patches["stdio_client"]),
patch("turnstone.core.mcp_client.ClientSession", patches["ClientSession"]),
):
_run(loop, mgr._connect_one_locked("srv", mgr._server_configs["srv"]))
state = mgr._static_servers["srv"]
assert state.session is session
assert state.owner_task is not None and not state.owner_task.done()
assert mgr.is_mcp_tool("mcp__srv__echo") is True
assert {r["uri"] for r in mgr.get_resources()} == {"res://one"}
_run(loop, mgr._teardown_static_session("srv"))
def test_failed_add_is_atomic_and_rejects_message_only_method_match(
self, running_loop_mgr
) -> None:
"""A later discovery failure cannot leave a callable ghost tool.
The error deliberately says ``Method not found`` but carries
INVALID_PARAMS: compatibility is classified by JSON-RPC code only.
"""
mgr, _loop, _ = running_loop_mgr
patches: dict[str, Any] = {}
fake = _fake_transport_and_session(patches)
session = fake["session"]
session.get_server_capabilities = MagicMock(
return_value=mcp_types.ServerCapabilities(
tools=mcp_types.ToolsCapability(listChanged=False),
resources=mcp_types.ResourcesCapability(listChanged=False),
)
)
session.list_tools = AsyncMock(
return_value=mcp_types.ListToolsResult(
tools=[
mcp_types.Tool(
name="ghost",
description="Must never publish",
inputSchema={"type": "object", "properties": {}},
)
]
)
)
session.list_resources = AsyncMock(
side_effect=McpError(
mcp_types.ErrorData(
code=mcp_types.INVALID_PARAMS,
message="Method not found",
)
)
)
with (
patch("turnstone.core.mcp_client.stdio_client", patches["stdio_client"]),
patch("turnstone.core.mcp_client.ClientSession", patches["ClientSession"]),
):
result = mgr.add_server_sync(
"new",
{"type": "stdio", "command": "fake-cmd"},
timeout=5,
)
assert result["connected"] is False
assert "Method not found" in result["error"]
assert "new" not in mgr._server_configs
state = mgr._static_servers["new"]
assert state.session is None
assert state.owner_task is None
assert state.close_requested is None
assert mgr.is_mcp_tool("mcp__new__ghost") is False
assert mgr.get_tools() == []
assert mgr.get_resources() == []
assert fake["events"] == [
"transport_enter",
"session_enter",
"session_exit",
"transport_exit",
]
def test_timed_out_add_waits_for_atomic_rollback(self, running_loop_mgr) -> None:
"""Returning timeout must not strand an unconfigured live transport."""
mgr, _loop, _ = running_loop_mgr
patches: dict[str, Any] = {}
fake = _fake_transport_and_session(patches)
session = fake["session"]
session.get_server_capabilities = MagicMock(
return_value=mcp_types.ServerCapabilities(
tools=mcp_types.ToolsCapability(listChanged=False),
resources=mcp_types.ResourcesCapability(listChanged=False),
)
)
session.list_tools = AsyncMock(
return_value=mcp_types.ListToolsResult(
tools=[
mcp_types.Tool(
name="ghost",
description="Must never publish",
inputSchema={"type": "object", "properties": {}},
)
]
)
)
async def _park_resources() -> Any:
await asyncio.sleep(3600)
session.list_resources = _park_resources
with (
patch("turnstone.core.mcp_client.stdio_client", patches["stdio_client"]),
patch("turnstone.core.mcp_client.ClientSession", patches["ClientSession"]),
):
result = mgr.add_server_sync(
"new",
{"type": "stdio", "command": "fake-cmd"},
timeout=0.05,
)
assert result == {
"connected": False,
"tools": 0,
"resources": 0,
"prompts": 0,
"error": "MCP server 'new' registration timed out",
}
assert "new" not in mgr._server_configs
state = mgr._static_servers["new"]
assert state.session is None
assert state.owner_task is None
assert state.close_requested is None
assert state.tools == []
assert state.resources == []
assert state.prompts == []
assert mgr.is_mcp_tool("mcp__new__ghost") is False
assert fake["events"] == [
"transport_enter",
"session_enter",
"session_exit",
"transport_exit",
]
def test_add_reports_late_success_when_connect_suppresses_timeout(
self, running_loop_mgr
) -> None:
"""The sync result reflects the definitive loop-side outcome.
A dependency can suppress cancellation. In that case ``asyncio.timeout``
cannot honestly report a timeout, so the registration is a late success
rather than a failed result with a live ghost catalog.
"""
mgr, _loop, _ = running_loop_mgr
async def _late_success(name: str, _cfg: dict[str, Any]) -> None:
# Model the completion-vs-timeout race: the connect finishes and
# commits after the sync caller's deadline but before its
# cancellation can make the operation fail.
with contextlib.suppress(asyncio.CancelledError):
await asyncio.sleep(3600)
state = mgr._ensure_static_state(name)
state.session = MagicMock()
state.tools = [
{
"type": "function",
"function": {
"name": f"mcp__{name}__late",
"description": "late",
"parameters": {"type": "object", "properties": {}},
},
}
]
mgr._rebuild_tools()
with patch.object(mgr, "_connect_one_locked", side_effect=_late_success):
result = mgr.add_server_sync(
"new",
{"type": "stdio", "command": "fake-cmd"},
timeout=0.05,
)
assert result == {
"connected": True,
"tools": 1,
"resources": 0,
"prompts": 0,
"error": "",
}
assert mgr._server_configs["new"] == {"type": "stdio", "command": "fake-cmd"}
state = mgr._static_servers["new"]
assert state.session is not None
assert len(state.tools) == 1
assert mgr.is_mcp_tool("mcp__new__late") is True
def test_blocked_loop_cannot_return_failure_before_add_outcome(self, running_loop_mgr) -> None:
"""A synchronous listener stall may delay success, never expose a ghost."""
mgr, _loop, _ = running_loop_mgr
async def _publish_then_notify(name: str, _cfg: dict[str, Any]) -> None:
state = mgr._ensure_static_state(name)
state.session = MagicMock()
state.tools = [
{
"type": "function",
"function": {
"name": f"mcp__{name}__live",
"description": "live",
"parameters": {"type": "object", "properties": {}},
},
}
]
mgr._rebuild_tools()
mgr.add_listener(lambda: time.sleep(0.1))
started = time.monotonic()
with patch.object(mgr, "_connect_one_locked", side_effect=_publish_then_notify):
result = mgr.add_server_sync(
"new",
{"type": "stdio", "command": "fake-cmd"},
timeout=0.01,
)
elapsed = time.monotonic() - started
assert elapsed >= 0.08
assert result["connected"] is True
assert result["error"] == ""
assert "new" in mgr._server_configs
assert mgr._static_servers["new"].session is not None
assert mgr.is_mcp_tool("mcp__new__live") is True
def test_owner_death_evicts_session(self, running_loop_mgr) -> None:
"""Trigger-A observer: the transport collapsing under a live session
(owner task dies without a requested close) evicts the session so the
@@ -236,6 +580,54 @@ class TestTransportOwnerLifecycle:
# The owner unwound its cms despite dying mid-discovery.
assert fake["events"][-2:] == ["session_exit", "transport_exit"]
def test_owner_death_same_turn_as_discovery_cannot_publish_catalog(
self, running_loop_mgr
) -> None:
"""When discovery and owner death tie, transport death wins."""
mgr, _loop, _ = running_loop_mgr
patches: dict[str, Any] = {}
fake = _fake_transport_and_session(patches)
async def _cancel_owner_and_return_tool() -> mcp_types.ListToolsResult:
owner = mgr._static_servers["new"].owner_task
assert owner is not None
owner.cancel()
# Let the owner consume cancellation before this discovery task
# returns, putting both futures in the completed set observed by
# ``_await_owner_discovery``.
await asyncio.sleep(0)
return mcp_types.ListToolsResult(
tools=[
mcp_types.Tool(
name="ghost",
description="Must never publish",
inputSchema={"type": "object", "properties": {}},
)
]
)
fake["session"].list_tools = AsyncMock(side_effect=_cancel_owner_and_return_tool)
with (
patch("turnstone.core.mcp_client.stdio_client", patches["stdio_client"]),
patch("turnstone.core.mcp_client.ClientSession", patches["ClientSession"]),
):
result = mgr.add_server_sync(
"new",
{"type": "stdio", "command": "fake-cmd"},
timeout=5,
)
assert result["connected"] is False
assert "died during discovery" in result["error"]
assert "new" not in mgr._server_configs
state = mgr._static_servers["new"]
assert state.session is None
assert state.owner_task is None
assert state.tools == []
assert mgr.is_mcp_tool("mcp__new__ghost") is False
assert fake["events"][-2:] == ["session_exit", "transport_exit"]
def test_base_exception_escape_resolves_waiter_and_propagates(self, running_loop_mgr) -> None:
"""A BaseException-derived escape that is neither CancelledError nor
Exception/group (a library control-flow escape; SystemExit and
+61
View File
@@ -1414,6 +1414,67 @@ def test_pool_resource_discovery_on_connect_via_real_streamable_http(
assert {r["uri"] for r in mgr._user_resources["user-1"]} == {"res://test/1", "res://test/2"}
@pytest.mark.parametrize(
("unsupported_method", "expected_uri", "is_template"),
[
("templates", "res://concrete", False),
("resources", "res://items/{id}", True),
],
)
def test_pool_resource_discovery_accepts_either_list_method_unsupported(
running_loop_mgr: Any,
monkeypatch: pytest.MonkeyPatch,
unsupported_method: str,
expected_uri: str,
is_template: bool,
) -> None:
"""Real SDK boundary: either half of resource discovery may be absent.
The server advertises the aggregate resources capability and exposes a
tool plus exactly one resource-list method. A wire-level -32601 from the
other method is an empty half-catalog, not a failed pool registration.
"""
mgr, loop, _ = running_loop_mgr
unsupported = {
"jsonrpc": "2.0",
"id": 0,
"error": {"code": -32601, "message": "Method not found"},
}
concrete = _list_resources_payload([_resource_spec("res://concrete")])
templates = {
"jsonrpc": "2.0",
"id": 3,
"result": {
"resourceTemplates": [
{
"uriTemplate": "res://items/{id}",
"name": "item",
"mimeType": "text/plain",
}
]
},
}
handler = _make_jsonrpc_handler(
init_response=_init_response_with_caps(resources=True),
list_tools_response=_list_tools_payload([_tool_spec("echo")]),
list_resources_response=unsupported if unsupported_method == "resources" else concrete,
list_resource_templates_response=(
unsupported if unsupported_method == "templates" else templates
),
)
_build_mock_transport_factory(mgr, monkeypatch, handler)
_patch_tcp_probe(mgr, monkeypatch)
entry = _connect_pool(mgr, loop, user_id="user-1", server_name="pool-srv")
assert entry.session is not None
assert mgr.is_mcp_tool("mcp__pool-srv__echo", user_id="user-1") is True
assert entry.resources is not None
assert [(r["uri"], bool(r.get("template"))) for r in entry.resources] == [
(expected_uri, is_template)
]
def test_pool_prompt_discovery_on_connect_via_real_streamable_http(
running_loop_mgr: Any, monkeypatch: pytest.MonkeyPatch
) -> None:
+133 -18
View File
@@ -121,7 +121,7 @@ def _seed_memory(storage, name="test_key", content="test content", **kw):
storage.create_structured_memory(
mid,
name,
kw.get("description", ""),
kw.get("description", "Seeded memory"),
kw.get("mem_type", "general"),
kw.get("scope", "global"),
kw.get("scope_id", ""),
@@ -130,6 +130,20 @@ def _seed_memory(storage, name="test_key", content="test content", **kw):
return mid
def _seed_workstream(storage, ws_id: str = "ws1", user_id: str = "test-user") -> None:
storage.register_workstream(ws_id, user_id=user_id)
def _save_body(name: str, content: str, **overrides: Any) -> dict[str, Any]:
body: dict[str, Any] = {
"name": name,
"content": content,
"description": f"Description for {name}",
}
body.update(overrides)
return body
# ===========================================================================
# Server endpoint tests
# ===========================================================================
@@ -158,6 +172,7 @@ class TestServerListMemories:
assert r.json()["memories"][0]["name"] == "a"
def test_filter_by_scope(self, server_client, storage):
_seed_workstream(storage)
_seed_memory(storage, "a", "x", scope="global")
_seed_memory(storage, "b", "y", scope="workstream", scope_id="ws1")
r = server_client.get("/v1/api/memories?scope=workstream&scope_id=ws1")
@@ -174,12 +189,38 @@ class TestServerListMemories:
r = server_client.get("/v1/api/memories?limit=abc")
assert r.status_code == 400
def test_unscoped_list_is_caller_bound(self, server_client, storage):
_seed_memory(storage, "global_visible", "g")
_seed_memory(storage, "own_visible", "u", scope="user", scope_id="test-user")
_seed_memory(storage, "victim_user", "secret", scope="user", scope_id="victim")
_seed_memory(storage, "victim_coord", "secret", scope="coordinator", scope_id="victim")
_seed_memory(storage, "private_project", "secret", scope="project", scope_id="p1")
r = server_client.get("/v1/api/memories")
assert r.status_code == 200
assert {row["name"] for row in r.json()["memories"]} == {
"global_visible",
"own_visible",
}
def test_internal_scopes_are_rejected(self, server_client):
for scope in ("coordinator", "project", "bogus"):
r = server_client.get(f"/v1/api/memories?scope={scope}&scope_id=victim")
assert r.status_code == 400
def test_workstream_scope_is_owner_bound(self, server_client, storage):
_seed_workstream(storage, "victim-ws", "victim")
_seed_memory(storage, "secret", "x", scope="workstream", scope_id="victim-ws")
r = server_client.get("/v1/api/memories?scope=workstream&scope_id=victim-ws")
assert r.status_code == 403
class TestServerSaveMemory:
def test_create(self, server_client):
r = server_client.post(
"/v1/api/memories",
json={"name": "my_key", "content": "my content"},
json=_save_body("my_key", "my content"),
)
assert r.status_code == 201
data = r.json()
@@ -191,21 +232,23 @@ class TestServerSaveMemory:
def test_upsert(self, server_client):
server_client.post(
"/v1/api/memories",
json={"name": "key", "content": "v1"},
json=_save_body("key", "v1"),
)
r = server_client.post(
"/v1/api/memories",
json={"name": "key", "content": "v2"},
json=_save_body("key", "v2", description="Updated key description"),
)
assert r.status_code == 200
assert r.json()["content"] == "v2"
def test_with_type_and_scope(self, server_client):
def test_with_type_and_scope(self, server_client, storage):
_seed_workstream(storage)
r = server_client.post(
"/v1/api/memories",
json={
"name": "feedback_key",
"content": "data",
"description": "Feedback memory",
"type": "feedback",
"scope": "workstream",
"scope_id": "ws1",
@@ -216,17 +259,30 @@ class TestServerSaveMemory:
assert r.json()["scope"] == "workstream"
def test_missing_name(self, server_client):
r = server_client.post("/v1/api/memories", json={"content": "data"})
r = server_client.post(
"/v1/api/memories", json={"content": "data", "description": "Missing name"}
)
assert r.status_code == 400
def test_missing_content(self, server_client):
r = server_client.post("/v1/api/memories", json={"name": "k"})
r = server_client.post(
"/v1/api/memories", json={"name": "k", "description": "Missing content"}
)
assert r.status_code == 400
@pytest.mark.parametrize("description", [None, "", " "])
def test_missing_or_empty_description(self, server_client, description):
r = server_client.post(
"/v1/api/memories",
json={"name": "k", "content": "c", "description": description},
)
assert r.status_code == 400
assert "description is required" in r.json()["error"]
def test_invalid_type(self, server_client):
r = server_client.post(
"/v1/api/memories",
json={"name": "k", "content": "c", "type": "bogus"},
json=_save_body("k", "c", type="bogus"),
)
assert r.status_code == 400
assert "invalid type" in r.json()["error"]
@@ -234,7 +290,7 @@ class TestServerSaveMemory:
def test_invalid_scope(self, server_client):
r = server_client.post(
"/v1/api/memories",
json={"name": "k", "content": "c", "scope": "bogus"},
json=_save_body("k", "c", scope="bogus"),
)
assert r.status_code == 400
assert "invalid scope" in r.json()["error"]
@@ -242,7 +298,7 @@ class TestServerSaveMemory:
def test_content_too_large(self, server_client):
r = server_client.post(
"/v1/api/memories",
json={"name": "k", "content": "x" * 70000},
json=_save_body("k", "x" * 70000),
)
assert r.status_code == 400
assert "limit" in r.json()["error"]
@@ -250,18 +306,32 @@ class TestServerSaveMemory:
def test_name_normalisation(self, server_client):
r = server_client.post(
"/v1/api/memories",
json={"name": "My-Key Name", "content": "data"},
json=_save_body("My-Key Name", "data"),
)
assert r.status_code == 201
assert r.json()["name"] == "my_key_name"
def test_create_and_update_are_audited(self, server_client, storage):
first = server_client.post(
"/v1/api/memories",
json=_save_body("audit_me", "v1"),
)
second = server_client.post(
"/v1/api/memories",
json=_save_body("audit_me", "v2", description="Updated audit memory"),
)
assert first.status_code == 201
assert second.status_code == 200
assert len(storage.list_audit_events(action="memory.save", user_id="test-user")) == 1
assert len(storage.list_audit_events(action="memory.update", user_id="test-user")) == 1
class TestServerUserScopeSecurity:
def test_user_scope_binds_to_auth(self, server_client):
"""User scope auto-resolves scope_id from authenticated user."""
r = server_client.post(
"/v1/api/memories",
json={"name": "priv", "content": "secret", "scope": "user"},
json=_save_body("priv", "secret", scope="user"),
)
assert r.status_code == 201
assert r.json()["scope_id"] == "test-user"
@@ -270,7 +340,7 @@ class TestServerUserScopeSecurity:
"""Cannot access another user's memories via scope_id."""
r = server_client.post(
"/v1/api/memories",
json={"name": "x", "content": "y", "scope": "user", "scope_id": "other-user"},
json=_save_body("x", "y", scope="user", scope_id="other-user"),
)
assert r.status_code == 403
@@ -278,7 +348,7 @@ class TestServerUserScopeSecurity:
"""Passing own user_id as scope_id is allowed."""
r = server_client.post(
"/v1/api/memories",
json={"name": "x", "content": "y", "scope": "user", "scope_id": "test-user"},
json=_save_body("x", "y", scope="user", scope_id="test-user"),
)
assert r.status_code == 201
@@ -298,7 +368,7 @@ class TestServerScopeScopeIdValidation:
def test_save_global_with_scope_id_rejected(self, server_client):
r = server_client.post(
"/v1/api/memories",
json={"name": "k", "content": "c", "scope": "global", "scope_id": "ws1"},
json=_save_body("k", "c", scope="global", scope_id="ws1"),
)
assert r.status_code == 400
assert "scope_id" in r.json()["error"]
@@ -306,15 +376,16 @@ class TestServerScopeScopeIdValidation:
def test_save_workstream_without_scope_id_rejected(self, server_client):
r = server_client.post(
"/v1/api/memories",
json={"name": "k", "content": "c", "scope": "workstream"},
json=_save_body("k", "c", scope="workstream"),
)
assert r.status_code == 400
assert "scope_id is required" in r.json()["error"]
def test_save_workstream_with_scope_id_ok(self, server_client):
def test_save_workstream_with_scope_id_ok(self, server_client, storage):
_seed_workstream(storage)
r = server_client.post(
"/v1/api/memories",
json={"name": "k", "content": "c", "scope": "workstream", "scope_id": "ws1"},
json=_save_body("k", "c", scope="workstream", scope_id="ws1"),
)
assert r.status_code == 201
@@ -380,6 +451,21 @@ class TestServerSearchMemories:
r = server_client.post("/v1/api/memories/search", json={})
assert r.status_code == 400
def test_unscoped_search_is_caller_bound(self, server_client, storage):
_seed_memory(storage, "own", "needle", scope="user", scope_id="test-user")
_seed_memory(storage, "victim", "needle", scope="user", scope_id="victim")
_seed_memory(storage, "project", "needle", scope="project", scope_id="p1")
r = server_client.post("/v1/api/memories/search", json={"query": "needle"})
assert r.status_code == 200
assert {row["name"] for row in r.json()["memories"]} == {"own"}
def test_internal_scope_is_rejected(self, server_client):
r = server_client.post(
"/v1/api/memories/search",
json={"query": "x", "scope": "project", "scope_id": "p1"},
)
assert r.status_code == 400
class TestServerDeleteMemory:
def test_delete(self, server_client, storage):
@@ -393,6 +479,7 @@ class TestServerDeleteMemory:
assert r.status_code == 404
def test_delete_scoped(self, server_client, storage):
_seed_workstream(storage)
_seed_memory(storage, "k", "data", scope="workstream", scope_id="ws1")
# Wrong scope → not found
r = server_client.delete("/v1/api/memories/k")
@@ -405,6 +492,14 @@ class TestServerDeleteMemory:
r = server_client.delete("/v1/api/memories/k?scope=bogus")
assert r.status_code == 400
def test_delete_is_audited(self, server_client, storage):
mid = _seed_memory(storage, "audited")
r = server_client.delete("/v1/api/memories/audited")
assert r.status_code == 200
events = storage.list_audit_events(action="memory.delete", user_id="test-user")
assert len(events) == 1
assert events[0]["resource_id"] == mid
# ===========================================================================
# Console admin endpoint tests
@@ -492,6 +587,26 @@ class TestAdminDeleteMemory:
r = admin_client.delete("/v1/api/admin/memories/nonexistent-id")
assert r.status_code == 404
def test_no_audit_or_success_when_atomic_delete_misses(
self, admin_client, storage, monkeypatch
):
mid = _seed_memory(storage, "still_here")
monkeypatch.setattr(storage, "delete_structured_memory_by_id_returning", lambda _mid: None)
r = admin_client.delete(f"/v1/api/admin/memories/{mid}")
assert r.status_code == 404
assert storage.get_structured_memory(mid) is not None
assert storage.list_audit_events(action="memory.delete") == []
def test_storage_failure_is_500(self, admin_client, storage, monkeypatch):
def _raise(_memory_id):
raise RuntimeError("db down")
monkeypatch.setattr(storage, "delete_structured_memory_by_id_returning", _raise)
r = admin_client.delete("/v1/api/admin/memories/m1")
assert r.status_code == 500
# ===========================================================================
# Storage: delete_structured_memory_by_id
+42 -6
View File
@@ -1,7 +1,9 @@
"""Tests for turnstone.core.memory_relevance — scoring, formatting, context extraction."""
from typing import Any
from unittest.mock import patch
from turnstone.core import auth
from turnstone.core.memory_relevance import (
MemoryConfig,
build_memory_context,
@@ -298,6 +300,11 @@ def _make_session(fetch_limit: int = 5, relevance_k: int = 3, **kwargs: object):
)
def _execute_prepared_tool(session: Any, item: dict[str, Any]) -> tuple[str, str]:
item.setdefault("_principal_id", session._tool_prepare_principal_id())
return item["execute"](item)
class TestCompositionCandidateSelection:
"""Verify the query-aware candidate set in _init_system_messages."""
@@ -520,9 +527,11 @@ class TestMemorySearchToolExecution:
"""Multi-word query returns rows where ANY term matches — not all."""
from turnstone.core.memory import save_structured_memory
save_structured_memory("postgres_notes", "host=localhost port=5432")
save_structured_memory("redis_notes", "host=redis port=6379")
save_structured_memory("unrelated", "completely different")
save_structured_memory(
"postgres_notes", "host=localhost port=5432", description="Postgres notes"
)
save_structured_memory("redis_notes", "host=redis port=6379", description="Redis notes")
save_structured_memory("unrelated", "completely different", description="Unrelated notes")
session = _make_session()
item = session._prepare_memory(
@@ -532,12 +541,39 @@ class TestMemorySearchToolExecution:
# Sanity: prepare returned a search-ready dispatch (not an error item)
assert item.get("action") == "search"
call_id, msg = session._exec_memory(item)
call_id, msg = _execute_prepared_tool(session, item)
assert call_id == "call-1"
assert "postgres_notes" in msg
# Other memories don't match any query term
assert "unrelated" not in msg
def test_search_and_list_guidance_carries_the_displayed_scope(self, tmp_db, monkeypatch):
"""Follow-up guidance must not drop a project result's scope."""
from turnstone.core.memory import save_structured_memory
save_structured_memory(
"july_digest",
"project day digest",
description="July project digest",
scope="project",
scope_id="p1",
)
monkeypatch.setattr(
auth,
"resolve_project_access",
lambda *_a, **_k: auth.ProjectAccess(True, True, "P", "active"),
)
session = _make_session(user_id="u1", project_id="p1")
for args in (
{"action": "search", "query": "digest"},
{"action": "list"},
):
item = session._prepare_memory("call-1", args)
_, msg = _execute_prepared_tool(session, item)
assert "[general:project] july_digest" in msg
assert "call memory(action='get') with the displayed name and scope" in msg
class TestPerTurnSearchCache:
"""The per-turn cache spares redundant SQL across mid-turn rebuilds."""
@@ -545,7 +581,7 @@ class TestPerTurnSearchCache:
def test_repeated_search_in_same_turn_hits_cache(self, tmp_db):
from turnstone.core.memory import save_structured_memory
save_structured_memory("hello_mem", "alpha beta gamma")
save_structured_memory("hello_mem", "alpha beta gamma", description="Greeting memory")
session = _make_session()
with patch(
"turnstone.core.session.search_visible_structured_memories",
@@ -560,7 +596,7 @@ class TestPerTurnSearchCache:
def test_user_turn_invalidates_cache(self, tmp_db):
from turnstone.core.memory import save_structured_memory
save_structured_memory("hello_mem", "alpha")
save_structured_memory("hello_mem", "alpha", description="Greeting memory")
session = _make_session()
with patch(
"turnstone.core.session.search_visible_structured_memories",
+390 -81
View File
@@ -1,11 +1,4 @@
"""Phase 4: the ``project`` memory scope.
Covers construction-time access resolution (``_project_id`` / ``_project_writable``)
and its effect on recall ``_visible_scopes`` / ``_resolve_scope_id`` /
``_validate_scope`` for both interactive and coordinator sessions. The ACL is
monkeypatched (it is unit-tested in ``test_project_storage.py``); here we assert
the session wiring around it.
"""
"""Actor-scoped, live ``project`` memory authorization."""
from __future__ import annotations
@@ -35,10 +28,32 @@ def _session(**kwargs: Any) -> ChatSession:
return ChatSession(**defaults)
class TestConstructionResolvesProjectAccess:
"""Construction resolves the attached project through a single
``resolve_project_access`` call; recall is gated on read access AND a
non-archived project."""
def _execute_prepared_tool(
session: ChatSession,
item: dict[str, Any],
) -> tuple[str, str | list[dict[str, Any]]]:
item.setdefault("_principal_id", session._tool_prepare_principal_id())
return item["execute"](item)
def _project_session(
monkeypatch: pytest.MonkeyPatch,
*,
writable: bool = True,
kind: WorkstreamKind = WorkstreamKind.INTERACTIVE,
user_id: str = "u1",
) -> ChatSession:
"""Construct an attached session whose live ACL stays controllable."""
monkeypatch.setattr(
auth,
"resolve_project_access",
lambda *_a, **_k: auth.ProjectAccess(True, writable, "P", "active"),
)
return _session(user_id=user_id, ws_id="ws1", kind=kind, project_id="p1")
class TestLiveProjectAccess:
"""Each access snapshot resolves the attachment for the current actor."""
def _access(self, can_read: bool, can_write: bool, state: str = "active") -> object:
return auth.ProjectAccess(can_read, can_write, "P", state)
@@ -48,9 +63,10 @@ class TestConstructionResolvesProjectAccess:
auth, "resolve_project_access", lambda *a, **k: self._access(True, True)
)
s = _session(user_id="u1", project_id="p1")
assert s._project_id == "p1"
assert s._project_writable is True
assert s._project_name == "P"
access = s._memory_access()
assert access.project_id == "p1"
assert access.project_writable is True
assert access.project_name == "P"
def test_read_only_member(self, monkeypatch: pytest.MonkeyPatch) -> None:
# Read access but no write (e.g. a non-member reading a public project).
@@ -58,16 +74,19 @@ class TestConstructionResolvesProjectAccess:
auth, "resolve_project_access", lambda *a, **k: self._access(True, False)
)
s = _session(user_id="u1", project_id="p1")
assert s._project_id == "p1"
assert s._project_writable is False
access = s._memory_access()
assert access.project_id == "p1"
assert access.project_writable is False
def test_denied_without_access(self, monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.setattr(
auth, "resolve_project_access", lambda *a, **k: self._access(False, False)
)
s = _session(user_id="u1", project_id="p1")
assert s._project_id == ""
assert s._project_writable is False
access = s._memory_access()
assert access.attached_project_id == "p1"
assert access.project_id == ""
assert access.project_writable is False
def test_archived_project_not_recalled(self, monkeypatch: pytest.MonkeyPatch) -> None:
# Full access but archived → not recalled (the owner still reaches it via
@@ -76,13 +95,17 @@ class TestConstructionResolvesProjectAccess:
auth, "resolve_project_access", lambda *a, **k: self._access(True, True, "archived")
)
s = _session(user_id="u1", project_id="p1")
assert s._project_id == ""
assert s._project_writable is False
access = s._memory_access()
assert access.attached_project_id == "p1"
assert access.project_id == ""
assert access.project_writable is False
def test_no_project_id_is_inert(self) -> None:
s = _session(user_id="u1")
assert s._project_id == ""
assert s._project_writable is False
access = s._memory_access()
assert access.attached_project_id == ""
assert access.project_id == ""
assert access.project_writable is False
def test_unauthenticated_never_resolves(self, monkeypatch: pytest.MonkeyPatch) -> None:
# Even if the ACL would allow it, an empty user_id short-circuits before
@@ -91,13 +114,49 @@ class TestConstructionResolvesProjectAccess:
auth, "resolve_project_access", lambda *a, **k: self._access(True, True)
)
s = _session(user_id="", project_id="p1")
assert s._project_id == ""
access = s._memory_access()
assert access.attached_project_id == "p1"
assert access.project_id == ""
def test_project_display_name_uses_explicit_principal(
self, monkeypatch: pytest.MonkeyPatch
) -> None:
seen: list[tuple[str, str]] = []
def _resolve(principal_id: str, project_id: str) -> object:
seen.append((principal_id, project_id))
return self._access(True, True)
monkeypatch.setattr(auth, "resolve_project_access", _resolve)
s = _session(user_id="owner", project_id="p1")
s._acting_user_id = "stale-turn-actor"
seen.clear() # Ignore constructor-time system-context composition.
assert s.project_name_for_principal("reconnecting-viewer") == "P"
assert seen == [("reconnecting-viewer", "p1")]
def test_project_display_name_does_not_fallback_for_empty_principal(
self, monkeypatch: pytest.MonkeyPatch
) -> None:
seen: list[tuple[str, str]] = []
def _resolve(principal_id: str, project_id: str) -> object:
seen.append((principal_id, project_id))
return self._access(True, True)
monkeypatch.setattr(auth, "resolve_project_access", _resolve)
s = _session(user_id="owner", project_id="p1")
seen.clear()
assert s.project_name_for_principal("") == ""
assert seen == []
class TestProjectRecall:
def test_interactive_visible_scopes_includes_project(self) -> None:
s = _session(user_id="u1", ws_id="ws1")
s._project_id = "p1"
def test_interactive_visible_scopes_includes_project(
self, monkeypatch: pytest.MonkeyPatch
) -> None:
s = _project_session(monkeypatch)
scopes = s._visible_scopes()
assert ("project", "p1") in scopes
assert ("global", "") in scopes
@@ -107,9 +166,10 @@ class TestProjectRecall:
s = _session(user_id="u1", ws_id="ws1")
assert all(scope != "project" for scope, _ in s._visible_scopes())
def test_coordinator_adds_project_keeps_isolation(self) -> None:
s = _session(user_id="u1", kind=WorkstreamKind.COORDINATOR)
s._project_id = "p1"
def test_coordinator_adds_project_keeps_isolation(
self, monkeypatch: pytest.MonkeyPatch
) -> None:
s = _project_session(monkeypatch, kind=WorkstreamKind.COORDINATOR)
scopes = s._visible_scopes()
assert ("coordinator", "u1") in scopes
assert ("project", "p1") in scopes
@@ -118,25 +178,24 @@ class TestProjectRecall:
def test_visible_scopes_omits_empty_project(self) -> None:
s = _session(user_id="u1", ws_id="ws1")
s._project_id = ""
assert all(scope != "project" for scope, _ in s._visible_scopes())
class TestProjectScopeResolutionAndValidation:
def test_resolve_scope_id_project(self) -> None:
s = _session(user_id="u1")
s._project_id = "p1"
def test_resolve_scope_id_project(self, monkeypatch: pytest.MonkeyPatch) -> None:
s = _project_session(monkeypatch)
assert s._resolve_scope_id("project") == "p1"
def test_validate_requires_attachment(self) -> None:
def test_validate_requires_attachment(self, monkeypatch: pytest.MonkeyPatch) -> None:
s = _session(user_id="u1")
assert s._validate_scope("project", "cid") is not None # not attached → rejected
s._project_id = "p1"
assert s._validate_scope("project", "cid") is None
attached = _project_session(monkeypatch)
assert attached._validate_scope("project", "cid") is None
def test_coordinator_allows_project_rejects_global(self) -> None:
s = _session(user_id="u1", kind=WorkstreamKind.COORDINATOR)
s._project_id = "p1"
def test_coordinator_allows_project_rejects_global(
self, monkeypatch: pytest.MonkeyPatch
) -> None:
s = _project_session(monkeypatch, kind=WorkstreamKind.COORDINATOR)
assert s._validate_scope("project", "cid") is None # project allowed for coord
assert s._validate_scope("global", "cid") is not None # global still rejected
@@ -172,78 +231,328 @@ class TestProjectInSystemContext:
class TestProjectWriteGate:
"""The save AND delete memory paths block writes to a project the session
can read but not write (a read-only member of a public project). Construction
resolves ``_project_writable``; these drive the preparer to assert the gate
actually fires (the resolution-level check lives in
``TestConstructionResolvesProjectAccess``)."""
resolves live access; these drive the preparer to assert the gate actually
fires (the resolution-level check lives in ``TestLiveProjectAccess``)."""
def _attached(self, *, writable: bool) -> ChatSession:
s = _session(user_id="u1")
s._project_id = "p1"
s._project_writable = writable
return s
def _attached(self, monkeypatch: pytest.MonkeyPatch, *, writable: bool) -> ChatSession:
return _project_session(monkeypatch, writable=writable)
def test_save_blocked_when_read_only(self) -> None:
s = self._attached(writable=False)
def test_save_blocked_when_read_only(self, monkeypatch: pytest.MonkeyPatch) -> None:
s = self._attached(monkeypatch, writable=False)
out = s._prepare_memory(
"cid", {"action": "save", "scope": "project", "name": "k", "content": "v"}
"cid",
{
"action": "save",
"scope": "project",
"name": "k",
"content": "v",
"description": "Test memory",
},
)
assert "read-only access to this project" in out.get("error", "")
assert "read-only access to the attached project" in out.get("error", "")
def test_save_allowed_when_writable(self) -> None:
s = self._attached(writable=True)
def test_save_allowed_when_writable(self, monkeypatch: pytest.MonkeyPatch) -> None:
s = self._attached(monkeypatch, writable=True)
out = s._prepare_memory(
"cid", {"action": "save", "scope": "project", "name": "k", "content": "v"}
"cid",
{
"action": "save",
"scope": "project",
"name": "k",
"content": "v",
"description": "Test memory",
},
)
assert "error" not in out
assert out.get("execute") is not None # would proceed to the save exec
def test_delete_blocked_when_read_only(self) -> None:
s = self._attached(writable=False)
def test_delete_blocked_when_read_only(self, monkeypatch: pytest.MonkeyPatch) -> None:
s = self._attached(monkeypatch, writable=False)
out = s._prepare_memory("cid", {"action": "delete", "scope": "project", "name": "k"})
assert "read-only access to this project" in out.get("error", "")
assert "read-only access to the attached project" in out.get("error", "")
def test_delete_allowed_when_writable(self) -> None:
s = self._attached(writable=True)
def test_delete_allowed_when_writable(self, monkeypatch: pytest.MonkeyPatch) -> None:
s = self._attached(monkeypatch, writable=True)
out = s._prepare_memory("cid", {"action": "delete", "scope": "project", "name": "k"})
assert "error" not in out
assert out.get("execute") is not None
class TestProjectDefaultSaveScope:
"""A writable attached project becomes the DEFAULT save scope (both kinds);
a read-only or unattached session keeps the kind default."""
class TestActingPrincipalProjectAuthority:
@staticmethod
def _tool_call(call_id: str, **arguments: Any) -> dict[str, Any]:
import json
def test_writable_project_is_default(self) -> None:
s = _session(user_id="u1")
s._project_id = "p1"
s._project_writable = True
return {
"id": call_id,
"function": {"name": "memory", "arguments": json.dumps(arguments)},
}
def test_guest_cannot_inherit_owner_project_access(
self, tmp_db: str, monkeypatch: pytest.MonkeyPatch
) -> None:
def resolve(user_id: str, _project_id: str, **_kwargs: Any) -> auth.ProjectAccess:
if user_id == "owner":
return auth.ProjectAccess(True, True, "Owner Project", "active")
return auth.ProjectAccess(False, False, "", "")
monkeypatch.setattr(auth, "resolve_project_access", resolve)
session = _session(user_id="owner", ws_id="shared", project_id="p1")
session.bind_acting_user("guest")
assert all(scope != "project" for scope, _ in session._visible_scopes())
for action in ("get", "save", "delete"):
arguments: dict[str, Any] = {
"action": action,
"name": "owner_secret",
"scope": "project",
}
if action == "save":
arguments["content"] = "guest write"
arguments["description"] = "Guest write attempt"
item = session._prepare_tool(self._tool_call(action, **arguments))
assert item["_principal_id"] == "guest"
assert "error" in item
assert "acting user cannot access" in item["error"]
def test_project_delete_revalidates_prepared_principal_and_live_acl(
self, tmp_db: str, monkeypatch: pytest.MonkeyPatch
) -> None:
from turnstone.core.memory import (
get_structured_memory_by_name,
save_structured_memory,
)
guest_access = {"value": auth.ProjectAccess(True, True, "Shared", "active")}
def resolve(user_id: str, _project_id: str, **_kwargs: Any) -> auth.ProjectAccess:
if user_id == "guest":
return guest_access["value"]
return auth.ProjectAccess(True, True, "Shared", "active")
monkeypatch.setattr(auth, "resolve_project_access", resolve)
save_structured_memory(
"shared_secret",
"keep",
description="Shared project secret",
scope="project",
scope_id="p1",
)
session = _session(user_id="owner", ws_id="shared", project_id="p1")
session.bind_acting_user("guest")
item = session._prepare_tool(
self._tool_call(
"delete",
action="delete",
name="shared_secret",
scope="project",
)
)
assert "error" not in item
assert item["_principal_id"] == "guest"
guest_access["value"] = auth.ProjectAccess(False, False, "", "")
session.bind_acting_user("owner")
_, message = _execute_prepared_tool(session, item)
assert "acting user cannot access" in message
assert get_structured_memory_by_name("shared_secret", "project", "p1") is not None
def test_archived_project_is_removed_from_live_visibility(
self, tmp_db: str, monkeypatch: pytest.MonkeyPatch
) -> None:
project_state = {"value": "active"}
monkeypatch.setattr(
auth,
"resolve_project_access",
lambda *_args, **_kwargs: auth.ProjectAccess(
True, True, "Shared", project_state["value"]
),
)
session = _session(user_id="owner", ws_id="shared", project_id="p1")
assert ("project", "p1") in session._visible_scopes()
project_state["value"] = "archived"
assert all(scope != "project" for scope, _ in session._visible_scopes())
item = session._prepare_memory(
"get", {"action": "get", "name": "anything", "scope": "project"}
)
assert "error" in item
assert "active attached project" in item["error"]
class TestProjectDefaultSaveScope:
"""An attachment is the inherited target even when it is read-only."""
def test_writable_project_is_default(self, monkeypatch: pytest.MonkeyPatch) -> None:
s = _project_session(monkeypatch)
assert s._default_memory_scope() == "project"
def test_read_only_project_keeps_kind_default(self) -> None:
s = _session(user_id="u1")
s._project_id = "p1"
s._project_writable = False
assert s._default_memory_scope() == "global"
def test_read_only_project_remains_inherited_target(
self, monkeypatch: pytest.MonkeyPatch
) -> None:
s = _project_session(monkeypatch, writable=False)
assert s._default_memory_scope() == "project"
def test_no_project_keeps_kind_default(self) -> None:
assert _session(user_id="u1")._default_memory_scope() == "global"
def test_coordinator_writable_project_is_default(self) -> None:
s = _session(user_id="u1", kind=WorkstreamKind.COORDINATOR)
s._project_id = "p1"
s._project_writable = True
def test_coordinator_writable_project_is_default(self, monkeypatch: pytest.MonkeyPatch) -> None:
s = _project_session(monkeypatch, kind=WorkstreamKind.COORDINATOR)
assert s._default_memory_scope() == "project"
def test_coordinator_without_project_is_coordinator(self) -> None:
s = _session(user_id="u1", kind=WorkstreamKind.COORDINATOR)
assert s._default_memory_scope() == "coordinator"
def test_save_without_scope_lands_in_project(self) -> None:
def test_save_without_scope_lands_in_project(self, monkeypatch: pytest.MonkeyPatch) -> None:
# End-to-end: an unscoped save in a writable-project session resolves to
# scope=project / scope_id=project_id (not the global default).
s = _session(user_id="u1")
s._project_id = "p1"
s._project_writable = True
out = s._prepare_memory("cid", {"action": "save", "name": "k", "content": "v"})
s = _project_session(monkeypatch)
out = s._prepare_memory(
"cid",
{
"action": "save",
"name": "k",
"content": "v",
"description": "Test memory",
},
)
assert out.get("scope") == "project"
assert out.get("scope_id") == "p1"
class TestProjectDefaultGetDeleteScope:
"""An attached project is the inherited get/delete target.
This aligns the name-based lifecycle: a memory saved without an explicit
scope can be fetched or removed the same way while the workstream remains
attached to that project.
"""
@staticmethod
def _attached(
monkeypatch: pytest.MonkeyPatch,
*,
writable: bool = True,
kind: WorkstreamKind = WorkstreamKind.INTERACTIVE,
) -> ChatSession:
return _project_session(monkeypatch, writable=writable, kind=kind)
def test_get_without_scope_targets_project(self, monkeypatch: pytest.MonkeyPatch) -> None:
# Read access is sufficient for the inherited get target; writability
# only controls save/delete.
s = self._attached(monkeypatch, writable=False)
item = s._prepare_memory("cid", {"action": "get", "name": "k"})
assert item["scopes_to_try"] == [("project", "p1")]
def test_delete_without_scope_targets_writable_project(
self, monkeypatch: pytest.MonkeyPatch
) -> None:
s = self._attached(monkeypatch)
item = s._prepare_memory("cid", {"action": "delete", "name": "k"})
assert item["scopes_to_try"] == [("project", "p1")]
def test_delete_without_scope_rejects_read_only_project(
self, monkeypatch: pytest.MonkeyPatch
) -> None:
s = self._attached(monkeypatch, writable=False)
item = s._prepare_memory("cid", {"action": "delete", "name": "k"})
assert "read-only access to the attached project" in item.get("error", "")
def test_read_only_project_does_not_block_explicit_other_scope(
self, monkeypatch: pytest.MonkeyPatch
) -> None:
s = self._attached(monkeypatch, writable=False)
item = s._prepare_memory(
"cid",
{"action": "delete", "name": "k", "scope": "global"},
)
assert "error" not in item
assert item["scopes_to_try"] == [("global", "")]
def test_coordinator_inherits_project_too(self, monkeypatch: pytest.MonkeyPatch) -> None:
s = self._attached(monkeypatch, kind=WorkstreamKind.COORDINATOR)
get_item = s._prepare_memory("get", {"action": "get", "name": "k"})
delete_item = s._prepare_memory("delete", {"action": "delete", "name": "k"})
assert get_item["scopes_to_try"] == [("project", "p1")]
assert delete_item["scopes_to_try"] == [("project", "p1")]
def test_unscoped_get_and_delete_round_trip_project_memory(
self, tmp_db: str, monkeypatch: pytest.MonkeyPatch
) -> None:
from turnstone.core.memory import (
get_structured_memory_by_name,
save_structured_memory,
)
row, _ = save_structured_memory(
"july_digest",
"full digest",
description="July digest",
scope="project",
scope_id="p1",
)
assert row is not None
s = self._attached(monkeypatch)
get_item = s._prepare_memory("get", {"action": "get", "name": "july_digest"})
_, get_msg = _execute_prepared_tool(s, get_item)
assert "[general:project] july_digest" in get_msg
assert "full digest" in get_msg
delete_item = s._prepare_memory("delete", {"action": "delete", "name": "july_digest"})
_, delete_msg = _execute_prepared_tool(s, delete_item)
assert "Deleted memory 'july_digest' (scope=project)" in delete_msg
assert get_structured_memory_by_name("july_digest", "project", "p1") is None
def test_wrong_explicit_scope_hints_at_attached_project(
self, tmp_db: str, monkeypatch: pytest.MonkeyPatch
) -> None:
from turnstone.core.memory import save_structured_memory
row, _ = save_structured_memory(
"july_digest",
"full digest",
description="July digest",
scope="project",
scope_id="p1",
)
assert row is not None
s = self._attached(monkeypatch)
for action in ("get", "delete"):
item = s._prepare_memory(
action,
{"action": action, "name": "july_digest", "scope": "global"},
)
_, msg = _execute_prepared_tool(s, item)
assert "not found (scope=global)" in msg
assert "exists in scope='project'" in msg
assert "retry with scope='project'" in msg
def test_project_default_miss_hints_at_other_visible_scope(
self, tmp_db: str, monkeypatch: pytest.MonkeyPatch
) -> None:
from turnstone.core.memory import save_structured_memory
row, _ = save_structured_memory(
"shared_runbook",
"global content",
description="Shared runbook",
scope="global",
)
assert row is not None
s = self._attached(monkeypatch)
for action in ("get", "delete"):
item = s._prepare_memory(
action,
{"action": action, "name": "shared_runbook"},
)
_, msg = _execute_prepared_tool(s, item)
assert "not found (scope=project)" in msg
assert "exists in scope='global'" in msg
assert "retry with scope='global'" in msg
+3 -3
View File
@@ -68,9 +68,9 @@ class TestProjectStore:
# a sibling project's nor other scopes' rows.
backend.create_project("p1", "A", "u1")
backend.create_project("p2", "B", "u1")
backend.create_structured_memory("m1", "k", "", "general", "project", "p1", "v")
backend.create_structured_memory("m2", "k", "", "general", "project", "p2", "v")
backend.create_structured_memory("m3", "k", "", "general", "user", "u1", "v")
backend.create_structured_memory("m1", "k", "Test memory", "general", "project", "p1", "v")
backend.create_structured_memory("m2", "k", "Test memory", "general", "project", "p2", "v")
backend.create_structured_memory("m3", "k", "Test memory", "general", "user", "u1", "v")
assert backend.delete_project("p1")
assert backend.get_structured_memory("m1") is None # purged
assert backend.get_structured_memory("m2") is not None # sibling project intact
+56
View File
@@ -361,6 +361,62 @@ async def test_logout():
assert resp.status == "ok"
# ---------------------------------------------------------------------------
# Memories
# ---------------------------------------------------------------------------
@pytest.mark.anyio
async def test_save_memory_requires_and_sends_description():
captured: dict = {}
def handler(request: httpx.Request) -> httpx.Response:
captured.update(json.loads(request.content))
return _json_response(
{
"memory_id": "m1",
"name": "deployment_process",
"description": captured["description"],
"type": "general",
"scope": "global",
"scope_id": "",
"content": "Deploy from main",
"created": "2026-08-11T00:00:00",
"updated": "2026-08-11T00:00:00",
},
status=201,
)
transport = httpx.MockTransport(handler)
async with httpx.AsyncClient(transport=transport, base_url="http://test") as hc:
client = AsyncTurnstoneServer(httpx_client=hc)
memory = await client.save_memory(
"deployment_process",
"Deploy from main",
description=" Production deployment workflow ",
)
assert captured["description"] == "Production deployment workflow"
assert memory.description == "Production deployment workflow"
@pytest.mark.anyio
@pytest.mark.parametrize("description", [None, "", " "])
async def test_save_memory_rejects_empty_description(description):
def unexpected_request(_request: httpx.Request) -> httpx.Response:
raise AssertionError("invalid memory must not reach the server")
transport = httpx.MockTransport(unexpected_request)
async with httpx.AsyncClient(transport=transport, base_url="http://test") as hc:
client = AsyncTurnstoneServer(httpx_client=hc)
with pytest.raises(ValueError, match="description is required"):
await client.save_memory(
"deployment_process",
"Deploy from main",
description=description, # type: ignore[arg-type]
)
# ---------------------------------------------------------------------------
# Health
# ---------------------------------------------------------------------------
+219 -56
View File
@@ -1,17 +1,19 @@
"""Offline pins of the SDK boundary behaviors the #937 retry design rests on.
Three facts, each probed against the REAL SDKs over mock/loopback
Four facts, each probed against the REAL SDKs over mock/loopback
transports (no network, no live backend):
1. The OpenAI SDK's ``max_retries`` covers request time only — a
mid-BODY death produces no re-request, and the raw ``httpx.ReadError``
escapes the chunk iterator unwrapped.
2. The Anthropic ``messages.stream()`` helper propagates the same shape.
3. Closing an httpx-backed SDK client from another thread while a read is
blocked (the ``ModelRegistry.reload()`` shape) surfaces as an
``httpx.TransportError`` on the blocked ``next()`` which is why
``_stream_response``'s mid-stream re-issue ladder re-resolves the
registry binding before re-creating.
1. OpenAI v3's ``max_retries`` covers request time only — a mid-BODY death
produces no re-request, and the raw ``httpx2.ReadError`` escapes both Chat
Completions and Responses chunk iterators unwrapped.
2. OpenAI v3's runtime-only legacy-client path preserves the old ``httpx``
exception family when an application explicitly injects that client.
3. The Anthropic ``messages.stream()`` helper propagates the ``httpx`` shape.
4. Closing an OpenAI v3 default client from another thread while a read is
blocked (the ``ModelRegistry.reload()`` shape) completes safely; a later
wire release surfaces as an ``httpx2.TransportError`` on the blocked
``next()``. The production ``transport_guarded`` seam must normalize it
before the retry gate.
If an SDK/httpx upgrade changes any of these, the ``transport_guarded``
conversion (and the retry gate consuming it) must be re-verified these
@@ -27,6 +29,7 @@ import time
import anthropic
import httpx
import httpx2
import openai
import pytest
@@ -39,6 +42,12 @@ CHAT_CHUNK = (
'"finish_reason":null}]}' + LF + LF
)
RESPONSES_EVENT = (
'data: {"type":"response.output_text.delta","sequence_number":0,'
'"item_id":"item_1","output_index":0,"content_index":0,'
'"delta":"hello","logprobs":[]}' + LF + LF
)
ANTHROPIC_EVENTS = (
"event: message_start"
+ LF
@@ -71,6 +80,17 @@ class _DyingStream(httpx.SyncByteStream):
raise httpx.ReadError("[SSL] record layer failure (_ssl.c:2590)")
class _Httpx2DyingStream(httpx2.SyncByteStream):
"""HTTPX2 response body: one SSE payload, then a wire death."""
def __init__(self, payload: bytes) -> None:
self._payload = payload
def __iter__(self):
yield self._payload
raise httpx2.ReadError("[SSL] record layer failure (_ssl.c:2590)")
def _dying_transport(payload: str, requests: list) -> httpx.MockTransport:
def handler(request: httpx.Request) -> httpx.Response:
requests.append(request)
@@ -84,12 +104,70 @@ def _dying_transport(payload: str, requests: list) -> httpx.MockTransport:
return httpx.MockTransport(handler)
def test_openai_midbody_death_is_unwrapped_readerror_and_no_rerequest():
def _httpx2_dying_transport(payload: str, requests: list) -> httpx2.MockTransport:
def handler(request: httpx2.Request) -> httpx2.Response:
requests.append(request)
return httpx2.Response(
200,
headers={"content-type": "text/event-stream"},
stream=_Httpx2DyingStream(payload.encode()),
request=request,
)
return httpx2.MockTransport(handler)
def test_openai_v3_chat_midbody_death_is_unwrapped_httpx2_error_and_no_rerequest():
requests: list = []
client = openai.OpenAI(
api_key="probe",
base_url="http://probe.invalid/v1",
http_client=httpx.Client(transport=_dying_transport(CHAT_CHUNK, requests)),
http_client=httpx2.Client(transport=_httpx2_dying_transport(CHAT_CHUNK, requests)),
max_retries=2,
)
stream = client.chat.completions.create(
model="m", messages=[{"role": "user", "content": "hi"}], stream=True
)
texts = []
with pytest.raises(httpx2.ReadError) as excinfo:
for chunk in stream:
if chunk.choices and chunk.choices[0].delta.content:
texts.append(chunk.choices[0].delta.content)
# The retry gate matches on the class NAME; pin the exact identity the
# SDK lets escape, and that it is the HTTPX2 transport family.
assert type(excinfo.value).__name__ == "ReadError"
assert isinstance(excinfo.value, httpx2.TransportError)
assert texts == ["hello"] # the request succeeded; the BODY died
assert len(requests) == 1 # max_retries never re-requested mid-body
def test_openai_v3_responses_midbody_death_is_unwrapped_httpx2_error_and_no_rerequest():
requests: list = []
client = openai.OpenAI(
api_key="probe",
base_url="http://probe.invalid/v1",
http_client=httpx2.Client(transport=_httpx2_dying_transport(RESPONSES_EVENT, requests)),
max_retries=2,
)
stream = client.responses.create(model="m", input="hi", stream=True)
texts = []
with pytest.raises(httpx2.ReadError) as excinfo:
for event in stream:
if event.type == "response.output_text.delta":
texts.append(event.delta)
assert type(excinfo.value).__name__ == "ReadError"
assert isinstance(excinfo.value, httpx2.TransportError)
assert texts == ["hello"]
assert len(requests) == 1
def test_openai_v3_legacy_httpx_midbody_death_keeps_legacy_error_family():
requests: list = []
legacy_http_client = httpx.Client(transport=_dying_transport(CHAT_CHUNK, requests))
client = openai.OpenAI(
api_key="probe",
base_url="http://probe.invalid/v1",
http_client=legacy_http_client, # type: ignore[arg-type]
max_retries=2,
)
stream = client.chat.completions.create(
@@ -100,12 +178,10 @@ def test_openai_midbody_death_is_unwrapped_readerror_and_no_rerequest():
for chunk in stream:
if chunk.choices and chunk.choices[0].delta.content:
texts.append(chunk.choices[0].delta.content)
# The retry gate matches on the class NAME; pin the exact identity the
# SDK lets escape, and that it is the httpx transport family.
assert type(excinfo.value).__name__ == "ReadError"
assert isinstance(excinfo.value, httpx.TransportError)
assert texts == ["hello"] # the request succeeded; the BODY died
assert len(requests) == 1 # max_retries never re-requested mid-body
assert texts == ["hello"]
assert len(requests) == 1
def test_anthropic_midbody_death_is_unwrapped_readerror_and_no_rerequest():
@@ -136,13 +212,59 @@ def test_anthropic_midbody_death_is_unwrapped_readerror_and_no_rerequest():
assert len(requests) == 1
def test_cross_thread_client_close_surfaces_transport_error_to_blocked_read():
"""The ``ModelRegistry.reload()`` shape: an admin-thread ``client.close()``
under a worker blocked in ``next()`` must surface as an
``httpx.TransportError`` (so ``transport_guarded`` converts it and the
retry gate passes) not as a plain ``RuntimeError`` the gate would
treat as fatal."""
body = f"{len(CHAT_CHUNK):x}" + CRLF + CHAT_CHUNK + CRLF
@pytest.mark.parametrize("surface", ["chat", "responses"])
def test_closed_v3_default_client_creation_stays_sdk_wrapped_and_unarmed(surface: str):
"""A re-create on the client closed by reload is still a creation error.
OpenAI v3 must wrap the default HTTPX2 client's ``RuntimeError`` as its
retryable ``APIConnectionError`` before either adapter can arm the stream.
This preserves the creation-vs-mid-stream classifier while the original
normalized stream death remains available to the outer re-issue ladder.
"""
from turnstone.core.providers import create_provider
client = openai.OpenAI(
api_key="probe",
base_url="http://probe.invalid/v1",
max_retries=0,
)
client.close()
cancel_ref: list = []
provider = create_provider("openai-compatible", api_surface=surface)
with pytest.raises(openai.APIConnectionError) as excinfo:
provider.create_streaming(
client=client,
model="m",
messages=[{"role": "user", "content": "hi"}],
cancel_ref=cancel_ref,
)
assert cancel_ref == []
assert type(excinfo.value).__name__ in provider.retryable_error_names
assert type(excinfo.value.__cause__) is RuntimeError
@pytest.mark.parametrize(
("surface", "payload"),
[("chat", CHAT_CHUNK), ("responses", RESPONSES_EVENT)],
)
def test_cross_thread_v3_default_client_close_then_wire_release_is_normalized(
surface: str, payload: str
):
"""The ``ModelRegistry.reload()`` shape stays safe at the retry seam.
OpenAI v3's synchronous HTTPX2 client does not promise that cross-thread
``close()`` itself interrupts a blocked body read. Pin the behavior
Turnstone needs instead: closing from the admin thread completes safely
while a worker is in ``next()``, and the subsequent wire release reaches
``transport_guarded`` as a provider-retryable ``IncompleteStreamError``
for both OpenAI streaming adapters.
"""
from turnstone.core.providers import create_provider, transport_guarded
from turnstone.core.providers._protocol import IncompleteStreamError
body = f"{len(payload):x}" + CRLF + payload + CRLF
response_head = (
"HTTP/1.1 200 OK"
+ CRLF
@@ -153,59 +275,100 @@ def test_cross_thread_client_close_surfaces_transport_error_to_blocked_read():
+ CRLF
)
listener = socket.create_server(("127.0.0.1", 0))
listener.settimeout(10.0)
port = listener.getsockname()[1]
client_closed = threading.Event()
release_peer = threading.Event()
reader_blocked = threading.Event()
close_done = threading.Event()
first_content: list[str] = []
reader_errors: list[BaseException] = []
closer_errors: list[BaseException] = []
server_errors: list[BaseException] = []
def serve() -> None:
conn, _ = listener.accept()
conn.recv(65536)
conn.sendall((response_head + body).encode())
# Hold the connection (no second chunk) so the reader blocks, and
# release only once the client has been closed under it.
client_closed.wait(timeout=10.0)
conn.close()
try:
conn, _ = listener.accept()
with conn:
conn.settimeout(10.0)
conn.recv(65536)
conn.sendall((response_head + body).encode())
# Keep the peer open through close_done: any reader error
# before release_peer is therefore caused by close(), not EOF.
if not release_peer.wait(timeout=15.0):
raise AssertionError("peer release was never signalled")
except BaseException as exc:
server_errors.append(exc)
client = openai.OpenAI(api_key="probe", base_url=f"http://127.0.0.1:{port}/v1", max_retries=0)
client = openai.OpenAI(
api_key="probe",
base_url=f"http://127.0.0.1:{port}/v1",
max_retries=0,
timeout=5.0,
)
def read_stream() -> None:
try:
provider = create_provider("openai-compatible", api_surface=surface)
chunks = provider.create_streaming(
client=client,
model="m",
messages=[{"role": "user", "content": "hi"}],
)
it = transport_guarded(chunks)
first_content.append(next(it).content_delta)
reader_blocked.set()
next(it)
except BaseException as exc:
reader_errors.append(exc)
def closer() -> None:
reader_blocked.wait(timeout=10.0)
time.sleep(0.5) # let the reader enter the blocking socket read
client.close()
client_closed.set()
try:
if not reader_blocked.wait(timeout=10.0):
raise AssertionError("reader never reached the blocked body read")
time.sleep(0.5) # let the reader enter the blocking socket read
client.close()
except BaseException as exc:
closer_errors.append(exc)
finally:
close_done.set()
server_thread = threading.Thread(target=serve)
reader_thread = threading.Thread(target=read_stream)
closer_thread = threading.Thread(target=closer)
server_thread.start()
reader_thread.start()
closer_thread.start()
try:
stream = client.chat.completions.create(
model="m", messages=[{"role": "user", "content": "hi"}], stream=True
)
it = iter(stream)
first = next(it) # the one sent chunk arrives; the wire then idles
assert first.choices[0].delta.content == "hello"
reader_blocked.set()
with pytest.raises(httpx.TransportError) as excinfo:
next(it) # blocked read, killed by the cross-thread close()
# Platform/timing-dependent: the killed read surfaces as ReadError
# (EBADF from the blocked recv) or, where the reader observes EOF
# first, RemoteProtocolError (chunked body never terminated). Both
# are TransportError members of _BACKEND_STREAM_EXC_NAMES, so
# transport_guarded converts either and the retry gate passes — the
# property this pin exists for.
assert type(excinfo.value).__name__ in {"ReadError", "RemoteProtocolError"}
assert reader_blocked.wait(timeout=10.0)
assert close_done.wait(timeout=10.0)
assert closer_errors == []
# The server has not closed its peer yet, proving close() completed
# safely rather than merely returning after an EOF unblocked it.
assert server_errors == []
release_peer.set()
reader_thread.join(timeout=10.0)
finally:
# Unblock and join both threads on every exit path so nothing
# Unblock and join every thread on every exit path so nothing
# outlives the test (leaked-thread guard).
reader_blocked.set()
client_closed.set()
release_peer.set()
closer_thread.join(timeout=10.0)
reader_thread.join(timeout=10.0)
server_thread.join(timeout=10.0)
listener.close()
with contextlib.suppress(Exception):
client.close()
assert first_content == ["hello"]
assert len(reader_errors) == 1
exc = reader_errors[0]
assert isinstance(exc, IncompleteStreamError)
assert any(name in str(exc) for name in ("ReadError", "RemoteProtocolError"))
cause = exc.__cause__
assert isinstance(cause, httpx2.TransportError)
assert type(cause).__name__ in {"ReadError", "RemoteProtocolError"}
assert server_errors == []
assert not closer_thread.is_alive()
assert not reader_thread.is_alive()
assert not server_thread.is_alive()
@@ -240,7 +403,7 @@ class TestEagerAppendContract:
requests: list = []
client = openai.OpenAI(
api_key="probe",
http_client=httpx.Client(transport=_dying_transport(CHAT_CHUNK, requests)),
http_client=httpx2.Client(transport=_httpx2_dying_transport(CHAT_CHUNK, requests)),
)
self._armed_at_return(OpenAIChatCompletionsProvider(), client)
assert len(requests) == 1 # the HTTP call happened inside create
@@ -251,7 +414,7 @@ class TestEagerAppendContract:
requests: list = []
client = openai.OpenAI(
api_key="probe",
http_client=httpx.Client(transport=_dying_transport(CHAT_CHUNK, requests)),
http_client=httpx2.Client(transport=_httpx2_dying_transport(RESPONSES_EVENT, requests)),
)
self._armed_at_return(OpenAIResponsesProvider(), client)
assert len(requests) == 1
+9 -11
View File
@@ -1435,28 +1435,26 @@ class TestInteractiveEventsLifted:
section.
"""
def test_events_replay_yields_connected_first(self):
"""Pre-lift ``events_sse`` yielded a ``connected`` event
first (model + skip_permissions). The lifted callback
preserves the order so client SSE handlers that key on
the connected event for state setup keep working."""
from turnstone.server import _interactive_events_replay
def test_shared_preamble_yields_connected_first(self):
"""The shared handler's preamble preserves connected-event shape."""
from turnstone.core.session_replay import session_replay_preamble
ws, ui, request = _make_interactive_replay_mocks()
out = list(_interactive_events_replay(ws, ui, request))
out = list(session_replay_preamble(ws.session, ui, project_name="Visible Project"))
assert out[0]["type"] == "connected"
assert out[0]["model"] == "gpt-5"
assert out[0]["model_alias"] == "default"
assert out[0]["project_name"] == "Visible Project"
assert out[0]["skip_permissions"] is False
def test_events_replay_includes_status_only_when_last_usage_present(self):
def test_shared_preamble_includes_status_only_when_last_usage_present(self):
"""The ``status`` event populates the per-tab token-usage
bar on resume. Skipped when ``session._last_usage`` is None
(a freshly-created workstream that hasn't completed a turn)."""
from turnstone.server import _interactive_events_replay
from turnstone.core.session_replay import session_replay_preamble
ws, ui, request = _make_interactive_replay_mocks()
out = list(_interactive_events_replay(ws, ui, request))
ws, ui, _request = _make_interactive_replay_mocks()
out = list(session_replay_preamble(ws.session, ui))
assert "status" not in {ev["type"] for ev in out}
def test_events_replay_yields_pending_approval_then_verdicts(self):
+403 -72
View File
@@ -29,7 +29,12 @@ from turnstone.core.model_turn import (
provider_extra_params,
serialized_tool_chars,
)
from turnstone.core.session import _IMAGE_EXTENSIONS, _IMAGE_SIZE_CAP, ChatSession
from turnstone.core.session import (
_IMAGE_EXTENSIONS,
_IMAGE_SIZE_CAP,
_MEMORY_MIXED_BATCH_ERROR,
ChatSession,
)
from turnstone.core.trajectory import (
Role,
Turn,
@@ -129,6 +134,15 @@ def _make_session(
return session
def _execute_prepared_tool(
session: ChatSession,
item: dict[str, Any],
) -> tuple[str, str | list[dict[str, Any]]]:
"""Mirror the dispatch boundary for tests that call a preparer directly."""
item.setdefault("_principal_id", session._tool_prepare_principal_id())
return item["execute"](item)
@contextlib.contextmanager
def _send_with_mocks(session, responses, mock_execute, **extra_patches):
"""Stand up the mock context that the queued-message ``send()`` tests share.
@@ -458,12 +472,20 @@ class TestTaskExec:
"func_name": "task_agent",
"needs_approval": True,
"execute": execute,
"_needs_origin_context": True,
"_requires_fresh_system_prefix": True,
}
def prepare(_tool_call):
seen["prepare"] = session._tool_prepare_principal_id()
return item
judge = MagicMock(side_effect=evaluate_intent)
with (
patch.object(session, "_safe_prepare_tool", return_value=item),
patch.object(session, "_safe_prepare_tool", side_effect=prepare),
patch.object(session, "_evaluate_intent", judge),
patch.object(session.ui, "approve_tools", side_effect=approve_tools),
patch.object(session, "_ensure_system_prefix_fresh") as ensure_system_prefix,
):
session._execute_tools(
[{"id": "c1", "function": {"name": "task_agent", "arguments": "{}"}}],
@@ -471,9 +493,14 @@ class TestTaskExec:
my_generation=generation,
)
assert seen["prepare"] == "user-a"
assert seen["worker"] == "user-a"
assert seen["generation"] == generation
assert seen["event"] is generation_event
ensure_system_prefix.assert_called_once_with(
principal_id="user-a",
origin_generation=generation,
)
assert judge.call_args.kwargs["principal_id"] == "user-a"
assert seen["execution_item"] is not item
approval_witness = seen["approval_item"]["_approval_cancel_witness"]
@@ -6199,6 +6226,96 @@ class TestSafePrepareTool:
assert "RuntimeError" in output
class TestToolBatchPolicy:
@staticmethod
def _tool_calls(count: int) -> list[dict[str, Any]]:
return [
{
"id": f"call_{index}",
"function": {"name": "state_tool", "arguments": "{}"},
}
for index in range(count)
]
def test_mixed_read_write_batch_is_rejected(self, tmp_db):
session = _make_session()
executed: list[str] = []
def execute(item):
executed.append(item["call_id"])
return item["call_id"], "unexpected"
items = [
{
"call_id": "call_0",
"func_name": "state_tool",
"execute": execute,
"needs_approval": False,
"_batch_policy": {
"group": "state",
"access": "write",
"mixed_access_error": "Error: state reads and writes cannot run together",
"serialize": True,
},
},
{
"call_id": "call_1",
"func_name": "state_tool",
"execute": execute,
"needs_approval": False,
"_batch_policy": {
"group": "state",
"access": "read",
"mixed_access_error": "Error: state reads and writes cannot run together",
},
},
]
with (
patch.object(session, "_safe_prepare_tool", side_effect=items),
patch.object(session.ui, "approve_tools", return_value=(True, None)),
):
results, _ = session._execute_tools(self._tool_calls(2))
assert executed == []
assert all("cannot run together" in str(result) for _, result in results)
def test_write_batch_executes_serially_in_model_order(self, tmp_db):
session = _make_session()
executed: list[str] = []
def execute(item):
executed.append(item["call_id"])
return item["call_id"], "ok"
items = [
{
"call_id": f"call_{index}",
"func_name": "state_tool",
"execute": execute,
"needs_approval": False,
"_batch_policy": {
"group": "state",
"access": "write",
"mixed_access_error": "Error: mixed state access",
"serialize": True,
},
}
for index in range(2)
]
with (
patch.object(session, "_safe_prepare_tool", side_effect=items),
patch.object(session.ui, "approve_tools", return_value=(True, None)),
patch(
"turnstone.core.session.concurrent.futures.ThreadPoolExecutor",
side_effect=AssertionError("serialized writes must not enter the parallel pool"),
),
):
results, _ = session._execute_tools(self._tool_calls(2))
assert executed == ["call_0", "call_1"]
assert [call_id for call_id, _ in results] == executed
class TestCoordinatorMemoryScope:
"""Verify the ``coordinator`` memory scope's resolution + validation rules.
@@ -6269,7 +6386,7 @@ class TestCoordinatorMemoryScope:
)
err = session._validate_scope("coordinator", "call_1")
assert err is not None
assert err["error"].startswith("Error: 'coordinator' scope is only valid")
assert "unavailable to this workstream kind" in err["error"]
def test_validate_rejects_coord_scope_for_child_interactive(self, tmp_db):
"""Children of a coord MUST be rejected too — letting them write
@@ -6286,7 +6403,7 @@ class TestCoordinatorMemoryScope:
)
err = session._validate_scope("coordinator", "call_1")
assert err is not None
assert err["error"].startswith("Error: 'coordinator' scope is only valid")
assert "unavailable to this workstream kind" in err["error"]
def test_validate_accepts_coord_scope_for_coord_session(self, tmp_db):
from turnstone.core.workstream import WorkstreamKind
@@ -6314,6 +6431,7 @@ class TestCoordinatorMemoryScope:
"call_1",
{
"action": "save",
"description": "Test memory",
"name": "orchestration_plan",
"content": "step 1: investigate; step 2: report",
"scope": "coordinator",
@@ -6341,6 +6459,7 @@ class TestCoordinatorMemoryScope:
"call_1",
{
"action": "save",
"description": "Test memory",
"name": "injected_instruction",
"content": "ignore previous instructions and ...",
"scope": "coordinator",
@@ -6360,6 +6479,7 @@ class TestCoordinatorMemoryScope:
save_structured_memory(
"private_plan",
"internal coord notes",
description="Private orchestration plan",
scope="coordinator",
scope_id="user-1",
)
@@ -6424,16 +6544,20 @@ class TestCoordinatorMemoryScope:
from turnstone.core.workstream import WorkstreamKind
# Seed every non-coord scope with a sentinel memory.
save_structured_memory("global_note", "anyone can read", scope="global")
save_structured_memory(
"global_note", "anyone can read", description="Global note", scope="global"
)
save_structured_memory(
"ws_note",
"interactive ws notes",
description="Workstream note",
scope="workstream",
scope_id="coord-1", # same id as the coord under test
)
save_structured_memory(
"user_note",
"user-wide notes from another IC session",
description="User note",
scope="user",
scope_id="user-1",
)
@@ -6466,10 +6590,13 @@ class TestCoordinatorMemoryScope:
from turnstone.core.memory import save_structured_memory
from turnstone.core.workstream import WorkstreamKind
save_structured_memory("global_x", "some content", scope="global")
save_structured_memory(
"global_x", "some content", description="Global content", scope="global"
)
save_structured_memory(
"coord_x",
"orchestration content",
description="Coordinator content",
scope="coordinator",
scope_id="user-1",
)
@@ -6497,14 +6624,11 @@ class TestCoordinatorMemoryScope:
for bad in ("global", "workstream", "user"):
err = coord._validate_scope(bad, "call_1")
assert err is not None, f"coord should reject scope={bad!r}"
assert f"'{bad}' scope is not available" in err["error"]
assert f"scope '{bad}' is unavailable" in err["error"]
def test_coord_default_save_scope_is_coordinator(self, tmp_db):
"""Coord sessions calling memory(action='save') without an
explicit scope default to 'coordinator' anything else would
either land in a namespace the coord can't read back from
(workstream/user) or fall back to global which the new
visibility rules also exclude."""
explicit scope target the coordinator namespace."""
from turnstone.core.workstream import WorkstreamKind
coord = _make_session(
@@ -6514,17 +6638,20 @@ class TestCoordinatorMemoryScope:
)
item = coord._prepare_memory(
"call_1",
{"action": "save", "name": "auto_scope", "content": "x"},
{
"action": "save",
"name": "auto_scope",
"content": "x",
"description": "Automatic scope test",
},
)
assert "error" not in item
assert item["scope"] == "coordinator"
assert item["scope_id"] == "user-1"
def test_coord_implicit_walk_only_coordinator(self, tmp_db):
def test_coord_inherited_get_targets_only_coordinator(self, tmp_db):
"""Coord ``memory(action='get')`` with no explicit scope must
walk only the coordinator scope the IC walk
(workstream user global) would be wasted lookups against
rows the coord can't see."""
target only the coordinator scope."""
from turnstone.core.workstream import WorkstreamKind
coord = _make_session(
@@ -6539,10 +6666,8 @@ class TestCoordinatorMemoryScope:
assert "error" not in item
assert [s for s, _ in item["scopes_to_try"]] == ["coordinator"]
def test_ic_implicit_walk_unchanged(self, tmp_db):
"""Interactive sessions retain the narrowest-to-widest walk:
workstream user global. Coord scope is excluded IC
sessions can't see/write it anyway."""
def test_ic_unscoped_get_uses_single_global_target(self, tmp_db):
"""Without a project, interactive save/get/delete all inherit global."""
from turnstone.core.workstream import WorkstreamKind
ic = _make_session(
@@ -6555,8 +6680,7 @@ class TestCoordinatorMemoryScope:
{"action": "get", "name": "anything"},
)
assert "error" not in item
scopes = [s for s, _ in item["scopes_to_try"]]
assert scopes == ["workstream", "user", "global"]
assert item["scopes_to_try"] == [("global", "")]
def test_coord_memory_persists_across_sessions(self, tmp_db):
"""End-to-end through the real save lane: a memory saved by one
@@ -6575,13 +6699,14 @@ class TestCoordinatorMemoryScope:
"call_1",
{
"action": "save",
"description": "Test memory",
"name": "deploy_runbook",
"content": "drain node before rotating certs",
"scope": "coordinator",
},
)
assert "error" not in item
result = item["execute"](item)
result = _execute_prepared_tool(first, item)
assert "Saved" in str(result) or "saved" in str(result).lower()
# Brand-new coordinator session, new ws_id, same user.
@@ -6595,7 +6720,7 @@ class TestCoordinatorMemoryScope:
{"action": "get", "name": "deploy_runbook"},
)
assert "error" not in get_item
out = str(get_item["execute"](get_item))
out = str(_execute_prepared_tool(second, get_item))
assert "drain node before rotating certs" in out
def test_coordinator_session_requires_user_id(self, tmp_db):
@@ -6636,11 +6761,17 @@ class TestCoordinatorMemoryScope:
coord._user_id = "" # simulate a constructor-bypassing double
err = coord._validate_scope("coordinator", "call_1")
assert err is not None
assert "requires authenticated user identity" in err["error"]
assert "requires an authenticated acting user" in err["error"]
assert coord._coordinator_scope_id() == ""
item = coord._prepare_memory(
"call_1",
{"action": "save", "name": "x", "content": "y", "scope": "coordinator"},
{
"action": "save",
"name": "x",
"content": "y",
"description": "Authentication backstop test",
"scope": "coordinator",
},
)
assert "error" in item
@@ -6654,6 +6785,7 @@ class TestCoordinatorMemoryScope:
save_structured_memory(
"other_users_row",
"must not leak",
description="Another user's row",
scope="coordinator",
scope_id="user-9",
)
@@ -6682,7 +6814,48 @@ class TestMemoryToolAudit:
return get_storage().list_audit_events(action=action)
def test_save_new_emits_memory_save(self, tmp_db):
def test_preparer_declares_generic_batch_policy(self, tmp_db):
session = _make_session(ws_id="ws-1", user_id="user-1")
saved = session._prepare_memory(
"save-call",
{
"action": "save",
"name": "fact_one",
"content": "alpha content",
"description": "Alpha fact",
},
)
fetched = session._prepare_memory(
"get-call",
{"action": "get", "name": "fact_one"},
)
assert saved["_batch_policy"] == {
"group": "memory",
"access": "write",
"mixed_access_error": _MEMORY_MIXED_BATCH_ERROR,
"serialize": True,
}
assert fetched["_batch_policy"] == {
"group": "memory",
"access": "read",
"mixed_access_error": _MEMORY_MIXED_BATCH_ERROR,
"serialize": False,
}
def test_unstamped_executor_does_not_inherit_session_actor(self, tmp_db):
session = _make_session(ws_id="ws-1", user_id="user-1")
item = session._prepare_memory(
"get-call",
{"action": "get", "name": "private_fact", "scope": "user"},
)
_call_id, message = item["execute"](item)
assert "requires an authenticated acting user" in message
@pytest.mark.parametrize("description", [None, "", " "])
def test_save_requires_non_empty_description(self, tmp_db, description):
session = _make_session(ws_id="ws-1", user_id="user-1")
item = session._prepare_memory(
"call_1",
@@ -6690,12 +6863,27 @@ class TestMemoryToolAudit:
"action": "save",
"name": "fact_one",
"content": "alpha content",
"description": description,
},
)
assert "error" in item
assert "description' must be non-empty" in item["error"]
def test_save_new_emits_memory_save(self, tmp_db):
session = _make_session(ws_id="ws-1", user_id="user-1")
item = session._prepare_memory(
"call_1",
{
"action": "save",
"description": "Test memory",
"name": "fact_one",
"content": "alpha content",
"scope": "user",
"type": "reference",
},
)
assert "error" not in item
session._exec_memory(item)
_execute_prepared_tool(session, item)
rows = self._audit_rows("memory.save")
assert len(rows) == 1
@@ -6712,6 +6900,71 @@ class TestMemoryToolAudit:
# The "create" path must NOT also stamp an update row.
assert self._audit_rows("memory.update") == []
def test_prepared_user_save_stays_bound_to_acting_principal(self, tmp_db):
from turnstone.core.memory import get_structured_memory_by_name
session = _make_session(ws_id="shared", user_id="owner")
session.bind_acting_user("guest")
item = session._prepare_tool(
{
"id": "call_1",
"function": {
"name": "memory",
"arguments": json.dumps(
{
"action": "save",
"description": "Test memory",
"name": "private_note",
"content": "guest content",
"scope": "user",
}
),
},
}
)
assert item["_principal_id"] == "guest"
assert item["scope_id"] == "guest"
session.bind_acting_user("owner")
_, message = _execute_prepared_tool(session, item)
assert "Saved memory" in message
assert get_structured_memory_by_name("private_note", "user", "guest") is not None
assert get_structured_memory_by_name("private_note", "user", "owner") is None
rows = self._audit_rows("memory.save")
assert len(rows) == 1
assert rows[0]["user_id"] == "guest"
def test_guest_user_get_does_not_probe_owner_namespace(self, tmp_db):
from turnstone.core.memory import save_structured_memory
save_structured_memory(
"owner_secret",
"must not leak",
description="Owner secret",
scope="user",
scope_id="owner",
)
session = _make_session(ws_id="shared", user_id="owner")
session.bind_acting_user("guest")
item = session._prepare_tool(
{
"id": "call_1",
"function": {
"name": "memory",
"arguments": json.dumps(
{"action": "get", "name": "owner_secret", "scope": "user"}
),
},
}
)
_, message = _execute_prepared_tool(session, item)
assert "not found" in message
assert "must not leak" not in message
assert "exists in scope" not in message
def test_save_global_scope_emits_empty_scope_id(self, tmp_db):
"""Global memories have no scope_id — the audit row's detail
must still carry the key (with value ``""``) so a forensic
@@ -6722,13 +6975,14 @@ class TestMemoryToolAudit:
"call_1",
{
"action": "save",
"description": "Test memory",
"name": "fact_global",
"content": "shared content",
"scope": "global",
},
)
assert "error" not in item
session._exec_memory(item)
_execute_prepared_tool(session, item)
rows = self._audit_rows("memory.save")
assert len(rows) == 1
@@ -6744,13 +6998,14 @@ class TestMemoryToolAudit:
"call_x",
{
"action": "save",
"description": "Test memory",
"name": "fact_one",
"content": content,
"scope": "user",
"type": "reference",
},
)
session._exec_memory(item)
_execute_prepared_tool(session, item)
saves = self._audit_rows("memory.save")
updates = self._audit_rows("memory.update")
@@ -6765,20 +7020,21 @@ class TestMemoryToolAudit:
"call_1",
{
"action": "save",
"description": "Test memory",
"name": "fact_one",
"content": "alpha",
"scope": "user",
"type": "reference",
},
)
session._exec_memory(save_item)
_execute_prepared_tool(session, save_item)
saved_memory_id = self._audit_rows("memory.save")[0]["resource_id"]
delete_item = session._prepare_memory(
"call_2",
{"action": "delete", "name": "fact_one", "scope": "user"},
)
_, msg = session._exec_memory(delete_item)
_, msg = _execute_prepared_tool(session, delete_item)
assert "Deleted memory" in msg
rows = self._audit_rows("memory.delete")
@@ -6797,23 +7053,70 @@ class TestMemoryToolAudit:
"call_1",
{"action": "delete", "name": "no_such_mem", "scope": "user"},
)
_, msg = session._exec_memory(delete_item)
_, msg = _execute_prepared_tool(session, delete_item)
assert "not found" in msg
assert self._audit_rows("memory.delete") == []
def test_committed_delete_is_truthful_and_next_prefix_refresh_fails_closed(self, tmp_db):
from turnstone.core.memory import get_structured_memory_by_name, save_structured_memory
save_structured_memory("doomed", "value", description="Memory to delete", scope="global")
session = _make_session(ws_id="ws-1", user_id="user-1")
item = session._prepare_memory(
"call_1", {"action": "delete", "name": "doomed", "scope": "global"}
)
_, message = _execute_prepared_tool(session, item)
assert "Deleted memory 'doomed'" in message
assert get_structured_memory_by_name("doomed", "global", "") is None
assert len(self._audit_rows("memory.delete")) == 1
assert session._system_prefix_dirty is True
with (
patch.object(
session,
"_init_system_messages",
side_effect=RuntimeError("composition failed"),
),
pytest.raises(RuntimeError, match="composition failed"),
):
session._ensure_system_prefix_fresh()
def test_storage_failure_is_not_reported_as_not_found(self, tmp_db):
from turnstone.core.storage import get_storage
session = _make_session(ws_id="ws-1", user_id="user-1")
storage = get_storage()
operations = (
(
"get_structured_memory_by_name",
{"action": "get", "name": "key", "scope": "global"},
),
(
"delete_structured_memory_returning",
{"action": "delete", "name": "key", "scope": "global"},
),
)
for method_name, arguments in operations:
item = session._prepare_memory("call_1", arguments)
with patch.object(storage, method_name, side_effect=RuntimeError("db down")):
_, message = _execute_prepared_tool(session, item)
assert "storage operation failed" in message
assert "not found" not in message
def test_reads_emit_no_audit(self, tmp_db):
session = _make_session(ws_id="ws-1", user_id="user-1")
session._exec_memory(
session._prepare_memory(
"call_save",
{
"action": "save",
"name": "fact_one",
"content": "alpha",
"scope": "user",
},
)
save_item = session._prepare_memory(
"call_save",
{
"action": "save",
"description": "Test memory",
"name": "fact_one",
"content": "alpha",
"scope": "user",
},
)
_execute_prepared_tool(session, save_item)
for spec in (
{"action": "get", "name": "fact_one", "scope": "user"},
@@ -6822,7 +7125,7 @@ class TestMemoryToolAudit:
):
item = session._prepare_memory("call_read", spec)
assert "error" not in item
session._exec_memory(item)
_execute_prepared_tool(session, item)
# Only the save above should have audited.
save_count = len(self._audit_rows("memory.save"))
@@ -6842,6 +7145,7 @@ class TestMemoryToolAudit:
"call_1",
{
"action": "save",
"description": "Test memory",
"name": "fact_one",
"content": "alpha",
"scope": "user",
@@ -6851,7 +7155,7 @@ class TestMemoryToolAudit:
"turnstone.core.audit.record_audit",
side_effect=RuntimeError("audit storage exploded"),
):
_, msg = session._exec_memory(item)
_, msg = _execute_prepared_tool(session, item)
assert "Saved memory 'fact_one'" in msg
# The save itself still landed.
from turnstone.core.memory import get_structured_memory_by_name
@@ -6863,10 +7167,10 @@ class TestPerKindToolVariants:
"""Verify the ``kind_variants`` metadata applies per-kind tool overrides.
Each kind sees only the tool surface it can actually use the
coord sees ``scope`` enum ``["coordinator"]`` and a coord-flavored
description; the IC sees ``["global", "workstream", "user"]`` and
the existing IC-flavored description. The union ``TOOLS`` list
keeps the full schema for introspection / docs / eval catalogs.
coord sees coordinator/project scopes and a coord-flavored description;
the IC sees global/workstream/user/project and the IC-flavored description.
The union ``TOOLS`` list keeps the full schema for introspection / docs /
eval catalogs.
"""
def test_coord_memory_tool_has_coord_only_scope_enum(self):
@@ -6877,6 +7181,10 @@ class TestPerKindToolVariants:
# v1.7: a coordinator attached to a project also reads/writes the shared
# 'project' scope, alongside its isolated 'coordinator' namespace.
assert scope["enum"] == ["coordinator", "project"]
scope_desc = scope["description"]
assert "Save/get/delete without scope target project when attached" in scope_desc
assert "otherwise coordinator" in scope_desc
assert "valid explicit scope selects exactly that scope" in scope_desc
def test_coord_memory_tool_description_mentions_orchestration(self):
from turnstone.core.tools import COORDINATOR_TOOLS
@@ -6896,6 +7204,10 @@ class TestPerKindToolVariants:
scope = memory["function"]["parameters"]["properties"]["scope"]
# v1.7: 'project' is offered (usable when the workstream is attached).
assert scope["enum"] == ["global", "workstream", "user", "project"]
scope_desc = scope["description"]
assert "Save/get/delete without scope target project when attached" in scope_desc
assert "otherwise global" in scope_desc
assert "valid explicit scope selects exactly that scope" in scope_desc
def test_ic_memory_tool_description_omits_coord_scope(self):
from turnstone.core.tools import INTERACTIVE_TOOLS
@@ -7024,7 +7336,12 @@ class TestMemoryAccessTouch:
def _save(name: str, content: str) -> None:
from turnstone.core.memory import save_structured_memory
save_structured_memory(name, content, scope="global")
save_structured_memory(
name,
content,
description=f"Test memory for {name}",
scope="global",
)
@staticmethod
def _empty_session() -> ChatSession:
@@ -7134,7 +7451,7 @@ class TestMemoryAccessTouch:
self._save("kafka_runbook", "restart the kafka broker pods")
item = session._prepare_memory("call_1", {"action": "search", "query": "kafka"})
assert "error" not in item
session._exec_memory(item)
_execute_prepared_tool(session, item)
assert self._access_count("kafka_runbook") == 1
def test_get_action_touches_fetched_memory(self, tmp_db):
@@ -7144,7 +7461,7 @@ class TestMemoryAccessTouch:
"call_1", {"action": "get", "name": "kafka_runbook", "scope": "global"}
)
assert "error" not in item
session._exec_memory(item)
_execute_prepared_tool(session, item)
assert self._access_count("kafka_runbook") == 1
def test_get_miss_touches_nothing(self, tmp_db):
@@ -7153,7 +7470,7 @@ class TestMemoryAccessTouch:
item = session._prepare_memory(
"call_1", {"action": "get", "name": "no_such_mem", "scope": "global"}
)
_, msg = session._exec_memory(item)
_, msg = _execute_prepared_tool(session, item)
assert "not found" in msg
# The existing row must not be collaterally touched by a miss.
assert self._access_count("kafka_runbook") == 0
@@ -7162,7 +7479,7 @@ class TestMemoryAccessTouch:
session = self._empty_session()
self._save("kafka_runbook", "restart the kafka broker pods")
item = session._prepare_memory("call_1", {"action": "list"})
session._exec_memory(item)
_execute_prepared_tool(session, item)
assert self._access_count("kafka_runbook") == 0
def test_save_action_does_not_touch_access_count(self, tmp_db):
@@ -7174,9 +7491,15 @@ class TestMemoryAccessTouch:
session = self._empty_session()
item = session._prepare_memory(
"call_1",
{"action": "save", "name": "kafka_runbook", "content": "x", "scope": "global"},
{
"action": "save",
"name": "kafka_runbook",
"content": "x",
"description": "Kafka runbook",
"scope": "global",
},
)
session._exec_memory(item)
_execute_prepared_tool(session, item)
assert self._access_count("kafka_runbook") == 0
def test_save_through_exec_does_not_recompose_prefix(self, tmp_db):
@@ -7202,13 +7525,14 @@ class TestMemoryAccessTouch:
"call_1",
{
"action": "save",
"description": "Test memory",
"name": "kafka_scaling",
"content": "restart kafka and scale the broker pods cluster",
"scope": "global",
},
)
assert "error" not in item
session._exec_memory(item)
_execute_prepared_tool(session, item)
# 1. Prefix byte-for-byte unchanged -> no prompt-cache bust.
after = "\n".join(m["content"] for m in session.system_messages if m["role"] == "system")
@@ -7226,11 +7550,8 @@ class TestMemoryAccessTouch:
)
assert '<memory name="kafka_scaling"' in recomposed
def test_save_through_tool_preserves_omitted_overwrites_explicit(self, tmp_db):
"""The None-sentinel flows through _prepare_memory -> _exec_memory: a
content-only re-save keeps the stored type/description, while an
explicit field overwrites it. Guards the _prepare_memory omit->None
logic that the storage-level tests don't exercise."""
def test_save_through_tool_requires_and_updates_description(self, tmp_db):
"""Every tool save describes the row; an omitted type stays preserved."""
from turnstone.core.memory import get_structured_memory_by_name
session = self._empty_session()
@@ -7246,48 +7567,58 @@ class TestMemoryAccessTouch:
},
)
assert "error" not in item
session._exec_memory(item)
_execute_prepared_tool(session, item)
# Content-only re-save (omits type/description) -> both preserved.
# An update supplies a fresh description while omitting type.
item2 = session._prepare_memory(
"c2", {"action": "save", "name": "digest", "content": "v2", "scope": "global"}
"c2",
{
"action": "save",
"name": "digest",
"content": "v2",
"description": "revised daily digest",
"scope": "global",
},
)
session._exec_memory(item2)
_execute_prepared_tool(session, item2)
mem = get_structured_memory_by_name("digest", "global", "")
assert mem is not None
assert mem["content"] == "v2"
assert mem["type"] == "reference"
assert mem["description"] == "daily digest"
assert mem["description"] == "revised daily digest"
# An invalid/typo'd type is treated as unset -> stored type preserved,
# not silently downgraded to "general".
# Invalid/typo'd types fail preparation and do not mutate the row.
item_bad = session._prepare_memory(
"c2b",
{
"action": "save",
"description": "Test memory",
"name": "digest",
"content": "v2b",
"type": "nonsense",
"scope": "global",
},
)
session._exec_memory(item_bad)
assert "error" in item_bad
assert "invalid memory type" in item_bad["error"]
mem = get_structured_memory_by_name("digest", "global", "")
assert mem is not None
assert mem["type"] == "reference" # invalid type ignored, not downgraded
assert mem["content"] == "v2"
assert mem["type"] == "reference"
# An explicit field -> overwrites (the behaviour the None-sentinel enables).
item3 = session._prepare_memory(
"c3",
{
"action": "save",
"description": "Test memory",
"name": "digest",
"content": "v3",
"type": "general",
"scope": "global",
},
)
session._exec_memory(item3)
_execute_prepared_tool(session, item3)
mem = get_structured_memory_by_name("digest", "global", "")
assert mem is not None
assert mem["type"] == "general"
+2 -2
View File
@@ -1,6 +1,6 @@
"""Tests for :meth:`ChatSession._format_backend_error`.
The helper turns bare backend-boundary exceptions (httpx ``ReadTimeout``,
The helper turns bare backend-boundary exceptions (HTTPX/HTTPX2 ``ReadTimeout``,
OpenAI SDK ``APITimeoutError`` / ``APIConnectionError`` /
``NotFoundError`` / ``RateLimitError`` / ``AuthenticationError``) into
operator-actionable messages that include the provider, base URL, and
@@ -225,7 +225,7 @@ def test_rate_limit_with_overflow_phrasing_is_not_mislabeled_overflow():
def _stream_death_exemplars() -> list[BaseException]:
"""One realistic instance per name in ``_BACKEND_STREAM_EXC_NAMES``:
the normalized shape the guarded iterators raise, plus the raw httpx
the normalized shape the guarded iterators raise, plus the raw HTTPX-family
names for any future unguarded path."""
from turnstone.core.providers import IncompleteStreamError
+4 -3
View File
@@ -124,9 +124,10 @@ def test_nonfork_resume_rebinds_project_memory_context_before_recomposition(tmp_
assert session.resume("target-ws") is True
assert session.ws_id == "target-ws"
assert session._project_id == "target-project"
assert session._project_name == "Target Project"
assert session._project_writable is True
access = session._memory_access()
assert access.project_id == "target-project"
assert access.project_name == "Target Project"
assert access.project_writable is True
assert ("project", "target-project") in session._visible_scopes()
assert ("project", "source-project") not in session._visible_scopes()
assert stale_cache_key not in session._mem_search_cache
+2
View File
@@ -46,6 +46,7 @@ _ESM_BUNDLES = [
_SHARED / "auth.js",
_SHARED / "renderer.js",
_SHARED / "status_bar.js",
_SHARED / "composer_paste_text.js",
_SHARED / "composer.js",
_SHARED / "composer_attachments.js",
_SHARED / "composer_queue.js",
@@ -73,6 +74,7 @@ _ESM_NO_VAR_BUNDLES = [
_SHARED / "toast.js",
_SHARED / "kb.js",
_SHARED / "auth.js",
_SHARED / "composer_paste_text.js",
_SHARED / "interactive.js",
_SHARED / "conversation.js",
_SHARED / "preview.js",
+9 -8
View File
@@ -6,6 +6,7 @@ hint pattern, and the skill catalog disclosure in system messages.
from __future__ import annotations
import threading
from typing import Any
from unittest.mock import MagicMock, patch
@@ -1375,15 +1376,14 @@ class TestSkillCatalogDisclosure:
session._tools = []
session._client_type = ClientType.CLI
session._username = ""
# _init_system_messages renders the attached project into the Session
# Context; this __new__-built session skips __init__'s project resolution,
# so seed the (unattached) defaults it reads.
session._project_name = ""
session._project_id = ""
session._project_writable = False
# This __new__-built session skips __init__'s attachment setup.
session._memory_attached_project_id = ""
session._system_prefix_lock = threading.RLock()
session._system_prefix_dirty = True
session._system_prefix_signature = None
session._kind = "interactive"
# Persona snapshot attrs (set by __init__, bypassed here) — legacy
# defaults: no override, unrestricted tools, MCP + memory on.
# Persona snapshot attrs (set by __init__, bypassed here): open
# defaults with no override, unrestricted tools, MCP + memory on.
session._persona_name = ""
session._persona_prompt = ""
session._persona_tools = None
@@ -1393,6 +1393,7 @@ class TestSkillCatalogDisclosure:
session._memory_config = MagicMock()
session._memory_config.fetch_limit = 0
session._user_id = "test-user"
session._acting_user_id = ""
# _init_system_messages -> _recompute_shared_state reads the session
# owner (_mcp_user_id) to decide shared-workstream framing; __init__
# normally sets it from user_id, so seed it here for the __new__ build.
+93 -6
View File
@@ -53,6 +53,7 @@ def _fake_request(
headers: dict[str, str] | None = None,
query: dict[str, str] | None = None,
path_params: dict[str, str] | None = None,
user_id: str = "viewer-1",
) -> Request:
"""Construct a Starlette ``Request`` for the events handler.
@@ -80,7 +81,9 @@ def _fake_request(
async def _recv() -> dict[str, Any]: # noqa: RUF029 — async signature required
return {"type": "http.disconnect"}
return Request(scope, receive=_recv)
request = Request(scope, receive=_recv)
request.state.auth_result = SimpleNS(user_id=user_id)
return request
# ---------------------------------------------------------------------------
@@ -673,6 +676,87 @@ def test_handler_emits_retry_on_first_yield() -> None:
assert 2500 <= retry <= 4500, f"retry {retry} outside jitter band [2500, 4500]"
def test_handler_resolves_project_off_loop_and_preserves_legacy_replay_callback() -> None:
"""Project metadata uses the viewer off-loop; replay keeps its 3-arg API."""
from turnstone.core.session_replay import (
request_replay_project_name,
session_replay_preamble,
)
event_loop_thread = threading.get_ident()
resolver_threads: list[int] = []
principals: list[str] = []
callback_calls: list[tuple[Any, Any, Any]] = []
session = MagicMock()
session.model = "test"
session.model_alias = ""
session._last_usage = None
def _project_name(principal_id: str) -> str:
resolver_threads.append(threading.get_ident())
principals.append(principal_id)
return "Visible Project"
session.project_name_for_principal.side_effect = _project_name
def _replay(ws: Any, ui: Any, request: Any) -> Any:
callback_calls.append((ws, ui, request))
yield from session_replay_preamble(
ws.session,
ui,
project_name=request_replay_project_name(request),
)
yield {"type": "tail"}
ui = _make_ui()
_, blob = _drain_handler_yields(
ui,
session=session,
events_replay=_replay,
max_yields=4,
)
assert principals == ["viewer-1"]
assert resolver_threads and resolver_threads[0] != event_loop_thread
assert len(callback_calls) == 1
assert '"type": "connected"' in blob
assert '"project_name": "Visible Project"' in blob
assert '"type": "tail"' in blob
def test_handler_project_lookup_failure_keeps_connected_event() -> None:
"""Optional project metadata fails closed without losing bootstrap."""
from turnstone.core.session_replay import (
request_replay_project_name,
session_replay_preamble,
)
session = MagicMock()
session.model = "test"
session.model_alias = ""
session._last_usage = None
session.project_name_for_principal.side_effect = RuntimeError("storage unavailable")
def _replay(ws: Any, ui: Any, request: Any) -> Any:
yield from session_replay_preamble(
ws.session,
ui,
project_name=request_replay_project_name(request),
)
ui = _make_ui()
_, blob = _drain_handler_yields(
ui,
session=session,
events_replay=_replay,
max_yields=3,
)
assert '"type": "connected"' in blob
assert '"project_name": ""' in blob
def test_handler_replay_ok_skips_snapshot_emits_id(monkeypatch: Any) -> None:
"""``Last-Event-ID`` + buffer covers gap → emit buffered events
with SSE ``id:`` field, SKIP the in-progress snapshot (it would
@@ -982,6 +1066,11 @@ class _HandoffSession:
self.context_window = 1000
self.reasoning_effort = "low"
self._last_usage = {"prompt_tokens": 12, "completion_tokens": 0}
self.project_name_principals: list[str] = []
def project_name_for_principal(self, principal_id: str) -> str:
self.project_name_principals.append(principal_id)
return ""
def register_listener_for_history_handoff(
self,
@@ -1093,11 +1182,8 @@ def test_handoff_cursor_replay_keeps_preamble_without_pending_control_duplicate(
session = _HandoffSession(ui)
def _full_replay(_ws: Any, _ui: Any, _request: Any) -> Any:
# The full replay's own preamble half is what the replay_ok path
# must NOT re-run; the lifted body calls the shared
# session_replay_preamble directly instead (no per-kind hook).
yield {"type": "connected", "model": "test"}
yield {"type": "status", "total_tokens": 12}
# The kind-specific tail must NOT run on replay_ok; the lifted body
# calls the shared preamble directly and the ring owns controls.
yield {"type": "approve_request", "items": [{"call_id": "duplicate"}]}
_, blob = _drain_handler_yields(
@@ -1110,6 +1196,7 @@ def test_handoff_cursor_replay_keeps_preamble_without_pending_control_duplicate(
)
assert session.calls == [(session.token, 0)]
assert session.project_name_principals == ["viewer-1"]
assert '"type": "connected"' in blob
assert '"type": "status"' in blob
assert '"type": "state_change"' in blob
+72 -30
View File
@@ -1,5 +1,7 @@
"""Tests for turnstone.core.memory — structured memory facade functions."""
import pytest
from turnstone.core.memory import (
count_structured_memories,
delete_structured_memory,
@@ -7,31 +9,71 @@ from turnstone.core.memory import (
list_structured_memories,
normalize_key,
save_structured_memory,
save_structured_memory_strict,
search_structured_memories,
)
def _save(name, content, **kwargs):
kwargs.setdefault("description", "test memory description")
return save_structured_memory(name, content, **kwargs)
class TestSaveStructuredMemory:
@pytest.mark.parametrize("save", [save_structured_memory, save_structured_memory_strict])
@pytest.mark.parametrize("description", [None, "", " "])
def test_description_is_required(self, tmp_db, description, save):
with pytest.raises(ValueError, match="description is required"):
save("test_key", "hello world", description=description)
def test_best_effort_save_still_swallows_storage_failures(self, tmp_db, monkeypatch):
from turnstone.core import memory as memory_mod
class _BoomStorage:
def upsert_structured_memory(self, *_args, **_kwargs):
raise RuntimeError("simulated storage failure")
monkeypatch.setattr(memory_mod, "get_storage", lambda: _BoomStorage())
assert save_structured_memory(
"test_key",
"hello world",
description="Test memory",
) == (None, False)
def test_best_effort_save_does_not_misclassify_backend_value_error(self, tmp_db, monkeypatch):
from turnstone.core import memory as memory_mod
class _BoomStorage:
def upsert_structured_memory(self, *_args, **_kwargs):
raise ValueError("backend decode failure")
monkeypatch.setattr(memory_mod, "get_storage", lambda: _BoomStorage())
assert save_structured_memory(
"test_key",
"hello world",
description="Test memory",
) == (None, False)
def test_save_new(self, tmp_db):
row, was_update = save_structured_memory("test_key", "hello world")
row, was_update = _save("test_key", "hello world")
assert row and row["memory_id"]
assert was_update is False
def test_save_upsert(self, tmp_db):
row1, was_update1 = save_structured_memory("test_key", "first")
row2, was_update2 = save_structured_memory("test_key", "second")
row1, was_update1 = _save("test_key", "first")
row2, was_update2 = _save("test_key", "second")
assert was_update1 is False
assert was_update2 is True
assert row2 and row1 and row2["memory_id"] == row1["memory_id"] # same row
assert row2["content"] == "second"
def test_save_normalizes_key(self, tmp_db):
save_structured_memory("My-Key", "value")
_save("My-Key", "value")
mems = list_structured_memories()
assert any(m["name"] == "my_key" for m in mems)
def test_save_with_type_and_scope(self, tmp_db):
save_structured_memory("k", "v", mem_type="user", scope="workstream", scope_id="ws1")
_save("k", "v", mem_type="user", scope="workstream", scope_id="ws1")
mems = list_structured_memories(scope="workstream", scope_id="ws1")
assert len(mems) == 1
assert mems[0]["type"] == "user"
@@ -39,14 +81,14 @@ class TestSaveStructuredMemory:
class TestDeleteStructuredMemory:
def test_delete_existing(self, tmp_db):
save_structured_memory("mykey", "val")
_save("mykey", "val")
assert delete_structured_memory("mykey")
def test_delete_nonexistent(self, tmp_db):
assert not delete_structured_memory("nope")
def test_delete_normalizes_key(self, tmp_db):
save_structured_memory("my_key", "val")
_save("my_key", "val")
assert delete_structured_memory("My-Key")
@@ -55,25 +97,25 @@ class TestListStructuredMemories:
assert list_structured_memories() == []
def test_list_returns_saved(self, tmp_db):
save_structured_memory("a", "alpha")
save_structured_memory("b", "beta")
_save("a", "alpha")
_save("b", "beta")
mems = list_structured_memories()
assert len(mems) == 2
class TestSearchStructuredMemories:
def test_search_finds_match(self, tmp_db):
save_structured_memory("db_host", "localhost", description="database hostname")
save_structured_memory("api_url", "http://example.com")
_save("db_host", "localhost", description="database hostname")
_save("api_url", "http://example.com")
results = search_structured_memories("database")
assert len(results) >= 1
assert any(r["name"] == "db_host" for r in results)
def test_multiword_or_matches_partial(self, tmp_db):
"""OR-of-terms: memory matching only 1 of 3 query terms is returned."""
save_structured_memory("postgres_config", "host=localhost port=5432")
save_structured_memory("redis_config", "host=redis port=6379")
save_structured_memory("unrelated", "nothing relevant here")
_save("postgres_config", "host=localhost port=5432")
_save("redis_config", "host=redis port=6379")
_save("unrelated", "nothing relevant here")
# "postgres missing_word_a missing_word_b": only postgres_config matches "postgres"
results = search_structured_memories("postgres missing_word_a missing_word_b")
@@ -83,9 +125,9 @@ class TestSearchStructuredMemories:
def test_multiword_or_multiple_partial_matches(self, tmp_db):
"""Multiple memories each matching different terms are all returned."""
save_structured_memory("key_alpha", "alpha content here")
save_structured_memory("key_beta", "beta content here")
save_structured_memory("key_other", "completely different")
_save("key_alpha", "alpha content here")
_save("key_beta", "beta content here")
_save("key_other", "completely different")
results = search_structured_memories("alpha beta")
names = {r["name"] for r in results}
@@ -95,9 +137,9 @@ class TestSearchStructuredMemories:
def test_search_scope_filtering_preserved(self, tmp_db):
"""Search with scope filter only returns memories in that scope."""
save_structured_memory("ws1_fact", "alpha info", scope="workstream", scope_id="ws1")
save_structured_memory("ws2_fact", "alpha info", scope="workstream", scope_id="ws2")
save_structured_memory("global_fact", "alpha info", scope="global")
_save("ws1_fact", "alpha info", scope="workstream", scope_id="ws1")
_save("ws2_fact", "alpha info", scope="workstream", scope_id="ws2")
_save("global_fact", "alpha info", scope="global")
results = search_structured_memories("alpha", scope="workstream", scope_id="ws1")
names = {r["name"] for r in results}
@@ -108,7 +150,7 @@ class TestSearchStructuredMemories:
class TestGetStructuredMemoryByName:
def test_get_existing(self, tmp_db):
save_structured_memory("my_mem", "full content here that is quite long")
_save("my_mem", "full content here that is quite long")
mem = get_structured_memory_by_name("my_mem", "global", "")
assert mem is not None
assert mem["content"] == "full content here that is quite long"
@@ -118,12 +160,12 @@ class TestGetStructuredMemoryByName:
assert get_structured_memory_by_name("nope", "global", "") is None
def test_get_wrong_scope(self, tmp_db):
save_structured_memory("ws_mem", "data", scope="workstream", scope_id="ws1")
_save("ws_mem", "data", scope="workstream", scope_id="ws1")
assert get_structured_memory_by_name("ws_mem", "global", "") is None
assert get_structured_memory_by_name("ws_mem", "workstream", "ws1") is not None
def test_get_normalizes_key(self, tmp_db):
save_structured_memory("My-Key", "value")
_save("My-Key", "value")
mem = get_structured_memory_by_name("My-Key", "global", "")
assert mem is not None
assert mem["name"] == "my_key"
@@ -134,8 +176,8 @@ class TestCountStructuredMemories:
assert count_structured_memories() == 0
def test_count_after_save(self, tmp_db):
save_structured_memory("a", "1")
save_structured_memory("b", "2")
_save("a", "1")
_save("b", "2")
assert count_structured_memories() == 2
@@ -154,11 +196,11 @@ class TestScopeIsolation:
def _seed(self):
"""Create memories across multiple scopes."""
save_structured_memory("global_note", "visible to all", scope="global")
save_structured_memory("ws1_note", "belongs to ws1", scope="workstream", scope_id="ws1")
save_structured_memory("ws2_note", "belongs to ws2", scope="workstream", scope_id="ws2")
save_structured_memory("u1_note", "belongs to user1", scope="user", scope_id="u1")
save_structured_memory("u2_note", "belongs to user2", scope="user", scope_id="u2")
_save("global_note", "visible to all", scope="global")
_save("ws1_note", "belongs to ws1", scope="workstream", scope_id="ws1")
_save("ws2_note", "belongs to ws2", scope="workstream", scope_id="ws2")
_save("u1_note", "belongs to user1", scope="user", scope_id="u1")
_save("u2_note", "belongs to user2", scope="user", scope_id="u2")
@staticmethod
def _list_visible(ws_id: str, user_id: str, mem_type: str = "", limit: int = 50):
+212 -54
View File
@@ -2,6 +2,15 @@
class TestCreateAndGet:
def test_create_requires_non_empty_description(self, backend):
import pytest
for description in (None, "", " "):
with pytest.raises(ValueError, match="description is required"):
backend.create_structured_memory(
"m1", "test_key", description, "general", "global", "", "data"
)
def test_create_and_get_by_id(self, backend):
backend.create_structured_memory("m1", "test_key", "desc", "general", "global", "", "data")
mem = backend.get_structured_memory("m1")
@@ -43,34 +52,44 @@ class TestSaveUpsert:
import pytest
import sqlalchemy as sa
backend.create_structured_memory("m1", "dup", "", "general", "global", "", "a")
backend.create_structured_memory("m1", "dup", "Test memory", "general", "global", "", "a")
with pytest.raises(sa.exc.IntegrityError):
backend.create_structured_memory("m2", "dup", "", "general", "global", "", "b")
backend.create_structured_memory(
"m2", "dup", "Test memory", "general", "global", "", "b"
)
def test_save_same_key_updates_in_place(self, backend):
from turnstone.core.memory import save_structured_memory
row1, was_update1 = save_structured_memory("upsert_key", "v1", scope="global")
row1, was_update1 = save_structured_memory(
"upsert_key", "v1", description="first description", scope="global"
)
assert row1 and was_update1 is False # inserted
row2, was_update2 = save_structured_memory("upsert_key", "v2", scope="global")
row2, was_update2 = save_structured_memory(
"upsert_key", "v2", description="updated description", scope="global"
)
assert row2 and was_update2 is True # updated in place
assert row2["memory_id"] == row1["memory_id"] # same row, not a duplicate
assert row2["content"] == "v2"
names = [r["name"] for r in backend.list_structured_memories(scope="global")]
assert names.count("upsert_key") == 1
def test_save_same_key_preserves_description_and_type_on_default_resave(self, backend):
from turnstone.core.memory import save_structured_memory
def test_save_same_key_requires_and_updates_description(self, backend):
from turnstone.core.memory import save_structured_memory, save_structured_memory_strict
save_structured_memory(
"meta_key", "c1", description="orig desc", mem_type="fact", scope="global"
)
# A re-save that omits description/type (defaults) must not clobber them.
save_structured_memory("meta_key", "c2", scope="global")
# Every update must describe the revised memory; type can still be omitted.
import pytest
with pytest.raises(ValueError, match="description is required"):
save_structured_memory_strict("meta_key", "c2", description=None, scope="global")
save_structured_memory("meta_key", "c2", description="revised description", scope="global")
row = backend.get_structured_memory_by_name("meta_key", "global", "")
assert row["content"] == "c2"
assert row["description"] == "orig desc"
assert row["description"] == "revised description"
assert row["type"] == "fact"
def test_upsert_method_updates_in_place_no_raise(self, backend):
@@ -89,20 +108,75 @@ class TestSaveUpsert:
names = [r["name"] for r in backend.list_structured_memories(scope="global")]
assert names.count("k") == 1
def test_upsert_none_preserves_explicit_overwrites(self, backend):
"""None description/type keep the stored value on conflict; an explicit
value (including "" / "general") overwrites it."""
def test_upsert_requires_description_and_preserves_omitted_type(self, backend):
"""Description is mandatory; an omitted type keeps the stored value."""
import pytest
backend.create_structured_memory("m1", "k", "keepdesc", "fact", "global", "", "v1")
# None -> preserve stored description/type (a content-only save).
row, _ = backend.upsert_structured_memory("m2", "k", None, None, "global", "", "v2")
with pytest.raises(ValueError, match="description is required"):
backend.upsert_structured_memory("m2", "k", None, None, "global", "", "v2")
with pytest.raises(ValueError, match="description is required"):
backend.upsert_structured_memory("m2", "k", " ", None, "global", "", "v2")
row, _ = backend.upsert_structured_memory(
"m2", "k", "new description", None, "global", "", "v2"
)
assert row["content"] == "v2"
assert row["description"] == "keepdesc"
assert row["description"] == "new description"
assert row["type"] == "fact"
# Explicit "" / "general" -> overwrite.
row2, _ = backend.upsert_structured_memory("m3", "k", "", "general", "global", "", "v3")
assert row2["description"] == ""
row2, _ = backend.upsert_structured_memory(
"m3", "k", "final description", "general", "global", "", "v3"
)
assert row2["description"] == "final description"
assert row2["type"] == "general"
def test_active_project_guard_accepts_only_active_project(self, backend):
import pytest
backend.create_project("active", "Active", "u1")
row, was_update = backend.upsert_structured_memory(
"m1",
"guarded",
"guarded description",
None,
"project",
"active",
"value",
require_active_project=True,
)
assert row["scope_id"] == "active"
assert was_update is False
backend.create_project("archived", "Archived", "u1", state="archived")
for project_id in ("archived", "missing"):
with pytest.raises(ValueError, match="missing, archived"):
backend.upsert_structured_memory(
f"m-{project_id}",
"guarded",
"guarded description",
None,
"project",
project_id,
"value",
require_active_project=True,
)
assert backend.get_structured_memory_by_name("guarded", "project", project_id) is None
def test_active_project_guard_rejects_non_project_scope(self, backend):
import pytest
with pytest.raises(ValueError, match="requires project scope"):
backend.upsert_structured_memory(
"m1",
"guarded",
"guarded description",
None,
"global",
"",
"value",
require_active_project=True,
)
class TestDelete:
def test_delete_existing(self, backend):
@@ -118,63 +192,119 @@ class TestDelete:
assert not backend.delete_structured_memory("k", "global", "")
assert backend.delete_structured_memory("k", "workstream", "ws1")
def test_delete_returning_is_atomic_and_truthful(self, backend):
backend.create_structured_memory(
"m1", "k", "description", "reference", "user", "u1", "data"
)
deleted = backend.delete_structured_memory_returning("k", "user", "u1")
assert deleted is not None
assert deleted["memory_id"] == "m1"
assert deleted["description"] == "description"
assert deleted["type"] == "reference"
assert backend.get_structured_memory("m1") is None
assert backend.delete_structured_memory_returning("k", "user", "u1") is None
def test_delete_by_id_returning_is_atomic_and_truthful(self, backend):
backend.create_structured_memory("m1", "k", "Test memory", "general", "global", "", "data")
deleted = backend.delete_structured_memory_by_id_returning("m1")
assert deleted is not None
assert deleted["name"] == "k"
assert backend.get_structured_memory("m1") is None
assert backend.delete_structured_memory_by_id_returning("m1") is None
class TestFindScopes:
def test_finds_only_requested_same_name_scopes(self, backend):
backend.create_structured_memory("m1", "same", "Test memory", "general", "global", "", "g")
backend.create_structured_memory(
"m2", "same", "Test memory", "general", "user", "u1", "own"
)
backend.create_structured_memory(
"m3", "same", "Test memory", "general", "user", "victim", "secret"
)
backend.create_structured_memory(
"m4", "other", "Test memory", "general", "workstream", "ws1", "other"
)
found = backend.find_structured_memory_scopes(
"same", [("global", ""), ("user", "u1"), ("workstream", "ws1")]
)
assert set(found) == {("global", ""), ("user", "u1")}
class TestList:
def test_list_all(self, backend):
backend.create_structured_memory("m1", "a", "", "general", "global", "", "1")
backend.create_structured_memory("m2", "b", "", "user", "global", "", "2")
backend.create_structured_memory("m1", "a", "Test memory", "general", "global", "", "1")
backend.create_structured_memory("m2", "b", "Test memory", "user", "global", "", "2")
mems = backend.list_structured_memories()
assert len(mems) == 2
def test_list_by_type(self, backend):
backend.create_structured_memory("m1", "a", "", "general", "global", "", "1")
backend.create_structured_memory("m2", "b", "", "user", "global", "", "2")
backend.create_structured_memory("m1", "a", "Test memory", "general", "global", "", "1")
backend.create_structured_memory("m2", "b", "Test memory", "user", "global", "", "2")
mems = backend.list_structured_memories(mem_type="user")
assert len(mems) == 1
assert mems[0]["name"] == "b"
def test_list_by_scope(self, backend):
backend.create_structured_memory("m1", "a", "", "general", "global", "", "1")
backend.create_structured_memory("m2", "b", "", "general", "workstream", "ws1", "2")
backend.create_structured_memory("m1", "a", "Test memory", "general", "global", "", "1")
backend.create_structured_memory(
"m2", "b", "Test memory", "general", "workstream", "ws1", "2"
)
mems = backend.list_structured_memories(scope="workstream")
assert len(mems) == 1
def test_list_respects_limit(self, backend):
for i in range(10):
backend.create_structured_memory(f"m{i}", f"k{i}", "", "general", "global", "", f"{i}")
backend.create_structured_memory(
f"m{i}", f"k{i}", "Test memory", "general", "global", "", f"{i}"
)
mems = backend.list_structured_memories(limit=3)
assert len(mems) == 3
class TestSearch:
def test_search_by_name(self, backend):
backend.create_structured_memory("m1", "database_config", "", "general", "global", "", "pg")
backend.create_structured_memory("m2", "api_key", "", "general", "global", "", "secret")
backend.create_structured_memory(
"m1", "database_config", "Test memory", "general", "global", "", "pg"
)
backend.create_structured_memory(
"m2", "api_key", "Test memory", "general", "global", "", "secret"
)
results = backend.search_structured_memories("database")
assert len(results) == 1
assert results[0]["name"] == "database_config"
def test_search_by_content(self, backend):
backend.create_structured_memory("m1", "a", "", "general", "global", "", "postgresql host")
backend.create_structured_memory(
"m1", "a", "Test memory", "general", "global", "", "postgresql host"
)
results = backend.search_structured_memories("postgresql")
assert len(results) == 1
def test_search_empty_lists_all(self, backend):
backend.create_structured_memory("m1", "a", "", "general", "global", "", "1")
backend.create_structured_memory("m2", "b", "", "general", "global", "", "2")
backend.create_structured_memory("m1", "a", "Test memory", "general", "global", "", "1")
backend.create_structured_memory("m2", "b", "Test memory", "general", "global", "", "2")
results = backend.search_structured_memories("")
assert len(results) == 2
class TestCount:
def test_count_all(self, backend):
backend.create_structured_memory("m1", "a", "", "general", "global", "", "1")
backend.create_structured_memory("m2", "b", "", "general", "global", "", "2")
backend.create_structured_memory("m1", "a", "Test memory", "general", "global", "", "1")
backend.create_structured_memory("m2", "b", "Test memory", "general", "global", "", "2")
assert backend.count_structured_memories() == 2
def test_count_by_scope(self, backend):
backend.create_structured_memory("m1", "a", "", "general", "global", "", "1")
backend.create_structured_memory("m2", "b", "", "general", "workstream", "ws1", "2")
backend.create_structured_memory("m1", "a", "Test memory", "general", "global", "", "1")
backend.create_structured_memory(
"m2", "b", "Test memory", "general", "workstream", "ws1", "2"
)
assert backend.count_structured_memories(scope="global") == 1
assert backend.count_structured_memories(scope="workstream") == 1
@@ -184,8 +314,12 @@ class TestSearchOrOfTerms:
def test_single_matching_term_in_multi_word_query(self, backend):
"""Memory with content 'apple' found when query is 'apple banana cherry'."""
backend.create_structured_memory("m1", "apple_mem", "", "general", "global", "", "apple")
backend.create_structured_memory("m2", "other_mem", "", "general", "global", "", "grape")
backend.create_structured_memory(
"m1", "apple_mem", "Test memory", "general", "global", "", "apple"
)
backend.create_structured_memory(
"m2", "other_mem", "Test memory", "general", "global", "", "grape"
)
results = backend.search_structured_memories("apple banana cherry")
names = {r["name"] for r in results}
@@ -194,10 +328,18 @@ class TestSearchOrOfTerms:
def test_partial_overlap_across_memories(self, backend):
"""Each memory matches one of three terms; all three are returned."""
backend.create_structured_memory("m1", "alpha_doc", "", "general", "global", "", "alpha")
backend.create_structured_memory("m2", "beta_doc", "", "general", "global", "", "beta")
backend.create_structured_memory("m3", "gamma_doc", "", "general", "global", "", "gamma")
backend.create_structured_memory("m4", "unrelated", "", "general", "global", "", "delta")
backend.create_structured_memory(
"m1", "alpha_doc", "Test memory", "general", "global", "", "alpha"
)
backend.create_structured_memory(
"m2", "beta_doc", "Test memory", "general", "global", "", "beta"
)
backend.create_structured_memory(
"m3", "gamma_doc", "Test memory", "general", "global", "", "gamma"
)
backend.create_structured_memory(
"m4", "unrelated", "Test memory", "general", "global", "", "delta"
)
results = backend.search_structured_memories("alpha beta gamma")
names = {r["name"] for r in results}
@@ -209,12 +351,14 @@ class TestSearchOrOfTerms:
def test_scope_filter_preserved(self, backend):
"""OR-of-terms search still respects scope / scope_id filters."""
backend.create_structured_memory(
"m1", "ws1_note", "", "general", "workstream", "ws1", "info"
"m1", "ws1_note", "Test memory", "general", "workstream", "ws1", "info"
)
backend.create_structured_memory(
"m2", "ws2_note", "", "general", "workstream", "ws2", "info"
"m2", "ws2_note", "Test memory", "general", "workstream", "ws2", "info"
)
backend.create_structured_memory(
"m3", "global_note", "Test memory", "general", "global", "", "info"
)
backend.create_structured_memory("m3", "global_note", "", "general", "global", "", "info")
results = backend.search_structured_memories("info", scope="workstream", scope_id="ws1")
names = {r["name"] for r in results}
@@ -224,9 +368,11 @@ class TestSearchOrOfTerms:
def test_term_cap_normalizes_unbounded_query(self, backend):
"""A multi-KB query collapses to <= MAX terms (de-dupe + length filter)."""
backend.create_structured_memory("m1", "alpha_doc", "", "general", "global", "", "alpha")
backend.create_structured_memory(
"m2", "other_doc", "", "general", "global", "", "irrelevant"
"m1", "alpha_doc", "Test memory", "general", "global", "", "alpha"
)
backend.create_structured_memory(
"m2", "other_doc", "Test memory", "general", "global", "", "irrelevant"
)
# Build a noisy query: same word repeated, plus 1-char tokens that
@@ -241,10 +387,18 @@ class TestVisibleStructuredMemories:
"""Single-query union helpers used by the composition path."""
def test_list_visible_unions_global_workstream_user(self, backend):
backend.create_structured_memory("m1", "g_note", "", "general", "global", "", "g")
backend.create_structured_memory("m2", "ws_note", "", "general", "workstream", "ws1", "w")
backend.create_structured_memory("m3", "u_note", "", "general", "user", "u1", "u")
backend.create_structured_memory("m4", "other_ws", "", "general", "workstream", "ws2", "x")
backend.create_structured_memory(
"m1", "g_note", "Test memory", "general", "global", "", "g"
)
backend.create_structured_memory(
"m2", "ws_note", "Test memory", "general", "workstream", "ws1", "w"
)
backend.create_structured_memory(
"m3", "u_note", "Test memory", "general", "user", "u1", "u"
)
backend.create_structured_memory(
"m4", "other_ws", "Test memory", "general", "workstream", "ws2", "x"
)
scopes = [("global", ""), ("workstream", "ws1"), ("user", "u1")]
rows = backend.list_visible_structured_memories(scopes)
@@ -252,12 +406,14 @@ class TestVisibleStructuredMemories:
assert names == {"g_note", "ws_note", "u_note"} # ws2 excluded
def test_search_visible_unions_scopes_and_terms(self, backend):
backend.create_structured_memory("m1", "g_alpha", "", "general", "global", "", "alpha")
backend.create_structured_memory(
"m2", "ws_beta", "", "general", "workstream", "ws1", "beta"
"m1", "g_alpha", "Test memory", "general", "global", "", "alpha"
)
backend.create_structured_memory(
"m3", "ws_other", "", "general", "workstream", "ws2", "alpha"
"m2", "ws_beta", "Test memory", "general", "workstream", "ws1", "beta"
)
backend.create_structured_memory(
"m3", "ws_other", "Test memory", "general", "workstream", "ws2", "alpha"
)
scopes = [("global", ""), ("workstream", "ws1")]
@@ -268,7 +424,9 @@ class TestVisibleStructuredMemories:
assert "ws_other" not in names # ws2 -> outside visibility
def test_visible_helpers_handle_empty_scopes(self, backend):
backend.create_structured_memory("m1", "anything", "", "general", "global", "", "x")
backend.create_structured_memory(
"m1", "anything", "Test memory", "general", "global", "", "x"
)
assert backend.list_visible_structured_memories([]) == []
assert backend.search_visible_structured_memories("x", []) == []
@@ -288,7 +446,7 @@ class TestStableOrderingOnTimestampTies:
# batch lands them in the same second.
for mid in ("zebra_id", "apple_id", "mango_id"):
backend.create_structured_memory(
mid, f"name_{mid}", "", "general", "global", "", "shared content"
mid, f"name_{mid}", "Test memory", "general", "global", "", "shared content"
)
import sqlalchemy as sa
+311 -6
View File
@@ -2,12 +2,12 @@
from __future__ import annotations
import ipaddress
import pytest
from turnstone.core.storage import get_storage, init_storage, reset_storage
lacme = pytest.importorskip("lacme")
@pytest.fixture(autouse=True)
def _storage(tmp_path):
@@ -83,9 +83,14 @@ def test_list_certs(tls_manager):
data = resp.json()
assert len(data["certs"]) >= 1
assert data["certs"][0]["domain"] == "test.internal"
assert data["certs"][0]["renewable"] is True
assert data["certs"][0]["deletable"] is False
def test_renew_cert(tls_manager):
from datetime import timedelta
from cryptography import x509
from starlette.testclient import TestClient
client = TestClient(_make_app(tls_manager))
@@ -93,6 +98,10 @@ def test_renew_cert(tls_manager):
assert resp.status_code == 200
data = resp.json()
assert data["domain"] == "test.internal"
renewed = tls_manager._store.load_cert("test.internal")
assert renewed is not None
leaf = x509.load_pem_x509_certificate(renewed.cert_pem)
assert leaf.not_valid_after_utc - leaf.not_valid_before_utc == timedelta(hours=48)
def test_renew_cert_not_found(tls_manager):
@@ -103,17 +112,44 @@ def test_renew_cert_not_found(tls_manager):
assert resp.status_code == 404
def test_delete_cert(tls_manager):
def test_renew_remote_cert_conflict(tls_manager):
from starlette.testclient import TestClient
remote = tls_manager._ca.issue(["remote.internal"])
old_key = remote.key_pem
client = TestClient(_make_app(tls_manager))
resp = client.delete("/certs/test.internal")
resp = client.post("/certs/remote.internal/renew")
assert resp.status_code == 409
assert tls_manager._store.load_cert("remote.internal").key_pem == old_key
def test_delete_expired_remote_cert(tls_manager):
from starlette.testclient import TestClient
tls_manager._ca.issue(["retired.internal"], validity_hours=0)
client = TestClient(_make_app(tls_manager))
resp = client.delete("/certs/retired.internal")
assert resp.status_code == 200
assert resp.json()["deleted"] == "test.internal"
assert resp.json()["deleted"] == "retired.internal"
# Verify it's gone
resp = client.get("/certs")
domains = [c["domain"] for c in resp.json()["certs"]]
assert "test.internal" not in domains
assert "retired.internal" not in domains
def test_delete_active_cert_conflict(tls_manager):
from starlette.testclient import TestClient
client = TestClient(_make_app(tls_manager))
resp = client.delete("/certs/test.internal")
assert resp.status_code == 409
assert tls_manager._store.load_cert("test.internal") is not None
def test_delete_cert_not_found(tls_manager):
@@ -268,6 +304,275 @@ def test_cli_bootstrap_no_issue(tmp_path):
assert len(list(certs_dir.iterdir())) == 0
def test_cli_bootstrap_uses_filestore_path_for_ipv6(tmp_path):
"""The CLI accepts IPv6 and lets FileStore encode its non-portable key."""
import argparse
import lacme
from cryptography import x509
from turnstone.admin import _cmd_tls_bootstrap
out = tmp_path / "certs"
args = argparse.Namespace(out=str(out), issue=["2001:0db8::10"])
_cmd_tls_bootstrap(args)
bundle = lacme.FileStore(out).load_cert("2001:db8::10")
assert bundle is not None
assert bundle.cert_path is not None
assert bundle.cert_path.parent != out / "certs" / "2001:db8::10"
sans = (
x509.load_pem_x509_certificate(bundle.cert_pem)
.extensions.get_extension_for_class(x509.SubjectAlternativeName)
.value
)
assert list(sans) == [x509.IPAddress(ipaddress.IPv6Address("2001:db8::10"))]
@pytest.mark.parametrize(
("domain", "sans", "expected"),
[
("node-1", ["node-1.internal"], ["node-1", "node-1.internal"]),
(
"2001:0db8::10",
["node-1.internal", "192.0.2.10"],
[
ipaddress.IPv6Address("2001:db8::10"),
"node-1.internal",
ipaddress.IPv4Address("192.0.2.10"),
],
),
],
ids=["dns", "mixed-ip"],
)
def test_cli_issue_closes_sync_client(monkeypatch, tmp_path, domain, sans, expected):
"""The synchronous HTTPX2-backed ACME client is closed after issuance."""
import argparse
from types import SimpleNamespace
import lacme
state: dict[str, object] = {}
monkeypatch.setenv("TURNSTONE_JWT_SECRET", "test-jwt-secret-minimum-32-chars!")
class FakeSyncClient:
def __init__(self, **kwargs):
state["kwargs"] = kwargs
def __enter__(self):
state["entered"] = True
return self
def __exit__(self, exc_type, exc_value, traceback):
import asyncio
state["exited"] = True
asyncio.run(state["kwargs"]["http_client"].aclose())
def issue(self, domains):
state["domains"] = domains
return SimpleNamespace(cert_pem=b"cert", fullchain_pem=b"chain", key_pem=b"key")
monkeypatch.setattr(lacme, "SyncClient", FakeSyncClient)
from turnstone.admin import _cmd_tls_issue
out = tmp_path / "issued"
args = argparse.Namespace(
console_url="http://console:8090",
domain=domain,
san=sans,
out=str(out),
)
_cmd_tls_issue(args)
kwargs = state.pop("kwargs")
assert kwargs["directory_url"] == "http://console:8090/acme/directory"
assert kwargs["allow_insecure"] is True
assert kwargs["http_client"].is_closed
assert state == {
"entered": True,
"domains": expected,
"exited": True,
}
assert (out / "cert.pem").read_bytes() == b"cert"
assert (out / "fullchain.pem").read_bytes() == b"chain"
assert (out / "key.pem").read_bytes() == b"key"
def test_cli_issue_closes_sync_client_on_failure(monkeypatch, tmp_path):
"""The synchronous client also closes when certificate issuance fails."""
import argparse
import lacme
state: dict[str, object] = {}
monkeypatch.setenv("TURNSTONE_JWT_SECRET", "test-jwt-secret-minimum-32-chars!")
class FailingSyncClient:
def __init__(self, **kwargs):
state["http_client"] = kwargs["http_client"]
def __enter__(self):
return self
def __exit__(self, exc_type, exc_value, traceback):
import asyncio
state["exited"] = True
asyncio.run(state["http_client"].aclose())
def issue(self, _domains):
raise RuntimeError("issuance failed")
monkeypatch.setattr(lacme, "SyncClient", FailingSyncClient)
from turnstone.admin import _cmd_tls_issue
args = argparse.Namespace(
console_url="http://console:8090",
domain="node-1",
san=[],
out=str(tmp_path / "issued"),
)
with pytest.raises(RuntimeError, match="issuance failed"):
_cmd_tls_issue(args)
assert state["exited"] is True
assert state["http_client"].is_closed
def test_cli_issue_against_authenticated_auto_approve_responder(monkeypatch, tmp_path):
"""The real lacme SyncClient completes mixed DNS/IP Turnstone issuance."""
import argparse
import asyncio
import httpx2
from cryptography import x509
from starlette.applications import Starlette
from starlette.middleware import Middleware
from starlette.routing import Mount
from turnstone.admin import _cmd_tls_issue
from turnstone.console.tls import TLSManager
from turnstone.core import tls as tls_module
from turnstone.core.auth import AUTH_COOKIE_CONSOLE, JWT_AUD_CONSOLE, AuthMiddleware
secret = "test-jwt-secret-minimum-32-chars!"
base = "http://console.test/acme"
manager = TLSManager(get_storage(), acme_external_url=base)
asyncio.run(manager.init_ca())
app = Starlette(
routes=[Mount("/acme", app=manager.get_responder())],
middleware=[
Middleware(
AuthMiddleware,
jwt_audience=JWT_AUD_CONSOLE,
cookie_name=AUTH_COOKIE_CONSOLE,
)
],
)
app.state.jwt_secret = secret
app.state.auth_storage = get_storage()
def build_client(
console_url,
*,
external_url="",
token_provider,
):
bases = [f"{console_url.rstrip('/')}/acme"]
if external_url:
bases.append(external_url.rstrip("/"))
return httpx2.AsyncClient(
transport=httpx2.ASGITransport(app=app),
auth=tls_module.ACMEHTTPAuth(bases, token_provider),
follow_redirects=False,
trust_env=False,
)
monkeypatch.setenv("TURNSTONE_JWT_SECRET", secret)
monkeypatch.delenv("TURNSTONE_ACME_EXTERNAL_URL", raising=False)
monkeypatch.setattr(tls_module, "build_acme_http_client", build_client)
out = tmp_path / "issued"
_cmd_tls_issue(
argparse.Namespace(
console_url="http://console.test",
domain="node.example",
san=["192.0.2.10", "2001:db8::10"],
out=str(out),
)
)
leaf = x509.load_pem_x509_certificate((out / "cert.pem").read_bytes())
sans = list(leaf.extensions.get_extension_for_class(x509.SubjectAlternativeName).value)
assert sans == [
x509.DNSName("node.example"),
x509.IPAddress(ipaddress.IPv4Address("192.0.2.10")),
x509.IPAddress(ipaddress.IPv6Address("2001:db8::10")),
]
assert (out / "key.pem").stat().st_mode & 0o777 == 0o600
def test_cli_ca_cert_preserves_trusted_https(monkeypatch, tmp_path):
import argparse
from types import SimpleNamespace
import httpx2
from turnstone.admin import _cmd_tls_ca_cert
seen = {}
def fake_get(url, **kwargs):
seen["url"] = url
seen["kwargs"] = kwargs
return SimpleNamespace(content=b"trusted-ca", raise_for_status=lambda: None)
monkeypatch.setattr(httpx2, "get", fake_get)
output = tmp_path / "ca.pem"
_cmd_tls_ca_cert(
argparse.Namespace(
console_url="https://console.example:8443",
out=str(output),
)
)
assert seen == {
"url": "https://console.example:8443/acme/ca.pem",
"kwargs": {"follow_redirects": False, "trust_env": False},
}
assert output.read_bytes() == b"trusted-ca"
def test_atomic_write_refuses_destination_symlink(tmp_path):
from turnstone.admin import _atomic_write_file
victim = tmp_path / "victim"
victim.write_bytes(b"preserve")
link = tmp_path / "key.pem"
link.symlink_to(victim)
with pytest.raises(RuntimeError, match="Refusing to replace symlink"):
_atomic_write_file(link, b"secret", mode=0o600)
assert victim.read_bytes() == b"preserve"
def test_atomic_write_refuses_symlinked_output_directory(tmp_path):
from turnstone.admin import _atomic_write_file
real_dir = tmp_path / "real"
real_dir.mkdir()
linked_dir = tmp_path / "linked"
linked_dir.symlink_to(real_dir, target_is_directory=True)
with pytest.raises(RuntimeError, match="symlink components"):
_atomic_write_file(linked_dir / "key.pem", b"secret", mode=0o600)
assert not (real_dir / "key.pem").exists()
# ── Config parsing ────────────────────────────────────────────────────────────
+318 -2
View File
@@ -2,6 +2,7 @@
from __future__ import annotations
import ipaddress
from pathlib import Path
from unittest.mock import MagicMock
@@ -9,8 +10,6 @@ import pytest
from turnstone.core.storage import get_storage, init_storage, reset_storage
lacme = pytest.importorskip("lacme")
@pytest.fixture(autouse=True)
def _storage(tmp_path):
@@ -58,6 +57,49 @@ def test_explicit_console_url_skips_discovery():
assert client._console_url == "http://explicit:9090"
@pytest.mark.anyio
async def test_ca_fetch_preserves_explicit_https(monkeypatch):
"""A trusted HTTPS bootstrap proxy must not be silently downgraded."""
from turnstone.core import tls as tls_module
seen: dict[str, object] = {}
class FakeResponse:
content = b"ca-pem"
def raise_for_status(self):
pass
class FakeHTTPClient:
def __init__(self, **kwargs):
seen["kwargs"] = kwargs
async def __aenter__(self):
return self
async def __aexit__(self, *_args):
pass
async def get(self, url):
seen["url"] = url
return FakeResponse()
monkeypatch.setattr(tls_module.httpx2, "AsyncClient", FakeHTTPClient)
client = tls_module.TLSClient(
storage=get_storage(),
console_url="https://console.example:8443",
hostnames=["node-1"],
)
await client._fetch_ca_cert()
assert seen == {
"kwargs": {"trust_env": False},
"url": "https://console.example:8443/acme/ca.pem",
}
assert client.ca_pem == b"ca-pem"
# ── SSL context construction ─────────────────────────────────────────────────
@@ -76,6 +118,280 @@ async def test_ssl_contexts_none_before_init():
assert not client.initialized
@pytest.mark.anyio
async def test_request_reissues_legacy_dns_ip_as_ip_san(monkeypatch):
"""A legacy DNS:<ip> row is not reused for a typed IP node identity."""
import lacme
from cryptography import x509
from turnstone.core import tls as tls_module
ca = lacme.CertificateAuthority()
ca.init()
legacy = ca.issue(["192.0.2.10"])
replacement = ca.issue([ipaddress.IPv4Address("192.0.2.10")])
captured = {}
class FakeClient:
def __init__(self, **kwargs):
self._store = kwargs["store"]
async def __aenter__(self):
return self
async def __aexit__(self, *_args):
pass
async def issue(self, identifiers):
captured["identifiers"] = identifiers
return self._store.save_cert(replacement)
monkeypatch.setattr(tls_module.lacme, "Client", FakeClient)
client = tls_module.TLSClient(
storage=get_storage(),
console_url="http://console:8090",
hostnames=["192.0.2.10"],
)
client._ca_pem = ca.root_cert_pem
client._store.save_cert(legacy)
await client._request_cert()
assert captured["identifiers"] == [ipaddress.IPv4Address("192.0.2.10")]
assert client.bundle is not None
assert client.bundle.cert_pem == replacement.cert_pem
sans = (
x509.load_pem_x509_certificate(client.bundle.cert_pem)
.extensions.get_extension_for_class(x509.SubjectAlternativeName)
.value
)
assert list(sans) == [x509.IPAddress(ipaddress.IPv4Address("192.0.2.10"))]
assert client._store.load_cert("192.0.2.10").cert_pem == replacement.cert_pem
# ── Renewal client lifecycle ─────────────────────────────────────────────────────
@pytest.mark.anyio
async def test_renewal_uses_public_stop_and_close(monkeypatch):
"""The manager and HTTPX2-backed lacme client both release their resources."""
from turnstone.core import tls as tls_module
state: dict[str, bool] = {}
class FakeClient:
def __init__(self, **_kwargs):
state["client_created"] = True
async def close(self):
state["client_closed"] = True
class FakeRenewalManager:
def __init__(self, **kwargs):
state["manager_created"] = True
state["renewal_policy"] = {
"interval_hours": kwargs["interval_hours"],
"days_before_expiry": kwargs["days_before_expiry"],
"max_jitter_seconds": kwargs["max_jitter_seconds"],
}
def start(self):
state["manager_started"] = True
async def stop(self):
state["manager_stopped"] = True
monkeypatch.setattr(tls_module.lacme, "Client", FakeClient)
monkeypatch.setattr(tls_module.lacme, "RenewalManager", FakeRenewalManager)
client = tls_module.TLSClient(
storage=get_storage(),
console_url="http://console:8090",
hostnames=["node-1"],
)
await client.start_renewal()
http_client = client._renewal_http_client
assert http_client is not None
await client.stop_renewal()
assert state == {
"client_created": True,
"manager_created": True,
"renewal_policy": {
"interval_hours": 12,
"days_before_expiry": 1,
"max_jitter_seconds": 600,
},
"manager_started": True,
"manager_stopped": True,
"client_closed": True,
}
assert client._renewal_manager is None
assert client._renewal_client is None
assert client._renewal_http_client is None
assert http_client.is_closed
def test_renewal_policy_has_one_failure_headroom():
"""The next retry after one failed due sweep still precedes expiry."""
from turnstone.core.tls import (
CERT_VALIDITY_HOURS,
RENEW_INTERVAL_HOURS,
RENEW_MAX_JITTER_SECONDS,
)
max_sleep_seconds = RENEW_INTERVAL_HOURS * 3600 + RENEW_MAX_JITTER_SECONDS
assert 3 * max_sleep_seconds < CERT_VALIDITY_HOURS * 3600
@pytest.mark.anyio
async def test_renewal_start_failure_closes_client(monkeypatch):
"""A failed manager start must not leak its HTTPX2-backed lacme client."""
from turnstone.core import tls as tls_module
state: dict[str, bool] = {}
class FakeClient:
def __init__(self, **_kwargs):
pass
async def close(self):
state["client_closed"] = True
class FailingRenewalManager:
def __init__(self, **_kwargs):
pass
def start(self):
raise RuntimeError("could not start")
monkeypatch.setattr(tls_module.lacme, "Client", FakeClient)
monkeypatch.setattr(tls_module.lacme, "RenewalManager", FailingRenewalManager)
client = tls_module.TLSClient(
storage=get_storage(),
console_url="http://console:8090",
hostnames=["node-1"],
)
with pytest.raises(RuntimeError, match="could not start"):
await client.start_renewal()
assert state == {"client_closed": True}
assert client._renewal_manager is None
assert client._renewal_client is None
assert client._renewal_http_client is None
@pytest.mark.anyio
async def test_renewal_start_failure_preserves_original_when_close_fails(monkeypatch):
"""Cleanup errors must not replace the manager startup failure."""
from turnstone.core import tls as tls_module
class FakeClient:
def __init__(self, **_kwargs):
pass
async def close(self):
raise RuntimeError("close failed")
class FailingRenewalManager:
def __init__(self, **_kwargs):
pass
def start(self):
raise RuntimeError("could not start")
monkeypatch.setattr(tls_module.lacme, "Client", FakeClient)
monkeypatch.setattr(tls_module.lacme, "RenewalManager", FailingRenewalManager)
client = tls_module.TLSClient(
storage=get_storage(),
console_url="http://console:8090",
hostnames=["node-1"],
)
with pytest.raises(RuntimeError, match="could not start"):
await client.start_renewal()
@pytest.mark.anyio
@pytest.mark.parametrize("anyio_backend", ["asyncio"])
async def test_renewal_stop_finishes_cleanup_before_propagating_repeated_cancellation(
monkeypatch, anyio_backend
):
"""Manager, lacme client, and HTTPX2 transport close despite cancellation."""
import asyncio
from turnstone.core import tls as tls_module
manager_entered = asyncio.Event()
release_manager = asyncio.Event()
client_entered = asyncio.Event()
release_client = asyncio.Event()
events: list[str] = []
class FakeHTTPClient:
async def aclose(self):
events.append("http_closed")
class FakeClient:
def __init__(self, **_kwargs):
pass
async def close(self):
events.append("client_entered")
client_entered.set()
await release_client.wait()
events.append("client_closed")
class FakeRenewalManager:
def __init__(self, **_kwargs):
pass
def start(self):
pass
async def stop(self):
events.append("manager_entered")
manager_entered.set()
await release_manager.wait()
events.append("manager_stopped")
monkeypatch.setattr(tls_module, "build_acme_http_client", lambda *_a, **_kw: FakeHTTPClient())
monkeypatch.setattr(tls_module.lacme, "Client", FakeClient)
monkeypatch.setattr(tls_module.lacme, "RenewalManager", FakeRenewalManager)
client = tls_module.TLSClient(
storage=get_storage(),
console_url="http://console:8090",
hostnames=["node-1"],
)
await client.start_renewal()
stopping = asyncio.create_task(client.stop_renewal())
await manager_entered.wait()
stopping.cancel()
await asyncio.sleep(0)
assert not stopping.done()
release_manager.set()
await client_entered.wait()
stopping.cancel()
await asyncio.sleep(0)
assert not stopping.done()
release_client.set()
with pytest.raises(asyncio.CancelledError):
await stopping
assert events == [
"manager_entered",
"manager_stopped",
"client_entered",
"client_closed",
"http_closed",
]
assert client._renewal_manager is None
assert client._renewal_client is None
assert client._renewal_http_client is None
# ── Backward compatibility ───────────────────────────────────────────────────
+467 -2
View File
@@ -2,12 +2,12 @@
from __future__ import annotations
import ipaddress
import pytest
from turnstone.core.storage import get_storage, init_storage, reset_storage
lacme = pytest.importorskip("lacme")
@pytest.fixture(autouse=True)
def _storage(tmp_path):
@@ -69,6 +69,347 @@ async def test_get_responder(tls_manager):
assert callable(responder)
@pytest.mark.anyio
async def test_responder_advertises_external_url():
"""Directory links use the routable URL, not the request's container address."""
from starlette.applications import Starlette
from starlette.routing import Mount
from starlette.testclient import TestClient
from turnstone.console.tls import TLSManager
external_url = "http://192.0.2.10:8090/acme/"
manager = TLSManager(get_storage(), acme_external_url=external_url)
await manager.init_ca()
app = Starlette(routes=[Mount("/acme", app=manager.get_responder())])
with TestClient(app) as client:
response = client.get("/acme/directory")
assert response.status_code == 200
advertised = response.json()
required = {
"newNonce": "http://192.0.2.10:8090/acme/new-nonce",
"newAccount": "http://192.0.2.10:8090/acme/new-account",
"newOrder": "http://192.0.2.10:8090/acme/new-order",
"revokeCert": "http://192.0.2.10:8090/acme/revoke-cert",
"keyChange": "http://192.0.2.10:8090/acme/key-change",
}
assert required.items() <= advertised.items()
@pytest.mark.anyio
async def test_responder_without_external_url_uses_request_address():
"""Unset configuration preserves same-host and in-network enrollment."""
from starlette.applications import Starlette
from starlette.routing import Mount
from starlette.testclient import TestClient
from turnstone.console.tls import TLSManager
manager = TLSManager(get_storage())
await manager.init_ca()
app = Starlette(routes=[Mount("/acme", app=manager.get_responder())])
with TestClient(app, base_url="http://console.internal:8090") as client:
response = client.get("/acme/directory")
assert response.status_code == 200
assert response.json()["newOrder"] == "http://console.internal:8090/acme/new-order"
@pytest.mark.anyio
async def test_responder_external_url_requires_acme_mount():
"""Catch the easy-to-miss deployment error before advertising bad links."""
from turnstone.console.tls import TLSManager
manager = TLSManager(
get_storage(),
acme_external_url="http://192.0.2.10:8090",
)
await manager.init_ca()
with pytest.raises(ValueError, match="must include.* /acme mount"):
manager.get_responder()
def _authenticated_acme_app(manager, secret: str):
from starlette.applications import Starlette
from starlette.middleware import Middleware
from starlette.routing import Mount
from turnstone.core.auth import AUTH_COOKIE_CONSOLE, JWT_AUD_CONSOLE, AuthMiddleware
app = Starlette(
routes=[Mount("/acme", app=manager.get_responder())],
middleware=[
Middleware(
AuthMiddleware,
jwt_audience=JWT_AUD_CONSOLE,
cookie_name=AUTH_COOKIE_CONSOLE,
)
],
)
app.state.jwt_secret = secret
app.state.auth_storage = get_storage()
return app
@pytest.mark.anyio
async def test_acme_signing_routes_require_enrollment_token():
"""Discovery stays public, but the auto-approving signer does not."""
import httpx2
from turnstone.console.tls import TLSManager
secret = "test-jwt-secret-minimum-32-chars!"
manager = TLSManager(get_storage())
await manager.init_ca()
transport = httpx2.ASGITransport(app=_authenticated_acme_app(manager, secret))
async with httpx2.AsyncClient(transport=transport, base_url="http://console.test") as client:
assert (await client.get("/acme/directory")).status_code == 200
assert (await client.head("/acme/new-nonce")).status_code == 200
assert (await client.get("/acme/ca.pem")).status_code == 200
for path in (
"/acme/new-account",
"/acme/new-order",
"/acme/authz/1",
"/acme/chall/1",
"/acme/finalize/1",
"/acme/order/1",
"/acme/cert/1",
"/acme/key-change",
"/acme/revoke-cert",
):
assert (await client.post(path)).status_code == 401
@pytest.mark.anyio
@pytest.mark.parametrize(
"identifiers",
[
[ipaddress.IPv4Address("192.0.2.10")],
[ipaddress.IPv6Address("2001:db8::10")],
[
"Node.Example",
ipaddress.IPv4Address("192.0.2.10"),
ipaddress.IPv6Address("2001:db8::10"),
],
],
ids=["ipv4", "ipv6", "mixed"],
)
async def test_authenticated_full_issuance_follows_external_urls(identifiers):
"""Turnstone auth and external routing preserve DNS/IP identifier types."""
import httpx2
import lacme
from cryptography import x509
from cryptography.hazmat.primitives import hashes
from lacme.events import CACertificateIssued
from turnstone.console.tls import TLSManager
from turnstone.core.auth import (
JWT_AUD_CONSOLE,
TLS_ACME_TOKEN_SOURCE,
ServiceTokenManager,
)
from turnstone.core.tls import ACMEHTTPAuth
class NoopChallengeHandler:
def __init__(self):
self.provisioned = []
self.deprovisioned = []
async def provision(self, domain, _token, _key_authorization):
self.provisioned.append(domain)
async def deprovision(self, domain, _token):
self.deprovisioned.append(domain)
secret = "test-jwt-secret-minimum-32-chars!"
internal_base = "http://console.internal:8090/acme"
external_base = "http://ca.example:8090/acme"
manager = TLSManager(get_storage(), acme_external_url=external_base)
await manager.init_ca()
ca_events = []
manager._event_dispatcher.subscribe(ca_events.append, event_type=CACertificateIssued)
app = _authenticated_acme_app(manager, secret)
tokens = ServiceTokenManager(
user_id="node-1",
scopes=frozenset({"service"}),
source=TLS_ACME_TOKEN_SOURCE,
secret=secret,
audience=JWT_AUD_CONSOLE,
)
challenge_handler = NoopChallengeHandler()
transport = httpx2.ASGITransport(app=app)
async with (
httpx2.AsyncClient(
transport=transport,
auth=ACMEHTTPAuth([internal_base, external_base], lambda: tokens.token),
follow_redirects=False,
trust_env=False,
) as http_client,
lacme.Client(
directory_url=f"{internal_base}/directory",
store=manager._store,
challenge_handler=challenge_handler,
http_client=http_client,
poll_interval=0.001,
allow_insecure=True,
) as client,
):
bundle = await client.issue(identifiers)
root = x509.load_pem_x509_certificate(manager.get_root_cert_pem())
chain = x509.load_pem_x509_certificates(bundle.fullchain_pem)
from datetime import timedelta
assert chain[0].issuer == root.subject
assert chain[-1].fingerprint(hashes.SHA256()) == root.fingerprint(hashes.SHA256())
assert chain[0].not_valid_after_utc - chain[0].not_valid_before_utc == timedelta(hours=48)
assert bundle.key_pem
identifier_strings = tuple(str(value) for value in identifiers)
assert bundle.domain == identifier_strings[0]
assert bundle.domains == identifier_strings
assert manager._store.load_cert(identifier_strings[0]) == bundle
assert challenge_handler.provisioned == list(identifier_strings)
assert challenge_handler.deprovisioned == list(identifier_strings)
assert len(ca_events) == 1
assert ca_events[0].name == identifier_strings[0]
assert ca_events[0].names == identifier_strings
@pytest.mark.anyio
async def test_acme_http_auth_rejects_hostile_directory_destination():
"""Absolute URLs from a directory cannot redirect the enrollment JWT."""
import json
import httpx2
import lacme
from turnstone.core.tls import ACMEHTTPAuth
seen: list[httpx2.Request] = []
def handler(request: httpx2.Request) -> httpx2.Response:
seen.append(request)
directory = {
"newNonce": "http://attacker.example/acme/new-nonce",
"newAccount": "http://console.test/acme/new-account",
"newOrder": "http://console.test/acme/new-order",
"revokeCert": "http://console.test/acme/revoke-cert",
"keyChange": "http://console.test/acme/key-change",
}
return httpx2.Response(200, content=json.dumps(directory), request=request)
async with (
httpx2.AsyncClient(
transport=httpx2.MockTransport(handler),
auth=ACMEHTTPAuth(["http://console.test/acme"], lambda: "sensitive-token"),
) as http_client,
lacme.Client(
directory_url="http://console.test/acme/directory",
http_client=http_client,
allow_insecure=True,
) as client,
):
with pytest.raises(RuntimeError, match="outside configured responder bases"):
await client.create_account()
assert len(seen) == 1
assert seen[0].url == httpx2.URL("http://console.test/acme/directory")
assert "Authorization" not in seen[0].headers
@pytest.mark.anyio
@pytest.mark.parametrize(
"url",
[
"http://console.test/acme/cert/%2e%2e/%2e%2e/v1/api/admin",
"http://console.test/acme/cert/%2Fv1%2Fapi%2Fadmin",
"http://console.test/acme/cert//v1/api/admin",
"http://console.test/acme/cert/1?next=/v1/api/admin",
"http://user@console.test/acme/cert/1",
],
)
async def test_acme_http_auth_rejects_noncanonical_protected_urls(url):
"""Enrollment credentials never cross ambiguous proxy path boundaries."""
import httpx2
from turnstone.core.tls import ACMEHTTPAuth
seen: list[httpx2.Request] = []
def handler(request: httpx2.Request) -> httpx2.Response:
seen.append(request)
return httpx2.Response(200, request=request)
async with httpx2.AsyncClient(
transport=httpx2.MockTransport(handler),
auth=ACMEHTTPAuth(["http://console.test/acme"], lambda: "sensitive-token"),
) as client:
with pytest.raises(RuntimeError, match="non-canonical|unknown ACME"):
await client.post(url)
assert seen == []
@pytest.mark.anyio
async def test_acme_http_auth_accepts_exact_dynamic_resource():
import httpx2
from turnstone.core.tls import ACMEHTTPAuth
seen: list[httpx2.Request] = []
def handler(request: httpx2.Request) -> httpx2.Response:
seen.append(request)
return httpx2.Response(200, request=request)
async with httpx2.AsyncClient(
transport=httpx2.MockTransport(handler),
auth=ACMEHTTPAuth(["http://console.test/acme"], lambda: "sensitive-token"),
) as client:
response = await client.post("http://console.test/acme/cert/order_1.example-2~x")
assert response.status_code == 200
assert len(seen) == 1
assert seen[0].headers["Authorization"] == "Bearer sensitive-token"
@pytest.mark.anyio
@pytest.mark.parametrize("anyio_backend", ["asyncio"])
async def test_console_renewal_stop_cleans_up_before_cancellation(anyio_backend):
import asyncio
from turnstone.console.tls import TLSManager
entered = asyncio.Event()
release = asyncio.Event()
stopped = False
class FakeManager:
async def stop(self):
nonlocal stopped
entered.set()
await release.wait()
stopped = True
manager = TLSManager(get_storage())
manager._renewal_manager = FakeManager()
task = asyncio.create_task(manager.stop_renewal())
await entered.wait()
task.cancel()
await asyncio.sleep(0)
assert not task.done()
release.set()
with pytest.raises(asyncio.CancelledError):
await task
assert stopped is True
assert manager._renewal_manager is None
# ── Cert issuance ─────────────────────────────────────────────────────────────
@@ -101,6 +442,130 @@ async def test_issue_console_certs_persists(tls_manager):
assert bundle1.cert_pem == bundle2.cert_pem
@pytest.mark.anyio
async def test_frontend_and_internal_same_domain_use_separate_stores(monkeypatch):
"""External frontend issuance cannot replace the console mTLS identity."""
import lacme
from turnstone.console.tls import TLSManager
manager = TLSManager(get_storage())
await manager.init_ca()
await manager.issue_console_certs(["console.example"])
internal = manager.internal_bundle
assert internal is not None
external_ca = lacme.CertificateAuthority()
external_ca.init(cn="External CA")
frontend = external_ca.issue(["console.example"])
class FakeExternalClient:
def __init__(self, **kwargs):
self._store = kwargs["store"]
async def __aenter__(self):
return self
async def __aexit__(self, *_args):
return None
async def issue(self, _identifiers):
return self._store.save_cert(frontend)
monkeypatch.setattr(lacme, "Client", FakeExternalClient)
await manager._issue_frontend_cert(
["console.example"],
"https://external-ca.example/directory",
)
assert manager._store.load_cert("console.example") == internal
assert manager._frontend_store is not None
assert manager._frontend_store.load_cert("console.example") == frontend
restarted = TLSManager(get_storage())
await restarted.init_ca()
await restarted.issue_console_certs(["console.example"])
assert restarted.internal_bundle is not None
assert restarted.internal_bundle.cert_pem == internal.cert_pem
@pytest.mark.anyio
async def test_console_rejects_same_san_bundle_from_wrong_root(tls_manager):
"""Legacy same-domain frontend rows are repaired, not reused for mTLS."""
import lacme
from cryptography import x509
from cryptography.hazmat.primitives import hashes
await tls_manager.init_ca()
external_ca = lacme.CertificateAuthority()
external_ca.init(cn="Turnstone CA")
wrong_root_bundle = external_ca.issue(["console.example"])
tls_manager._store.save_cert(wrong_root_bundle)
await tls_manager.issue_console_certs(["console.example"])
replacement = tls_manager.internal_bundle
assert replacement is not None
active_root = x509.load_pem_x509_certificate(tls_manager.get_root_cert_pem())
replacement_chain = x509.load_pem_x509_certificates(replacement.fullchain_pem)
assert replacement.cert_pem != wrong_root_bundle.cert_pem
assert replacement_chain[-1].fingerprint(hashes.SHA256()) == active_root.fingerprint(
hashes.SHA256()
)
@pytest.mark.anyio
async def test_cluster_bundle_validation_rejects_injected_chain_member(tls_manager):
from dataclasses import replace
import lacme
from turnstone.core.tls import validate_cluster_certificate_bundle
await tls_manager.init_ca()
valid = tls_manager._ca.issue(["console.example"])
unrelated_ca = lacme.CertificateAuthority()
unrelated_ca.init(cn="Unrelated CA")
malformed = replace(
valid,
fullchain_pem=valid.cert_pem + unrelated_ca.root_cert_pem + tls_manager.get_root_cert_pem(),
)
with pytest.raises(ValueError, match="chain.*invalid"):
validate_cluster_certificate_bundle(
malformed,
["console.example"],
tls_manager.get_root_cert_pem(),
)
@pytest.mark.anyio
async def test_console_reissues_legacy_dns_ip_as_ip_san(tls_manager):
"""An unexpired DNS:192.0.2.10 bundle cannot satisfy an IP identity."""
from cryptography import x509
await tls_manager.init_ca()
legacy = tls_manager._ca.issue(["192.0.2.10"])
legacy_sans = (
x509.load_pem_x509_certificate(legacy.cert_pem)
.extensions.get_extension_for_class(x509.SubjectAlternativeName)
.value
)
assert list(legacy_sans) == [x509.DNSName("192.0.2.10")]
await tls_manager.issue_console_certs([ipaddress.IPv4Address("192.0.2.10")])
replacement = tls_manager.internal_bundle
assert replacement is not None
assert replacement.cert_pem != legacy.cert_pem
replacement_sans = (
x509.load_pem_x509_certificate(replacement.cert_pem)
.extensions.get_extension_for_class(x509.SubjectAlternativeName)
.value
)
assert list(replacement_sans) == [x509.IPAddress(ipaddress.IPv4Address("192.0.2.10"))]
# ── SSL contexts ──────────────────────────────────────────────────────────────
+250 -36
View File
@@ -10,14 +10,14 @@ Regression coverage for the cluster-wide mTLS breakage where:
from __future__ import annotations
import ipaddress
import socket
import lacme
import pytest
from turnstone.core.storage import get_storage, init_storage, reset_storage
lacme = pytest.importorskip("lacme")
@pytest.fixture(autouse=True)
def _storage(tmp_path):
@@ -38,7 +38,7 @@ def test_advertised_host_is_primary():
names = build_cert_hostnames("http://server-1:8080", bind_host="0.0.0.0")
assert names[0] == "server-1"
assert "localhost" in names
assert "127.0.0.1" in names
assert ipaddress.IPv4Address("127.0.0.1") in names
# 0.0.0.0 is a wildcard bind and must not become a SAN
assert "0.0.0.0" not in names
@@ -67,17 +67,92 @@ def test_fallback_to_os_hostname_when_no_advertise_url():
assert build_cert_hostnames("")[0] == socket.gethostname()
def test_extra_sans_rejects_wildcard_and_unspecified():
"""A stray wildcard / unspecified-address SAN must not reach the cert."""
def test_extra_sans_rejects_wildcard():
"""A stray wildcard SAN must not reach the cert."""
from turnstone.core.tls import build_cert_hostnames
names = build_cert_hostnames("http://server-1:8080", extra_sans="*, 0.0.0.0, ::, edge")
names = build_cert_hostnames("http://server-1:8080", extra_sans="*, edge")
assert "*" not in names
assert "0.0.0.0" not in names
assert "::" not in names
assert "edge" in names
@pytest.mark.parametrize(
("url", "expected"),
[
("http://192.0.2.10:8080", ipaddress.IPv4Address("192.0.2.10")),
("http://[2001:db8::10]:8080", ipaddress.IPv6Address("2001:db8::10")),
],
)
def test_advertised_ip_literal_is_typed_primary(url, expected):
"""URL IP literals cross the lacme boundary as typed IP identifiers."""
from turnstone.core.tls import build_cert_hostnames
assert build_cert_hostnames(url)[0] == expected
def test_extra_ip_literal_sans_are_typed():
from turnstone.core.tls import build_cert_hostnames
names = build_cert_hostnames(
"http://server-1:8080",
extra_sans="192.0.2.10, 2001:db8::10",
)
assert ipaddress.IPv4Address("192.0.2.10") in names
assert ipaddress.IPv6Address("2001:db8::10") in names
@pytest.mark.parametrize(
("value", "message"),
[
("0.0.0.0", "Unspecified address"),
("::", "Unspecified address"),
("fe80::1%eth0", "Scoped IPv6 address"),
],
)
def test_unusable_extra_ip_identity_rejected(value, message):
from turnstone.core.tls import build_cert_hostnames
with pytest.raises(ValueError, match=message):
build_cert_hostnames("http://server-1:8080", extra_sans=value)
@pytest.mark.parametrize(
"url",
[
"http://0.0.0.0:8080",
"http://[::]:8080",
"http://[fe80::1%25eth0]:8080",
],
)
def test_unusable_advertised_ip_identity_rejected(url):
from turnstone.core.tls import build_cert_hostnames
with pytest.raises(ValueError, match="not a valid certificate identity"):
build_cert_hostnames(url)
def test_identifier_normalization_is_typed_and_order_preserving():
"""Semantic duplicates collapse without changing DNS presentation."""
from turnstone.core.tls import normalize_certificate_identifiers
identifiers = normalize_certificate_identifiers(
[
"Node.Example",
"node.example",
"192.0.2.10",
ipaddress.IPv4Address("192.0.2.10"),
"2001:0db8:0:0::10",
ipaddress.IPv6Address("2001:db8::10"),
]
)
assert identifiers == [
"Node.Example",
ipaddress.IPv4Address("192.0.2.10"),
ipaddress.IPv6Address("2001:db8::10"),
]
# ── _SingleDomainStore ────────────────────────────────────────────────────────
@@ -86,30 +161,38 @@ def _san_values(cert_pem: bytes) -> list[str]:
cert = x509.load_pem_x509_certificate(cert_pem)
san = cert.extensions.get_extension_for_class(x509.SubjectAlternativeName).value
return [g.value for g in san]
return [str(g.value) for g in san]
def _sans(cert_pem: bytes):
from cryptography import x509
cert = x509.load_pem_x509_certificate(cert_pem)
return list(cert.extensions.get_extension_for_class(x509.SubjectAlternativeName).value)
@pytest.mark.anyio
async def test_single_domain_store_filters_and_delegates():
"""list_certs exposes only the wrapped domain; other ops delegate."""
from turnstone.console.tls import TLSManager
from turnstone.core.tls import _SingleDomainStore
from turnstone.core.tls_store import RenewalStoreView
mgr = TLSManager(get_storage())
await mgr.init_ca()
for dom in ("server-1", "server-2", "server-3"):
mgr._store.save_cert(mgr._ca.issue([dom]))
wrapped = _SingleDomainStore(mgr._store, "server-2")
wrapped = RenewalStoreView(mgr._store, "server-2")
assert isinstance(wrapped, lacme.Store)
listed = wrapped.list_certs()
assert [b.domain for b in listed] == ["server-2"]
# __getattr__ delegation still reaches the real store
# Explicit Store delegation still reaches the real store.
assert wrapped.load_cert("server-1") is not None
assert wrapped.delete_cert("server-3") is True
assert len(mgr._store.list_certs()) == 2
# An empty domain (missing identity) matches nothing — the safe fallback
# that prevents an unscoped sweep of the whole shared store.
assert _SingleDomainStore(mgr._store, "").list_certs() == []
assert RenewalStoreView(mgr._store, "").list_certs() == []
# ── End-to-end SAN identity ───────────────────────────────────────────────────
@@ -131,6 +214,33 @@ async def test_issued_cert_covers_advertised_host():
assert "server-1" in _san_values(bundle.cert_pem)
@pytest.mark.anyio
async def test_issued_cert_preserves_dns_and_ip_san_types():
"""Typed IP inputs become x509.IPAddress while DNS stays x509.DNSName."""
from cryptography import x509
from turnstone.console.tls import TLSManager
mgr = TLSManager(get_storage())
await mgr.init_ca()
bundle = mgr._ca.issue(
[
"Node.Example",
ipaddress.IPv4Address("192.0.2.10"),
ipaddress.IPv6Address("2001:db8::10"),
]
)
sans = _sans(bundle.cert_pem)
assert sans == [
x509.DNSName("Node.Example"),
x509.IPAddress(ipaddress.IPv4Address("192.0.2.10")),
x509.IPAddress(ipaddress.IPv6Address("2001:db8::10")),
]
assert bundle.domain == "Node.Example"
assert bundle.domains == ("Node.Example", "192.0.2.10", "2001:db8::10")
# ── Renewal scoping (the storm fix) ───────────────────────────────────────────
@@ -138,7 +248,7 @@ async def test_issued_cert_covers_advertised_host():
async def test_renewal_sweep_only_touches_own_domain():
"""A scoped sweep renews this node's cert and leaves siblings alone."""
from turnstone.console.tls import TLSManager
from turnstone.core.tls import _SingleDomainStore
from turnstone.core.tls_store import RenewalStoreView
mgr = TLSManager(get_storage())
await mgr.init_ca()
@@ -149,11 +259,101 @@ async def test_renewal_sweep_only_touches_own_domain():
# keeps the sweep from renewing siblings.
rm = lacme.RenewalManager(
ca=mgr._ca,
store=_SingleDomainStore(mgr._store, "server-1"),
store=RenewalStoreView(mgr._store, "server-1"),
days_before_expiry=99999,
)
renewed = await rm.check_and_renew()
assert {b.domain for b in renewed} == {"server-1"}
# Turnstone's CA policy applies to RenewalManager's implicit CA.issue()
# call, and the fresh 48-hour leaf is no longer immediately due.
from datetime import timedelta
from cryptography import x509
leaf = x509.load_pem_x509_certificate(renewed[0].cert_pem)
assert leaf.not_valid_after_utc - leaf.not_valid_before_utc == timedelta(hours=48)
normal_policy = lacme.RenewalManager(
ca=mgr._ca,
store=RenewalStoreView(mgr._store, "server-1"),
days_before_expiry=1,
)
assert await normal_policy.check_and_renew() == []
@pytest.mark.anyio
async def test_renewal_uses_signed_leaf_expiry_not_row_metadata():
"""Mutable DB timestamps cannot suppress or spuriously force renewal."""
from datetime import UTC, datetime, timedelta
from turnstone.console.tls import TLSManager
from turnstone.core.tls_store import RenewalStoreView
mgr = TLSManager(get_storage())
await mgr.init_ca()
live = mgr._ca.issue(["live-node"])
expired = mgr._ca.issue(["expired-node"], validity_hours=0)
future = (datetime.now(UTC) + timedelta(days=365)).isoformat()
past = (datetime.now(UTC) - timedelta(days=365)).isoformat()
get_storage().save_tls_cert(
domain="live-node",
cert_pem=live.cert_pem.decode(),
fullchain_pem=live.fullchain_pem.decode(),
key_pem=live.key_pem.decode(),
issued_at=past,
expires_at=past,
meta='{"domain": "live-node", "domains": ["live-node"], "namespace": ""}',
)
get_storage().save_tls_cert(
domain="expired-node",
cert_pem=expired.cert_pem.decode(),
fullchain_pem=expired.fullchain_pem.decode(),
key_pem=expired.key_pem.decode(),
issued_at=future,
expires_at=future,
meta='{"domain": "expired-node", "domains": ["expired-node"], "namespace": ""}',
)
live_renewal = lacme.RenewalManager(
ca=mgr._ca,
store=RenewalStoreView(mgr._store, "live-node"),
days_before_expiry=1,
)
expired_renewal = lacme.RenewalManager(
ca=mgr._ca,
store=RenewalStoreView(mgr._store, "expired-node"),
days_before_expiry=1,
)
assert await live_renewal.check_and_renew() == []
renewed = await expired_renewal.check_and_renew()
assert [bundle.domain for bundle in renewed] == ["expired-node"]
@pytest.mark.anyio
async def test_renewal_recovers_ip_identifier_types_from_sans():
"""DB metadata stays strings while renewal preserves authoritative IP SANs."""
from cryptography import x509
from turnstone.console.tls import TLSManager
mgr = TLSManager(get_storage())
await mgr.init_ca()
await mgr.issue_console_certs(
[
ipaddress.IPv6Address("2001:0db8::10"),
ipaddress.IPv4Address("192.0.2.10"),
]
)
renewed = mgr.renew_cert("2001:db8::10")
assert renewed.domain == "2001:db8::10"
assert renewed.domains == ("2001:db8::10", "192.0.2.10")
assert _sans(renewed.cert_pem) == [
x509.IPAddress(ipaddress.IPv6Address("2001:db8::10")),
x509.IPAddress(ipaddress.IPv4Address("192.0.2.10")),
]
assert mgr._store.load_cert("2001:db8::10") == renewed
# ── Orphan GC ─────────────────────────────────────────────────────────────────
@@ -161,7 +361,7 @@ async def test_renewal_sweep_only_touches_own_domain():
@pytest.mark.anyio
async def test_gc_removes_only_long_expired_certs():
"""GC reclaims certs expired past the cutoff and keeps live ones."""
"""GC trusts signed leaf expiry, not mutable database timestamps."""
from datetime import UTC, datetime, timedelta
from turnstone.console.tls import TLSManager
@@ -169,22 +369,21 @@ async def test_gc_removes_only_long_expired_certs():
mgr = TLSManager(get_storage())
await mgr.init_ca()
live = mgr._ca.issue(["server-1"])
mgr._store.save_cert(live)
# A decommissioned node's row: reuse real PEMs but stamp it expired-long-ago.
dead = mgr._ca.issue(["dead-node"])
# A stale/tampered metadata timestamp cannot delete a still-valid identity.
old = (datetime.now(UTC) - timedelta(days=30)).isoformat()
get_storage().save_tls_cert(
domain="dead-node",
cert_pem=dead.cert_pem.decode(),
fullchain_pem=dead.fullchain_pem.decode(),
key_pem=dead.key_pem.decode(),
domain="server-1",
cert_pem=live.cert_pem.decode(),
fullchain_pem=live.fullchain_pem.decode(),
key_pem=live.key_pem.decode(),
issued_at=old,
expires_at=old,
meta="{}",
meta='{"domain": "server-1", "domains": ["server-1"], "namespace": ""}',
)
# validity_hours=0 produces a genuinely expired signed leaf.
mgr._ca.issue(["dead-node"], validity_hours=0)
removed = mgr.gc_expired_certs(max_age_days=7)
removed = mgr.gc_expired_certs(max_age_days=0)
assert removed == 1
domains = {b.domain for b in mgr._store.list_certs()}
assert domains == {"server-1"}
@@ -215,7 +414,8 @@ async def test_client_ctx_cached_and_reloaded_in_place():
# ── Server-side renewal → reload-hook wiring ──────────────────────────────────
def test_renew_callback_updates_bundle_and_runs_reload_hook():
@pytest.mark.anyio
async def test_renew_callback_updates_bundle_and_runs_reload_hook():
"""The renewal callback caches the new bundle and fires the reload hook."""
from types import SimpleNamespace
@@ -226,27 +426,41 @@ def test_renew_callback_updates_bundle_and_runs_reload_hook():
client.set_cert_reload_hook(seen.append)
bundle = SimpleNamespace(domain="server-1")
client._handle_renewed(bundle)
await client._handle_renewed(bundle)
assert client.bundle is bundle
assert seen == [bundle]
def test_renew_callback_swallows_reload_hook_errors():
"""A failing reload hook must not abort the renewal callback."""
from types import SimpleNamespace
@pytest.mark.anyio
async def test_renew_callback_rolls_back_persistence_on_reload_error():
"""A failed live reload leaves the previous DB/runtime identity authoritative."""
import lacme
from turnstone.core.tls import TLSClient
client = TLSClient(storage=get_storage(), hostnames=["server-1"])
ca = lacme.CertificateAuthority()
ca.init()
previous = ca.issue(["server-1"])
replacement = ca.issue(["server-1"])
client._store.save_cert(previous)
client._bundle = previous
def _boom(_bundle: object) -> None:
raise RuntimeError("listener swap failed")
reloaded = []
def _boom(bundle: object) -> None:
reloaded.append(bundle)
if bundle is replacement:
raise RuntimeError("listener swap failed")
client.set_cert_reload_hook(_boom)
bundle = SimpleNamespace(domain="server-1")
client._handle_renewed(bundle) # must not raise
assert client.bundle is bundle
with pytest.raises(RuntimeError, match="listener swap failed"):
await client._handle_renewed(replacement)
assert client.bundle is previous
assert client._store.load_cert("server-1") == previous
assert reloaded == [replacement, previous]
# ── swap_context_cert (shared listener/client hot-swap) ───────────────────────
+178 -3
View File
@@ -5,6 +5,7 @@ from __future__ import annotations
import json
from datetime import UTC, datetime
import lacme
import pytest
from turnstone.core.storage import get_storage, init_storage, reset_storage
@@ -165,7 +166,6 @@ def test_adapter_load_ca_missing(store_adapter):
def test_adapter_save_load_cert(store_adapter):
lacme = pytest.importorskip("lacme")
now = datetime.now(UTC)
bundle = lacme.CertBundle(
domain="test.internal",
@@ -186,8 +186,82 @@ def test_adapter_save_load_cert(store_adapter):
assert loaded.key_pem == b"key"
def test_adapter_keyless_save_does_not_create_managed_identity(store_adapter):
"""Responder-side CSR results are not usable identities without a key."""
now = datetime.now(UTC)
bundle = lacme.CertBundle(
domain="new.internal",
domains=("new.internal",),
cert_pem=b"new-cert",
fullchain_pem=b"new-chain",
key_pem=b"",
issued_at=now,
expires_at=now,
)
assert store_adapter.save_cert(bundle) is bundle
assert store_adapter.load_cert("new.internal") is None
assert get_storage().load_tls_cert("new.internal") is None
def test_adapter_keyless_save_preserves_existing_identity(store_adapter):
"""Interrupted reenrollment cannot replace a working cert and private key."""
now = datetime.now(UTC)
existing = lacme.CertBundle(
domain="node.internal",
domains=("node.internal",),
cert_pem=b"old-cert",
fullchain_pem=b"old-chain",
key_pem=b"old-key",
issued_at=now,
expires_at=now,
)
keyless = lacme.CertBundle(
domain="node.internal",
domains=("node.internal",),
cert_pem=b"new-cert",
fullchain_pem=b"new-chain",
key_pem=b"",
issued_at=now,
expires_at=now,
)
store_adapter.save_cert(existing)
store_adapter.save_cert(keyless)
loaded = store_adapter.load_cert("node.internal")
assert loaded == existing
def test_adapter_hides_and_heals_legacy_keyless_rows(store_adapter):
"""Old poisoned rows are ignored until a complete enrollment replaces them."""
now = datetime.now(UTC)
get_storage().save_tls_cert(
domain="legacy.internal",
cert_pem="keyless-cert",
fullchain_pem="keyless-chain",
key_pem="",
issued_at=now.isoformat(),
expires_at=now.isoformat(),
meta='{"domains": ["legacy.internal"]}',
)
assert store_adapter.load_cert("legacy.internal") is None
assert store_adapter.list_certs() == []
complete = lacme.CertBundle(
domain="legacy.internal",
domains=("legacy.internal",),
cert_pem=b"complete-cert",
fullchain_pem=b"complete-chain",
key_pem=b"complete-key",
issued_at=now,
expires_at=now,
)
store_adapter.save_cert(complete)
assert store_adapter.load_cert("legacy.internal") == complete
def test_adapter_list_certs(store_adapter):
lacme = pytest.importorskip("lacme")
now = datetime.now(UTC)
for name in ["alpha", "beta"]:
bundle = lacme.CertBundle(
@@ -211,7 +285,6 @@ def test_adapter_load_cert_missing(store_adapter):
def test_adapter_account_key_roundtrip(store_adapter):
"""Test account key save/load with real cryptography objects."""
pytest.importorskip("lacme")
from cryptography.hazmat.primitives.asymmetric import ec
key = ec.generate_private_key(ec.SECP256R1())
@@ -224,3 +297,105 @@ def test_adapter_account_key_roundtrip(store_adapter):
def test_adapter_account_key_missing(store_adapter):
assert store_adapter.load_account_key() is None
def test_adapter_namespaces_same_domain_certificates():
from turnstone.core.tls_store import StorageStore
now = datetime.now(UTC)
internal = lacme.CertBundle(
domain="console.example",
domains=("console.example",),
cert_pem=b"internal-cert",
fullchain_pem=b"internal-chain",
key_pem=b"internal-key",
issued_at=now,
expires_at=now,
)
frontend = lacme.CertBundle(
domain="console.example",
domains=("console.example",),
cert_pem=b"frontend-cert",
fullchain_pem=b"frontend-chain",
key_pem=b"frontend-key",
issued_at=now,
expires_at=now,
)
internal_store = StorageStore(get_storage())
frontend_store = StorageStore(
get_storage(),
namespace="console-frontend",
account_key_id="console-frontend:test-ca",
)
internal_store.save_cert(internal)
frontend_store.save_cert(frontend)
assert internal_store.load_cert("console.example") == internal
assert frontend_store.load_cert("console.example") == frontend
assert internal_store.list_certs() == [internal]
assert frontend_store.list_certs() == [frontend]
rows = get_storage().list_tls_certs()
assert {row["domain"] for row in rows} == {
"console.example",
"turnstone-scope:console-frontend:console.example",
}
def test_adapter_namespaces_account_keys():
from cryptography.hazmat.primitives.asymmetric import ec
from turnstone.core.tls_store import StorageStore
internal_key = ec.generate_private_key(ec.SECP256R1())
frontend_key = ec.generate_private_key(ec.SECP256R1())
internal_store = StorageStore(get_storage())
frontend_store = StorageStore(
get_storage(),
namespace="console-frontend",
account_key_id="console-frontend:test-ca",
)
internal_store.save_account_key(internal_key)
frontend_store.save_account_key(frontend_key)
loaded_internal = internal_store.load_account_key()
loaded_frontend = frontend_store.load_account_key()
assert loaded_internal is not None
assert loaded_frontend is not None
assert loaded_internal.private_numbers() == internal_key.private_numbers()
assert loaded_frontend.private_numbers() == frontend_key.private_numbers()
def test_validation_store_preserves_existing_bundle_on_rejection(store_adapter):
from turnstone.core.tls_store import CertificateValidationStore
now = datetime.now(UTC)
existing = lacme.CertBundle(
domain="node.internal",
domains=("node.internal",),
cert_pem=b"old-cert",
fullchain_pem=b"old-chain",
key_pem=b"old-key",
issued_at=now,
expires_at=now,
)
rejected = lacme.CertBundle(
domain="node.internal",
domains=("node.internal",),
cert_pem=b"wrong-ca-cert",
fullchain_pem=b"wrong-ca-chain",
key_pem=b"new-key",
issued_at=now,
expires_at=now,
)
store_adapter.save_cert(existing)
validating = CertificateValidationStore(
store_adapter,
lambda _bundle: (_ for _ in ()).throw(ValueError("wrong CA")),
)
with pytest.raises(ValueError, match="wrong CA"):
validating.save_cert(rejected)
assert store_adapter.load_cert("node.internal") == existing
+1
View File
@@ -225,6 +225,7 @@ class TestContextOverflowRecovery:
patch.object(session, "_stream_response", side_effect=mock_stream_response),
patch.object(session, "_compact_messages", compact_mock),
patch.object(session, "_full_messages", return_value=[]),
patch.object(session, "_over_soft", return_value=False),
patch.object(session, "_update_token_table"),
patch.object(session, "_print_status_line"),
patch.object(session, "_emit_state"),
+145
View File
@@ -2,6 +2,10 @@
from __future__ import annotations
import os
import pytest
class TestVersionHtml:
def test_app_css_gets_version(self):
@@ -54,6 +58,13 @@ class TestVersionHtml:
result = version_html(html)
assert result == html # unchanged
def test_vendor_prefix_lookalike_is_still_versioned(self):
from turnstone.core.web_helpers import version_html
html = '<script src="/shared/hls-2-player.js"></script>'
result = version_html(html)
assert "/shared/hls-2-player.js?v=" in result
def test_external_urls_not_modified(self):
from turnstone.core.web_helpers import version_html
@@ -116,6 +127,140 @@ class TestVersionHtml:
assert result == html # unchanged — already has query string
class TestRevalidatingStaticFiles:
def test_same_versioned_url_revalidates_after_asset_changes(self, tmp_path):
from starlette.applications import Starlette
from starlette.routing import Mount
from starlette.testclient import TestClient
from turnstone import __version__
from turnstone.core.web_helpers import RevalidatingStaticFiles
asset = tmp_path / "app.js"
old_body = b"export const generation = 'old';"
new_body = b"export const generation = 'new';"
assert len(old_body) == len(new_body)
asset.write_bytes(old_body)
original_stat = asset.stat()
app = Starlette(
routes=[Mount("/static", app=RevalidatingStaticFiles(directory=str(tmp_path)))]
)
with TestClient(app) as client:
url = f"/static/app.js?v={__version__}"
first = client.get(url)
assert first.status_code == 200
assert first.headers["cache-control"] == "no-cache"
assert "last-modified" not in first.headers
old_etag = first.headers["etag"]
asset.write_bytes(new_body)
os.utime(
asset,
ns=(original_stat.st_atime_ns, original_stat.st_mtime_ns),
)
changed_stat = asset.stat()
assert changed_stat.st_size == original_stat.st_size
assert changed_stat.st_mtime_ns == original_stat.st_mtime_ns
changed = client.get(url, headers={"If-None-Match": old_etag})
assert changed.status_code == 200
assert changed.content == new_body
assert changed.headers["etag"] != old_etag
assert changed.headers["cache-control"] == "no-cache"
assert "last-modified" not in changed.headers
stale_date_only = client.get(
url,
headers={"If-Modified-Since": "Wed, 31 Dec 9999 23:59:59 GMT"},
)
assert stale_date_only.status_code == 200
assert stale_date_only.content == new_body
unchanged = client.get(url, headers={"If-None-Match": changed.headers["etag"]})
assert unchanged.status_code == 304
assert unchanged.headers["cache-control"] == "no-cache"
def test_version_named_vendor_asset_is_immutable(self, tmp_path):
from starlette.applications import Starlette
from starlette.routing import Mount
from starlette.testclient import TestClient
from turnstone.core.web_helpers import RevalidatingStaticFiles
vendor_dir = tmp_path / "katex-0.18.4"
vendor_dir.mkdir()
(vendor_dir / "katex.min.css").write_text(".katex {}", encoding="utf-8")
app = Starlette(
routes=[Mount("/shared", app=RevalidatingStaticFiles(directory=str(tmp_path)))]
)
with TestClient(app) as client:
resp = client.get("/shared/katex-0.18.4/katex.min.css")
assert resp.status_code == 200
assert resp.headers["cache-control"] == "public, max-age=31536000, immutable"
assert resp.headers["etag"]
def test_missing_asset_is_not_cached(self, tmp_path):
from starlette.applications import Starlette
from starlette.routing import Mount
from starlette.testclient import TestClient
from turnstone.core.web_helpers import RevalidatingStaticFiles
app = Starlette(
routes=[Mount("/shared", app=RevalidatingStaticFiles(directory=str(tmp_path)))]
)
with TestClient(app) as client:
resp = client.get("/shared/not-deployed-yet.js")
assert resp.status_code == 404
assert resp.headers["cache-control"] == "no-store"
class TestStaticAssetCacheControl:
@pytest.mark.parametrize(
("path", "expected"),
[
("interactive.js", "no-cache"),
("hls-2-player.js", "no-cache"),
("katex-0.18.4/katex.min.css", "public, max-age=31536000, immutable"),
("hljs-11.11.1/highlight.min.js", "public, max-age=31536000, immutable"),
("katex-0.18.4/../private.json", "no-store"),
(r"katex-0.18.4\..\private.json", "no-store"),
("nested//asset.js", "no-store"),
],
)
def test_policy_requires_a_canonical_exact_vendor_path(self, path, expected):
from turnstone.core.web_helpers import static_asset_cache_control
assert static_asset_cache_control(path) == expected
def test_every_packaged_versioned_vendor_directory_uses_the_shared_policy(self):
import re
from pathlib import Path
import turnstone
from turnstone.core.web_helpers import static_asset_cache_control, version_html
shared_dir = Path(turnstone.__file__).resolve().parent / "shared_static"
versioned_dir = re.compile(r"^[a-z][a-z0-9_-]*-\d+(?:\.\d+)+$")
vendor_dirs = sorted(
path.name
for path in shared_dir.iterdir()
if path.is_dir() and versioned_dir.fullmatch(path.name)
)
assert vendor_dirs
for directory in vendor_dirs:
asset_path = f"{directory}/asset.js"
assert static_asset_cache_control(asset_path) == "public, max-age=31536000, immutable"
html = f'<script src="/shared/{asset_path}"></script>'
assert version_html(html) == html
class TestLatin1SafeFilename:
"""Content-Disposition filename sanitizer — must yield a value that is
both latin-1 encodable (Starlette) and control-char free (h11)."""
+116 -41
View File
@@ -253,18 +253,59 @@ def _cmd_revoke_token(args: argparse.Namespace) -> None:
# ---------------------------------------------------------------------------
def _atomic_write_file(path: str | os.PathLike[str], data: bytes, *, mode: int) -> None:
"""Atomically replace *path* without exposing partial or permissive data."""
import contextlib
import stat
import tempfile
from pathlib import Path
target = Path(path)
target.parent.mkdir(parents=True, exist_ok=True)
absolute_parent = target.parent.absolute()
if absolute_parent.resolve(strict=True) != absolute_parent:
raise RuntimeError(f"Refusing output directory with symlink components: {target.parent}")
parent_stat = target.parent.lstat()
if (
stat.S_ISLNK(parent_stat.st_mode)
or not stat.S_ISDIR(parent_stat.st_mode)
or parent_stat.st_uid != os.geteuid()
):
raise RuntimeError(f"Refusing unsafe output directory: {target.parent}")
if target.is_symlink():
raise RuntimeError(f"Refusing to replace symlink: {target}")
fd, temporary_name = tempfile.mkstemp(prefix=f".{target.name}.", dir=target.parent)
temporary = Path(temporary_name)
try:
os.fchmod(fd, mode)
with os.fdopen(fd, "wb") as handle:
fd = -1
handle.write(data)
handle.flush()
os.fsync(handle.fileno())
os.replace(temporary, target)
directory_flags = os.O_RDONLY | getattr(os, "O_DIRECTORY", 0)
directory_fd = os.open(target.parent, directory_flags)
try:
os.fsync(directory_fd)
finally:
os.close(directory_fd)
finally:
if fd >= 0:
os.close(fd)
with contextlib.suppress(FileNotFoundError):
temporary.unlink()
def _cmd_tls_bootstrap(args: argparse.Namespace) -> None:
"""Initialize CA and issue certs offline."""
try:
from lacme import CertificateAuthority, FileStore
except ImportError:
print("lacme not installed. Run: pip install turnstone[tls]", file=sys.stderr)
sys.exit(1)
import contextlib
import os
from pathlib import Path
from lacme import CertificateAuthority, FileStore
out_dir = Path(args.out)
out_dir.mkdir(parents=True, exist_ok=True)
with contextlib.suppress(PermissionError):
@@ -282,12 +323,17 @@ def _cmd_tls_bootstrap(args: argparse.Namespace) -> None:
os.chmod(ca_cert_path, 0o644)
print(f"CA cert: {ca_cert_path}")
# Issue certs for requested domains
for domain in args.issue:
bundle = ca.issue([domain], validity_hours=48)
store.save_cert(bundle)
cert_dir = out_dir / "certs" / domain
print(f"Issued: {domain} -> {cert_dir}")
from turnstone.core.tls import parse_certificate_identifier
# Issue certs for requested DNS names or IP literals. CertificateAuthority
# saves through FileStore and returns its authoritative encoded paths; IPv6
# and other non-portable keys must never be projected into a guessed path.
for raw_identifier in args.issue:
identifier = parse_certificate_identifier(raw_identifier)
bundle = ca.issue([identifier], validity_hours=48)
if bundle.cert_path is None:
raise RuntimeError("FileStore did not return the issued certificate path")
print(f"Issued: {bundle.domain} -> {bundle.cert_path.parent}")
print(f"\nBootstrap complete. {len(args.issue)} cert(s) issued.")
print(f"CA and certs written to: {out_dir}")
@@ -295,35 +341,58 @@ def _cmd_tls_bootstrap(args: argparse.Namespace) -> None:
def _cmd_tls_issue(args: argparse.Namespace) -> None:
"""Request a cert from the console's ACME endpoint."""
try:
from lacme import SyncClient
except ImportError:
print("lacme not installed. Run: pip install turnstone[tls]", file=sys.stderr)
sys.exit(1)
import os
from pathlib import Path
from lacme import SyncClient
from turnstone.core.auth import (
JWT_AUD_CONSOLE,
TLS_ACME_TOKEN_SOURCE,
ServiceTokenManager,
load_jwt_secret,
)
from turnstone.core.tls import (
TurnstoneAutoApproveChallengeHandler,
build_acme_http_client,
normalize_certificate_identifiers,
)
console_url = args.console_url
if not console_url:
console_url = _discover_console_url()
domains = [args.domain] + args.san
identifiers = normalize_certificate_identifiers([args.domain, *args.san])
identifier_strings = [str(value) for value in identifiers]
directory_url = f"{console_url}/acme/directory"
print(f"Requesting cert for {domains} from {directory_url}")
print(f"Requesting cert for {identifier_strings} from {directory_url}")
client = SyncClient(
directory_url=directory_url,
allow_insecure=True,
enrollment_tokens = ServiceTokenManager(
user_id="admin-cli",
scopes=frozenset({"service"}),
source=TLS_ACME_TOKEN_SOURCE,
secret=load_jwt_secret(),
audience=JWT_AUD_CONSOLE,
)
bundle = client.issue(domains)
http_client = build_acme_http_client(
console_url,
external_url=os.environ.get("TURNSTONE_ACME_EXTERNAL_URL", ""),
token_provider=lambda: enrollment_tokens.token,
)
with SyncClient(
directory_url=directory_url,
http_client=http_client,
challenge_handler=TurnstoneAutoApproveChallengeHandler(),
allow_insecure=True,
) as client:
bundle = client.issue(identifiers)
out_dir = Path(args.out)
out_dir.mkdir(parents=True, exist_ok=True)
(out_dir / "cert.pem").write_bytes(bundle.cert_pem)
(out_dir / "fullchain.pem").write_bytes(bundle.fullchain_pem)
(out_dir / "key.pem").write_bytes(bundle.key_pem)
os.chmod(out_dir / "key.pem", 0o600)
_atomic_write_file(out_dir / "cert.pem", bundle.cert_pem, mode=0o644)
_atomic_write_file(out_dir / "fullchain.pem", bundle.fullchain_pem, mode=0o644)
_atomic_write_file(out_dir / "key.pem", bundle.key_pem, mode=0o600)
print(f"Certificate written to {out_dir}/")
print(" cert.pem (leaf certificate)")
@@ -333,20 +402,21 @@ def _cmd_tls_issue(args: argparse.Namespace) -> None:
def _cmd_tls_ca_cert(args: argparse.Namespace) -> None:
"""Download the CA root certificate from the console."""
import httpx
import httpx2
console_url = args.console_url
if not console_url:
console_url = _discover_console_url()
# Use plain HTTP for bootstrap (node may not have CA cert yet)
# WARNING: This is trust-on-first-use (TOFU) — verify the fingerprint
base = console_url.replace("https://", "http://")
# Preserve an explicitly trusted HTTPS bootstrap endpoint. Plain HTTP is
# still supported for direct trusted-network deployments and remains TOFU.
base = console_url.rstrip("/")
url = f"{base}/acme/ca.pem"
print(f"Fetching CA cert from {url}")
print("WARNING: Fetching over plain HTTP — verify the fingerprint below")
if url.startswith("http://"):
print("WARNING: Fetching over plain HTTP — verify the fingerprint below")
resp = httpx.get(url)
resp = httpx2.get(url, follow_redirects=False, trust_env=False)
resp.raise_for_status()
# Show fingerprint for out-of-band verification
@@ -357,13 +427,13 @@ def _cmd_tls_ca_cert(args: argparse.Namespace) -> None:
from pathlib import Path
Path(args.out).write_bytes(resp.content)
_atomic_write_file(Path(args.out), resp.content, mode=0o644)
print(f"CA cert written to {args.out}")
def _cmd_tls_list(args: argparse.Namespace) -> None:
"""List certificates from the console."""
import httpx
import httpx2
console_url = args.console_url
if not console_url:
@@ -384,7 +454,7 @@ def _cmd_tls_list(args: argparse.Namespace) -> None:
audience=JWT_AUD_CONSOLE,
)
headers["Authorization"] = f"Bearer {mgr.token}"
resp = httpx.get(url, headers=headers)
resp = httpx2.get(url, headers=headers)
resp.raise_for_status()
data = resp.json()
@@ -699,12 +769,17 @@ def main() -> None:
"--issue",
action="append",
default=[],
help="Domain to issue cert for (repeatable)",
help="DNS name or IP address to issue a cert for (repeatable)",
)
p_issue = sub.add_parser("tls-issue", help="Request cert from console ACME")
p_issue.add_argument("domain", help="Primary domain for the certificate")
p_issue.add_argument("--san", action="append", default=[], help="Additional SAN (repeatable)")
p_issue.add_argument("domain", help="Primary DNS name or IP address for the certificate")
p_issue.add_argument(
"--san",
action="append",
default=[],
help="Additional DNS or IP SAN (repeatable)",
)
p_issue.add_argument("--out", default=".", help="Output directory for PEM files")
p_issue.add_argument(
"--console-url", default="", help="Console URL (discovered from DB if empty)"
@@ -714,7 +789,7 @@ def main() -> None:
p_cacert.add_argument("--out", default="ca.pem", help="Output file path")
p_cacert.add_argument("--console-url", default="", help="Console URL")
p_tlslist = sub.add_parser("tls-list", help="List issued certificates")
p_tlslist = sub.add_parser("tls-list", help="List managed certificates")
p_tlslist.add_argument("--console-url", default="", help="Console URL")
# Node metadata commands
+1 -1
View File
@@ -1172,7 +1172,7 @@ CONSOLE_ENDPOINTS: list[EndpointSpec] = [
EndpointSpec(
"/v1/api/admin/tls/certs",
"GET",
"List all issued TLS certificates",
"List all managed TLS certificates",
tags=["Admin"],
),
EndpointSpec(
+19 -5
View File
@@ -753,10 +753,20 @@ MemoryScope = Literal["global", "workstream", "user"]
class SaveMemoryRequest(BaseModel):
name: str = Field(description="Memory identifier (normalized to snake_case)")
content: str = Field(description="Memory content", max_length=65536)
description: str = Field(default="", description="Short description for relevance matching")
type: MemoryType = Field(default="general", description="Memory type")
name: str = Field(
description="Memory identifier (normalized to snake_case)",
min_length=1,
max_length=256,
)
content: str = Field(description="Memory content", min_length=1, max_length=65536)
description: str = Field(
description="Required non-empty description used for relevance matching",
min_length=1,
)
type: MemoryType | None = Field(
default=None,
description="Memory type; omission preserves it on update and defaults on insert",
)
scope: MemoryScope = Field(default="global", description="Memory scope")
scope_id: str = Field(
default="",
@@ -765,6 +775,8 @@ class SaveMemoryRequest(BaseModel):
@model_validator(mode="after")
def _validate_scope_scope_id(self) -> SaveMemoryRequest:
if not self.description.strip():
raise ValueError("description is required and must be non-empty")
scope_id = self.scope_id.strip()
if self.scope == "global" and scope_id:
raise ValueError("scope_id is not allowed with global scope")
@@ -795,7 +807,7 @@ MemoryScopeFilter = Literal["", "global", "workstream", "user"]
class SearchMemoriesRequest(BaseModel):
query: str = Field(description="Search query text")
query: str = Field(description="Search query text", min_length=1)
type: MemoryTypeFilter = Field(default="", description="Filter by memory type")
scope: MemoryScopeFilter = Field(default="", description="Filter by scope")
scope_id: str = Field(default="", description="Filter by scope_id")
@@ -808,6 +820,8 @@ class SearchMemoriesRequest(BaseModel):
raise ValueError("scope_id is not allowed with global scope")
if scope_id and not self.scope:
raise ValueError("scope is required when scope_id is provided")
if self.scope == "workstream" and not scope_id:
raise ValueError("scope_id is required for workstream scope")
return self
+7 -5
View File
@@ -496,16 +496,17 @@ SERVER_ENDPOINTS: list[EndpointSpec] = [
EndpointSpec(
"/v1/api/memories",
"GET",
"List structured memories",
"List structured memories. Without a scope, returns global plus the authenticated user's memories; workstream scope is owner-bound.",
response_model=ListMemoriesResponse,
query_params=[
QueryParam("type", "Filter by memory type"),
QueryParam("scope", "Filter by scope"),
QueryParam("scope", "Filter by public scope: global, workstream, or user"),
QueryParam("scope_id", "Filter by scope identifier"),
QueryParam(
"limit", "Max results (default 100, max 200)", schema_type="integer", default=100
),
],
error_codes=[400, 403, 404, 500],
tags=["Memories"],
),
EndpointSpec(
@@ -514,15 +515,16 @@ SERVER_ENDPOINTS: list[EndpointSpec] = [
"Save (upsert) a structured memory",
request_model=SaveMemoryRequest,
response_model=MemoryInfo,
error_codes=[400],
error_codes=[400, 403, 404, 500],
tags=["Memories"],
),
EndpointSpec(
"/v1/api/memories/search",
"POST",
"Search structured memories by query",
"Search structured memories by query. Without a scope, searches global plus the authenticated user's memories.",
request_model=SearchMemoriesRequest,
response_model=ListMemoriesResponse,
error_codes=[400, 403, 404, 500],
tags=["Memories"],
),
EndpointSpec(
@@ -534,7 +536,7 @@ SERVER_ENDPOINTS: list[EndpointSpec] = [
QueryParam("scope", "Scope (default: global)"),
QueryParam("scope_id", "Scope identifier"),
],
error_codes=[404],
error_codes=[400, 403, 404, 500],
tags=["Memories"],
),
# --- Admin settings ---
+107 -64
View File
@@ -14,6 +14,7 @@ import argparse
import asyncio
import contextlib
import functools
import hashlib
import json
import logging
import math
@@ -38,7 +39,6 @@ from starlette.background import BackgroundTask
from starlette.middleware import Middleware
from starlette.responses import HTMLResponse, JSONResponse, Response, StreamingResponse
from starlette.routing import Mount, Route
from starlette.staticfiles import StaticFiles
from turnstone.api.console_spec import build_console_spec
from turnstone.api.docs import make_docs_handler, make_openapi_handler
@@ -76,7 +76,10 @@ from turnstone.core.model_registry import (
from turnstone.core.model_registry import MODEL_AUTH_MODES as _MODEL_AUTH_MODES
from turnstone.core.rendezvous import NoAvailableNodeError, NodeRef
from turnstone.core.rerank_calibrate import canonical_caps_value
from turnstone.core.session_replay import session_replay_preamble
from turnstone.core.session_replay import (
request_replay_project_name,
session_replay_preamble,
)
from turnstone.core.session_routes import (
AttachmentUploadHelpers,
CoordOnlyVerbHandlers,
@@ -107,8 +110,12 @@ from turnstone.core.skill_field_validation import SKILL_RUNTIME_CONFIG_FIELDS
from turnstone.core.skill_kind import SkillKind
from turnstone.core.skill_parser import MAX_SKILL_DESCRIPTION_LEN
from turnstone.core.web_helpers import (
RevalidatingStaticFiles,
is_safe_static_asset_path,
read_json_or_400,
require_storage_or_503,
static_asset_cache_control,
version_html,
)
from turnstone.core.workstream import (
Workstream,
@@ -144,10 +151,6 @@ _HTML_ETAG = ""
def _load_static() -> None:
import hashlib
from turnstone.core.web_helpers import version_html
global _HTML, _HTML_ETAG
_HTML = version_html((_STATIC_DIR / "index.html").read_text(encoding="utf-8"))
_HTML_ETAG = '"' + hashlib.md5(_HTML.encode()).hexdigest()[:16] + '"' # noqa: S324
@@ -3029,52 +3032,66 @@ async def proxy_index(request: Request) -> Response:
return JSONResponse({"error": "Node unreachable"}, status_code=502)
async def proxy_static(request: Request) -> Response:
"""GET /node/{node_id}/static/{path} — proxy static files."""
def _proxy_static_request_headers(request: Request) -> dict[str, str]:
"""Build upstream headers for a cache-aware static asset request."""
headers = _proxy_auth_headers(request)
for name in ("if-none-match", "if-modified-since"):
value = request.headers.get(name)
if value:
headers[name] = value
return headers
def _proxy_static_response(resp: httpx.Response, path: str) -> Response:
"""Preserve upstream validators and apply the local static cache policy."""
cache_control = "no-store"
if resp.status_code in (200, 304):
cache_control = resp.headers.get("cache-control") or static_asset_cache_control(path)
headers = {"Cache-Control": cache_control}
for name in ("content-type", "etag", "last-modified"):
value = resp.headers.get(name)
if value:
headers[name] = value
return Response(content=resp.content, status_code=resp.status_code, headers=headers)
def _static_proxy_error(message: str, status_code: int) -> JSONResponse:
return JSONResponse(
{"error": message}, status_code=status_code, headers={"Cache-Control": "no-store"}
)
async def _proxy_static_mount(request: Request, mount: str) -> Response:
"""Proxy one validated static mount without URL-normalization ambiguity."""
node_id = request.path_params["node_id"]
path = request.path_params["path"]
if not is_safe_static_asset_path(path):
return _static_proxy_error("Invalid static asset path", 400)
server_url = _get_server_url(request, node_id)
if not server_url:
return JSONResponse({"error": "Node not found"}, status_code=404)
return _static_proxy_error("Node not found", 404)
encoded_path = "/".join(urllib.parse.quote(segment, safe="") for segment in path.split("/"))
client: httpx.AsyncClient = request.app.state.proxy_client
try:
resp = await client.get(
f"{server_url}/static/{path}",
headers=_proxy_auth_headers(request),
)
return Response(
content=resp.content,
status_code=resp.status_code,
media_type=resp.headers.get("content-type", "application/octet-stream"),
f"{server_url}/{mount}/{encoded_path}",
headers=_proxy_static_request_headers(request),
)
return _proxy_static_response(resp, path)
except httpx.HTTPError as exc:
log.debug("Proxy static error for %s/%s: %s", node_id, path, exc)
return JSONResponse({"error": "Node unreachable"}, status_code=502)
log.debug("Proxy %s error for %s/%s: %s", mount, node_id, path, exc)
return _static_proxy_error("Node unreachable", 502)
async def proxy_static(request: Request) -> Response:
"""GET /node/{node_id}/static/{path} — proxy static files."""
return await _proxy_static_mount(request, "static")
async def proxy_shared_static(request: Request) -> Response:
"""GET /node/{node_id}/shared/{path} — proxy shared static files."""
node_id = request.path_params["node_id"]
path = request.path_params["path"]
server_url = _get_server_url(request, node_id)
if not server_url:
return JSONResponse({"error": "Node not found"}, status_code=404)
client: httpx.AsyncClient = request.app.state.proxy_client
try:
resp = await client.get(
f"{server_url}/shared/{path}",
headers=_proxy_auth_headers(request),
)
return Response(
content=resp.content,
status_code=resp.status_code,
media_type=resp.headers.get("content-type", "application/octet-stream"),
)
except httpx.HTTPError as exc:
log.debug("Proxy shared static error for %s/%s: %s", node_id, path, exc)
return JSONResponse({"error": "Node unreachable"}, status_code=502)
return await _proxy_static_mount(request, "shared")
# Auth endpoints the console handles locally instead of forwarding to
@@ -3687,20 +3704,15 @@ def _audit_retry_coordinator(
def _coord_events_replay(
ws: Workstream,
ui: Any,
request: Request, # noqa: ARG001 — coord replay doesn't need request context
request: Request,
) -> Iterable[dict[str, Any]]:
"""Initial SSE replay payload for coord ``events`` connections.
Yields, in order:
1. ``connected`` + optional ``status`` via the shared
:func:`turnstone.core.session_replay.session_replay_preamble`
so the dashboard's status bar populates before any live tick.
Same payload shape interactive uses.
2. Pending approval prompt (if any) and the cached LLM verdicts
that fired since it surfaced. Without this replay a refresh
loses the judge chip on the pending approval until the
operator re-invokes the action.
Yields ``connected`` plus optional ``status``, then the pending approval
prompt (if any) and cached LLM verdicts that fired since it surfaced. The
shared handler resolves viewer-specific project metadata off-loop before
invoking this callback. Without the control replay a refresh loses the
judge chip until the operator re-invokes the action.
Coord still skips conversation history the dashboard fetches it
via a separate ``GET /history`` endpoint and doesn't want a
@@ -3708,7 +3720,11 @@ def _coord_events_replay(
Pure read never mutates ``ui`` / ``ws`` / ``session``.
"""
yield from session_replay_preamble(ws.session, ui)
yield from session_replay_preamble(
ws.session,
ui,
project_name=request_replay_project_name(request),
)
# EVERY live approval cycle replays (parallel task agents can have
# several outstanding), each card followed once by the cached LLM
@@ -3963,14 +3979,18 @@ async def coordinator_page(request: Request) -> Response:
if not template_path.is_file():
return JSONResponse({"error": "coordinator UI template missing"}, status_code=500)
try:
body = template_path.read_text(encoding="utf-8")
body = version_html(template_path.read_text(encoding="utf-8"))
except OSError:
return JSONResponse({"error": "failed to read coordinator UI template"}, status_code=500)
# Inject the ws_id as an HTML attribute. ws_id passed the
# ``_VALID_WS_ID_RE`` gate above (hex only) so there's nothing
# to HTML-escape; leave the replacement simple.
body = body.replace("{{WS_ID}}", ws_id)
return Response(body, media_type="text/html; charset=utf-8")
etag = '"' + hashlib.md5(body.encode()).hexdigest()[:16] + '"' # noqa: S324
headers = {"Cache-Control": "no-cache", "ETag": etag}
if request.headers.get("If-None-Match") == etag:
return Response(status_code=304, headers=headers)
return HTMLResponse(body, headers=headers)
_CHILDREN_PAGE_LIMIT = 200
@@ -9544,12 +9564,14 @@ async def admin_delete_memory(request: Request) -> JSONResponse:
return err
memory_id = request.path_params["memory_id"]
existing = storage.get_structured_memory(memory_id)
try:
existing = storage.delete_structured_memory_by_id_returning(memory_id)
except Exception:
log.warning("memory.admin_delete_failed memory_id=%s", memory_id, exc_info=True)
return JSONResponse({"error": "Failed to delete memory"}, status_code=500)
if not existing:
return JSONResponse({"error": "Memory not found"}, status_code=404)
storage.delete_structured_memory_by_id(memory_id)
audit_uid, ip = _audit_context(request)
record_audit(
storage,
@@ -15564,7 +15586,7 @@ async def tls_ca_status(request: Request) -> JSONResponse:
async def tls_list_certs(request: Request) -> JSONResponse:
"""GET /v1/api/admin/tls/certs — List issued certificates."""
"""GET /v1/api/admin/tls/certs — List managed certificates."""
from turnstone.core.auth import require_permission
err = require_permission(request, "admin.settings")
@@ -15582,6 +15604,8 @@ async def tls_list_certs(request: Request) -> JSONResponse:
"domains": list(c.domains),
"issued_at": c.issued_at.isoformat(),
"expires_at": c.expires_at.isoformat(),
"renewable": mgr.can_renew_cert(c.domain),
"deletable": mgr.can_delete_cert(c.domain),
}
for c in certs
],
@@ -15591,6 +15615,7 @@ async def tls_list_certs(request: Request) -> JSONResponse:
async def tls_renew_cert(request: Request) -> JSONResponse:
"""POST /v1/api/admin/tls/certs/{domain}/renew — Force cert renewal."""
from turnstone.console.tls import CertificateNotLocallyManagedError
from turnstone.core.auth import require_permission
err = require_permission(request, "admin.settings")
@@ -15609,6 +15634,8 @@ async def tls_renew_cert(request: Request) -> JSONResponse:
"expires_at": bundle.expires_at.isoformat(),
},
)
except CertificateNotLocallyManagedError as e:
return JSONResponse({"error": str(e)}, status_code=409)
except ValueError as e:
return JSONResponse({"error": str(e)}, status_code=404)
except Exception as e:
@@ -15617,6 +15644,7 @@ async def tls_renew_cert(request: Request) -> JSONResponse:
async def tls_delete_cert(request: Request) -> JSONResponse:
"""DELETE /v1/api/admin/tls/certs/{domain} — Delete a certificate."""
from turnstone.console.tls import CertificateNotLocallyManagedError
from turnstone.core.auth import require_permission
err = require_permission(request, "admin.settings")
@@ -15626,7 +15654,11 @@ async def tls_delete_cert(request: Request) -> JSONResponse:
if mgr is None or not mgr.ca_initialized:
return JSONResponse({"error": "TLS not enabled"}, status_code=404)
domain = request.path_params["domain"]
if not mgr.delete_cert(domain):
try:
deleted = mgr.delete_cert(domain)
except CertificateNotLocallyManagedError as e:
return JSONResponse({"error": str(e)}, status_code=409)
if not deleted:
return JSONResponse({"error": f"No cert for {domain}"}, status_code=404)
return JSONResponse({"deleted": domain})
@@ -16467,8 +16499,16 @@ def create_app(
Route("/metrics", console_metrics_endpoint),
Route("/openapi.json", _openapi_handler),
Route("/docs", _docs_handler),
Mount("/static", app=StaticFiles(directory=str(_STATIC_DIR)), name="static"),
Mount("/shared", app=StaticFiles(directory=str(_SHARED_DIR)), name="shared"),
Mount(
"/static",
app=RevalidatingStaticFiles(directory=str(_STATIC_DIR)),
name="static",
),
Mount(
"/shared",
app=RevalidatingStaticFiles(directory=str(_SHARED_DIR)),
name="shared",
),
# Coordinator one-pane UI — the route serves a single
# index.html template with the ws_id injected via data-ws-id
# so coordinator.js can pull it without an extra round-trip.
@@ -16510,7 +16550,9 @@ def create_app(
app.state.console_metrics = console_metrics or ConsoleMetrics()
# Mount ACME responder whenever a TLS manager is configured.
# ACMEResponder (lacme 1.0.2+) serves /ca.pem natively.
# ACMEResponder serves /ca.pem natively. AuthMiddleware leaves only
# directory/nonce/CA bootstrap resources public; every signing route needs
# the dedicated service enrollment JWT.
if tls_manager is not None:
from starlette.routing import Mount as RouteMount
@@ -16717,7 +16759,11 @@ def main() -> None:
if _cs.get("tls.enabled"):
from turnstone.console.tls import TLSManager
tls_mgr = TLSManager(auth_storage, config_store=_cs)
tls_mgr = TLSManager(
auth_storage,
config_store=_cs,
acme_external_url=os.environ.get("TURNSTONE_ACME_EXTERNAL_URL") or None,
)
# Init CA before create_app so ACME responder can be mounted
import asyncio
@@ -16728,9 +16774,6 @@ def main() -> None:
# / docs/tls.md); rewriting the scheme to https here would
# advertise an ACME URL nodes can't reach.
log.info("TLS enabled")
except ImportError:
log.warning("TLS enabled but lacme not installed — pip install turnstone[tls]")
tls_mgr = None
except Exception:
log.warning("TLS initialization failed", exc_info=True)
tls_mgr = None
+14 -7
View File
@@ -3887,24 +3887,31 @@ function loadTlsCerts() {
const colActions = document.createElement("span");
colActions.className = "admin-col admin-col-actions";
const kebab = _kebabMenuEl([
{
const actions = [];
if (c.renewable) {
actions.push({
label: "Renew",
attrs: {
"data-tls-renew": c.domain,
"aria-label": "Renew certificate for " + c.domain,
},
},
{
});
}
if (c.deletable) {
actions.push({
label: "Delete",
kind: "danger",
attrs: {
"data-tls-delete": c.domain,
"aria-label": "Delete certificate for " + c.domain,
},
},
]);
colActions.appendChild(kebab);
});
}
if (actions.length > 0) {
colActions.appendChild(_kebabMenuEl(actions));
} else {
colActions.textContent = "Managed by node";
}
row.appendChild(colDomain);
row.appendChild(colSans);
+15 -8
View File
@@ -1491,12 +1491,20 @@ function _homeRenderChips() {
}
function _homeStageFile(file) {
if (!file) return;
if (!file) return false;
const paste = window.TurnstonePasteText;
if (
paste &&
paste.isDuplicatePastedTextFile &&
paste.isDuplicatePastedTextFile(file, _homeStagedFiles)
) {
return true;
}
if (_homeStagedFiles.length >= _HOME_MAX_FILES) {
_homeShowError(
"At most " + _HOME_MAX_FILES + " attachments per coordinator",
);
return;
return false;
}
if (!_homeIsAttachmentAllowed(file)) {
_homeShowError(
@@ -1504,17 +1512,18 @@ function _homeStageFile(file) {
file.name +
" (allowed: png/jpeg/gif/webp images, text)",
);
return;
return false;
}
const isImage = (file.type || "").indexOf("image/") === 0;
const cap = isImage ? _HOME_IMAGE_CAP : _HOME_TEXT_CAP;
if (file.size > cap) {
_homeShowError(file.name + " exceeds the " + _homeFormatSize(cap) + " cap");
return;
return false;
}
_homeShowError("");
_homeStagedFiles.push(file);
_homeRenderChips();
return true;
}
function _homeClearStagedFiles() {
@@ -1807,7 +1816,7 @@ function _mountHomeCoordComposer() {
},
attachments: {
onAttach: function (file) {
_homeStageFile(file);
return _homeStageFile(file);
},
},
dragDrop: { targetEl: mount, dropClass: "home-coord-drop" },
@@ -1962,9 +1971,7 @@ function submitHomeCoord(textFromComposer) {
// pending storage rows until the GC sweep. Require text whenever
// attachments are staged so the first turn always picks them up.
if (files.length > 0 && !(task || "").trim()) {
_homeShowError(
"Add a task message — attachments need an initial turn to dispatch on.",
);
_homeShowError("Add a message to send with this attachment.");
return;
}
const shared = {
@@ -50,6 +50,7 @@ import {
acceptUserTurnEvent,
clientSendMaySettleForViewer,
createQueueController,
mergeRejectedComposerText,
mintClientSendId,
parsePriority,
postAndSettleSend,
@@ -2656,9 +2657,34 @@ function createCoordinatorPane(root, wsId, opts) {
function coordSend() {
const text = composer.value;
const trimmed = (text || "").trim();
if (!trimmed) return false;
if (!trimmed) {
if (!attachments.isEmpty()) {
appendText("info", "Add a message to send with this attachment.", {
label: "info",
});
}
return false;
}
// The live-turn interjection queue is text-only. Keep the input and chips
// intact instead of optimistically clearing them before attachments_busy.
if (busy && !attachments.isEmpty()) {
appendText(
"info",
"Attachments can't be sent while the assistant is working. Wait for it to finish, then send again.",
{ label: "info" },
);
return false;
}
const snap = attachments.snapshot();
if (snap.uploading) {
appendText(
"info",
"Wait for attachments to finish uploading before sending.",
{ label: "info" },
);
return false;
}
let queuedEl = null;
let optimisticEl = null;
@@ -2734,6 +2760,9 @@ function createCoordinatorPane(root, wsId, opts) {
setBusy: (b) => setBusy(b),
busyIsOptimistic: () => busy && busySource === "optimistic",
paneIsBusy: () => busy,
restoreInput: () => {
composer.value = mergeRejectedComposerText(trimmed, composer.value);
},
renderError: (msg) => appendText("error", msg, { label: "error" }),
consumeAttachments: (attached, droppedIds) =>
attachments.consume(attached, droppedIds),
@@ -4278,6 +4307,12 @@ function createCoordinatorPane(root, wsId, opts) {
setBusy: (value) => setBusy(value),
busyIsOptimistic: () => busy && busySource === "optimistic",
paneIsBusy: () => busy,
restoreInput: () => {
composer.value = mergeRejectedComposerText(
editText,
composer.value,
);
},
renderError: (message) =>
appendText("error", message, { label: "error" }),
consumeAttachments: () => {},
@@ -43,6 +43,7 @@
<script type="module" src="/shared/toast.js"></script>
<script type="module" src="/shared/auth.js"></script>
<script type="module" src="/shared/kb.js"></script>
<script type="module" src="/shared/composer_paste_text.js"></script>
<script type="module" src="/shared/composer.js"></script>
<script type="module" src="/shared/composer_attachments.js"></script>
<script type="module" src="/shared/composer_queue.js"></script>
+1
View File
@@ -4116,6 +4116,7 @@
<script type="module" src="/shared/models.js"></script>
<script type="module" src="/shared/skills.js"></script>
<script type="module" src="/shared/project_creator.js"></script>
<script type="module" src="/shared/composer_paste_text.js"></script>
<script type="module" src="/shared/composer.js"></script>
<!-- coordinator-pane deps (step 4): the controller builds its chrome + uses
the shared composer/renderer stack; load before the shell module -->
+280 -127
View File
@@ -7,12 +7,37 @@ CA and ACME server, issuing short-lived mTLS certificates to all services.
from __future__ import annotations
from typing import TYPE_CHECKING, Any
import hashlib
from datetime import UTC, datetime
from typing import TYPE_CHECKING
from urllib.parse import urlsplit
import lacme
import structlog
from lacme import (
ACMEResponder,
CertBundle,
CertificateAuthority,
IdentifierValue,
RenewalManager,
)
from lacme.events import (
CertificateExpiring,
CertificateIssued,
CertificateRenewed,
ChallengeFailed,
)
from turnstone.core.tls import (
CERT_VALIDITY_HOURS,
RENEW_BEFORE_EXPIRY_DAYS,
RENEW_INTERVAL_HOURS,
RENEW_MAX_JITTER_SECONDS,
)
if TYPE_CHECKING:
import ssl
from collections.abc import Sequence
from starlette.types import ASGIApp
@@ -25,19 +50,66 @@ log = structlog.get_logger(__name__)
_CA_CN = "Turnstone CA"
_CA_NAME = "turnstone" # Store key for save_ca/load_ca
_CA_VALIDITY_DAYS = 3650 # 10 years
_CERT_VALIDITY_HOURS = 48
_RENEW_INTERVAL_HOURS = 24
_RENEW_BEFORE_EXPIRY_DAYS = 1
def _require_lacme() -> Any:
class CertificateNotLocallyManagedError(RuntimeError):
"""Raised when an admin action would take ownership from another issuer."""
class _TurnstoneCertificateAuthority(CertificateAuthority):
"""Pin lacme's implicit responder/renewal lifetime to Turnstone policy.
lacme's public CA methods default to 24-hour leaves, and its responder and
CA-mode RenewalManager intentionally call those methods without a lifetime.
Turnstone advertises and schedules around 48-hour cluster identities, so
the CA owned by this manager supplies that default at the shared chokepoint.
Explicit non-default lifetimes remain available to callers.
"""
def issue(
self,
names: IdentifierValue | Sequence[IdentifierValue],
*,
client: bool = False,
validity_days: int = CERT_VALIDITY_HOURS // 24,
validity_hours: int | None = None,
) -> CertBundle:
return super().issue(
names,
client=client,
validity_days=validity_days,
validity_hours=validity_hours,
)
def issue_from_csr(
self,
csr_der: bytes,
*,
validated_identifiers: Sequence[IdentifierValue] | None = None,
client: bool = False,
validity_days: int = CERT_VALIDITY_HOURS // 24,
validity_hours: int | None = None,
) -> CertBundle:
return super().issue_from_csr(
csr_der,
validated_identifiers=validated_identifiers,
client=client,
validity_days=validity_days,
validity_hours=validity_hours,
)
def _certificate_not_valid_after(bundle: CertBundle) -> datetime:
"""Read expiry from the signed leaf rather than mutable store metadata."""
from cryptography import x509
try:
import lacme
except ImportError:
raise ImportError(
"lacme is required for TLS support. Install with: pip install turnstone[tls]",
) from None
return lacme
certificates = x509.load_pem_x509_certificates(bundle.cert_pem)
except (TypeError, ValueError) as exc:
raise ValueError(f"Certificate {bundle.domain!r} is malformed") from exc
if len(certificates) != 1:
raise ValueError(f"Certificate {bundle.domain!r} does not contain one leaf")
return certificates[0].not_valid_after_utc
class TLSManager:
@@ -49,27 +121,28 @@ class TLSManager:
await mgr.init_ca()
responder = mgr.get_responder() # Mount at /acme
await mgr.issue_console_certs() # Self-issue for this node
mgr.start_renewal() # Background auto-renewal
await mgr.start_renewal() # Background auto-renewal
"""
def __init__(
self,
storage: StorageBackend,
config_store: ConfigStore | None = None,
acme_external_url: str | None = None,
) -> None:
lacme = _require_lacme()
from turnstone.core.tls_store import StorageStore
self._storage = storage
self._store = StorageStore(storage)
self._frontend_store: StorageStore | None = None
self._config_store = config_store
self._acme_external_url = acme_external_url
self._event_dispatcher = lacme.EventDispatcher()
self._ca: Any | None = None
self._responder: Any | None = None
self._renewal_task: Any | None = None
self._renewal_manager: Any | None = None
self._internal_bundle: Any | None = None
self._frontend_bundle: Any | None = None
self._ca: _TurnstoneCertificateAuthority | None = None
self._responder: ACMEResponder | None = None
self._renewal_manager: RenewalManager | None = None
self._internal_bundle: CertBundle | None = None
self._frontend_bundle: CertBundle | None = None
# Cached mTLS client context, mutated in place on renewal so the
# proxy/collector httpx clients pick up the renewed client cert
# without being rebuilt.
@@ -93,29 +166,18 @@ class TLSManager:
def _subscribe_events(self) -> None:
"""Subscribe structlog handlers to lacme lifecycle events."""
_require_lacme()
from lacme.events import (
CertificateExpiring,
CertificateIssued,
CertificateRenewed,
ChallengeFailed,
)
def _on_issued(event: Any) -> None:
if isinstance(event, CertificateIssued):
log.info("tls.cert.issued", domain=event.domain)
def _on_issued(event: CertificateIssued) -> None:
log.info("tls.cert.issued", domain=event.domain)
def _on_renewed(event: Any) -> None:
if isinstance(event, CertificateRenewed):
log.info("tls.cert.renewed", domain=event.domain)
def _on_renewed(event: CertificateRenewed) -> None:
log.info("tls.cert.renewed", domain=event.domain)
def _on_expiring(event: Any) -> None:
if isinstance(event, CertificateExpiring):
log.warning("tls.cert.expiring", domain=event.domain)
def _on_expiring(event: CertificateExpiring) -> None:
log.warning("tls.cert.expiring", domain=event.domain)
def _on_failed(event: Any) -> None:
if isinstance(event, ChallengeFailed):
log.error("tls.challenge.failed", domain=getattr(event, "domain", "unknown"))
def _on_failed(event: ChallengeFailed) -> None:
log.error("tls.challenge.failed", domain=event.domain)
self._event_dispatcher.subscribe(_on_issued, event_type=CertificateIssued)
self._event_dispatcher.subscribe(_on_renewed, event_type=CertificateRenewed)
@@ -131,12 +193,10 @@ class TLSManager:
into the database store first so the console uses the same CA that
signed the infrastructure certs.
"""
lacme = _require_lacme()
# Import bootstrap CA from well-known volume path if not already in DB
self._import_bootstrap_ca()
self._ca = lacme.CertificateAuthority(
self._ca = _TurnstoneCertificateAuthority(
self._store,
name=_CA_NAME,
event_dispatcher=self._event_dispatcher,
@@ -176,23 +236,29 @@ class TLSManager:
"""Return the ACME responder ASGI app for mounting."""
if self._ca is None:
raise RuntimeError("CA not initialized — call init_ca() first")
lacme = _require_lacme()
if self._responder is None:
if self._acme_external_url is not None:
path = urlsplit(self._acme_external_url).path.rstrip("/")
if not path.endswith("/acme"):
raise ValueError(
"TURNSTONE_ACME_EXTERNAL_URL must include the responder's /acme mount"
)
self._responder = lacme.ACMEResponder(
ca=self._ca,
auto_approve=True,
external_url=self._acme_external_url,
)
return self._responder # type: ignore[no-any-return]
return self._responder
def get_root_cert_pem(self) -> bytes:
"""Return the CA root certificate in PEM format."""
if self._ca is None:
raise RuntimeError("CA not initialized — call init_ca() first")
return self._ca.root_cert_pem # type: ignore[no-any-return]
return self._ca.root_cert_pem
# -- Cert issuance ---------------------------------------------------------
async def issue_console_certs(self, hostnames: list[str]) -> None:
async def issue_console_certs(self, hostnames: Sequence[IdentifierValue]) -> None:
"""Issue certificates for the console node.
Raises ValueError if hostnames is empty.
@@ -201,11 +267,12 @@ class TLSManager:
- Internal cert: always from the internal CA (for mTLS with cluster)
- Frontend cert: from external ACME CA if configured, else internal CA
"""
if not hostnames:
raise ValueError("issue_console_certs requires at least one hostname")
from turnstone.core.tls import normalize_certificate_identifiers
identifiers = normalize_certificate_identifiers(hostnames)
# Internal cert — always from our own CA
await self._issue_internal_cert(hostnames)
await self._issue_internal_cert(identifiers)
# Frontend cert — external CA if configured
acme_directory = ""
@@ -213,128 +280,157 @@ class TLSManager:
acme_directory = self._config_store.get("tls.acme_directory") or ""
if acme_directory:
await self._issue_frontend_cert(hostnames, acme_directory)
await self._issue_frontend_cert(identifiers, acme_directory)
else:
# Self-issue from internal CA (behind reverse proxy or internal only)
self._frontend_store = None
self._frontend_bundle = self._internal_bundle
log.info("tls.frontend.self_issued", hostnames=hostnames)
log.info("tls.frontend.self_issued", hostnames=[str(value) for value in identifiers])
async def _issue_internal_cert(self, hostnames: list[str]) -> None:
async def _issue_internal_cert(self, identifiers: Sequence[IdentifierValue]) -> None:
"""Issue an internal mTLS cert from the internal CA."""
if self._ca is None:
raise RuntimeError("CA not initialized")
# Check for existing cert in store (skip if expired)
existing = self._store.load_cert(hostnames[0])
from turnstone.core.tls import validate_cluster_certificate_bundle
primary = str(identifiers[0])
# Reuse only when the complete identity is valid for this exact active
# CA and typed SAN set. A legacy bundle keyed by "192.0.2.10" may hold
# DNS:192.0.2.10, and a pre-namespace frontend bundle may be signed by
# a public CA despite sharing the same string key.
existing = self._store.load_cert(primary)
if existing is not None:
from datetime import UTC, datetime
if existing.expires_at > datetime.now(UTC):
try:
validate_cluster_certificate_bundle(
existing,
identifiers,
self.get_root_cert_pem(),
)
except ValueError as exc:
log.warning(
"tls.internal.invalid_existing_identity",
domain=primary,
error=str(exc),
)
else:
self._internal_bundle = existing
log.info("tls.internal.loaded", domain=hostnames[0])
log.info("tls.internal.loaded", domain=primary)
return
log.info("tls.internal.expired", domain=hostnames[0])
self._store.delete_cert(hostnames[0])
# Issue new cert
# CA issuance saves the complete bundle through StorageStore, replacing
# the old row only after the new key and certificate both exist.
bundle = self._ca.issue(
hostnames,
validity_hours=_CERT_VALIDITY_HOURS,
identifiers,
validity_hours=CERT_VALIDITY_HOURS,
)
self._store.save_cert(bundle)
self._internal_bundle = bundle
log.info("tls.internal.issued", domain=hostnames[0])
log.info("tls.internal.issued", domain=primary)
async def _issue_frontend_cert(
self,
hostnames: list[str],
identifiers: Sequence[IdentifierValue],
acme_directory: str,
) -> None:
"""Issue a frontend cert from an external ACME CA."""
lacme = _require_lacme()
from lacme.challenges.http01 import HTTP01Handler
from turnstone.core.tls_store import StorageStore
handler = HTTP01Handler()
profile = hashlib.sha256(acme_directory.rstrip("/").encode()).hexdigest()
frontend_store = StorageStore(
self._storage,
namespace="console-frontend",
account_key_id=f"console-frontend:{profile}",
)
async with lacme.Client(
directory_url=acme_directory,
store=self._store,
store=frontend_store,
challenge_handler=handler,
event_dispatcher=self._event_dispatcher,
) as client:
self._frontend_bundle = await client.issue(hostnames)
self._store.save_cert(self._frontend_bundle)
self._frontend_bundle = await client.issue(identifiers)
self._frontend_store = frontend_store
log.info(
"tls.frontend.issued",
domain=hostnames[0],
domain=str(identifiers[0]),
ca=acme_directory,
)
# -- Auto-renewal ----------------------------------------------------------
async def start_renewal(self) -> None:
"""Start background auto-renewal for all stored certificates.
"""Start CA-direct auto-renewal for the console's internal certificate.
Uses CA-direct mode (lacme 1.0.2+) signs directly via the CA
without going through ACME. No loopback client, no network,
no startup ordering dependency.
Other nodes renew their own identities, and an externally issued
frontend certificate is deliberately excluded. This signs directly via
the CA without a loopback client or startup-ordering dependency.
"""
if self._ca is None:
raise RuntimeError("CA not initialized")
lacme = _require_lacme()
from turnstone.core.tls import _SingleDomainStore
if self._renewal_manager is not None:
raise RuntimeError("TLS renewal is already running")
from turnstone.core.tls_store import RenewalStoreView
def _on_renewed(bundle: Any) -> None:
# Update our cached bundles if the renewed domain matches
def _on_renewed(bundle: CertBundle) -> None:
# Only the internal scoped store reaches this callback. A
# same-domain frontend identity lives in a separate store and must
# never be replaced by the cluster CA.
if self._internal_bundle and bundle.domain == self._internal_bundle.domain:
self._internal_bundle = bundle
# Swap the renewed material into the live mTLS client context so
# proxy/collector connections present the new cert. Contain
# failures (matching the node-side reload hook) so a reload
# error can't abort the callback before the frontend-bundle
# update below or perturb the renewal sweep.
previous = self._internal_bundle
try:
self._reload_client_ctx(bundle)
except Exception:
try:
self._reload_client_ctx(previous)
except Exception:
log.error("tls.client_ctx.rollback_failed", exc_info=True)
self._store.save_cert(previous)
log.warning("tls.client_ctx.reload_failed", exc_info=True)
if self._frontend_bundle and bundle.domain == self._frontend_bundle.domain:
raise
self._internal_bundle = bundle
if self._frontend_store is None and self._frontend_bundle:
self._frontend_bundle = bundle
# Scope the sweep to the console's own cert; the store is shared, so an
# unscoped CA-direct sweep would re-sign every node's cert. Empty domain
# → renew nothing (never the whole store). An external-ACME frontend
# cert has a different domain and is intentionally excluded (re-signing
# an externally-issued cert with the internal CA would break it).
# cert is held in its own namespace and intentionally excluded even
# when it has the same primary domain; the cluster CA must not re-sign it.
own_domain = self._internal_bundle.domain if self._internal_bundle is not None else ""
renewal_store = _SingleDomainStore(self._store, own_domain)
renewal_store = RenewalStoreView(self._store, own_domain)
self._renewal_manager = lacme.RenewalManager(
manager = lacme.RenewalManager(
ca=self._ca,
store=renewal_store,
interval_hours=_RENEW_INTERVAL_HOURS,
days_before_expiry=_RENEW_BEFORE_EXPIRY_DAYS,
interval_hours=RENEW_INTERVAL_HOURS,
days_before_expiry=RENEW_BEFORE_EXPIRY_DAYS,
max_jitter_seconds=RENEW_MAX_JITTER_SECONDS,
on_renewed=_on_renewed,
event_dispatcher=self._event_dispatcher,
)
self._renewal_task = self._renewal_manager.start()
manager.start()
self._renewal_manager = manager
log.info(
"tls.renewal.started",
interval_hours=_RENEW_INTERVAL_HOURS,
interval_hours=RENEW_INTERVAL_HOURS,
)
async def stop_renewal(self) -> None:
"""Stop the background renewal task."""
if self._renewal_task is not None:
import asyncio
import contextlib
manager, self._renewal_manager = self._renewal_manager, None
if manager is not None:
from turnstone.core.tls import complete_tls_cleanup
self._renewal_task.cancel()
try:
with contextlib.suppress(asyncio.CancelledError):
await self._renewal_task
except Exception:
log.exception("tls.renewal.stop_error")
self._renewal_task = None
async def _cleanup() -> None:
try:
await manager.stop()
except Exception:
log.exception("tls.renewal.stop_error")
await complete_tls_cleanup(_cleanup())
# -- SSL contexts ----------------------------------------------------------
@@ -346,10 +442,9 @@ class TLSManager:
"""
if self._frontend_bundle is None:
return None
_require_lacme()
from lacme.mtls import server_ssl_context
return server_ssl_context( # type: ignore[no-any-return,unused-ignore]
return server_ssl_context(
cert_pem=self._frontend_bundle.fullchain_pem,
key_pem=self._frontend_bundle.key_pem,
ca_cert_pem=self.get_root_cert_pem(),
@@ -364,7 +459,6 @@ class TLSManager:
if self._internal_bundle is None:
return None
if self._client_ctx is None:
_require_lacme()
from lacme.mtls import client_ssl_context
self._client_ctx = client_ssl_context(
@@ -374,7 +468,7 @@ class TLSManager:
)
return self._client_ctx
def _reload_client_ctx(self, bundle: Any) -> None:
def _reload_client_ctx(self, bundle: CertBundle) -> None:
"""Load a renewed bundle into the cached mTLS client context in place.
httpx clients built with this context present the new client cert on
@@ -390,16 +484,25 @@ class TLSManager:
def gc_expired_certs(self, max_age_days: int = 7) -> int:
"""Delete stored certs that expired more than ``max_age_days`` ago.
A live node keeps its cert's ``expires_at`` in the future, so only
decommissioned-node (or legacy container-ID) rows go stale; deleting
them well past expiry is safe. Returns the number of rows removed.
The signed leaf, rather than mutable database metadata, determines the
cutoff. Only decommissioned-node (or legacy container-ID) rows go stale;
deleting them well past expiry is safe. Returns the number removed.
"""
from datetime import UTC, datetime, timedelta
cutoff = datetime.now(UTC) - timedelta(days=max_age_days)
removed = 0
for bundle in self._store.list_certs():
if bundle.expires_at < cutoff and self._store.delete_cert(bundle.domain):
try:
not_valid_after = _certificate_not_valid_after(bundle)
except ValueError as exc:
log.warning(
"tls.certs.gc_invalid_identity",
domain=bundle.domain,
error=str(exc),
)
continue
if not_valid_after < cutoff and self._store.delete_cert(bundle.domain):
removed += 1
if removed:
log.info("tls.certs.gc", removed=removed, max_age_days=max_age_days)
@@ -407,40 +510,90 @@ class TLSManager:
# -- Properties ------------------------------------------------------------
def list_certs(self) -> list[Any]:
"""List all stored certificate bundles."""
def list_certs(self) -> list[CertBundle]:
"""List all managed certificate identities."""
return self._store.list_certs()
def renew_cert(self, domain: str) -> Any:
"""Force-renew a certificate by domain. Returns the new bundle."""
def renew_cert(self, domain: str) -> CertBundle:
"""Force-renew the console-owned internal identity."""
if self._ca is None:
raise RuntimeError("CA not initialized")
existing = self._store.load_cert(domain)
if existing is None:
raise ValueError(f"No certificate for {domain}")
# Issue new cert first, then delete old (safe if issuance fails)
bundle = self._ca.issue(list(existing.domains))
self._store.delete_cert(domain)
self._store.save_cert(bundle)
# Update in-memory bundles if this is the console's own cert
if self._internal_bundle and bundle.domain == self._internal_bundle.domain:
self._internal_bundle = bundle
if self._frontend_bundle and bundle.domain == self._frontend_bundle.domain:
if self._internal_bundle is None or domain != self._internal_bundle.domain:
raise CertificateNotLocallyManagedError(
f"Certificate {domain!r} is owned by a remote node or external ACME issuer"
)
# CA issuance saves the complete replacement atomically through the
# store, preserving the previous usable row if issuance fails.
from turnstone.core.tls import certificate_bundle_identifiers
identifiers = certificate_bundle_identifiers(existing)
frontend_is_internal = self._frontend_store is None
try:
bundle = self._ca.issue(identifiers, validity_hours=CERT_VALIDITY_HOURS)
self._reload_client_ctx(bundle)
except BaseException:
# CA.issue persists before returning. Restore the prior complete
# identity if live-context installation fails so the DB does not
# claim a rotation the process could not adopt.
try:
self._reload_client_ctx(existing)
except Exception:
log.error("tls.client_ctx.rollback_failed", exc_info=True)
self._store.save_cert(existing)
raise
self._internal_bundle = bundle
if frontend_is_internal:
self._frontend_bundle = bundle
return bundle
def delete_cert(self, domain: str) -> bool:
"""Delete a certificate by domain."""
"""Delete an expired, remotely managed certificate by domain."""
existing = self._store.load_cert(domain)
if existing is None:
return False
if self._internal_bundle is not None and domain == self._internal_bundle.domain:
raise CertificateNotLocallyManagedError(
"The console's active internal certificate cannot be deleted"
)
try:
expired = _certificate_not_valid_after(existing) <= datetime.now(UTC)
except ValueError as exc:
raise CertificateNotLocallyManagedError(
f"Certificate {domain!r} is malformed and cannot be safely deleted"
) from exc
if not expired:
raise CertificateNotLocallyManagedError(
f"Active certificate {domain!r} must be rotated or removed by its owning node"
)
return self._store.delete_cert(domain)
def can_renew_cert(self, domain: str) -> bool:
"""Return whether this process owns and can hot-reload *domain*."""
return self._internal_bundle is not None and domain == self._internal_bundle.domain
def can_delete_cert(self, domain: str) -> bool:
"""Return whether *domain* is an expired remote identity."""
if self.can_renew_cert(domain):
return False
existing = self._store.load_cert(domain)
if existing is None:
return False
try:
return _certificate_not_valid_after(existing) <= datetime.now(UTC)
except ValueError:
return False
@property
def ca_initialized(self) -> bool:
return self._ca is not None
@property
def internal_bundle(self) -> Any | None:
def internal_bundle(self) -> CertBundle | None:
return self._internal_bundle
@property
def frontend_bundle(self) -> Any | None:
def frontend_bundle(self) -> CertBundle | None:
return self._frontend_bundle
+42 -2
View File
@@ -71,6 +71,7 @@ JWT_ISSUER = "turnstone"
JWT_AUD_SERVER = "turnstone-server"
JWT_AUD_CONSOLE = "turnstone-console"
JWT_AUD_CHANNEL = "turnstone-channel"
TLS_ACME_TOKEN_SOURCE = "tls-acme-enrollment"
_MIN_SECRET_LENGTH = 32 # 256 bits minimum for HMAC-SHA256
VALID_SCOPES: frozenset[str] = frozenset({"read", "write", "approve", "service"})
@@ -685,7 +686,19 @@ PUBLIC_PATHS: frozenset[str] = frozenset(
"/api/auth/oidc/callback",
}
)
PUBLIC_PREFIXES: tuple[str, ...] = ("/static/", "/shared/", "/acme/")
PUBLIC_PREFIXES: tuple[str, ...] = ("/static/", "/shared/")
# ACME discovery/bootstrap material is public because a node does not have a
# cluster identity yet. Everything capable of creating or changing an order
# is authenticated separately below: lacme's lightweight responder deliberately
# does not validate ACME JWS signatures or nonces itself.
ACME_PUBLIC_PATHS: frozenset[str] = frozenset(
{
"/acme/directory",
"/acme/new-nonce",
"/acme/ca.pem",
}
)
WRITE_PATHS: frozenset[str] = frozenset(
{
@@ -719,6 +732,14 @@ def _strip_version_prefix(path: str) -> str:
return path
def _is_protected_acme_path(path: str) -> bool:
"""Return whether *path* is an authenticated ACME responder resource."""
normalized = _strip_version_prefix(path)
return (
normalized == "/acme" or normalized.startswith("/acme/")
) and normalized not in ACME_PUBLIC_PATHS
# ---------------------------------------------------------------------------
# AuthResult
# ---------------------------------------------------------------------------
@@ -1011,7 +1032,7 @@ def validate_jwt(token: str, secret: str, audience: str = "") -> AuthResult | No
def is_public_path(path: str) -> bool:
"""Return *True* if the path should be accessible without authentication."""
normalized = _strip_version_prefix(path)
if normalized in PUBLIC_PATHS:
if normalized in PUBLIC_PATHS or normalized in ACME_PUBLIC_PATHS:
return True
if any(normalized.startswith(prefix) for prefix in PUBLIC_PREFIXES):
return True
@@ -1039,6 +1060,13 @@ def required_scope(method: str, path: str) -> str:
normalized = _strip_version_prefix(path)
normalized = normalized.rstrip("/") if normalized != "/" else normalized
# The responder auto-approves challenges, so every non-bootstrap ACME route
# is a cluster-CA signing surface. ``service`` cannot be granted through
# user-facing token mints; check_request additionally pins the token source
# to the dedicated enrollment identity.
if normalized == "/acme" or normalized.startswith("/acme/"):
return "service"
# Admin endpoints require approve scope
if normalized.startswith(ADMIN_PREFIX):
return "approve"
@@ -1229,6 +1257,13 @@ def check_request(
if result is None:
return False, 401, "Unauthorized: missing or invalid token", None
is_protected_acme = _is_protected_acme_path(path)
is_enrollment_token = result.token_source == TLS_ACME_TOKEN_SOURCE
if is_protected_acme and not is_enrollment_token:
return False, 403, "Forbidden: token is not valid for ACME enrollment", None
if is_enrollment_token and not is_protected_acme:
return False, 403, "Forbidden: ACME enrollment token is not valid for this resource", None
# Version gate — reject tokens minted by a different major.minor.
# Tokens without a ``ver`` claim are accepted (backward compat).
if jwt_version and result.token_version and result.token_version != jwt_version:
@@ -1631,6 +1666,11 @@ async def handle_auth_login(request: Request, audience: str, cookie_name: str) -
jwt_audience=audience,
storage=storage,
)
# Enrollment JWTs are capabilities for the protected ACME responder,
# not general service credentials. In particular, never let the
# legacy token-exchange path extend one into a fresh 24-hour session.
if result is not None and result.token_source == TLS_ACME_TOKEN_SOURCE:
result = None
if result is None:
# Record failed attempt for rate limiting
+256 -111
View File
@@ -507,6 +507,45 @@ def _cap_server_prompts(server_name: str, prompts: list[Any]) -> list[Any]:
return prompts[:_MAX_PROMPTS_PER_SERVER]
async def _list_resources_compatible(session: Any, server_name: str) -> Any:
"""List concrete resources, treating an unsupported method as empty.
MCP exposes one aggregate ``resources`` capability for both concrete
resources and resource templates. Servers may legitimately implement only
one of the two list methods, so the capability bit alone cannot tell us
which request is supported. Only the protocol's exact METHOD_NOT_FOUND code
is normalized; every other error remains a real discovery failure.
"""
try:
return await session.list_resources()
except McpError as exc:
if exc.error.code != mcp_types.METHOD_NOT_FOUND:
raise
log.debug(
"MCP server '%s' does not implement resources/list; treating it as empty",
server_name,
)
return mcp_types.ListResourcesResult(resources=[])
async def _list_resource_templates_compatible(session: Any, server_name: str) -> Any:
"""List resource templates, treating an unsupported method as empty.
See :func:`_list_resources_compatible` for why the aggregate capability
requires per-method probing and exact JSON-RPC error classification.
"""
try:
return await session.list_resource_templates()
except McpError as exc:
if exc.error.code != mcp_types.METHOD_NOT_FOUND:
raise
log.debug(
"MCP server '%s' does not implement resources/templates/list; treating it as empty",
server_name,
)
return mcp_types.ListResourceTemplatesResult(resourceTemplates=[])
# ---------------------------------------------------------------------------
# Per-server state containers
# ---------------------------------------------------------------------------
@@ -2336,96 +2375,127 @@ class MCPClientManager:
state.close_requested = close_requested
state.session = session
# Check push notification support for each capability
caps = session.get_server_capabilities()
# Capability flags and all three catalogs are STAGED until discovery
# succeeds in full. A later resource/prompt failure must not publish a
# callable tool backed by a registration that add_server_sync reports
# as failed. The live owner/session must likewise be torn down before
# the error escapes.
try:
caps = session.get_server_capabilities()
tools_cap = getattr(caps, "tools", None) if caps else None
state.supports_list_changed = bool(getattr(tools_cap, "listChanged", False))
tools_cap = getattr(caps, "tools", None) if caps else None
supports_list_changed = bool(getattr(tools_cap, "listChanged", False))
resources_cap = getattr(caps, "resources", None) if caps else None
state.supports_resources = resources_cap is not None
state.supports_resource_list_changed = bool(getattr(resources_cap, "listChanged", False))
resources_cap = getattr(caps, "resources", None) if caps else None
supports_resources = resources_cap is not None
supports_resource_list_changed = bool(getattr(resources_cap, "listChanged", False))
prompts_cap = getattr(caps, "prompts", None) if caps else None
state.supports_prompts = prompts_cap is not None
state.supports_prompt_list_changed = bool(getattr(prompts_cap, "listChanged", False))
prompts_cap = getattr(caps, "prompts", None) if caps else None
supports_prompts = prompts_cap is not None
supports_prompt_list_changed = bool(getattr(prompts_cap, "listChanged", False))
# Discover tools. Discovery runs in THIS caller task while the
# transport is hosted by the owner, so a transport collapse
# mid-discovery cancels the OWNER, not us — ``_await_owner_discovery``
# races the owner so that death surfaces as a prompt ConnectionError
# instead of hanging to the caller-side attempt timeout.
result = await self._await_owner_discovery(owner, session.list_tools())
capped = _cap_server_tools(name, result.tools)
server_tools: list[dict[str, Any]] = [_mcp_to_openai(name, tool) for tool in capped]
# Discover tools. Discovery runs in THIS caller task while the
# transport is hosted by the owner, so a transport collapse
# mid-discovery cancels the OWNER, not us — ``_await_owner_discovery``
# races the owner so that death surfaces as a prompt ConnectionError
# instead of hanging to the caller-side attempt timeout.
result = await self._await_owner_discovery(owner, session.list_tools())
capped = _cap_server_tools(name, result.tools)
server_tools: list[dict[str, Any]] = [_mcp_to_openai(name, tool) for tool in capped]
state.tools = server_tools
self._rebuild_tools()
# Discover resources
resource_count = 0
if resources_cap is not None:
# Discover resources. The protocol advertises the pair with one
# aggregate capability, but either list method may independently be
# absent; the compatibility wrappers normalize only -32601.
server_resources: list[dict[str, Any]] = []
res_result = await self._await_owner_discovery(owner, session.list_resources())
# Capped like the tools list above (and like the pool twins): a
# misbehaving server must not balloon the shared node's merged
# catalogs.
for r in _cap_server_resources(name, res_result.resources):
server_resources.append(
{
"uri": str(r.uri),
"name": r.name or "",
"description": r.description or "",
"mimeType": r.mimeType or "",
"server": name,
}
if resources_cap is not None:
res_result = await self._await_owner_discovery(
owner, _list_resources_compatible(session, name)
)
# Also include resource templates (catalog-only — not directly
# readable via read_resource since they contain URI placeholders)
tmpl_result = await self._await_owner_discovery(
owner, session.list_resource_templates()
)
for t in _cap_server_resource_templates(name, tmpl_result.resourceTemplates):
server_resources.append(
{
"uri": str(t.uriTemplate),
"name": t.name or "",
"description": t.description or "",
"mimeType": t.mimeType or "",
"server": name,
"template": True,
}
# Capped like the tools list above (and like the pool twins): a
# misbehaving server must not balloon the shared node's merged
# catalogs.
for r in _cap_server_resources(name, res_result.resources):
server_resources.append(
{
"uri": str(r.uri),
"name": r.name or "",
"description": r.description or "",
"mimeType": r.mimeType or "",
"server": name,
}
)
# Templates are catalog-only — not directly readable via
# read_resource since they contain URI placeholders.
tmpl_result = await self._await_owner_discovery(
owner, _list_resource_templates_compatible(session, name)
)
resource_count = len(server_resources)
state.resources = server_resources
self._rebuild_resources()
for t in _cap_server_resource_templates(name, tmpl_result.resourceTemplates):
server_resources.append(
{
"uri": str(t.uriTemplate),
"name": t.name or "",
"description": t.description or "",
"mimeType": t.mimeType or "",
"server": name,
"template": True,
}
)
# Discover prompts
prompt_count = 0
if prompts_cap is not None:
# Discover prompts.
server_prompts: list[dict[str, Any]] = []
prompt_result = await self._await_owner_discovery(owner, session.list_prompts())
for p in _cap_server_prompts(name, prompt_result.prompts):
server_prompts.append(
{
"name": f"mcp__{name}__{p.name}",
"original_name": p.name,
"server": name,
"description": p.description or "",
"arguments": [
{
"name": a.name,
"description": a.description or "",
"required": a.required or False,
}
for a in (p.arguments or [])
],
}
)
prompt_count = len(server_prompts)
state.prompts = server_prompts
if prompts_cap is not None:
prompt_result = await self._await_owner_discovery(owner, session.list_prompts())
for p in _cap_server_prompts(name, prompt_result.prompts):
server_prompts.append(
{
"name": f"mcp__{name}__{p.name}",
"original_name": p.name,
"server": name,
"description": p.description or "",
"arguments": [
{
"name": a.name,
"description": a.description or "",
"required": a.required or False,
}
for a in (p.arguments or [])
],
}
)
# The owner done-callback can evict the session in the same loop
# turn that the final discovery call completes. Revalidate the
# exact wiring immediately before commit; there are no awaits from
# this check through publication, so a callable catalog can never
# be installed behind a dead/replaced transport.
if owner.done() or state.owner_task is not owner or state.session is not session:
raise ConnectionError(f"MCP server '{name}' transport died before catalog commit")
except BaseException:
await self._teardown_static_session(name)
raise
# Publish the staged connection and catalogs only after every enabled
# discovery method succeeded. Successful reconnects also clear a stale
# resource/prompt catalog when the server drops that capability.
resources_changed = state.resources != server_resources
prompts_changed = state.prompts != server_prompts
state.supports_list_changed = supports_list_changed
state.supports_resources = supports_resources
state.supports_resource_list_changed = supports_resource_list_changed
state.supports_prompts = supports_prompts
state.supports_prompt_list_changed = supports_prompt_list_changed
state.tools = server_tools
state.resources = server_resources
state.prompts = server_prompts
self._rebuild_tools()
if resources_changed:
self._rebuild_resources()
if prompts_changed:
self._rebuild_prompts()
resource_count = len(server_resources)
prompt_count = len(server_prompts)
push_parts: list[str] = []
if state.supports_list_changed:
push_parts.append("tools")
@@ -2776,23 +2846,27 @@ class MCPClientManager:
call.cancel()
await asyncio.gather(call, return_exceptions=True)
raise
if call.done():
if call.cancelled():
# The discovery future was cancelled out from under us (an
# SDK-internal cancellation shape, not this method's own reap)
# — the transport can no longer answer, which is the same
# failure class as the owner dying. Convert instead of leaking
# a bare CancelledError the caller would misread as its own
# cancellation.
raise ConnectionError("MCP discovery request cancelled by transport failure")
return call.result() # normal result, or the real discovery error
# Owner finished first — the transport died under discovery. Reap the
# discovery future (gather absorbs its cancellation outcome; a caller
# cancellation arriving DURING the reap propagates instead — honoring
# the cancel beats reporting the dead transport).
call.cancel()
await asyncio.gather(call, return_exceptions=True)
raise ConnectionError("MCP transport owner died during discovery")
# If both futures completed in the same scheduling turn, owner death
# wins. Returning a successful list result after its backing transport
# has already exited would let the caller publish a catalog with no
# live session. Gathering a completed call also retrieves any exception
# it carried, avoiding an unobserved-task warning.
if owner.done():
if not call.done():
call.cancel()
await asyncio.gather(call, return_exceptions=True)
raise ConnectionError("MCP transport owner died during discovery")
# FIRST_COMPLETED returned normally and the owner is not done, so the
# discovery future is necessarily the completed member of the pair.
if call.cancelled():
# The discovery future was cancelled out from under us (an
# SDK-internal cancellation shape, not this method's own reap)
# — the transport can no longer answer, which is the same
# failure class as the owner dying. Convert instead of leaking
# a bare CancelledError the caller would misread as its own
# cancellation.
raise ConnectionError("MCP discovery request cancelled by transport failure")
return call.result() # normal result, or the real discovery error
async def _connect_one_pool(
self,
@@ -2985,10 +3059,7 @@ class MCPClientManager:
# ordering is irrelevant.
res_result, tmpl_result = await self._await_owner_discovery(
owner,
asyncio.gather(
session.list_resources(),
session.list_resource_templates(),
),
self._list_resource_pair(session, server_name),
)
except asyncio.CancelledError:
task = asyncio.current_task()
@@ -3068,6 +3139,15 @@ class MCPClientManager:
}
)
# Mirror the static commit guard. A transport owner can finish in the
# same scheduling turn as the final discovery response; never publish
# that response into the per-user maps after its session was evicted.
if owner.done() or entry.owner_task is not owner or entry.session is not session:
await self._teardown_pool_entry(key)
raise ConnectionError(
f"MCP pool server '{server_name}' transport died before catalog commit"
)
entry.tools = server_tools
entry.resources = server_resources if resources_cap is not None else None
entry.prompts = server_prompts if prompts_cap is not None else None
@@ -4320,7 +4400,7 @@ class MCPClientManager:
)
return added, removed
async def _list_resource_pair(self, session: Any) -> tuple[Any, Any]:
async def _list_resource_pair(self, session: Any, server_name: str) -> tuple[Any, Any]:
"""``list_resources`` + ``list_resource_templates`` in one bounded RTT.
The ONE copy of the paired-list protocol for both refresh twins
@@ -4329,8 +4409,10 @@ class MCPClientManager:
timeout budget and target disjoint catalogs (resources vs.
templates), so ordering is irrelevant.
Fail-FAST on the first real error a fast, meaningful failure
(an auth or method rejection) must surface as ITSELF, not be
Each method independently treats an exact JSON-RPC METHOD_NOT_FOUND as
an empty half-catalog. Fail-FAST on the first remaining real error a
fast, meaningful failure (for example auth or invalid params) must
surface as ITSELF, not be
masked behind a hung sibling's eventual ``TimeoutError`` — but
with the surviving sibling CANCELLED and REAPED inside this
scope before the error re-raises: bare fail-fast ``gather``
@@ -4350,8 +4432,10 @@ class MCPClientManager:
task, GC-reaped) rather than held onto.
"""
async with asyncio.timeout(self._CONNECT_TIMEOUT):
res_task = asyncio.create_task(session.list_resources())
tmpl_task = asyncio.create_task(session.list_resource_templates())
res_task = asyncio.create_task(_list_resources_compatible(session, server_name))
tmpl_task = asyncio.create_task(
_list_resource_templates_compatible(session, server_name)
)
try:
res_result, tmpl_result = await asyncio.gather(res_task, tmpl_task)
except BaseException:
@@ -4424,7 +4508,7 @@ class MCPClientManager:
user_id, server_name = key
old_uris = {r["uri"] for r in (entry.resources or []) if not r.get("template")}
res_result, tmpl_result = await self._list_resource_pair(session)
res_result, tmpl_result = await self._list_resource_pair(session, server_name)
if self._user_pool_entries.get(key) is not entry:
# Entry replaced mid-flight — stale result, discard.
return [], []
@@ -5035,7 +5119,7 @@ class MCPClientManager:
# turn the second list_resource_templates() call into AttributeError.
session = state.session
res_result, tmpl_result = await self._list_resource_pair(session)
res_result, tmpl_result = await self._list_resource_pair(session, name)
server_resources: list[dict[str, Any]] = []
# Capped like the pool twin — a misbehaving server's push must not
@@ -5687,15 +5771,76 @@ class MCPClientManager:
"error": "MCP event loop not running",
}
# Add to config so _refresh_all can reconnect on failure
# Add to config so _refresh_all can reconnect on failure.
self._server_configs[name] = cfg
future = asyncio.run_coroutine_threadsafe(self._connect_one(name, cfg), self._loop)
def _clear_failed_add_state() -> None:
"""Remove this registration's config and every public catalog."""
if self._server_configs.get(name) is cfg:
self._server_configs.pop(name, None)
failed_state = self._static_servers.get(name)
tools_changed = bool(failed_state and failed_state.tools)
resources_changed = bool(failed_state and failed_state.resources)
prompts_changed = bool(failed_state and failed_state.prompts)
if failed_state is not None:
failed_state.tools = []
failed_state.resources = []
failed_state.prompts = []
failed_state.supports_list_changed = False
failed_state.supports_resources = False
failed_state.supports_resource_list_changed = False
failed_state.supports_prompts = False
failed_state.supports_prompt_list_changed = False
if tools_changed:
self._rebuild_tools()
if resources_changed:
self._rebuild_resources()
if prompts_changed:
self._rebuild_prompts()
async def _add() -> None:
# Keep failure rollback under the SAME per-name lock as connect.
# The timeout is loop-owned: the sync bridge waits for a definitive
# outcome instead of racing a second wall-clock deadline against
# rollback. If the loop is temporarily blocked, this may report a
# late success, but it can never report failure while a published
# session/catalog remains live.
async with self._static_connect_lock_for(name):
try:
async with asyncio.timeout(timeout):
await self._connect_one_locked(name, cfg)
except BaseException as exc:
try:
await self._teardown_static_session(name)
finally:
# A newer concurrent add may already have replaced the
# config while waiting on this lock. Remove only the
# registration this coroutine owns; its successor will
# run after this cleanup releases the lock.
_clear_failed_add_state()
if isinstance(exc, TimeoutError):
raise TimeoutError(f"MCP server '{name}' registration timed out") from None
raise
future = asyncio.run_coroutine_threadsafe(_add(), self._loop)
try:
future.result(timeout=timeout)
# ``_add`` owns both its deadline and rollback. A caller-side
# timeout here could return a failed result while cleanup is merely
# queued on a stalled loop, exposing a live unconfigured tool.
future.result()
except concurrent.futures.TimeoutError:
return {
"connected": False,
"tools": 0,
"resources": 0,
"prompts": 0,
"error": f"MCP server '{name}' registration timed out",
}
except Exception as exc:
# Remove from configs on failure
self._server_configs.pop(name, None)
# The loop-side rollback normally removed this exact registration;
# retain a defensive cleanup for failures before _add could start.
if self._server_configs.get(name) is cfg:
self._server_configs.pop(name, None)
return {"connected": False, "tools": 0, "resources": 0, "prompts": 0, "error": str(exc)}
state = self._static_servers.get(name)
+98 -10
View File
@@ -855,13 +855,22 @@ def search_history_recent(limit: int = 20, *, user_id: str | None = None) -> lis
# -- Structured memories -------------------------------------------------------
def _require_memory_description(description: str) -> str:
"""Return a normalized description or raise the public validation error."""
if not isinstance(description, str) or not (normalized := description.strip()):
raise ValueError("memory description is required and must be non-empty")
return normalized
def save_structured_memory(
name: str,
content: str,
description: str | None = None,
description: str,
mem_type: str | None = None,
scope: str = "global",
scope_id: str = "",
*,
require_active_project: bool = False,
) -> tuple[dict[str, str] | None, bool]:
"""Save a structured memory as a single atomic upsert by name+scope+scope_id.
@@ -871,23 +880,65 @@ def save_structured_memory(
:meth:`StorageBackend.upsert_structured_memory`) -- no preceding read, no
IntegrityError round-trip, no TOCTOU window. ``(row, was_update)`` comes
straight from that upsert (this passes a fresh ``memory_id``, so a differing
returned id means an existing row was updated in place). A ``None``
description / ``mem_type`` means "leave unset" -- the column default applies
on insert and the stored value is kept on conflict.
returned id means an existing row was updated in place). ``description``
is required and must contain non-whitespace text for both inserts and
updates. A ``None`` ``mem_type`` keeps the stored value on an update and
uses the column default on insert.
"""
import uuid
name = normalize_key(name)
# Validate outside the best-effort storage boundary. Backend/driver
# ``ValueError`` instances remain operational failures; only this explicit
# caller-input check propagates.
normalized_description = _require_memory_description(description)
try:
row, was_update = get_storage().upsert_structured_memory(
str(uuid.uuid4()), name, description, mem_type, scope, scope_id, content
return save_structured_memory_strict(
name,
content,
description=normalized_description,
mem_type=mem_type,
scope=scope,
scope_id=scope_id,
require_active_project=require_active_project,
)
return (row, was_update) if row else (None, False)
except Exception:
log.warning("Failed to save structured memory name=%s", name, exc_info=True)
return None, False
def save_structured_memory_strict(
name: str,
content: str,
description: str,
mem_type: str | None = None,
scope: str = "global",
scope_id: str = "",
*,
require_active_project: bool = False,
) -> tuple[dict[str, str], bool]:
"""Strict structured-memory upsert for mutation-facing boundaries.
Unlike :func:`save_structured_memory`, storage failures propagate so an
API or tool cannot report a database outage as an ordinary failed/not-found
result. Best-effort internal callers keep using the facade.
"""
import uuid
normalized = normalize_key(name)
normalized_description = _require_memory_description(description)
row, was_update = get_storage().upsert_structured_memory(
str(uuid.uuid4()),
normalized,
normalized_description,
mem_type,
scope,
scope_id,
content,
require_active_project=require_active_project,
)
if not row:
raise RuntimeError("structured memory upsert returned no row")
return row, was_update
def get_structured_memory_by_name(
name: str, scope: str = "global", scope_id: str = ""
) -> dict[str, str] | None:
@@ -900,6 +951,13 @@ def get_structured_memory_by_name(
return None
def get_structured_memory_by_name_strict(
name: str, scope: str = "global", scope_id: str = ""
) -> dict[str, str] | None:
"""Strict scoped-name lookup; storage failures propagate."""
return get_storage().get_structured_memory_by_name(normalize_key(name), scope, scope_id)
def delete_structured_memory(name: str, scope: str = "global", scope_id: str = "") -> bool:
"""Delete a structured memory by name+scope. Returns True if existed."""
name = normalize_key(name)
@@ -919,6 +977,36 @@ def delete_structured_memory_by_id(memory_id: str) -> bool:
return False
def delete_structured_memory_returning_strict(
name: str, scope: str = "global", scope_id: str = ""
) -> dict[str, str] | None:
"""Atomically delete and return one scoped-name memory.
Storage failures propagate. A ``None`` return therefore means only that
no matching row existed at the mutation point.
"""
return get_storage().delete_structured_memory_returning(normalize_key(name), scope, scope_id)
def delete_structured_memory_by_id_returning_strict(
memory_id: str,
) -> dict[str, str] | None:
"""Atomically delete and return one memory by id; failures propagate."""
return get_storage().delete_structured_memory_by_id_returning(memory_id)
def find_structured_memory_scopes(
name: str,
scopes: list[tuple[str, str]],
) -> list[tuple[str, str]]:
"""Find visible same-name scope pairs in one metadata-only query."""
try:
return get_storage().find_structured_memory_scopes(normalize_key(name), scopes)
except Exception:
log.warning("Failed to find structured memory scopes name=%s", name, exc_info=True)
return []
def list_structured_memories(
mem_type: str = "",
scope: str = "",
+5 -5
View File
@@ -452,11 +452,11 @@ class OpenAIChatCompletionsProvider:
# operator-declared ``finish_reason_optional`` capability: on a
# server that never sends finish reasons, a stream that ended
# CLEANLY — the SDK ends iteration on [DONE]; an abrupt connection
# death raises httpx.TransportError out of this generator — after
# delivering output is a completed generation. Everywhere else a
# clean finish-less end is indistinguishable from a generation
# that died behind a clean-closing proxy/ASGI layer, so the shim
# stays DISARMED and the drain's complete-or-error gate raises
# death raises the active HTTP client's TransportError out of this
# generator — after delivering output is a completed generation.
# Everywhere else a clean finish-less end is indistinguishable
# from a generation that died behind a clean-closing proxy/ASGI layer,
# so the shim stays DISARMED and the drain's complete-or-error gate raises
# (retryable) instead of blessing possibly-truncated text.
# Reasoning counts as delivered output — a thinking model that
# spent its budget before emitting content is still a completed
+13 -7
View File
@@ -228,10 +228,12 @@ def transport_guarded(chunks: Iterator[StreamChunk]) -> Iterator[StreamChunk]:
Streaming moves the body read out of the SDK's
``APIConnectionError``-wrapped request into raw iteration, so a
mid-body wire death (connection drop, TLS record failure, read
timeout) surfaces as a bare ``httpx.TransportError`` no retry
predicate recognizes. This wrapper is that conversion rule made
reusable for consumers that keep streaming semantics (the
interactive loop); :func:`drain_stream` applies the same rule for
timeout) surfaces as a bare transport error no retry predicate
recognizes. OpenAI v3's default client raises ``httpx2`` errors;
Anthropic, Turnstone's own HTTP clients, and the OpenAI v3 legacy-client
escape hatch raise ``httpx`` errors. This wrapper is the one conversion
rule for both families, reusable by consumers that keep streaming
semantics (the interactive loop); :func:`drain_stream` applies it for
the single-shot lanes.
- A ``TransportError`` BEFORE any finish reason re-raises (chained)
@@ -243,7 +245,11 @@ def transport_guarded(chunks: Iterator[StreamChunk]) -> Iterator[StreamChunk]:
- Everything else chunks, exhaustion, non-transport exceptions
passes through untouched.
"""
import httpx # noqa: PLC0415 — heavyweight; deferred off the type-module import path
# Both libraries are heavyweight; keep them off this module's dataclass-
# only import path. ``httpx`` remains Turnstone's application transport,
# while ``httpx2`` is the OpenAI v3 default transport.
import httpx # noqa: PLC0415
import httpx2 # noqa: PLC0415
finish_seen = False
usage_seen = False
@@ -253,7 +259,7 @@ def transport_guarded(chunks: Iterator[StreamChunk]) -> Iterator[StreamChunk]:
sc = next(iterator)
except StopIteration:
return
except httpx.TransportError as exc:
except (httpx.TransportError, httpx2.TransportError) as exc:
if finish_seen:
# usage_captured distinguishes "completed result kept but
# its spend went missing from usage accounting" (the chat
@@ -348,7 +354,7 @@ def drain_stream(
Raises whatever the underlying stream raises retry/deadline/fallback
policy stays with the caller, exactly as with the old non-streaming
transport EXCEPT httpx transport failures, normalized by
transport EXCEPT HTTPX/HTTPX2 transport failures, normalized by
:func:`transport_guarded` (the one conversion rule, shared with the
interactive loop): a mid-body death before the finish reason is
re-raised (chained) as :class:`IncompleteStreamError`, restoring the
+714 -542
View File
File diff suppressed because it is too large Load Diff
+13 -4
View File
@@ -22,9 +22,17 @@ if TYPE_CHECKING:
from turnstone.core.session import ChatSession
def request_replay_project_name(request: Any) -> str:
"""Return project metadata pre-resolved by the shared SSE route."""
value = getattr(getattr(request, "state", None), "_session_replay_project_name", "")
return value if isinstance(value, str) else ""
def session_replay_preamble(
session: ChatSession | None,
ui: Any,
*,
project_name: str = "",
) -> Iterable[dict[str, Any]]:
"""Yield ``connected`` + optional ``status`` events for an SSE replay.
@@ -33,7 +41,8 @@ def session_replay_preamble(
through to the kind-specific tail.
- ``connected`` carries ``model`` / ``model_alias`` / ``skip_permissions``
so the per-tab status bar populates the model cell before any
history arrives.
history arrives. The caller supplies the project display name already
resolved for the authenticated connection principal.
- ``status`` only fires when ``session._last_usage`` exists (a
session that has completed at least one turn). The payload shape
matches :meth:`SessionUI.on_status` so live ticks and replays use
@@ -48,9 +57,9 @@ def session_replay_preamble(
"type": "connected",
"model": session.model,
"model_alias": session.model_alias or "",
# The attached project's display name (""=none) so the composer can
# paint its "has a project" badge on connect, beside the model chip.
"project_name": getattr(session, "_project_name", "") or "",
# The visible attached-project display name (""=none) so the composer
# can paint its project badge on connect, beside the model chip.
"project_name": project_name,
"skip_permissions": getattr(ui, "auto_approve", False),
}
+52 -27
View File
@@ -173,15 +173,11 @@ class EventsReplay(Protocol):
live event loop starts. Each yielded dict gets JSON-serialised
and sent as a single ``data:`` line to the client.
Interactive yields four things on connect: ``connected`` (model +
skip_permissions), ``status`` (token usage + context %, only when
``session._last_usage`` exists), ``history`` (replayed conversation),
and ``pending_approval`` + cached intent verdicts. Coord yields
just one: ``pending_approval`` (the rest aren't needed because
coord's dashboard fetches history via a separate ``/history``
endpoint and doesn't render the per-tab status bar). Kinds that
don't need any pre-replay wire ``None`` and the live loop starts
immediately.
The shared handler pre-resolves viewer-specific project metadata onto the
request before calling this stable three-argument callback. Production
callbacks emit ``connected`` plus optional ``status``, then pending controls
and cached verdicts. Conversation history stays on the separate REST
endpoint. Kinds without replay wire ``None``.
"""
def __call__(self, ws: Workstream, ui: Any, request: Request) -> Iterable[dict[str, Any]]:
@@ -444,13 +440,11 @@ class SessionEndpointConfig:
# wires ``None`` and lets the cluster collector handle the
# transition via ``CoordinatorAdapter.emit_rehydrated``.
open_post_load: OpenPostLoad | None = None
# (ws, ui, request) -> Iterable[dict]. Kind-specific initial
# SSE replay payload the lifted ``events`` body yields after
# registering the per-UI listener queue but before the live
# event loop. Both production kinds yield connected + optional status,
# followed by pending approval controls and cached verdicts. Conversation
# history stays on the separate REST ``/history`` bootstrap. Kinds that
# don't need pre-replay wire ``None``.
# (ws, ui, request) -> Iterable[dict]. Kind-specific initial SSE replay
# payload the lifted ``events`` body yields before the live event loop.
# The shared handler pins viewer-specific project metadata on ``request``;
# production callbacks combine it with connected/status and pending
# controls. Conversation history stays on the separate REST bootstrap.
events_replay: EventsReplay | None = None
# (request) -> Executor for the SSE live-loop's blocking
# ``queue.get`` wait. Interactive returns the dedicated
@@ -2298,15 +2292,15 @@ def make_open_handler(
def make_events_handler(cfg: SessionEndpointConfig) -> Handler:
"""Lifted body for ``GET {prefix}/{ws_id}/events`` — per-workstream SSE.
Both kinds share the SSE plumbing: register the per-UI listener
queue, run the kind-specific initial replay (``cfg.events_replay``,
typically ``connected`` + ``status`` + ``history`` + pending
approval / plan on interactive; just pending approval / plan on
coord), then drain the queue forever until either the workstream
closes (``ws_closed`` event) or the client disconnects.
Both kinds share the SSE plumbing: resolve viewer-specific project
metadata off-loop, register the per-UI listener queue, run the configured
initial replay (``cfg.events_replay``), then drain the queue forever until
either the workstream closes (``ws_closed`` event) or the client
disconnects. Cursor-only replay emits the shared connected/status preamble
directly because it deliberately skips kind-specific pending controls.
The kind-specific divergence is captured entirely by
``cfg.events_replay``. The live-loop body, the listener
The kind-specific replay divergence is captured by
``cfg.events_replay``. The cursor preamble, live-loop body, listener
registration, the ``ws_closed`` exit, the disconnect detection,
and the SSE-connect/disconnect metric recording are uniform.
@@ -2432,6 +2426,33 @@ def make_events_handler(cfg: SessionEndpointConfig) -> Handler:
# ``ui`` is a ``SessionUIBase`` subclass, so the cast is
# tightening the type, not weakening it.
ui_base = cast("SessionUIBase", ui)
from turnstone.core.web_helpers import auth_user_id
# Pin the authenticated viewer once per connection. Resolve optional
# project presentation metadata off the event loop and before listener
# registration, so database latency cannot stall unrelated async work
# or let this listener's queue accumulate while it waits.
replay_principal_id = auth_user_id(request).strip()
replay_project_name = ""
session = getattr(ws, "session", None)
project_name_for_principal = getattr(session, "project_name_for_principal", None)
if callable(project_name_for_principal):
try:
resolved_project_name = await asyncio.to_thread(
project_name_for_principal,
replay_principal_id,
)
if isinstance(resolved_project_name, str):
replay_project_name = resolved_project_name
except Exception:
# Project metadata is optional presentation context. Fail
# closed to no badge without dropping the SSE bootstrap.
log.debug(
"ws.events.project_name_failed ws=%s",
ws_id[:8],
exc_info=True,
)
request.state._session_replay_project_name = replay_project_name
replay_status: str
replay_events: list[dict[str, Any]] = []
lost_count = 0
@@ -2675,7 +2696,11 @@ def make_events_handler(cfg: SessionEndpointConfig) -> Handler:
# contract lives in session_replay_preamble alone (it
# no-ops on a detached session internally).
try:
for ev in session_replay_preamble(ws.session, ui):
for ev in session_replay_preamble(
ws.session,
ui,
project_name=replay_project_name,
):
yield {"data": json.dumps(ev)}
except Exception:
log.debug(
@@ -2745,8 +2770,8 @@ def make_events_handler(cfg: SessionEndpointConfig) -> Handler:
)
}
# Replay phase — stream the kind-specific initial
# payload one event at a time so the client sees
# Replay phase — stream the kind-specific initial payload
# one event at a time so the client sees
# the first byte immediately. Pre-building into
# a list would block time-to-first-byte until the
# entire replay materialized AND let the listener
+1 -1
View File
@@ -855,7 +855,7 @@ def _build_registry() -> dict[str, SettingDef]:
restart_required=True,
help="When enabled, the console runs an internal Certificate Authority and "
"ACME server. All cluster services (servers, channels) auto-provision "
"short-lived certificates for mutual TLS. Requires lacme: pip install turnstone[tls]",
"short-lived certificates for mutual TLS. ACME support is included with Turnstone.",
),
SettingDef(
"tls.acme_directory",
+85 -20
View File
@@ -4746,6 +4746,9 @@ class PostgreSQLBackend(_KeyedAttachmentSaveWrappers):
scope_id: str,
content: str,
) -> None:
if description is None or not description.strip():
raise ValueError("memory description is required and must be non-empty")
description = description.strip()
now = datetime.now(UTC).strftime("%Y-%m-%dT%H:%M:%S")
with self._conn() as conn:
conn.execute(
@@ -4770,19 +4773,24 @@ class PostgreSQLBackend(_KeyedAttachmentSaveWrappers):
self,
memory_id: str,
name: str,
description: str | None,
description: str,
mem_type: str | None,
scope: str,
scope_id: str,
content: str,
*,
require_active_project: bool = False,
) -> tuple[dict[str, str], bool]:
from sqlalchemy.dialects.postgresql import insert as pg_insert
if description is None or not description.strip():
raise ValueError("memory description is required and must be non-empty")
description = description.strip()
now = datetime.now(UTC).strftime("%Y-%m-%dT%H:%M:%S")
insert_stmt = pg_insert(structured_memories).values(
memory_id=memory_id,
name=name,
description="" if description is None else description,
description=description,
type="general" if mem_type is None else mem_type,
scope=scope,
scope_id=scope_id,
@@ -4792,16 +4800,14 @@ class PostgreSQLBackend(_KeyedAttachmentSaveWrappers):
last_accessed=now,
access_count=0,
)
# On conflict, refresh content + timestamps. description/type are
# overwritten only when the caller supplied them; None means "unset" ->
# keep the stored value. created and access_count are left untouched.
# On conflict, refresh content, description, and timestamps. A None type means
# "unset" -> keep the stored value. created/access_count stay untouched.
set_: dict[str, Any] = {
"content": insert_stmt.excluded.content,
"updated": now,
"last_accessed": now,
}
if description is not None:
set_["description"] = insert_stmt.excluded.description
set_["description"] = insert_stmt.excluded.description
if mem_type is not None:
set_["type"] = insert_stmt.excluded.type
stmt = insert_stmt.on_conflict_do_update(
@@ -4809,6 +4815,22 @@ class PostgreSQLBackend(_KeyedAttachmentSaveWrappers):
set_=set_,
).returning(structured_memories)
with self._conn() as conn:
if require_active_project:
if scope != "project" or not scope_id:
raise ValueError("active-project guard requires project scope")
project = conn.execute(
sa.select(projects.c.project_id)
.where(
sa.and_(
projects.c.project_id == scope_id,
projects.c.state == "active",
)
)
.with_for_update(read=True)
).fetchone()
if project is None:
conn.rollback()
raise ValueError("project is missing, archived, or no longer writable")
row = conn.execute(stmt).fetchone()
conn.commit()
if row is None: # unreachable: ON CONFLICT DO UPDATE returns one row
@@ -4841,26 +4863,58 @@ class PostgreSQLBackend(_KeyedAttachmentSaveWrappers):
def delete_structured_memory(
self, name: str, scope: str = "global", scope_id: str = ""
) -> bool:
with self._conn() as conn:
result = conn.execute(
sa.delete(structured_memories).where(
sa.and_(
structured_memories.c.name == name,
structured_memories.c.scope == scope,
structured_memories.c.scope_id == scope_id,
)
return self.delete_structured_memory_returning(name, scope, scope_id) is not None
def delete_structured_memory_returning(
self, name: str, scope: str = "global", scope_id: str = ""
) -> dict[str, str] | None:
stmt = (
sa.delete(structured_memories)
.where(
sa.and_(
structured_memories.c.name == name,
structured_memories.c.scope == scope,
structured_memories.c.scope_id == scope_id,
)
)
.returning(structured_memories)
)
with self._conn() as conn:
row = conn.execute(stmt).fetchone()
conn.commit()
return result.rowcount > 0
return dict(row._mapping) if row is not None else None
def delete_structured_memory_by_id(self, memory_id: str) -> bool:
return self.delete_structured_memory_by_id_returning(memory_id) is not None
def delete_structured_memory_by_id_returning(self, memory_id: str) -> dict[str, str] | None:
with self._conn() as conn:
result = conn.execute(
sa.delete(structured_memories).where(structured_memories.c.memory_id == memory_id)
)
row = conn.execute(
sa.delete(structured_memories)
.where(structured_memories.c.memory_id == memory_id)
.returning(structured_memories)
).fetchone()
conn.commit()
return result.rowcount > 0
return dict(row._mapping) if row is not None else None
def find_structured_memory_scopes(
self,
name: str,
scopes: list[tuple[str, str]],
) -> list[tuple[str, str]]:
if not scopes:
return []
with self._conn() as conn:
scope_clauses, params = self._build_scope_or_clause(scopes)
rows = conn.execute(
sa.text(
"SELECT scope, scope_id FROM structured_memories "
f"WHERE name = :name AND ({scope_clauses}) "
"ORDER BY scope, scope_id"
),
{**params, "name": name},
).fetchall()
return [(str(row.scope), str(row.scope_id)) for row in rows]
def list_structured_memories(
self,
@@ -5905,6 +5959,17 @@ class PostgreSQLBackend(_KeyedAttachmentSaveWrappers):
def delete_project(self, project_id: str) -> bool:
with self._conn() as conn:
# Serialize with guarded project-memory upserts. If a writer got
# the row first, its memory is committed before our purge; if this
# delete wins, the later writer's active-project check finds no row.
project = conn.execute(
sa.select(projects.c.project_id)
.where(projects.c.project_id == project_id)
.with_for_update()
).fetchone()
if project is None:
conn.rollback()
return False
# No FK cascade in the schema family, so purge the project's scoped
# memory + member rows explicitly (same transaction) before the
# project row — honouring the "destroys the container AND its scoped
+31 -6
View File
@@ -829,34 +829,41 @@ class StorageBackend(Protocol):
scope_id: str,
content: str,
) -> None:
"""Create a structured memory record."""
"""Create a structured memory record with a non-empty description."""
...
def upsert_structured_memory(
self,
memory_id: str,
name: str,
description: str | None,
description: str,
mem_type: str | None,
scope: str,
scope_id: str,
content: str,
*,
require_active_project: bool = False,
) -> tuple[dict[str, str], bool]:
"""Insert a structured memory, or update it in place on a
``(name, scope, scope_id)`` conflict.
Atomic ``INSERT ... ON CONFLICT DO UPDATE ... RETURNING`` no
IntegrityError round-trip, race-safe under concurrent saves of the same
key. ``description`` / ``mem_type`` of ``None`` mean "unset": the
column default ("" / "general") is used on insert and the stored value
is kept on conflict; a non-``None`` value (including "" or "general") is
written.
key. ``description`` must contain non-whitespace text on every insert
or update. A ``mem_type`` of ``None`` means "unset": the column default
is used on insert and the stored value is kept on conflict.
Returns ``(row, was_update)`` (like Django's ``update_or_create``): the
full saved row, and ``True`` when an existing row was updated rather
than inserted. Callers MUST supply a fresh unique ``memory_id`` it is
compared against the returned row's id to tell INSERT from UPDATE, so a
reused id would report ``was_update=False`` on a real update.
When ``require_active_project`` is true, ``scope`` must be
``"project"`` and the backend must verify that the referenced project
still exists and is active in the same transaction as the upsert. The
project row is locked where the backend supports row locks so a
concurrent project delete cannot leave an orphaned memory behind.
"""
...
@@ -876,10 +883,28 @@ class StorageBackend(Protocol):
"""Delete a structured memory by (name, scope, scope_id). Returns True if existed."""
...
def delete_structured_memory_returning(
self, name: str, scope: str = "global", scope_id: str = ""
) -> dict[str, str] | None:
"""Atomically delete and return a memory selected by its scoped name."""
...
def delete_structured_memory_by_id(self, memory_id: str) -> bool:
"""Delete a structured memory by its primary key. Returns True if existed."""
...
def delete_structured_memory_by_id_returning(self, memory_id: str) -> dict[str, str] | None:
"""Atomically delete and return a memory selected by primary key."""
...
def find_structured_memory_scopes(
self,
name: str,
scopes: list[tuple[str, str]],
) -> list[tuple[str, str]]:
"""Return visible scope pairs containing ``name`` in one small query."""
...
def list_structured_memories(
self,
mem_type: str = "",
+79 -20
View File
@@ -4818,6 +4818,9 @@ class SQLiteBackend(_KeyedAttachmentSaveWrappers):
scope_id: str,
content: str,
) -> None:
if description is None or not description.strip():
raise ValueError("memory description is required and must be non-empty")
description = description.strip()
now = datetime.now(UTC).strftime("%Y-%m-%dT%H:%M:%S")
with self._conn() as conn:
conn.execute(
@@ -4842,19 +4845,24 @@ class SQLiteBackend(_KeyedAttachmentSaveWrappers):
self,
memory_id: str,
name: str,
description: str | None,
description: str,
mem_type: str | None,
scope: str,
scope_id: str,
content: str,
*,
require_active_project: bool = False,
) -> tuple[dict[str, str], bool]:
from sqlalchemy.dialects.sqlite import insert as sqlite_insert
if description is None or not description.strip():
raise ValueError("memory description is required and must be non-empty")
description = description.strip()
now = datetime.now(UTC).strftime("%Y-%m-%dT%H:%M:%S")
insert_stmt = sqlite_insert(structured_memories).values(
memory_id=memory_id,
name=name,
description="" if description is None else description,
description=description,
type="general" if mem_type is None else mem_type,
scope=scope,
scope_id=scope_id,
@@ -4864,16 +4872,14 @@ class SQLiteBackend(_KeyedAttachmentSaveWrappers):
last_accessed=now,
access_count=0,
)
# On conflict, refresh content + timestamps. description/type are
# overwritten only when the caller supplied them; None means "unset" ->
# keep the stored value. created and access_count are left untouched.
# On conflict, refresh content, description, and timestamps. A None type means
# "unset" -> keep the stored value. created/access_count stay untouched.
set_: dict[str, Any] = {
"content": insert_stmt.excluded.content,
"updated": now,
"last_accessed": now,
}
if description is not None:
set_["description"] = insert_stmt.excluded.description
set_["description"] = insert_stmt.excluded.description
if mem_type is not None:
set_["type"] = insert_stmt.excluded.type
stmt = insert_stmt.on_conflict_do_update(
@@ -4881,6 +4887,24 @@ class SQLiteBackend(_KeyedAttachmentSaveWrappers):
set_=set_,
).returning(structured_memories)
with self._conn() as conn:
if require_active_project:
if scope != "project" or not scope_id:
raise ValueError("active-project guard requires project scope")
# SQLite has no row locks. Taking the writer lock before the
# existence check serializes this transaction with
# ``delete_project`` (which uses the same prologue).
conn.execute(sa.text("BEGIN IMMEDIATE"))
project = conn.execute(
sa.select(projects.c.project_id).where(
sa.and_(
projects.c.project_id == scope_id,
projects.c.state == "active",
)
)
).fetchone()
if project is None:
conn.rollback()
raise ValueError("project is missing, archived, or no longer writable")
row = conn.execute(stmt).fetchone()
conn.commit()
if row is None: # unreachable: ON CONFLICT DO UPDATE returns one row
@@ -4913,26 +4937,58 @@ class SQLiteBackend(_KeyedAttachmentSaveWrappers):
def delete_structured_memory(
self, name: str, scope: str = "global", scope_id: str = ""
) -> bool:
with self._conn() as conn:
result = conn.execute(
sa.delete(structured_memories).where(
sa.and_(
structured_memories.c.name == name,
structured_memories.c.scope == scope,
structured_memories.c.scope_id == scope_id,
)
return self.delete_structured_memory_returning(name, scope, scope_id) is not None
def delete_structured_memory_returning(
self, name: str, scope: str = "global", scope_id: str = ""
) -> dict[str, str] | None:
stmt = (
sa.delete(structured_memories)
.where(
sa.and_(
structured_memories.c.name == name,
structured_memories.c.scope == scope,
structured_memories.c.scope_id == scope_id,
)
)
.returning(structured_memories)
)
with self._conn() as conn:
row = conn.execute(stmt).fetchone()
conn.commit()
return result.rowcount > 0
return dict(row._mapping) if row is not None else None
def delete_structured_memory_by_id(self, memory_id: str) -> bool:
return self.delete_structured_memory_by_id_returning(memory_id) is not None
def delete_structured_memory_by_id_returning(self, memory_id: str) -> dict[str, str] | None:
with self._conn() as conn:
result = conn.execute(
sa.delete(structured_memories).where(structured_memories.c.memory_id == memory_id)
)
row = conn.execute(
sa.delete(structured_memories)
.where(structured_memories.c.memory_id == memory_id)
.returning(structured_memories)
).fetchone()
conn.commit()
return result.rowcount > 0
return dict(row._mapping) if row is not None else None
def find_structured_memory_scopes(
self,
name: str,
scopes: list[tuple[str, str]],
) -> list[tuple[str, str]]:
if not scopes:
return []
with self._conn() as conn:
scope_clauses, params = self._build_scope_or_clause(scopes)
rows = conn.execute(
sa.text(
"SELECT scope, scope_id FROM structured_memories "
f"WHERE name = :name AND ({scope_clauses}) "
"ORDER BY scope, scope_id"
),
{**params, "name": name},
).fetchall()
return [(str(row.scope), str(row.scope_id)) for row in rows]
def list_structured_memories(
self,
@@ -5961,6 +6017,9 @@ class SQLiteBackend(_KeyedAttachmentSaveWrappers):
def delete_project(self, project_id: str) -> bool:
with self._conn() as conn:
# Serialize with guarded project-memory upserts before inspecting
# or deleting the container.
conn.execute(sa.text("BEGIN IMMEDIATE"))
# No FK cascade in the schema family, so purge the project's scoped
# memory + member rows explicitly (same transaction) before the
# project row — honouring the "destroys the container AND its scoped
+562 -151
View File
@@ -7,18 +7,28 @@ SSL contexts for mTLS communication.
Flow:
1. Fetch CA root cert from console (plain HTTP, first boot)
2. Request service cert via ACME (plain HTTP, first boot)
3. Build SSL contexts for uvicorn (server) and httpx (client)
4. Start auto-renewal (uses existing cert for mTLS to console)
3. Build SSL contexts for the server listener and outbound HTTP clients
4. Start auto-renewal through the console's ACME responder
"""
from __future__ import annotations
import contextlib
import ipaddress
import re
from pathlib import Path
from typing import TYPE_CHECKING, Any
from typing import TYPE_CHECKING
import httpx2
import lacme
from lacme import CertBundle, IdentifierValue
if TYPE_CHECKING:
import ssl
from collections.abc import Callable
from collections.abc import Callable, Coroutine, Generator, Sequence
from typing import Any
from lacme.mtls import PemPaths
from turnstone.core.storage._protocol import StorageBackend
@@ -26,8 +36,10 @@ from turnstone.core.log import get_logger
log = get_logger(__name__)
_RENEW_INTERVAL_HOURS = 24
_RENEW_BEFORE_EXPIRY_DAYS = 1
CERT_VALIDITY_HOURS = 48
RENEW_INTERVAL_HOURS = 12
RENEW_BEFORE_EXPIRY_DAYS = 1
RENEW_MAX_JITTER_SECONDS = 600
# Boot-time init retry budget: 1+2+4+8+16 s ≈ 31 s of backoff. Sized to
# absorb a whole-stack restart, where every node races the console for the
@@ -36,6 +48,170 @@ _RENEW_BEFORE_EXPIRY_DAYS = 1
# connections.
TLS_INIT_RETRY_ATTEMPTS = 6
_ACME_PUBLIC_RESOURCES = frozenset({"/directory", "/new-nonce", "/ca.pem"})
_ACME_PROTECTED_RESOURCES = frozenset({"/new-account", "/new-order", "/key-change", "/revoke-cert"})
_ACME_PROTECTED_RESOURCE_RE = re.compile(r"^/(?:authz|chall|finalize|order|cert)/[A-Za-z0-9._~-]+$")
class TurnstoneAutoApproveChallengeHandler:
"""No-op challenge handler for Turnstone's authenticated ACME responder.
The responder intentionally auto-approves after the enrollment JWT has
authorized the signing route, so locally publishing an HTTP-01 response
would imply validation that never occurs. Do not use this with an external
ACME service; a validating CA requires a real challenge handler.
"""
async def provision(self, _domain: str, _token: str, _key_authorization: str) -> None:
return None
async def deprovision(self, _domain: str, _token: str) -> None:
return None
def _url_origin(url: httpx2.URL) -> tuple[str, str, int]:
"""Return a normalized origin tuple, including default ports."""
default_port = 443 if url.scheme == "https" else 80
return url.scheme, url.host, url.port or default_port
class ACMEHTTPAuth(httpx2.Auth):
"""Attach a rotating enrollment JWT only to configured ACME responders.
ACME directory entries are absolute URLs controlled by the responder. A
client-wide Authorization header would therefore leak a cluster service JWT
if a compromised directory named another host. This auth policy permits
only known responder origins + mount prefixes and authenticates only the
state-changing resources behind Turnstone's enrollment gate.
"""
def __init__(
self,
responder_bases: list[str],
token_provider: Callable[[], str] | None,
) -> None:
bases: list[tuple[tuple[str, str, int], str]] = []
for raw in responder_bases:
parsed = httpx2.URL(raw.rstrip("/"))
raw_path = parsed.raw_path
if (
not parsed.is_absolute_url
or parsed.scheme not in {"http", "https"}
or parsed.userinfo
or parsed.query
or parsed.fragment
or b"%" in raw_path
):
raise ValueError(f"ACME responder base is invalid: {raw!r}")
item = (_url_origin(parsed), parsed.path.rstrip("/"))
if item not in bases:
bases.append(item)
if not bases:
raise ValueError("At least one ACME responder base is required")
self._bases = tuple(bases)
self._token_provider = token_provider
def auth_flow(
self, request: httpx2.Request
) -> Generator[httpx2.Request, httpx2.Response, None]:
if request.url.userinfo or request.url.query or request.url.fragment:
raise RuntimeError(f"Refusing non-canonical ACME request URL: {request.url}")
raw_path = request.url.raw_path
if b"%" in raw_path:
# Reverse proxies disagree on whether encoded separators and dot
# segments are normalized before routing. ACME responder IDs use
# only URL-unreserved ASCII, so any escape is both unnecessary and
# an authorization-boundary ambiguity.
raise RuntimeError(f"Refusing non-canonical ACME request URL: {request.url}")
try:
request_path = raw_path.decode("ascii")
except UnicodeDecodeError as exc:
raise RuntimeError(f"Refusing non-canonical ACME request URL: {request.url}") from exc
suffix: str | None = None
request_origin = _url_origin(request.url)
for origin, base_path in self._bases:
if request_origin != origin:
continue
if request_path.startswith(f"{base_path}/"):
suffix = request_path[len(base_path) :]
break
if suffix is None:
raise RuntimeError(
f"Refusing ACME request outside configured responder bases: {request.url}"
)
if suffix in _ACME_PUBLIC_RESOURCES:
yield request
return
if suffix not in _ACME_PROTECTED_RESOURCES and not _ACME_PROTECTED_RESOURCE_RE.fullmatch(
suffix
):
raise RuntimeError(f"Refusing unknown ACME responder resource: {request.url}")
if self._token_provider is None:
raise RuntimeError("ACME enrollment credentials are not configured")
token = self._token_provider()
if not token:
raise RuntimeError("ACME enrollment token provider returned an empty token")
request.headers["Authorization"] = f"Bearer {token}"
yield request
def build_acme_http_client(
console_url: str,
*,
external_url: str = "",
token_provider: Callable[[], str] | None,
) -> httpx2.AsyncClient:
"""Build a fresh HTTPX2 client pinned to trusted ACME responder bases."""
bases = [f"{console_url.rstrip('/')}/acme"]
if external_url:
bases.append(external_url.rstrip("/"))
return httpx2.AsyncClient(
auth=ACMEHTTPAuth(bases, token_provider),
follow_redirects=False,
trust_env=False,
)
async def complete_tls_cleanup(cleanup: Coroutine[Any, Any, None]) -> None:
"""Finish resource cleanup before propagating caller cancellation.
A one-shot ``asyncio.shield`` still returns immediately when its caller is
cancelled. Retrying the shield keeps cleanup owned and observed even under
repeated cancellation, then restores the first cancellation to the caller.
"""
import asyncio
task = asyncio.create_task(cleanup)
cancellation: asyncio.CancelledError | None = None
while not task.done():
try:
await asyncio.shield(task)
except asyncio.CancelledError as exc:
# If the cleanup task itself was cancelled, preserve that result;
# otherwise remember the caller cancellation and keep draining.
if task.done() and task.cancelled():
break
if cancellation is None:
cancellation = exc
cleanup_error: BaseException | None = None
try:
task.result()
except BaseException as exc:
cleanup_error = exc
if cancellation is not None:
if cleanup_error is not None:
log.error(
"tls.cleanup.failed_during_cancellation",
error=f"{type(cleanup_error).__name__}: {cleanup_error}",
)
raise cancellation
if cleanup_error is not None:
raise cleanup_error
def tls_pem_runtime_dir() -> Path:
"""Parent directory for the boot-time PEM files.
@@ -86,7 +262,12 @@ def prepare_pem_runtime_dir() -> Path:
return root
def refresh_runtime_pems(bundle: Any, *, ca_pem: bytes | None, previous: Path | None) -> Any:
def refresh_runtime_pems(
bundle: CertBundle,
*,
ca_pem: bytes | None,
previous: Path | None,
) -> PemPaths:
"""Write a renewed bundle under the runtime root and drop the old dir.
Keeps the on-disk PEMs (the healthcheck's mTLS client identity) in
@@ -105,14 +286,196 @@ def refresh_runtime_pems(bundle: Any, *, ca_pem: bytes | None, previous: Path |
return new_paths
def _require_lacme() -> Any:
_IP_ADDRESS_TYPES = (ipaddress.IPv4Address, ipaddress.IPv6Address)
def parse_certificate_identifier(value: str) -> IdentifierValue:
"""Parse an operator-supplied DNS name or IP literal for lacme.
lacme deliberately treats every plain string as DNS, even when it looks
like an address. Turnstone's configuration/CLI boundary therefore converts
IP literals to typed objects while preserving DNS spelling. Unspecified and
scoped addresses are not usable peer identities and fail before enrollment.
"""
candidate = value.strip()
if not candidate:
raise ValueError("Certificate identifiers must be non-empty")
try:
import lacme
except ImportError:
raise ImportError(
"lacme is required for TLS support. Install with: pip install turnstone[tls]",
) from None
return lacme
address = ipaddress.ip_address(candidate)
except ValueError:
return candidate
if isinstance(address, ipaddress.IPv6Address) and address.scope_id is not None:
raise ValueError(f"Scoped IPv6 address {candidate!r} is not a valid certificate identity")
if address.is_unspecified:
raise ValueError(f"Unspecified address {candidate!r} is not a valid certificate identity")
return address
def _identifier_key(value: IdentifierValue) -> tuple[str, str]:
"""Return a case-insensitive, type-preserving identity key."""
if isinstance(value, _IP_ADDRESS_TYPES):
return "ip", str(value)
return "dns", value.lower()
def normalize_certificate_identifiers(
values: Sequence[IdentifierValue],
) -> list[IdentifierValue]:
"""Validate and de-duplicate Turnstone identities in first-seen order."""
if not values:
raise ValueError("At least one certificate identifier is required")
ordered: list[IdentifierValue] = []
seen: set[tuple[str, str]] = set()
for raw in values:
if isinstance(raw, str):
value = parse_certificate_identifier(raw)
elif isinstance(raw, _IP_ADDRESS_TYPES):
value = parse_certificate_identifier(str(raw))
else:
raise TypeError(
"Certificate identifiers must be str, IPv4Address, or IPv6Address, "
f"got {type(raw).__name__}"
)
key = _identifier_key(value)
if key not in seen:
seen.add(key)
ordered.append(value)
return ordered
def certificate_bundle_identifiers(bundle: CertBundle) -> list[IdentifierValue]:
"""Recover authoritative typed identities from a bundle's leaf SANs.
Bundle metadata and database keys intentionally remain strings. The leaf
certificate is authoritative for whether a numeric value is a legacy DNS
SAN or a real IP SAN, then metadata restores the original primary ordering.
"""
from cryptography import x509
if not bundle.domains or bundle.domain != bundle.domains[0]:
raise ValueError(f"Invalid certificate metadata for {bundle.domain!r}")
try:
certificates = x509.load_pem_x509_certificates(bundle.cert_pem)
except (TypeError, ValueError) as exc:
raise ValueError(f"Invalid leaf certificate for {bundle.domain!r}") from exc
if len(certificates) != 1:
raise ValueError(f"Expected one leaf certificate for {bundle.domain!r}")
try:
sans = certificates[0].extensions.get_extension_for_class(x509.SubjectAlternativeName).value
except x509.ExtensionNotFound as exc:
raise ValueError(f"Certificate for {bundle.domain!r} has no subjectAltName") from exc
remaining: list[IdentifierValue] = []
for san in sans:
if isinstance(san, x509.DNSName):
remaining.append(san.value)
elif isinstance(san, x509.IPAddress) and isinstance(san.value, _IP_ADDRESS_TYPES):
remaining.append(parse_certificate_identifier(str(san.value)))
else:
raise ValueError(
f"Certificate for {bundle.domain!r} has unsupported SAN {type(san).__name__}"
)
ordered: list[IdentifierValue] = []
for domain in bundle.domains:
match = next(
(
index
for index, value in enumerate(remaining)
if (
value.lower() == domain.lower()
if isinstance(value, str)
else str(value) == domain
)
),
None,
)
if match is None:
raise ValueError(f"Certificate SANs do not match stored metadata for {bundle.domain!r}")
ordered.append(remaining.pop(match))
if remaining:
raise ValueError(f"Certificate SANs do not match stored metadata for {bundle.domain!r}")
return ordered
def certificate_matches_identifiers(
bundle: CertBundle,
requested: Sequence[IdentifierValue],
) -> bool:
"""Return whether an existing leaf has exactly the requested typed SANs."""
actual = certificate_bundle_identifiers(bundle)
desired = normalize_certificate_identifiers(requested)
return len(actual) == len(desired) and {_identifier_key(value) for value in actual} == {
_identifier_key(value) for value in desired
}
def validate_cluster_certificate_bundle(
bundle: CertBundle,
requested: Sequence[IdentifierValue],
ca_pem: bytes,
) -> None:
"""Validate a persisted dual-purpose identity against the active CA.
Store metadata is only an index. Reuse is authorized by the actual leaf,
private key, typed SANs, validity window, EKU, and the exact active cluster
root. This also prevents a same-domain public frontend certificate from
being reused as the console's internal mTLS identity.
"""
from datetime import UTC, datetime
from cryptography import x509
from cryptography.hazmat.primitives import hashes, serialization
from cryptography.x509.oid import ExtendedKeyUsageOID
if not bundle.key_pem:
raise ValueError(f"Certificate for {bundle.domain!r} has no private key")
if not certificate_matches_identifiers(bundle, requested):
raise ValueError(f"Certificate identifiers do not match {bundle.domain!r}")
try:
leaf_certificates = x509.load_pem_x509_certificates(bundle.cert_pem)
chain = x509.load_pem_x509_certificates(bundle.fullchain_pem)
roots = x509.load_pem_x509_certificates(ca_pem)
private_key = serialization.load_pem_private_key(bundle.key_pem, password=None)
except (TypeError, ValueError) as exc:
raise ValueError(f"Certificate material for {bundle.domain!r} is invalid") from exc
if len(leaf_certificates) != 1 or len(chain) != 2 or len(roots) != 1:
raise ValueError(f"Certificate chain for {bundle.domain!r} is invalid")
leaf = leaf_certificates[0]
root = roots[0]
if chain[0].fingerprint(hashes.SHA256()) != leaf.fingerprint(hashes.SHA256()):
raise ValueError(f"Full chain for {bundle.domain!r} does not start with its leaf")
if chain[-1].fingerprint(hashes.SHA256()) != root.fingerprint(hashes.SHA256()):
raise ValueError(f"Certificate for {bundle.domain!r} is not chained to the active CA")
try:
leaf.verify_directly_issued_by(root)
except (TypeError, ValueError) as exc:
raise ValueError(
f"Certificate for {bundle.domain!r} was not issued by the active CA"
) from exc
public_format = serialization.PublicFormat.SubjectPublicKeyInfo
cert_public_key = leaf.public_key().public_bytes(serialization.Encoding.DER, public_format)
key_public_key = private_key.public_key().public_bytes(
serialization.Encoding.DER, public_format
)
if cert_public_key != key_public_key:
raise ValueError(f"Private key does not match certificate for {bundle.domain!r}")
now = datetime.now(UTC)
if leaf.not_valid_before_utc > now or leaf.not_valid_after_utc <= now:
raise ValueError(f"Certificate for {bundle.domain!r} is outside its validity window")
try:
eku = leaf.extensions.get_extension_for_class(x509.ExtendedKeyUsage).value
except x509.ExtensionNotFound as exc:
raise ValueError(f"Certificate for {bundle.domain!r} has no extended key usage") from exc
if not {
ExtendedKeyUsageOID.SERVER_AUTH,
ExtendedKeyUsageOID.CLIENT_AUTH,
}.issubset(set(eku)):
raise ValueError(f"Certificate for {bundle.domain!r} is not valid for cluster mTLS")
def build_cert_hostnames(
@@ -120,7 +483,7 @@ def build_cert_hostnames(
*,
bind_host: str = "",
extra_sans: str = "",
) -> list[str]:
) -> list[IdentifierValue]:
"""Build the ordered, de-duplicated SAN list for a service certificate.
The advertised host (the name peers dial) goes **first**, becoming the
@@ -133,61 +496,37 @@ def build_cert_hostnames(
import socket
from urllib.parse import urlsplit
names: list[str] = []
names: list[IdentifierValue] = []
if advertise_url:
host = urlsplit(advertise_url).hostname or ""
if host:
names.append(host)
names.append(parse_certificate_identifier(host))
# OS hostname (container ID under Docker) — keeps in-container self-dial
# working and provides a fallback primary on bare metal.
hostname = socket.gethostname()
names.append(hostname)
names.append(parse_certificate_identifier(hostname))
fqdn = socket.getfqdn()
if fqdn and fqdn != hostname:
names.append(fqdn)
names.extend(["localhost", "127.0.0.1"])
if bind_host and bind_host not in ("0.0.0.0", "::", ""):
names.append(bind_host)
# Reject wildcard / unspecified-address SANs so a stray TURNSTONE_TLS_SANS
# can't mint an over-broad cert the internal CA would have peers trust.
names.append(parse_certificate_identifier(fqdn))
names.extend(["localhost", ipaddress.IPv4Address("127.0.0.1")])
if bind_host and bind_host not in ("0.0.0.0", "::", "*"):
names.append(parse_certificate_identifier(bind_host))
# A bare wildcard is an invalid service identity. Unspecified IPs fail
# through the central parser rather than becoming unusable SANs.
for raw in extra_sans.split(","):
san = raw.strip()
if san and san not in ("0.0.0.0", "::", "*"):
names.append(san)
# De-duplicate, preserving first-seen order so the advertised host stays
# primary.
seen: set[str] = set()
ordered: list[str] = []
for name in names:
if name and name not in seen:
seen.add(name)
ordered.append(name)
return ordered
if not san or san == "*":
continue
names.append(parse_certificate_identifier(san))
return normalize_certificate_identifiers(names)
class _SingleDomainStore:
"""Store view exposing only one domain's cert to a renewal sweep.
lacme's RenewalManager renews everything ``list_certs()`` returns. The
store is shared cluster-wide, so an unscoped manager on each node renews
every other node's (and every dead container's) cert an N×M storm. This
wrapper limits the sweep to one domain; all other operations delegate to
the real store so renewed certs still persist to the shared database.
"""
def __init__(self, inner: Any, domain: str) -> None:
self._inner = inner
self._domain = domain
def list_certs(self) -> list[Any]:
cert = self._inner.load_cert(self._domain)
return [cert] if cert is not None else []
def __getattr__(self, name: str) -> Any:
return getattr(self._inner, name)
def swap_context_cert(ctx: ssl.SSLContext, bundle: Any, *, ca_pem: bytes | None = None) -> None:
def swap_context_cert(
ctx: ssl.SSLContext,
bundle: CertBundle,
*,
ca_pem: bytes | None = None,
) -> None:
"""Hot-swap a renewed bundle into a live :class:`ssl.SSLContext`.
Writes the bundle to short-lived PEM files, calls ``load_cert_chain`` (so
@@ -212,7 +551,7 @@ class TLSClient:
"""TLS client for service nodes.
Requests certificates from the console's ACME endpoint and provides
SSL contexts for server (uvicorn) and client (httpx) use.
SSL contexts for server (uvicorn) and HTTP clients.
Typical usage::
@@ -227,25 +566,29 @@ class TLSClient:
self,
storage: StorageBackend,
console_url: str = "",
hostnames: list[str] | None = None,
hostnames: Sequence[IdentifierValue] | None = None,
acme_external_url: str = "",
enrollment_token_provider: Callable[[], str] | None = None,
) -> None:
lacme = _require_lacme()
from turnstone.core.tls_store import StorageStore
from turnstone.core.tls_store import CertificateValidationStore, StorageStore
self._storage = storage
self._store = StorageStore(storage)
self._validated_store = CertificateValidationStore(self._store, self._validate_bundle)
self._console_url = console_url.rstrip("/") if console_url else ""
self._hostnames = hostnames or []
self._acme_external_url = acme_external_url.rstrip("/") if acme_external_url else ""
self._enrollment_token_provider = enrollment_token_provider
self._hostnames = normalize_certificate_identifiers(hostnames) if hostnames else []
self._event_dispatcher = lacme.EventDispatcher()
self._ca_pem: bytes | None = None
self._bundle: Any | None = None
self._renewal_task: Any | None = None
self._renewal_client: Any | None = None
self._bundle: CertBundle | None = None
self._renewal_manager: lacme.RenewalManager | None = None
self._renewal_client: lacme.Client | None = None
self._renewal_http_client: httpx2.AsyncClient | None = None
# Optional hook invoked with each renewed bundle so the live HTTPS
# listener can swap in the new cert (uvicorn never reloads its SSL
# context on its own — see ``set_cert_reload_hook``).
self._cert_reload_hook: Callable[[Any], None] | None = None
self._cert_reload_hook: Callable[[CertBundle], None] | None = None
# Wire Prometheus metrics
try:
@@ -260,7 +603,7 @@ class TLSClient:
else:
raise
def set_cert_reload_hook(self, hook: Callable[[Any], None]) -> None:
def set_cert_reload_hook(self, hook: Callable[[CertBundle], None]) -> None:
"""Register a callback that installs a renewed bundle into the listener.
Renewal updates the DB + ``self._bundle`` but not the running uvicorn
@@ -269,22 +612,41 @@ class TLSClient:
"""
self._cert_reload_hook = hook
def _handle_renewed(self, bundle: Any) -> None:
async def _handle_renewed(self, bundle: CertBundle) -> None:
"""Renewal callback: cache the new bundle and run the reload hook."""
previous = self._bundle
try:
if self._cert_reload_hook is not None:
self._cert_reload_hook(bundle)
except Exception:
# Client.issue saved before the callback. Restore the last identity
# so shared persistence cannot claim a rotation the live listener
# failed to adopt; the next sweep will retry.
if previous is not None:
try:
if self._cert_reload_hook is not None:
self._cert_reload_hook(previous)
except Exception:
log.error("tls.cert.reload_rollback_failed", exc_info=True)
self._store.save_cert(previous)
log.warning("tls.cert.reload_hook_failed", exc_info=True)
raise
self._bundle = bundle
log.info("tls.cert.renewed", domain=bundle.domain)
if self._cert_reload_hook is not None:
try:
self._cert_reload_hook(bundle)
except Exception:
log.warning("tls.cert.reload_hook_failed", exc_info=True)
def _validate_bundle(self, bundle: CertBundle) -> None:
"""Authorize a complete node identity before it reaches shared storage."""
if self._ca_pem is None:
raise ValueError("Cluster CA certificate is unavailable")
validate_cluster_certificate_bundle(bundle, self._hostnames, self._ca_pem)
async def init(self, *, attempts: int = 1, base_delay: float = 1.0) -> None:
"""Fetch CA root cert and request a service certificate.
If no console_url was provided, discovers it from the services
table. Performs initial cert provisioning over plain HTTP (ACME
protocol provides integrity via JWS).
table. Direct deployments provision over HTTP on a trusted network;
the dedicated enrollment JWT authenticates the node but does not make
plaintext transport confidential or resistant to an on-path attacker.
With ``attempts > 1``, failures are retried with exponential backoff
(``base_delay * 2**n``). A node restarted alongside the console loses
@@ -341,23 +703,20 @@ class TLSClient:
"Ensure the console is running and has registered, "
"or provide console_url explicitly."
)
url = consoles[0]["url"]
url = str(consoles[0]["url"])
log.info("tls.console.discovered", url=url)
return url
async def _fetch_ca_cert(self) -> None:
"""Fetch the CA root cert from the console.
Always uses plain HTTP for bootstrapping the node doesn't have
the CA cert yet, so it can't verify HTTPS.
Uses the configured scheme. Direct deployments normally use plain HTTP
(TOFU); an explicitly configured HTTPS proxy can provide independently
trusted server authentication before the cluster CA is available.
"""
import httpx
# Force HTTP for bootstrap (can't verify HTTPS without CA cert)
base = self._console_url.replace("https://", "http://")
url = f"{base}/acme/ca.pem"
url = f"{self._console_url}/acme/ca.pem"
try:
async with httpx.AsyncClient() as client:
async with httpx2.AsyncClient(trust_env=False) as client:
resp = await client.get(url)
resp.raise_for_status()
self._ca_pem = resp.content
@@ -374,87 +733,141 @@ class TLSClient:
if not self._hostnames:
raise ValueError("No hostnames configured for TLS cert request")
# Check for existing valid cert
from datetime import UTC, datetime
existing = self._store.load_cert(self._hostnames[0])
if existing is not None and existing.expires_at > datetime.now(UTC):
self._bundle = existing
log.info("tls.cert.loaded", domain=self._hostnames[0])
return
# Request new cert via ACME (plain HTTP for initial request)
lacme = _require_lacme()
from lacme.challenges.http01 import HTTP01Handler
primary = str(self._hostnames[0])
existing = self._store.load_cert(primary)
if existing is not None:
try:
self._validate_bundle(existing)
except ValueError as exc:
log.warning(
"tls.cert.invalid_existing_identity",
domain=primary,
error=str(exc),
)
else:
self._bundle = existing
log.info("tls.cert.loaded", domain=primary)
return
# Keep the old row until the complete client-side bundle is ready;
# StorageStore ignores the responder's intermediate keyless result,
# so successful issuance atomically replaces it without sacrificing
# the last usable identity on a transient failure.
log.info("tls.cert.identifiers_changed", domain=primary)
# Request a new cert through a responder-pinned, authenticated HTTPX2
# client. lacme deliberately leaves injected client ownership here.
directory_url = f"{self._console_url}/acme/directory"
http_client = build_acme_http_client(
self._console_url,
external_url=self._acme_external_url,
token_provider=self._enrollment_token_provider,
)
try:
async with lacme.Client(
directory_url=directory_url,
store=self._validated_store,
event_dispatcher=self._event_dispatcher,
challenge_handler=TurnstoneAutoApproveChallengeHandler(),
http_client=http_client,
allow_insecure=True,
) as client:
self._bundle = await client.issue(self._hostnames)
log.info("tls.cert.issued", domain=primary)
finally:
async with lacme.Client(
directory_url=directory_url,
store=self._store,
event_dispatcher=self._event_dispatcher,
challenge_handler=HTTP01Handler(),
allow_insecure=True,
) as client:
self._bundle = await client.issue(self._hostnames)
self._store.save_cert(self._bundle)
log.info("tls.cert.issued", domain=self._hostnames[0])
async def _close_http_client() -> None:
try:
await http_client.aclose()
except Exception:
log.exception("tls.enrollment.http_close_error")
await complete_tls_cleanup(_close_http_client())
# -- Auto-renewal ----------------------------------------------------------
async def start_renewal(self) -> None:
"""Start background auto-renewal via the console's ACME endpoint."""
lacme = _require_lacme()
from lacme.challenges.http01 import HTTP01Handler
if self._renewal_manager is not None:
raise RuntimeError("TLS renewal is already running")
directory_url = f"{self._console_url}/acme/directory"
client = lacme.Client(
directory_url=directory_url,
store=self._store,
event_dispatcher=self._event_dispatcher,
challenge_handler=HTTP01Handler(),
allow_insecure=True,
http_client = build_acme_http_client(
self._console_url,
external_url=self._acme_external_url,
token_provider=self._enrollment_token_provider,
)
await client.__aenter__()
client: lacme.Client | None = None
try:
client = lacme.Client(
directory_url=directory_url,
store=self._validated_store,
event_dispatcher=self._event_dispatcher,
challenge_handler=TurnstoneAutoApproveChallengeHandler(),
http_client=http_client,
allow_insecure=True,
)
# Scope the renewal sweep to this node's own certificate. The store
# is shared cluster-wide; an unscoped RenewalManager would renew every
# node's cert on every node (see :class:`_SingleDomainStore`). An
# empty domain matches nothing, so a missing hostname renews nothing
# rather than falling back to re-signing the whole cluster.
own_domain = self._hostnames[0] if self._hostnames else ""
renewal_store = _SingleDomainStore(self._store, own_domain)
manager = lacme.RenewalManager(
client=client,
store=renewal_store,
interval_hours=_RENEW_INTERVAL_HOURS,
days_before_expiry=_RENEW_BEFORE_EXPIRY_DAYS,
on_renewed=self._handle_renewed,
event_dispatcher=self._event_dispatcher,
)
self._renewal_task = manager.start()
# Scope the renewal sweep to this node's own certificate. The store
# is shared cluster-wide; an unscoped manager would renew every
# node's cert on every node (see RenewalStoreView). Empty domain
# matches nothing rather than re-signing the whole cluster.
own_domain = str(self._hostnames[0]) if self._hostnames else ""
from turnstone.core.tls_store import RenewalStoreView
renewal_store = RenewalStoreView(self._validated_store, own_domain)
manager = lacme.RenewalManager(
client=client,
store=renewal_store,
interval_hours=RENEW_INTERVAL_HOURS,
days_before_expiry=RENEW_BEFORE_EXPIRY_DAYS,
max_jitter_seconds=RENEW_MAX_JITTER_SECONDS,
on_renewed=self._handle_renewed,
event_dispatcher=self._event_dispatcher,
)
manager.start()
except BaseException:
async def _cleanup_failed_start() -> None:
if client is not None:
with contextlib.suppress(Exception):
await client.close()
with contextlib.suppress(Exception):
await http_client.aclose()
await complete_tls_cleanup(_cleanup_failed_start())
raise
self._renewal_manager = manager
self._renewal_client = client
self._renewal_http_client = http_client
log.info("tls.renewal.started", directory=directory_url)
async def stop_renewal(self) -> None:
"""Stop background renewal and close the ACME client."""
import contextlib
manager, self._renewal_manager = self._renewal_manager, None
client, self._renewal_client = self._renewal_client, None
http_client, self._renewal_http_client = self._renewal_http_client, None
if manager is None and client is None and http_client is None:
return
if self._renewal_task is not None:
import asyncio
self._renewal_task.cancel()
async def _cleanup() -> None:
try:
with contextlib.suppress(asyncio.CancelledError):
await self._renewal_task
if manager is not None:
await manager.stop()
except Exception:
log.exception("tls.renewal.stop_error")
self._renewal_task = None
if self._renewal_client is not None:
with contextlib.suppress(Exception):
await self._renewal_client.__aexit__(None, None, None)
self._renewal_client = None
finally:
if client is not None:
try:
await client.close()
except Exception:
log.exception("tls.renewal.client_close_error")
if http_client is not None:
try:
await http_client.aclose()
except Exception:
log.exception("tls.renewal.http_close_error")
await complete_tls_cleanup(_cleanup())
# -- SSL contexts ----------------------------------------------------------
@@ -462,10 +875,9 @@ class TLSClient:
"""Build SSL context for uvicorn HTTPS listener."""
if self._bundle is None or self._ca_pem is None:
return None
_require_lacme()
from lacme.mtls import server_ssl_context
return server_ssl_context( # type: ignore[no-any-return,unused-ignore]
return server_ssl_context(
cert_pem=self._bundle.fullchain_pem,
key_pem=self._bundle.key_pem,
ca_cert_pem=self._ca_pem,
@@ -475,10 +887,9 @@ class TLSClient:
"""Build mTLS client context for httpx connections."""
if self._bundle is None or self._ca_pem is None:
return None
_require_lacme()
from lacme.mtls import client_ssl_context
return client_ssl_context( # type: ignore[no-any-return,unused-ignore]
return client_ssl_context(
cert_pem=self._bundle.cert_pem,
key_pem=self._bundle.key_pem,
ca_cert_pem=self._ca_pem,
@@ -491,7 +902,7 @@ class TLSClient:
return self._ca_pem
@property
def bundle(self) -> Any | None:
def bundle(self) -> CertBundle | None:
return self._bundle
@property
+177 -34
View File
@@ -8,10 +8,18 @@ rather than the filesystem.
from __future__ import annotations
import json
import re
from dataclasses import replace
from datetime import UTC, datetime
from typing import TYPE_CHECKING, Any
from cryptography.hazmat.primitives.asymmetric import ec
from cryptography.hazmat.primitives.serialization import load_pem_private_key
from lacme import CertBundle, Store
if TYPE_CHECKING:
from collections.abc import Callable
from turnstone.core.storage._protocol import StorageBackend
@@ -23,30 +31,69 @@ def _parse_utc(iso: str) -> datetime:
return dt
def _ensure_lacme() -> Any:
"""Import lacme, raising a clear error if not installed."""
def _with_authoritative_leaf_times(bundle: CertBundle) -> CertBundle:
"""Replace mutable row timestamps with the signed leaf validity window."""
from cryptography import x509
try:
import lacme
except ImportError:
raise ImportError(
"lacme is required for TLS support. Install with: pip install turnstone[tls]",
) from None
return lacme
certificates = x509.load_pem_x509_certificates(bundle.cert_pem)
except (TypeError, ValueError) as exc:
raise ValueError(f"Certificate {bundle.domain!r} is malformed") from exc
if len(certificates) != 1:
raise ValueError(f"Certificate {bundle.domain!r} does not contain one leaf")
leaf = certificates[0]
return replace(
bundle,
issued_at=leaf.not_valid_before_utc,
expires_at=leaf.not_valid_after_utc,
)
_SCOPED_CERT_PREFIX = "turnstone-scope:"
_STORE_NAMESPACE_RE = re.compile(r"^[a-z0-9][a-z0-9-]*$")
class StorageStore:
"""lacme Store implementation backed by turnstone's database.
Implements the 7-method Store protocol that lacme's CertificateAuthority
Implements the 8-method Store protocol that lacme's CertificateAuthority
and Client use for persistence.
"""
def __init__(self, storage: StorageBackend) -> None:
def __init__(
self,
storage: StorageBackend,
*,
namespace: str = "",
account_key_id: str = "default",
) -> None:
if namespace and _STORE_NAMESPACE_RE.fullmatch(namespace) is None:
raise ValueError(f"Invalid TLS store namespace: {namespace!r}")
if not account_key_id:
raise ValueError("TLS account key ID must be non-empty")
self._storage = storage
self._namespace = namespace
self._account_key_id = account_key_id
def _stored_domain(self, domain: str) -> str:
if not self._namespace:
return domain
return f"{_SCOPED_CERT_PREFIX}{self._namespace}:{domain}"
def _owns_row(self, row: dict[str, Any]) -> bool:
domain = str(row["domain"])
meta = json.loads(row.get("meta") or "{}")
row_namespace = meta.get("namespace")
if not self._namespace:
return not domain.startswith(_SCOPED_CERT_PREFIX) and not row_namespace
return (
domain.startswith(f"{_SCOPED_CERT_PREFIX}{self._namespace}:")
and row_namespace == self._namespace
)
# -- Account key -----------------------------------------------------------
def save_account_key(self, key: Any) -> None:
def save_account_key(self, key: ec.EllipticCurvePrivateKey) -> None:
"""Persist the ACME account private key."""
from cryptography.hazmat.primitives.serialization import (
Encoding,
@@ -55,16 +102,19 @@ class StorageStore:
)
key_pem = key.private_bytes(Encoding.PEM, PrivateFormat.PKCS8, NoEncryption()).decode()
self._storage.save_tls_account_key("default", key_pem)
self._storage.save_tls_account_key(self._account_key_id, key_pem)
def load_account_key(self) -> Any | None:
def load_account_key(self) -> ec.EllipticCurvePrivateKey | None:
"""Load the ACME account private key, or None."""
from cryptography.hazmat.primitives.serialization import load_pem_private_key
pem = self._storage.load_tls_account_key("default")
pem = self._storage.load_tls_account_key(self._account_key_id)
if pem is None:
return None
return load_pem_private_key(pem.encode(), password=None)
key = load_pem_private_key(pem.encode(), password=None)
if not isinstance(key, ec.EllipticCurvePrivateKey):
raise TypeError(f"Expected EC private key, got {type(key).__name__}")
if not isinstance(key.curve, ec.SECP256R1):
raise TypeError(f"Expected P-256 key, got {key.curve.name}")
return key
# -- CA --------------------------------------------------------------------
@@ -81,11 +131,25 @@ class StorageStore:
# -- Certificates ----------------------------------------------------------
def save_cert(self, bundle: Any) -> Any:
"""Persist an issued certificate bundle."""
meta = json.dumps({"domains": list(bundle.domains)})
def save_cert(self, bundle: CertBundle) -> CertBundle:
"""Persist a usable, private-key-bearing certificate identity.
The ACME responder signs a client CSR and briefly produces a keyless
bundle before the client saves its complete identity. Persisting that
responder-side bundle would overwrite an existing private key with an
empty value if enrollment is interrupted.
"""
if not bundle.key_pem:
return bundle
meta = json.dumps(
{
"domain": bundle.domain,
"domains": list(bundle.domains),
"namespace": self._namespace,
}
)
self._storage.save_tls_cert(
domain=bundle.domain,
domain=self._stored_domain(bundle.domain),
cert_pem=bundle.cert_pem.decode(),
fullchain_pem=bundle.fullchain_pem.decode(),
key_pem=bundle.key_pem.decode(),
@@ -95,29 +159,36 @@ class StorageStore:
)
return bundle
def load_cert(self, domain: str) -> Any | None:
def load_cert(self, domain: str) -> CertBundle | None:
"""Load a certificate bundle by domain."""
row = self._storage.load_tls_cert(domain)
if row is None:
row = self._storage.load_tls_cert(self._stored_domain(domain))
if row is None or not row.get("key_pem"):
return None
return self._row_to_bundle(row)
bundle = self._row_to_bundle(row)
if bundle.domain != domain:
raise ValueError(
f"Stored TLS identity {bundle.domain!r} does not match lookup key {domain!r}"
)
return bundle
def list_certs(self) -> list[Any]:
"""List all stored certificate bundles."""
def list_certs(self) -> list[CertBundle]:
"""List all managed, private-key-bearing certificate bundles."""
rows = self._storage.list_tls_certs()
return [self._row_to_bundle(r) for r in rows]
return [
self._row_to_bundle(row) for row in rows if row.get("key_pem") and self._owns_row(row)
]
def delete_cert(self, domain: str) -> bool:
"""Delete a stored certificate bundle by domain."""
return self._storage.delete_tls_cert(domain)
return bool(self._storage.delete_tls_cert(self._stored_domain(domain)))
def _row_to_bundle(self, row: dict[str, Any]) -> Any:
def _row_to_bundle(self, row: dict[str, Any]) -> CertBundle:
"""Convert a storage row dict to a lacme CertBundle."""
lacme = _ensure_lacme()
meta = json.loads(row.get("meta") or "{}")
domains = tuple(meta.get("domains", [row["domain"]]))
return lacme.CertBundle(
domain=row["domain"],
domain = str(meta.get("domain", row["domain"]))
domains = tuple(meta.get("domains", [domain]))
return CertBundle(
domain=domain,
domains=domains,
cert_pem=row["cert_pem"].encode(),
fullchain_pem=row["fullchain_pem"].encode(),
@@ -125,3 +196,75 @@ class StorageStore:
issued_at=_parse_utc(row["issued_at"]),
expires_at=_parse_utc(row["expires_at"]),
)
class RenewalStoreView:
"""Store view exposing one managed identity to a renewal sweep.
Turnstone's certificate store is shared cluster-wide. lacme renews every
bundle returned by ``list_certs()``, so each service must receive a scoped
view while every other Store operation continues to delegate normally.
"""
def __init__(self, inner: Store, domain: str) -> None:
self._inner = inner
self._domain = domain
def save_account_key(self, key: ec.EllipticCurvePrivateKey) -> None:
self._inner.save_account_key(key)
def load_account_key(self) -> ec.EllipticCurvePrivateKey | None:
return self._inner.load_account_key()
def save_ca(self, name: str, cert_pem: bytes, key_pem: bytes) -> None:
self._inner.save_ca(name, cert_pem, key_pem)
def load_ca(self, name: str) -> tuple[bytes, bytes] | None:
return self._inner.load_ca(name)
def save_cert(self, bundle: CertBundle) -> CertBundle:
return self._inner.save_cert(bundle)
def load_cert(self, domain: str) -> CertBundle | None:
return self._inner.load_cert(domain)
def list_certs(self) -> list[CertBundle]:
cert = self._inner.load_cert(self._domain)
return [_with_authoritative_leaf_times(cert)] if cert is not None else []
def delete_cert(self, domain: str) -> bool:
return self._inner.delete_cert(domain)
class CertificateValidationStore:
"""Store wrapper that validates complete identities before persistence."""
def __init__(self, inner: Store, validator: Callable[[CertBundle], None]) -> None:
self._inner = inner
self._validator = validator
def save_account_key(self, key: ec.EllipticCurvePrivateKey) -> None:
self._inner.save_account_key(key)
def load_account_key(self) -> ec.EllipticCurvePrivateKey | None:
return self._inner.load_account_key()
def save_ca(self, name: str, cert_pem: bytes, key_pem: bytes) -> None:
self._inner.save_ca(name, cert_pem, key_pem)
def load_ca(self, name: str) -> tuple[bytes, bytes] | None:
return self._inner.load_ca(name)
def save_cert(self, bundle: CertBundle) -> CertBundle:
if bundle.key_pem:
self._validator(bundle)
return self._inner.save_cert(bundle)
def load_cert(self, domain: str) -> CertBundle | None:
return self._inner.load_cert(domain)
def list_certs(self) -> list[CertBundle]:
return self._inner.list_certs()
def delete_cert(self, domain: str) -> bool:
return self._inner.delete_cert(domain)
+5 -6
View File
@@ -64,12 +64,11 @@ def _apply_kind_variant(tool: dict[str, Any], kind: str, meta: dict[str, Any]) -
"""Return a kind-specific copy of ``tool`` with description / params overridden.
Each kind sees only the surface it can actually use for ``memory``,
coord sessions get a description + scope enum that mention only the
``coordinator`` scope, while interactive sessions get a description
+ scope enum that omit ``coordinator`` entirely. This keeps the
LLM contract tight: the model never sees enum values it can't use,
and never reads description sentences explaining why a scope is
forbidden.
coord sessions get ``coordinator`` plus the attach-dependent ``project``
scope, while interactive sessions get global/workstream/user plus
``project``. This keeps the LLM contract tight: the model never sees enum
values it can't use, and never reads description sentences explaining why
a scope is forbidden.
No-op (returns the input tool unchanged) when the tool has no
``kind_variants`` metadata or no entry for ``kind``. Otherwise
+107 -2
View File
@@ -3,15 +3,23 @@
from __future__ import annotations
import contextlib
import hashlib
import json
import os
import re
from functools import cached_property
from typing import TYPE_CHECKING, Any
from starlette.datastructures import Headers
from starlette.exceptions import HTTPException
from starlette.responses import FileResponse, PlainTextResponse, Response
from starlette.staticfiles import StaticFiles
if TYPE_CHECKING:
from starlette.middleware import Middleware
from starlette.requests import Request
from starlette.responses import JSONResponse
from starlette.types import Scope
def latin1_safe_filename(name: str, *, fallback: str = "attachment") -> str:
@@ -463,6 +471,103 @@ def cors_middleware(origins: list[str]) -> Middleware:
# Static asset cache-busting
# ---------------------------------------------------------------------------
# Keep this directory definition shared by the HTML rewriter and static response
# policy. These directories already carry their library version in the URL,
# so their contents can be cached immutably; every other first-party asset must
# revalidate because ES-module imports do not inherit the entry module's ?v=.
_VERSIONED_VENDOR_DIR = r"(?:katex|hljs|hls|mermaid)-\d+(?:\.\d+)+"
_VERSIONED_VENDOR_PATH_RE = re.compile(rf"^{_VERSIONED_VENDOR_DIR}/")
def is_safe_static_asset_path(path: str) -> bool:
"""Return whether *path* is a canonical mount-relative asset path.
Starlette decodes percent escapes before populating ``{path:path}``, and
HTTPX normalizes dot segments when it builds an outbound URL. Reject the
ambiguous forms before a console proxy can leave its static mount.
"""
return (
bool(path)
and "\\" not in path
and all(segment not in {"", ".", ".."} for segment in path.split("/"))
)
def static_asset_cache_control(path: str) -> str:
"""Return the cache policy for an asset path relative to its mount."""
if not is_safe_static_asset_path(path):
return "no-store"
if _VERSIONED_VENDOR_PATH_RE.match(path):
return "public, max-age=31536000, immutable"
return "no-cache"
class RevalidatingStaticFiles(StaticFiles):
"""StaticFiles with correctness-first caching for first-party assets.
First-party files use a content-derived ETag and no Last-Modified header.
``no-cache`` therefore revalidates by content on reload, even when two
builds share a package version, byte length, and filesystem timestamp.
Version-named vendor directories retain immutable caching.
"""
@cached_property
def _content_etags(self) -> dict[str, tuple[tuple[int, int, int], str]]:
"""Lazily allocate the bounded per-static-tree validator cache."""
return {}
def _content_etag(
self,
full_path: str | os.PathLike[str],
stat_result: os.stat_result,
) -> str:
path = os.fspath(full_path)
signature = (stat_result.st_mtime_ns, stat_result.st_ctime_ns, stat_result.st_size)
cached = self._content_etags.get(path)
if cached is not None and cached[0] == signature:
return cached[1]
with open(path, "rb") as asset:
digest = hashlib.file_digest(asset, "sha256").hexdigest()
etag = f'"sha256-{digest}"'
self._content_etags[path] = (signature, etag)
return etag
def file_response(
self,
full_path: str | os.PathLike[str],
stat_result: os.stat_result,
scope: Scope,
status_code: int = 200,
) -> Response:
asset_path = self.get_path(scope)
if static_asset_cache_control(asset_path) != "no-cache":
return super().file_response(full_path, stat_result, scope, status_code)
request_headers = Headers(scope=scope)
response = FileResponse(full_path, status_code=status_code, stat_result=stat_result)
etag = self._content_etag(full_path, stat_result)
response.headers["ETag"] = etag
del response.headers["Last-Modified"]
if self.is_not_modified(response.headers, request_headers):
return Response(status_code=304, headers={"ETag": etag})
return response
async def get_response(self, path: str, scope: Scope) -> Response:
try:
response = await super().get_response(path, scope)
except HTTPException as exc:
if exc.status_code != 404:
raise
return PlainTextResponse(
"Not Found", status_code=404, headers={"Cache-Control": "no-store"}
)
cache_control = (
"no-store" if response.status_code >= 400 else static_asset_cache_control(path)
)
response.headers["Cache-Control"] = cache_control
return response
# Matches src="/static/..." and href="/shared/..." (and vice-versa) but skips
# vendored libraries whose directory names already contain a version number
# (e.g. katex-0.16.44/, hljs-11.11.1/) and URLs that already have a query
@@ -470,7 +575,7 @@ def cors_middleware(origins: list[str]) -> Middleware:
_ASSET_RE = re.compile(
r'(?P<attr>(?:src|href)=")'
r"(?P<path>/(?:static|shared)/)"
r"(?!(?:katex|hljs|hls|mermaid)-\d)"
rf"(?!{_VERSIONED_VENDOR_DIR}/)"
r'(?P<file>[^"?]+)"'
)
@@ -480,7 +585,7 @@ def version_html(html: str) -> str:
Vendored libraries with version-bearing directory names are skipped.
URLs that already contain a query string are left unchanged.
Called once at startup when loading HTML into memory.
Safe to call either at startup or while rendering a dynamic template.
"""
from turnstone import __version__
+1
View File
@@ -86,6 +86,7 @@ _RELEVANT_ENV_VARS: tuple[str, ...] = (
"MODEL",
"TURNSTONE_HOST_IP",
"TURNSTONE_CONSOLE_URL",
"TURNSTONE_ACME_EXTERNAL_URL",
"TURNSTONE_SERVER_URL",
"TURNSTONE_ADVERTISE_URL",
"TURNSTONE_NODE_ID",
+8 -8
View File
@@ -232,7 +232,7 @@ class HeadlessSession(ChatSession):
- auto_approve is always True
- Tool calls are recorded into a structured log
- All stdout output is suppressed
- send_headless() uses non-streaming API
- send_headless() drains the production streaming provider path
"""
def __init__(
@@ -307,8 +307,8 @@ class HeadlessSession(ChatSession):
) -> list[dict[str, Any]]:
"""Run a complete conversation turn headlessly.
Uses non-streaming API calls. Captures all tool calls into
self.tool_call_log.
Drains production streaming provider calls into single-shot results.
Captures all tool calls into ``self.tool_call_log``.
Returns the tool call log: list of dicts with keys:
tool: str, args: dict, result: str (truncated), ok: bool,
@@ -533,9 +533,9 @@ def run_with_lifecycle(
past the teardown (which closes only the last one built).
* *drive* (returned by ``build_session``) submitted to a
one-worker executor and bounded by ``future.result(test_timeout)``.
The per-request httpx timeout cannot bound a STREAM: a trickling
response resets the read timeout indefinitely, so without the
wall clock a hung generation occupies a run slot forever. On
The per-request HTTP transport timeout cannot bound a STREAM: a
trickling response resets the read timeout indefinitely, so without
the wall clock a hung generation occupies a run slot forever. On
timeout the session is DROPPED, not closed: the shutdown did not
wait, so the worker is still inside the drive, and ``close()``'s
bounded shell-join would trade a bounded leak for a blocked
@@ -736,8 +736,8 @@ def _run_single_test(
)
def _build_client() -> Any:
# Per-attempt client with request-level timeout so httpx aborts
# the HTTP request itself — no zombie connections on the server.
# Per-attempt client with a per-read timeout. The executor wall clock
# above separately bounds a trickling stream that keeps resetting it.
return OpenAI(
base_url=client.base_url,
api_key=client.api_key,
+5 -6
View File
@@ -259,9 +259,8 @@ def _tasks_action_enum() -> frozenset[str]:
#
# Neither half is a literal here. The action VOCABULARY comes from the
# tool's own schema (:func:`_tasks_action_enum`) and the READ half is
# ``ChatSession._TASKS_READ_ACTIONS``, production's own classifier
# the one its parallel-batch guard and approval path rule on. Mutating
# is the remainder, so an action added to the schema counts as a
# ``_TASKS_READ_ACTIONS``, production's own preparer classifier.
# Mutating is the remainder, so an action added to the schema counts as a
# mutation until production classifies it as a read: a new write can
# never be silently dropped from the bookkeeping test, and the drift
# that IS possible (a new read) is caught statically by
@@ -824,7 +823,7 @@ def _seed_world(storage: Any, case: dict[str, Any]) -> None:
saved, _was_update = save_structured_memory(
row["name"],
row["content"],
row.get("description"),
row["description"],
row.get("type"),
scope=row.get("scope", "global"),
scope_id=row.get("scope_id", ""),
@@ -1969,7 +1968,7 @@ def _check_world_is_seedable(case: dict[str, Any]) -> str | None:
Recognized keys only (``memory`` / ``nodes``) an unrecognized key
is a silent no-op seed, which reads as "seeded" while leaving the
hollow world the block exists to fill. Memory rows need non-empty
string ``name`` and ``content`` (the production upsert's own
string ``name``, ``description``, and ``content`` (the production upsert's own
requirements, surfaced at authoring time); node rows need a
non-empty string ``node_id``.
"""
@@ -1984,7 +1983,7 @@ def _check_world_is_seedable(case: dict[str, Any]) -> str | None:
for i, row in enumerate(world.get("memory", ())):
if not isinstance(row, dict):
return f"world.memory[{i}] must be a dict"
for field in ("name", "content"):
for field in ("name", "content", "description"):
v = row.get(field)
if not isinstance(v, str) or not v.strip():
return f"world.memory[{i}].{field} must be a non-empty string"
+2
View File
@@ -95,6 +95,7 @@ NUDGE_CELLS: list[dict[str, Any]] = [
"memory": [
{
"name": "acme-api-project",
"description": "Acme API repository and deployment context",
"type": "reference",
"content": (
"acme-api: FastAPI service. Repo layout: "
@@ -105,6 +106,7 @@ NUDGE_CELLS: list[dict[str, Any]] = [
},
{
"name": "auth-backend-migration-status",
"description": "Current authentication migration status",
"content": (
"migrations/007_auth_backend.sql applied on the "
"staging replica; auth service suite green "
+6 -4
View File
@@ -525,19 +525,21 @@ class AsyncTurnstoneServer(_BaseClient):
name: str,
content: str,
*,
description: str = "",
description: str,
mem_type: str = "general",
scope: str = "global",
scope_id: str = "",
) -> MemoryInfo:
description = (description or "").strip()
if not description:
raise ValueError("memory description is required and must be non-empty")
body: dict[str, Any] = {
"name": name,
"content": content,
"description": description,
"type": mem_type,
"scope": scope,
}
if description:
body["description"] = description
if scope_id:
body["scope_id"] = scope_id
return await self._request(
@@ -868,7 +870,7 @@ class TurnstoneServer:
name: str,
content: str,
*,
description: str = "",
description: str,
mem_type: str = "general",
scope: str = "global",
scope_id: str = "",
+263 -79
View File
@@ -42,7 +42,6 @@ from starlette.middleware import Middleware
from starlette.requests import Request
from starlette.responses import HTMLResponse, JSONResponse, Response, StreamingResponse
from starlette.routing import Mount, Route
from starlette.staticfiles import StaticFiles
from turnstone import __version__
from turnstone.api.docs import make_docs_handler, make_openapi_handler
@@ -78,7 +77,10 @@ from turnstone.core.session_manager import (
STALE_CREATE_SWEEP_INTERVAL_SECONDS,
SessionManager,
)
from turnstone.core.session_replay import session_replay_preamble
from turnstone.core.session_replay import (
request_replay_project_name,
session_replay_preamble,
)
from turnstone.core.session_routes import (
AttachmentUploadHelpers,
CreatePreCommitError,
@@ -111,6 +113,7 @@ from turnstone.core.session_ui_base import (
)
from turnstone.core.tools import TOOLS # noqa: F401 — available for introspection
from turnstone.core.trajectory import final_assistant_text
from turnstone.core.web_helpers import RevalidatingStaticFiles
from turnstone.core.web_helpers import version_html as _version_html
from turnstone.core.workstream import (
Workstream,
@@ -801,12 +804,9 @@ def _interactive_events_replay(
) -> Iterable[dict[str, Any]]:
"""Initial SSE replay payload for interactive ``events`` connections.
Yields a ``connected`` event (model + skip_permissions), a
``status`` event with the workstream's last token usage + context %
(when a turn has completed), and the pending approval prompt + cached
intent verdicts (if a prompt is pending). The lifted
``make_events_handler`` body delegates that yield sequence to this
callback so the kind-specific shape stays in this module.
Yields the shared ``connected`` / ``status`` preamble, followed by pending
approval prompts and cached intent verdicts. The lifted handler resolves
viewer-specific project metadata off-loop before invoking this callback.
Conversation history is NOT replayed over SSE: the frontend fetches
it via ``GET /history`` on page load and re-fetches on the
@@ -821,10 +821,11 @@ def _interactive_events_replay(
# session can still be detached on the close-then-reopen path.
return
# Connected + status preamble — same shape coord replays and the
# lifted reconnect path use; one shared function, no per-kind
# wrapper, so a future field add cannot land on one surface only.
yield from session_replay_preamble(ws.session, ui)
yield from session_replay_preamble(
ws.session,
ui,
project_name=request_replay_project_name(request),
)
# Pending approval re-injection (so a reconnecting tab sees the
# prompt) + cached LLM verdicts received since the prompt fired.
@@ -3495,31 +3496,157 @@ def _resolve_user_scope_id(
return uid, None
def _resolve_workstream_memory_scope_id(
request: Request,
scope_id: str,
) -> tuple[str, JSONResponse | None]:
"""Bind REST workstream-memory access to the authenticated owner."""
from turnstone.core.storage._registry import get_storage
resolved = scope_id.strip()
if not resolved:
return "", JSONResponse(
{"error": "scope_id is required for workstream scope"},
status_code=400,
)
owner = get_storage().get_workstream_owner(resolved)
if owner is None:
return "", JSONResponse({"error": "Workstream not found"}, status_code=404)
caller = _auth_user_id(request)
if "service" not in _auth_scopes(request) and (not caller or owner != caller):
return "", JSONResponse(
{"error": "Cannot access another user's workstream memories"},
status_code=403,
)
return resolved, None
def _resolve_rest_memory_scope(
request: Request,
scope: str,
scope_id: str,
*,
allow_empty: bool,
) -> tuple[str, str, JSONResponse | None]:
"""Validate a public memory scope and bind its caller-controlled id."""
normalized_scope = scope.strip().lower()
normalized_id = scope_id.strip()
if not normalized_scope and allow_empty:
err = _validate_scope_scope_id(normalized_scope, normalized_id)
return normalized_scope, normalized_id, err
if normalized_scope not in _VALID_MEMORY_SCOPES:
return (
"",
"",
JSONResponse(
{
"error": (
f"invalid scope: {normalized_scope}; "
f"must be one of {sorted(_VALID_MEMORY_SCOPES)}"
)
},
status_code=400,
),
)
if normalized_scope == "user":
normalized_id, err = _resolve_user_scope_id(request, normalized_id)
if err:
return "", "", err
elif normalized_scope == "workstream":
normalized_id, err = _resolve_workstream_memory_scope_id(request, normalized_id)
if err:
return "", "", err
err = _validate_scope_scope_id(
normalized_scope,
normalized_id,
require_scope_id=True,
)
return normalized_scope, normalized_id, err
def _rest_visible_memory_scopes(request: Request) -> list[tuple[str, str]]:
"""Default public read envelope: global plus the caller's user scope."""
scopes = [("global", "")]
uid = _auth_user_id(request)
if uid:
scopes.append(("user", uid))
return scopes
def _audit_rest_memory_mutation(
request: Request,
action: str,
row: dict[str, str],
) -> None:
"""Record one authenticated REST/SDK memory mutation."""
from turnstone.core.audit import record_audit
from turnstone.core.storage._registry import get_storage
uid, ip = _audit_context(request)
record_audit(
get_storage(),
uid,
action,
"memory",
row["memory_id"],
{
"name": row["name"],
"scope": row["scope"],
"scope_id": row["scope_id"],
"type": row["type"],
"surface": "rest",
},
ip,
)
async def list_memories(request: Request) -> JSONResponse:
"""GET /v1/api/memories — list memories with optional filters."""
from turnstone.core.memory import list_structured_memories
from turnstone.core.storage._registry import get_storage
mem_type = request.query_params.get("type", "")
mem_type = request.query_params.get("type", "").strip().lower()
scope = request.query_params.get("scope", "")
scope_id = request.query_params.get("scope_id", "")
if mem_type and mem_type not in _VALID_MEMORY_TYPES:
return JSONResponse({"error": f"invalid type: {mem_type}"}, status_code=400)
try:
limit = min(int(request.query_params.get("limit", "100")), 200)
limit = int(request.query_params.get("limit", "100"))
except (ValueError, TypeError):
return JSONResponse({"error": "limit must be an integer"}, status_code=400)
err = _validate_scope_scope_id(scope, scope_id)
if not 1 <= limit <= 200:
return JSONResponse({"error": "limit must be between 1 and 200"}, status_code=400)
scope, scope_id, err = _resolve_rest_memory_scope(
request,
scope,
scope_id,
allow_empty=True,
)
if err:
return err
if scope == "user":
scope_id, err = _resolve_user_scope_id(request, scope_id)
if err:
return err
rows = list_structured_memories(mem_type=mem_type, scope=scope, scope_id=scope_id, limit=limit)
try:
storage = get_storage()
if scope:
rows = storage.list_structured_memories(
mem_type=mem_type,
scope=scope,
scope_id=scope_id,
limit=limit,
)
else:
rows = storage.list_visible_structured_memories(
_rest_visible_memory_scopes(request),
mem_type=mem_type,
limit=limit,
)
except Exception:
log.warning("memory.rest_list_failed", exc_info=True)
return JSONResponse({"error": "Memory storage unavailable"}, status_code=500)
return JSONResponse({"memories": rows, "total": len(rows)})
async def save_memory(request: Request) -> JSONResponse:
"""POST /v1/api/memories — save (upsert) a structured memory."""
from turnstone.core.memory import save_structured_memory
from turnstone.core.memory import save_structured_memory_strict
from turnstone.core.web_helpers import read_json_or_400
body = await read_json_or_400(request)
@@ -3536,12 +3663,15 @@ async def save_memory(request: Request) -> JSONResponse:
{"error": f"content exceeds {_MAX_MEMORY_CONTENT} character limit"},
status_code=400,
)
# None (field omitted) means "leave unset": the upsert keeps the stored
# value on update and defaults on insert; an explicit value overwrites.
raw_desc = body.get("description")
description = None if raw_desc is None else str(raw_desc)
description = "" if raw_desc is None else str(raw_desc).strip()
if not description:
return JSONResponse(
{"error": "description is required and must be non-empty"},
status_code=400,
)
raw_type = body.get("type")
mem_type = None if raw_type is None else str(raw_type)
mem_type = None if raw_type is None else str(raw_type).strip().lower()
scope = str(body.get("scope", "global"))
scope_id = str(body.get("scope_id", ""))
if mem_type is not None and mem_type not in _VALID_MEMORY_TYPES:
@@ -3549,24 +3679,31 @@ async def save_memory(request: Request) -> JSONResponse:
{"error": f"invalid type: {mem_type}; must be one of {sorted(_VALID_MEMORY_TYPES)}"},
status_code=400,
)
if scope not in _VALID_MEMORY_SCOPES:
return JSONResponse(
{"error": f"invalid scope: {scope}; must be one of {sorted(_VALID_MEMORY_SCOPES)}"},
status_code=400,
)
if scope == "user":
scope_id, err = _resolve_user_scope_id(request, scope_id)
if err:
return err
err = _validate_scope_scope_id(scope, scope_id, require_scope_id=True)
scope, scope_id, err = _resolve_rest_memory_scope(
request,
scope,
scope_id,
allow_empty=False,
)
if err:
return err
# The upsert RETURNINGs the full saved row, so no follow-up read is needed.
row, was_update = save_structured_memory(
name, content, description=description, mem_type=mem_type, scope=scope, scope_id=scope_id
)
if not row:
try:
row, was_update = save_structured_memory_strict(
name,
content,
description=description,
mem_type=mem_type,
scope=scope,
scope_id=scope_id,
)
except Exception:
log.warning("memory.rest_save_failed", name=name, exc_info=True)
return JSONResponse({"error": "Failed to save memory"}, status_code=500)
_audit_rest_memory_mutation(
request,
"memory.update" if was_update else "memory.save",
row,
)
return JSONResponse(row, status_code=200 if was_update else 201)
@@ -3575,7 +3712,7 @@ async def search_memories(request: Request) -> JSONResponse:
Uses POST for the request body but requires only read scope (non-mutating).
"""
from turnstone.core.memory import search_structured_memories as search_fn
from turnstone.core.storage._registry import get_storage
from turnstone.core.web_helpers import read_json_or_400
body = await read_json_or_400(request)
@@ -3584,46 +3721,72 @@ async def search_memories(request: Request) -> JSONResponse:
query = str(body.get("query", "")).strip()
if not query:
return JSONResponse({"error": "query is required"}, status_code=400)
mem_type = str(body.get("type", ""))
mem_type = str(body.get("type", "")).strip().lower()
scope = str(body.get("scope", ""))
scope_id = str(body.get("scope_id", ""))
try:
limit = min(int(body.get("limit", 20)), 50)
limit = int(body.get("limit", 20))
except (ValueError, TypeError):
return JSONResponse({"error": "limit must be an integer"}, status_code=400)
err = _validate_scope_scope_id(scope, scope_id)
if mem_type and mem_type not in _VALID_MEMORY_TYPES:
return JSONResponse({"error": f"invalid type: {mem_type}"}, status_code=400)
if not 1 <= limit <= 50:
return JSONResponse({"error": "limit must be between 1 and 50"}, status_code=400)
scope, scope_id, err = _resolve_rest_memory_scope(
request,
scope,
scope_id,
allow_empty=True,
)
if err:
return err
if scope == "user":
scope_id, err = _resolve_user_scope_id(request, scope_id)
if err:
return err
rows = search_fn(query, mem_type=mem_type, scope=scope, scope_id=scope_id, limit=limit)
try:
storage = get_storage()
if scope:
rows = storage.search_structured_memories(
query,
mem_type=mem_type,
scope=scope,
scope_id=scope_id,
limit=limit,
)
else:
rows = storage.search_visible_structured_memories(
query,
_rest_visible_memory_scopes(request),
mem_type=mem_type,
limit=limit,
)
except Exception:
log.warning("memory.rest_search_failed", exc_info=True)
return JSONResponse({"error": "Memory storage unavailable"}, status_code=500)
return JSONResponse({"memories": rows, "total": len(rows)})
async def delete_memory_endpoint(request: Request) -> JSONResponse:
"""DELETE /v1/api/memories/{name} — delete a memory by name and scope."""
from turnstone.core.memory import delete_structured_memory, normalize_key
from turnstone.core.memory import delete_structured_memory_returning_strict, normalize_key
name = normalize_key(request.path_params["name"])
scope = request.query_params.get("scope", "global")
if scope not in _VALID_MEMORY_SCOPES:
return JSONResponse(
{"error": f"invalid scope: {scope}; must be one of {sorted(_VALID_MEMORY_SCOPES)}"},
status_code=400,
)
scope_id = request.query_params.get("scope_id", "")
if scope == "user":
scope_id, err = _resolve_user_scope_id(request, scope_id)
if err:
return err
err = _validate_scope_scope_id(scope, scope_id, require_scope_id=True)
scope, scope_id, err = _resolve_rest_memory_scope(
request,
scope,
scope_id,
allow_empty=False,
)
if err:
return err
if delete_structured_memory(name, scope, scope_id):
return JSONResponse({"status": "ok", "name": name})
return JSONResponse({"error": f"Memory '{name}' not found"}, status_code=404)
try:
deleted = delete_structured_memory_returning_strict(name, scope, scope_id)
except Exception:
log.warning("memory.rest_delete_failed", name=name, exc_info=True)
return JSONResponse({"error": "Failed to delete memory"}, status_code=500)
if deleted is None:
return JSONResponse({"error": f"Memory '{name}' not found"}, status_code=404)
_audit_rest_memory_mutation(request, "memory.delete", deleted)
return JSONResponse({"status": "ok", "name": name})
# ---------------------------------------------------------------------------
@@ -5639,8 +5802,16 @@ def create_app(
Route("/metrics", metrics_endpoint),
Route("/openapi.json", _openapi_handler),
Route("/docs", _docs_handler),
Mount("/static", app=StaticFiles(directory=str(_STATIC_DIR)), name="static"),
Mount("/shared", app=StaticFiles(directory=str(_SHARED_DIR)), name="shared"),
Mount(
"/static",
app=RevalidatingStaticFiles(directory=str(_STATIC_DIR)),
name="static",
),
Mount(
"/shared",
app=RevalidatingStaticFiles(directory=str(_SHARED_DIR)),
name="shared",
),
],
middleware=_build_middleware(cors_origins),
lifespan=_lifespan,
@@ -6510,6 +6681,19 @@ def main() -> None:
bind_host=args.host,
extra_sans=os.environ.get("TURNSTONE_TLS_SANS", ""),
)
from turnstone.core.auth import (
JWT_AUD_CONSOLE,
TLS_ACME_TOKEN_SOURCE,
ServiceTokenManager,
)
enrollment_tokens = ServiceTokenManager(
user_id=_node_id,
scopes=frozenset({"service"}),
source=TLS_ACME_TOKEN_SOURCE,
secret=jwt_secret,
audience=JWT_AUD_CONSOLE,
)
tls_client = TLSClient(
storage=get_storage(),
hostnames=hostnames,
@@ -6518,6 +6702,11 @@ def main() -> None:
# explicit override pointing at the published ACME endpoint.
# Empty (the in-cluster default) falls back to service discovery.
console_url=os.environ.get("TURNSTONE_CONSOLE_URL", ""),
# In-cluster clients fetch the directory from ``console`` but a
# cross-host responder advertises its LAN/proxy URL. Trust that
# second credential destination only when operators configured it.
acme_external_url=os.environ.get("TURNSTONE_ACME_EXTERNAL_URL", ""),
enrollment_token_provider=lambda: enrollment_tokens.token,
)
asyncio.run(tls_client.init(attempts=TLS_INIT_RETRY_ATTEMPTS))
bundle = tls_client.bundle
@@ -6552,21 +6741,16 @@ def main() -> None:
"""
from turnstone.core.tls import refresh_runtime_pems, swap_context_cert
try:
new_paths = refresh_runtime_pems(
new_bundle,
ca_pem=tls_client.ca_pem,
previous=pem_dir_state["dir"],
)
pem_dir_state["dir"] = new_paths.cert.parent
except Exception:
log.warning("TLS runtime PEM refresh failed", exc_info=True)
cfg = getattr(app.state, "uvicorn_config", None)
live_ctx = getattr(cfg, "ssl", None) if cfg is not None else None
if live_ctx is None:
return # listener not started yet — boot cert still valid
swap_context_cert(live_ctx, new_bundle, ca_pem=tls_client.ca_pem)
if live_ctx is not None:
swap_context_cert(live_ctx, new_bundle, ca_pem=tls_client.ca_pem)
new_paths = refresh_runtime_pems(
new_bundle,
ca_pem=tls_client.ca_pem,
previous=pem_dir_state["dir"],
)
pem_dir_state["dir"] = new_paths.cert.parent
log.info("TLS cert reloaded into listener: %s", new_bundle.domain)
tls_client.set_cert_reload_hook(_reload_server_cert)

Some files were not shown because too many files have changed in this diff Show More