- The guard snapshotted live threads by `Thread.ident`, but idents are
recycled after a thread exits — a new leaked thread reusing an exited
thread's ident would be mistaken for pre-existing and missed (false
negative). Snapshot the Thread OBJECTS and compare by identity instead.
- Fix the `serve` fixture docstring: the factory returns the ephemeral
port, not the server.
Background daemons, event loops, and test servers that outlived their test
bled into later tests' captured output — an intermittent "I/O operation on
closed file" heisenbug, and the same class behind a past multi-day CI-hang
investigation.
- conftest: a fail-on-leak autouse guard (`_no_leaked_threads`) snapshots
threads at setup and fails any test that leaves one running past teardown,
with an `allow_thread_leak` opt-out — so the next leak is caught in minutes,
not days. Plus `logging.raiseExceptions = False` to mute the benign
logging-vs-capture-teardown race, and shared loop/server teardown helpers
(`stop_loop_thread`, `serve_until_exit`).
- collector (PRODUCT FIX): the node-discovery loop slept uninterruptibly, so
`ClusterCollector.stop()` couldn't join the `console-discovery` thread until
the full interval elapsed — a real shutdown hang in production (up to
`discovery_interval`). It now sleeps on an interruptible Event that `stop()`
sets and `start()` clears.
- test fixtures: docker_healthcheck's HTTP servers, the MCP background event
loops (shutdown_default_executor + close), and the FastMCP uvicorn upstreams
(timeout_graceful_shutdown=0 + force_exit) now tear down cleanly instead of
leaking.
Full non-live suite: 7456 passed, 0 closed-file errors, 0 leaked threads, and
~1.5 min faster (the leaks were dragging it).
A whole-stack restart races every node against the console for the CA
fetch (compose re-enforces depends_on ordering only on `up`): losers
logged one warning and served plain HTTP for their lifetime, while
winners served mTLS that the plain-HTTP container healthcheck could
never probe — leaving "healthy" plaintext nodes and "unhealthy"
working ones.
- TLSClient.init() grows attempts/base_delay retry (server passes 6
attempts, ~31 s backoff) absorbing the boot race; per-attempt CA-fetch
failures log warning + debug traceback instead of error tracebacks.
- healthcheck.py falls back to HTTPS when the plain probe fails,
presenting the node's own cert as the client cert with the cluster CA
pinned; dials localhost because the internal CA issues DNS SANs only.
Default plain-HTTP deployments are unchanged.
- The server writes boot PEMs under a fixed root (TURNSTONE_TLS_PEM_DIR,
default <tmpdir>/turnstone-tls) so the probe can find them; boot
clears stale dirs and refuses a symlinked/foreign-owned root; renewal
rewrites the PEM dir so the probe's client cert never outlives the
served cert.
- /health reports tls: "active"|"fallback" (absent when TLS is
disabled) so a silently downgraded node is observable.
Patrick Buckley