mirror of
https://github.com/turnstonelabs/turnstone.git
synced 2026-08-12 23:12:23 -06:00
Patrick Buckley
752fea0fdd
* feat(console): reactive node discovery via PG LISTEN/NOTIFY dispatcher Add a console-side `NotifyDispatcher` that holds a dedicated PostgreSQL `LISTEN` connection and fans wake-ups out to per-channel handlers on a separate dispatch thread. Cluster collector subscribes to a new `services` channel and runs node discovery reactively — new-node / graceful-deregister visibility drops from up-to-60 s to ~500 ms on Postgres, with the 60 s discovery loop retained as the backstop for crash-shaped node loss (NOTIFY only fires on real writes). Storage layer gains a uniform `notify` / `listen` API: - PostgreSQL: real `pg_notify` / `LISTEN` on a dedicated session-mode connection that bypasses pgbouncer (mandatory: pgbouncer is required in transaction-pool mode per docs, which is incompatible with LISTEN). - SQLite: in-process fan-out + synthetic-sweep fallback so consumer code is identical across backends. `TURNSTONE_DB_LISTEN_URL` (or `[database] listen_url` in config.toml) points the dispatcher's connection direct-to-Postgres. Defaults to the main DB URL when unset. Migration 053 installs the `services_notify` trigger; it filters heartbeat-only UPDATEs in-trigger so the 30 s × N-nodes heartbeat tick stays quiet, while INSERT, DELETE, and url/metadata-changing UPDATE still fire. Dispatcher detail: - Two threads: listener (drains stream → bounded queue) and dispatch (invokes handlers under exception suppression). Same-channel notifies coalesce per dispatch batch so an N-node deploy burst is one `_discover_nodes` per channel. - Reconnect uses exponential backoff (1 s → 30 s cap). After any successful reopen — whether the prior failure was a stream-poll error or a connect / initial-LISTEN error — one synthetic Notify with payload="reconcile" is enqueued per channel so handlers re-read on the same code path they use for real events. Future consumers (ConfigStore live reload, scheduler immediate dispatch, audit live-tail) plug in by adding their channel to the dispatcher's construction list. Tests: 22 dispatcher tests (incl. reconnect + coalescing under stub storage), 7 SQLite notify-stream tests, 4 PG-gated trigger-filter tests, 4 collector wire-in tests. All pass; ruff + mypy clean. * fix(notify): address Copilot review on #505 - _sqlite.py: SQLiteBackend.listen() now de-dupes channel names via dict.fromkeys before constructing the stream — duplicates would otherwise register the queue twice and double-deliver each notify. - _sqlite.py: SQLiteBackend.listen() gains a keyword-only sweep_interval parameter (defaults to _SQLITE_NOTIFY_SWEEP_INTERVAL) — matches what the comment at the constant already promised, and lets future consumers without their own polling timer pick a tighter cadence without reaching into private stream attributes. - _sqlite.py: documented the `except queue.Empty: pass` end-of-drain termination so it's not mistaken for swallowing an unexpected error. - _postgresql.py: docstring referenced :func:`_pg_listen_url` which was renamed to _resolve_pg_listen_url during PR development. - notify_dispatcher.py: module docstring referenced a non-existent _bootstrap_console_subsystem; wire-in is at console/server.py::main. Refuted (no change, false positives from github-code-quality bot): - 4× "Statement has no effect" on Protocol-method `...` ellipsis bodies (idiomatic Python Protocol declaration, not dead code). - 2× "Mixed import style" in tests — `import ... as nd_mod` is intentional to allow attribute assignment for monkey-patching the module's `_RECONNECT_BACKOFF_INITIAL` constant inside try/finally.