feat(compose): let bare-metal turnstone-servers join the cluster (incl. mTLS)

A turnstone-server running outside the compose network ("bare-metal", e.g. a
local-GPU box) couldn't fully join: it can't resolve the in-cluster console
(console:8090) to enroll its mTLS cert, and SearxNG was unreachable for
web_search. Only Postgres was published.

Publish the console's plain-HTTP ACME endpoint (:8090) and SearxNG (:8081)
alongside Postgres, all bound via one knob TURNSTONE_HOST_IP (default 127.0.0.1
-- nothing new on the LAN; set it to the host's LAN IP for a node on another
machine). Postgres keeps honoring the legacy POSTGRES_BIND as a fallback, so
existing .env files don't break.

The node's TLS client now honors TURNSTONE_CONSOLE_URL so a bare-metal node can
point at the published ACME endpoint instead of the unreachable in-cluster name
(empty = in-cluster service discovery, unchanged).

Docs (docker.md, tls.md), the run.sh-generated .env, and the bootstrap wizard
updated to match. The advertised host is the cert's primary SAN and the console
collector dials it back, so mTLS hostname verification holds both ways.

compose.yaml

@@ -29,10 +29,11 @@
 # Fewer nodes (lighter machines):
 #     docker compose up postgres console caddy channel node-1 node-2 node-3
 #
-# Join a bare-metal host: Postgres is published on 127.0.0.1:5432, so a
+# Join a bare-metal host: a turnstone-server running OUTSIDE compose (e.g. to use
-# turnstone-server running directly on this machine (e.g. to use a local GPU)
+# a local GPU) can join this cluster. Postgres, the console's ACME endpoint, and
-# can join the same cluster. Keep the secret + connection settings in
+# SearxNG are published on 127.0.0.1 so a node on THIS machine reaches them via
-# ~/.config/turnstone/config.toml (chmod 0600 — the loader warns otherwise):
+# localhost. Keep secrets in ~/.config/turnstone/config.toml (chmod 0600 — the
+# loader warns otherwise):
 #     [auth]
 #     jwt_secret = "dev-only-insecure-jwt-secret-change-me-for-real-deployments"
 #     [database]
@@ -41,10 +42,19 @@
 #     [api]
 #     base_url = "http://localhost:8000/v1"
 #     api_key = "dummy"
+#     [tls]              # only if the cluster runs mTLS
+#     enabled = true
 # then run (node identity isn't a secret, so it stays on the command line):
-#     TURNSTONE_NODE_ID=host-1 TURNSTONE_ADVERTISE_URL=http://host.docker.internal:8080 \
+#     TURNSTONE_NODE_ID=host-1 \
+#       TURNSTONE_ADVERTISE_URL=http://host.docker.internal:8080 \
+#       TURNSTONE_CONSOLE_URL=http://localhost:8090 \
+#       TURNSTONE_SEARXNG_URL=http://localhost:8081 \
 #       turnstone-server --host 0.0.0.0 --port 8080
-# It registers in Postgres and the console reaches it back via host.docker.internal.
+# The node registers in Postgres, auto-enrolls its mTLS cert from the console's
+# ACME endpoint (when the cluster runs mTLS), and the console collector reaches
+# it back via host.docker.internal. To join from ANOTHER machine, set
+# TURNSTONE_HOST_IP to this host's LAN IP and use it in the URLs above (and the
+# node's TURNSTONE_ADVERTISE_URL = the NODE host's IP) — see docs/docker.md.
 # =============================================================================
 
 name: turnstone
@@ -93,13 +103,14 @@ services:
       # INSECURE dev default — override POSTGRES_PASSWORD in .env for real use.
       POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:-turnstone}
       PGDATA: /var/lib/postgresql/data
-    # Published on localhost so a bare-metal turnstone-server running on THIS
+    # Published so a bare-metal turnstone-server can join the cluster (see "Join
-    # host can join the cluster (see "Join a bare-metal host" in the header).
+    # a bare-metal host" in the header). Bound to 127.0.0.1 by default (same-host
-    # Bound to 127.0.0.1 by default; set POSTGRES_BIND=0.0.0.0 to let another
+    # nodes only); set TURNSTONE_HOST_IP to this host's LAN IP to let another
-    # machine connect — but set a real POSTGRES_PASSWORD first, or you'll expose
+    # machine connect — but set a real POSTGRES_PASSWORD first, or you'll expose a
-    # a database with the insecure default password to your network.
+    # database with the insecure default password to your network. (The legacy
+    # POSTGRES_BIND is still honored as a fallback when TURNSTONE_HOST_IP is unset.)
     ports:
-      - "${POSTGRES_BIND:-127.0.0.1}:${POSTGRES_PORT:-5432}:5432"
+      - "${TURNSTONE_HOST_IP:-${POSTGRES_BIND:-127.0.0.1}}:${POSTGRES_PORT:-5432}:5432"
     volumes:
       - postgres-data:/var/lib/postgresql/data
     networks:
@@ -120,10 +131,12 @@ services:
   # turnstone-console — cluster dashboard. Reach it ONLY through Caddy at
   # https://localhost:8443 (see the caddy service below).
   #
-  # The console port (8090) is deliberately NOT published to the host: a plain
+  # Browsers must reach the dashboard through Caddy (https://localhost:8443): a
-  # HTTP/1.1 origin caps the browser at 6 connections, which starves the
+  # plain HTTP/1.1 origin caps the browser at 6 connections, which starves the
-  # dashboard's per-pane SSE streams. Caddy serves the browser over HTTP/2
+  # dashboard's per-pane SSE streams, whereas Caddy serves HTTP/2 (multiplexed)
-  # (multiplexed) and proxies to console:8090 internally, so the cap is gone.
+  # and proxies to console:8090 internally. The console's :8090 is published
+  # below ONLY so bare-metal nodes can reach the plain-HTTP ACME enrollment
+  # endpoint — don't point a browser at it.
   #
   # The single `build:` here produces the turnstone:local image every other
   # service reuses. extra_hosts lets the console reach a bare-metal server
@@ -138,6 +151,14 @@ services:
       - turnstone-console
       - --host=0.0.0.0
       - --port=8090
+    # Publishes the console's plain-HTTP listener so a bare-metal node can reach
+    # the ACME endpoint, fetch the CA, and enroll its cert (the console serves
+    # HTTP here even under mTLS). Bound to 127.0.0.1 by default; setting
+    # TURNSTONE_HOST_IP exposes the WHOLE console HTTP API — including the
+    # cert-issuing ACME endpoint — on that interface, so the JWT secret's
+    # strength is the only gate. Browsers use Caddy :8443, never this port.
+    ports:
+      - "${TURNSTONE_HOST_IP:-127.0.0.1}:8090:8090"
     environment:
       TURNSTONE_JWT_SECRET: *jwt-secret
       TURNSTONE_DB_BACKEND: *db-backend
@@ -219,6 +240,13 @@ services:
   # -------------------------------------------------------------------
   searxng:
     image: searxng/searxng:${SEARXNG_IMAGE_TAG:-latest}
+    # Published so a bare-metal node's web_search can reach it. SearxNG has NO
+    # auth, so it is bound to 127.0.0.1 by default; setting TURNSTONE_HOST_IP
+    # exposes it on that interface — an open search proxy on your LAN, which also
+    # triggers the SearxNG AGPL-3.0 §13 source-offer obligation (see docs/docker.md).
+    # In-compose nodes always use the internal http://searxng:8080 and ignore this.
+    ports:
+      - "${TURNSTONE_HOST_IP:-127.0.0.1}:${SEARXNG_API_PORT:-8081}:8080"
     volumes:
       - ./turnstone/deploy/searxng:/etc/searxng:ro
       - searxng-cache:/var/cache/searxng # favicon + internal SQLite cache (survives restarts)

docs/docker.md

@@ -59,9 +59,11 @@ is gone. Everything goes through `https://localhost:8443`.
 
 ## Join a bare-metal host
 
-PostgreSQL is published on `127.0.0.1:5432`, so a `turnstone-server` running
+PostgreSQL, the console's ACME endpoint (`:8090`), and SearxNG (`:8081`) are
-directly on the same machine — for example to use a local GPU — can join the
+published on `127.0.0.1`, so a `turnstone-server` running directly on the same
-same cluster and show up in the console alongside the containerized nodes.
+machine — for example to use a local GPU — can join the same cluster (enrolling
+its mTLS cert and running `web_search`) and show up in the console alongside the
+containerized nodes.
 
 Put the secret and connection settings in `~/.config/turnstone/config.toml`
 (secrets belong in this file, not the process environment — keep it `0600`,
@@ -85,18 +87,28 @@ command line:
 
 ```bash
 chmod 600 ~/.config/turnstone/config.toml
-TURNSTONE_NODE_ID=host-1 TURNSTONE_ADVERTISE_URL=http://host.docker.internal:8080 \
+TURNSTONE_NODE_ID=host-1 \
+  TURNSTONE_ADVERTISE_URL=http://host.docker.internal:8080 \
+  TURNSTONE_CONSOLE_URL=http://localhost:8090 \
+  TURNSTONE_SEARXNG_URL=http://localhost:8081 \
   turnstone-server --host 0.0.0.0 --port 8080
 ```
 
 The host server registers itself in PostgreSQL; the console reaches it back via
-`host.docker.internal`. The `jwt_secret` and DB credentials above are the
+`host.docker.internal`. `TURNSTONE_CONSOLE_URL` points the node at the console's
-dev-stack defaults — match whatever you set in `.env` if you changed them. To
+published ACME endpoint so it can enroll its mTLS certificate (needed only when
-let a **different** machine join, start the stack with `POSTGRES_BIND=0.0.0.0`
+the cluster runs mTLS; harmless otherwise), and `TURNSTONE_SEARXNG_URL` points
-and use the host's routable IP in the `url` and `TURNSTONE_ADVERTISE_URL` —
+`web_search` at the published SearxNG. The `jwt_secret` and DB credentials above
-but **set a strong `POSTGRES_PASSWORD` first**, or you'll expose a database with
+are the dev-stack defaults — match whatever you set in `.env` if you changed them.
-the insecure default password (and every user account + API-token hash in it) to
+
-your network.
+To let a server on a **different** machine join, start the stack with
+`TURNSTONE_HOST_IP=<this host's LAN IP>` — that binds PostgreSQL, the console
+ACME endpoint, and SearxNG to that interface. Then on the remote box set the
+three URLs above to that IP, and set `TURNSTONE_ADVERTISE_URL` to the **remote**
+box's own IP (the address the console dials back). **Set a strong
+`POSTGRES_PASSWORD` first** — `TURNSTONE_HOST_IP` exposes the database (and every
+user account + API-token hash in it), the console API, and the unauthenticated
+SearxNG to your network.
 
 ## Production stack
 
@@ -174,15 +186,18 @@ overrides.
 ### Ports
 
 Both stacks publish Caddy (dashboard) and PostgreSQL; the dev stack additionally
-publishes the SearxNG UI on localhost. Everything else is reached through Caddy or
+publishes the console's ACME endpoint and SearxNG on localhost so a bare-metal
-proxied by the console:
+node can enroll its cert and run `web_search`. Everything else is reached through
+Caddy or proxied by the console:
 
 | Variable | Default | Description |
 |----------|---------|-------------|
 | `CONSOLE_HTTPS_PORT` | `8443` | Host port for Caddy (dashboard HTTPS) |
 | `SEARXNG_HTTPS_PORT` | `8444` | Host port for the SearxNG UI via Caddy (dev: localhost-only; prod: opt-in) |
 | `POSTGRES_PORT` | `5432` | Host port for PostgreSQL (for bare-metal joins) |
-| `POSTGRES_BIND` | `127.0.0.1` | Interface PostgreSQL binds on; set `0.0.0.0` for LAN access |
+| `SEARXNG_API_PORT` | `8081` | Host port for the SearxNG API a bare-metal node's `web_search` dials (dev stack) |
+| `TURNSTONE_HOST_IP` | `127.0.0.1` | Interface PostgreSQL, the console ACME endpoint, and SearxNG bind on (dev stack). Set to this host's LAN IP so a bare-metal node on **another machine** can reach them — set a strong `POSTGRES_PASSWORD` first (it also exposes the DB and the unauthenticated SearxNG to your network). |
+| `POSTGRES_BIND` | `127.0.0.1` | Production stack (`turnstone/deploy/compose.yaml`) only: interface PostgreSQL binds on; set to the host's LAN IP for remote joins. |
 
 ### Channel gateway
 

docs/tls.md

@@ -244,7 +244,10 @@ const client = new TurnstoneServer({
 ### Node Bootstrap Flow
 
 1. Node starts, connects to shared database (plain connection)
-2. Discovers console URL from `services` table
+2. Discovers the console URL from the `services` table — or honors an explicit
+   `TURNSTONE_CONSOLE_URL` (a bare-metal node outside the compose network can't
+   resolve the in-cluster `console` name, so it points this at the console's
+   published ACME endpoint)
 3. Fetches CA root cert from `http://console/acme/ca.pem` (plain HTTP, TOFU)
 4. Requests a service cert via ACME (plain HTTP, JWS-signed). The cert's
    primary domain / SAN is the node's **advertised host** (the host of
@@ -291,7 +294,8 @@ cert's SANs don't include the dialed name.
 
 The console registers itself in the `services` table on startup. If the console
 hasn't started or the registration expired (1 hour TTL), nodes can't discover
-it. Use `--console-url` explicitly.
+it. Set `TURNSTONE_CONSOLE_URL` to a reachable console address (this is also how
+a bare-metal node that can't resolve the in-cluster `console` name enrolls).
 
 ### Browser HTTPS to the console
 

run.sh

@@ -333,7 +333,9 @@ prepare_env() {
 TURNSTONE_JWT_SECRET=$jwt
 POSTGRES_USER=turnstone
 POSTGRES_PASSWORD=$pgpw
-POSTGRES_BIND=127.0.0.1
+# Bind published bare-metal ports (Postgres, console ACME, SearxNG) to this
+# interface. 127.0.0.1 = same-host only; set your LAN IP to join from another box.
+TURNSTONE_HOST_IP=127.0.0.1
 POSTGRES_PORT=$PG_PORT
 CONSOLE_HTTPS_PORT=$CADDY_PORT
 EOF

turnstone/bootstrap.py

@@ -106,7 +106,10 @@ Generate with: `python -c "import secrets; print(secrets.token_hex(32))"`
 ### Ports / networking
 - `CONSOLE_HTTPS_PORT` — Caddy HTTPS port for the dashboard (default: 8443)
 - `POSTGRES_PORT` — PostgreSQL host port, for joining a bare-metal server (default: 5432)
-- `POSTGRES_BIND` — interface PostgreSQL binds on (default: 127.0.0.1; set 0.0.0.0 for LAN)
+- `TURNSTONE_HOST_IP` — interface the published bare-metal ports (Postgres, console
+  ACME, SearxNG) bind on (default: 127.0.0.1; set your host's LAN IP to join from
+  another machine). The legacy `POSTGRES_BIND` is still honored for Postgres.
+- `SEARXNG_API_PORT` — host port a bare-metal node's web_search dials SearxNG on (default: 8081)
 
 ### Channel Gateway (optional)
 - `TURNSTONE_DISCORD_TOKEN` — Discord bot token

turnstone/server.py

@@ -4667,6 +4667,11 @@ def main() -> None:
             tls_client = TLSClient(
                 storage=get_storage(),
                 hostnames=hostnames,
+                # A bare-metal node can't resolve the in-network console URL the
+                # services table advertises (http://console:8090), so honor an
+                # explicit override pointing at the published ACME endpoint.
+                # Empty (the in-cluster default) falls back to service discovery.
+                console_url=os.environ.get("TURNSTONE_CONSOLE_URL", ""),
             )
             asyncio.run(tls_client.init(attempts=TLS_INIT_RETRY_ATTEMPTS))
             bundle = tls_client.bundle