feat(deploy): systemd units for a bare-metal turnstone-server node

Hardened service + slice + node-identity drop-in template + a README for running a turnstone-server outside Docker that joins the compose cluster — the production-shaped counterpart to the one-liner in docs/docker.md. Secrets stay in config.toml; per-host identity + cluster URLs go in the drop-in. The README notes the cross-host mTLS caveat (turnstonelabs/lacme#22).
2026-08-12 23:12:23 -06:00 · 2026-06-15 03:39:34 -07:00
parent 1f61350545
commit 04b3a3abe4
5 changed files with 200 additions and 0 deletions
@@ -0,0 +1,72 @@
+# Running a bare-metal turnstone-server under systemd
+
+These units run a `turnstone-server` **outside** Docker (e.g. on a box with a
+local GPU) so it joins an existing cluster — typically the docker-compose stack
+in [`compose.yaml`](../../compose.yaml). They are the hardened, production-shaped
+counterpart to the quick `turnstone-server …` invocation in
+[`docs/docker.md`](../../docs/docker.md) ("Join a bare-metal host").
+
+| File | Purpose |
+|------|---------|
+| `turnstone-server.service` | The hardened server unit (sandboxed; secrets via `config.toml`). |
+| `turnstone.slice` | Shared memory/process budget for colocated Turnstone units. |
+| `turnstone-server.service.d/node.conf.example` | Per-host identity + cluster URLs drop-in (no secrets). |
+
+## Cluster-side prerequisite
+
+The compose stack must publish Postgres, the console's ACME endpoint, and SearxNG
+on an address the bare-metal host can reach. Start it with `TURNSTONE_HOST_IP`
+set to the compose host's LAN IP (default `127.0.0.1` keeps everything host-local):
+
+```bash
+TURNSTONE_HOST_IP=<compose-host-ip> docker compose up -d
+```
+
+## Install (run as root on the bare-metal host)
+
+```bash
+# 1. A dedicated, unprivileged user.
+useradd --system --no-create-home --shell /usr/sbin/nologin turnstone
+
+# 2. Install turnstone into a venv at /opt/turnstone-venv (lacme/mTLS is a core dep).
+uv venv /opt/turnstone-venv --python 3.12
+uv pip install --python /opt/turnstone-venv 'turnstone @ git+https://github.com/turnstonelabs/turnstone'
+#   …or from a local checkout:  uv pip install --python /opt/turnstone-venv /path/to/turnstone
+
+# 3. Secrets — match the cluster's JWT secret + DB credentials (kept out of env).
+install -d -m 750 -o turnstone -g turnstone /etc/turnstone
+cat > /etc/turnstone/config.toml <<'TOML'
+[auth]
+jwt_secret = "<same secret as the cluster>"
+[database]
+backend = "postgresql"
+url = "postgresql+psycopg://turnstone:<password>@<compose-host-ip>:5432/turnstone"
+[api]
+base_url = "http://localhost:8000/v1"   # a real model backend is configured in the console UI
+api_key = "dummy"
+TOML
+chown turnstone:turnstone /etc/turnstone/config.toml
+chmod 600 /etc/turnstone/config.toml
+
+# 4. Units + per-host drop-in.
+cp turnstone-server.service turnstone.slice /etc/systemd/system/
+install -d /etc/systemd/system/turnstone-server.service.d
+cp turnstone-server.service.d/node.conf.example \
+   /etc/systemd/system/turnstone-server.service.d/node.conf
+$EDITOR /etc/systemd/system/turnstone-server.service.d/node.conf   # set the addresses
+
+# 5. Go.
+systemctl daemon-reload
+systemctl enable --now turnstone-server.service
+journalctl -u turnstone-server -f          # watch it register + (if the cluster runs mTLS) enroll
+```
+
+`tls.enabled` is **not** set here — a joining node inherits it from the cluster's
+shared settings (the database). If the cluster runs mTLS, the node auto-enrolls a
+cert from the console's ACME endpoint and re-advertises itself over `https://`.
+
+> **mTLS + cross-host caveat:** a node on a *different* host than the console
+> currently can't complete ACME enrollment — the console advertises an
+> unroutable in-container address in its ACME directory
+> ([turnstonelabs/lacme#22](https://github.com/turnstonelabs/lacme/issues/22)).
+> Same-host bare-metal nodes, and any node in a non-mTLS cluster, are unaffected.
@@ -0,0 +1,85 @@
+# Run a bare-metal turnstone-server as a systemd service so it joins a cluster
+# (e.g. the docker-compose stack) from outside Docker — typically to use a local
+# GPU. Install steps + the cluster-side prerequisites are in deploy/systemd/README.md
+# and docs/docker.md ("Join a bare-metal host"). Per-host identity + the cluster
+# URLs go in a drop-in (see node.conf.example); secrets go in config.toml.
+[Unit]
+Description=Turnstone server (chat workstreams + LLM gateway)
+Documentation=https://github.com/turnstonelabs/turnstone
+# Postgres is required. After= orders against a colocated postgresql.service
+# when present and silently no-ops otherwise (the cluster DB is usually remote).
+After=network.target postgresql.service
+StartLimitIntervalSec=60
+StartLimitBurst=5
+
+[Service]
+Type=exec
+User=turnstone
+Group=turnstone
+
+# Secrets live in config.toml — JWT secret, Postgres URL+password, LLM API key —
+# kept out of os.environ so a prompt-injected tool can't dump them via `env`.
+Environment=TURNSTONE_CONFIG=/etc/turnstone/config.toml
+Environment=TURNSTONE_LOG_LEVEL=info
+
+Slice=turnstone.slice
+
+# Per-host node identity + cluster wiring (TURNSTONE_NODE_ID / _ADVERTISE_URL /
+# _CONSOLE_URL / _SEARXNG_URL) go in a drop-in, not here — see node.conf.example.
+
+StateDirectory=turnstone
+StateDirectoryMode=0750
+LogsDirectory=turnstone
+LogsDirectoryMode=0750
+WorkingDirectory=/var/lib/turnstone
+
+# --host 0.0.0.0 so the console collector + peer nodes can dial this node back
+# at its advertised address. (A single-node, Caddy-fronted install can use
+# 127.0.0.1 instead.) Rewrite --port if :8080 is already taken on the host.
+ExecStart=/opt/turnstone-venv/bin/turnstone-server --host 0.0.0.0 --port 8080
+
+Restart=on-failure
+RestartSec=5s
+TimeoutStartSec=120
+TimeoutStopSec=30
+KillSignal=SIGTERM
+KillMode=mixed
+
+# --- Resource limits ---
+# SSE keeps an fd per active workstream + outbound LLM stream + MCP stdio pipe.
+LimitNOFILE=65535
+LimitNPROC=8192
+TasksMax=8192
+LimitCORE=0
+
+# --- Hardening ---
+NoNewPrivileges=true
+CapabilityBoundingSet=
+AmbientCapabilities=
+UMask=0027
+PrivateTmp=true
+# PrivateDevices=true  — disabled: GPU access via /sys/class/drm
+ProtectSystem=strict
+ProtectHome=true
+ProtectKernelTunables=true
+ProtectKernelModules=true
+ProtectKernelLogs=true
+ProtectControlGroups=true
+ProtectClock=true
+ProtectHostname=true
+RestrictNamespaces=true
+RestrictRealtime=true
+RestrictSUIDSGID=true
+RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6
+LockPersonality=true
+MemoryDenyWriteExecute=true
+SystemCallArchitectures=native
+SystemCallFilter=@system-service
+SystemCallFilter=~@privileged @mount
+
+StandardOutput=journal
+StandardError=journal
+SyslogIdentifier=turnstone-server
+
+[Install]
+WantedBy=multi-user.target
@@ -0,0 +1,25 @@
+# Per-host node identity + cluster wiring for a bare-metal turnstone-server.
+# Copy to /etc/systemd/system/turnstone-server.service.d/node.conf and edit the
+# addresses, then `systemctl daemon-reload`. Identity + URLs are NOT secrets, so
+# they live here; the JWT secret + DB URL live in /etc/turnstone/config.toml.
+#
+# Addresses below use RFC 5737 documentation IPs — replace them:
+#   <this-host>     = the bare-metal host's own LAN IP (what the console dials back)
+#   <compose-host>  = the host running the cluster / docker-compose stack, started
+#                     with TURNSTONE_HOST_IP=<compose-host> so :8090 and :8081 are
+#                     published on its LAN interface (see docs/docker.md).
+[Service]
+# Unique node id (defaults to the hostname if unset).
+Environment=TURNSTONE_NODE_ID=host-1
+
+# The address peers + the console collector dial back. Auto-upgrades to https://
+# once the node enrolls its mTLS cert.
+Environment=TURNSTONE_ADVERTISE_URL=http://192.0.2.10:8080
+
+# The cluster console's reachable plain-HTTP ACME/API endpoint. A bare-metal node
+# can't resolve the in-cluster name (console:8090), so point it at the published
+# port; turnstone-server honors this for cert enrollment.
+Environment=TURNSTONE_CONSOLE_URL=http://192.0.2.1:8090
+
+# The cluster's published SearxNG, for the web_search tool.
+Environment=TURNSTONE_SEARXNG_URL=http://192.0.2.1:8081
@@ -0,0 +1,15 @@
+# Shared resource budget for the colocated Turnstone units. Without a slice each
+# unit's MemoryMax= is enforced independently — three units at 85% each can sum
+# to 255% of host RAM before any throttles. Under a shared slice the cap is
+# hierarchical: the slice ceiling is the real limit. (A bare-metal node that runs
+# only turnstone-server still benefits — and keeps the unit's Slice= reference
+# valid.) Adjust if the host runs other meaningful workloads alongside Turnstone.
+[Unit]
+Description=Turnstone services slice (server + console + channel)
+Documentation=https://github.com/turnstonelabs/turnstone
+Before=slices.target
+
+[Slice]
+MemoryHigh=70%
+MemoryMax=85%
+TasksMax=16384
@@ -110,6 +110,9 @@ box's own IP (the address the console dials back). **Set a strong
 user account + API-token hash in it), the console API, and the unauthenticated
 SearxNG to your network.

+To run the bare-metal node as a hardened, persistent service instead of by hand,
+use the systemd units in [`deploy/systemd/`](../deploy/systemd/).
+
 ## Production stack

 For a real deployment use the bundled stack, which pulls released images