diff --git a/deploy/systemd/README.md b/deploy/systemd/README.md new file mode 100644 index 00000000..6720c456 --- /dev/null +++ b/deploy/systemd/README.md @@ -0,0 +1,72 @@ +# Running a bare-metal turnstone-server under systemd + +These units run a `turnstone-server` **outside** Docker (e.g. on a box with a +local GPU) so it joins an existing cluster — typically the docker-compose stack +in [`compose.yaml`](../../compose.yaml). They are the hardened, production-shaped +counterpart to the quick `turnstone-server …` invocation in +[`docs/docker.md`](../../docs/docker.md) ("Join a bare-metal host"). + +| File | Purpose | +|------|---------| +| `turnstone-server.service` | The hardened server unit (sandboxed; secrets via `config.toml`). | +| `turnstone.slice` | Shared memory/process budget for colocated Turnstone units. | +| `turnstone-server.service.d/node.conf.example` | Per-host identity + cluster URLs drop-in (no secrets). | + +## Cluster-side prerequisite + +The compose stack must publish Postgres, the console's ACME endpoint, and SearxNG +on an address the bare-metal host can reach. Start it with `TURNSTONE_HOST_IP` +set to the compose host's LAN IP (default `127.0.0.1` keeps everything host-local): + +```bash +TURNSTONE_HOST_IP= docker compose up -d +``` + +## Install (run as root on the bare-metal host) + +```bash +# 1. A dedicated, unprivileged user. +useradd --system --no-create-home --shell /usr/sbin/nologin turnstone + +# 2. Install turnstone into a venv at /opt/turnstone-venv (lacme/mTLS is a core dep). +uv venv /opt/turnstone-venv --python 3.12 +uv pip install --python /opt/turnstone-venv 'turnstone @ git+https://github.com/turnstonelabs/turnstone' +# …or from a local checkout: uv pip install --python /opt/turnstone-venv /path/to/turnstone + +# 3. Secrets — match the cluster's JWT secret + DB credentials (kept out of env). +install -d -m 750 -o turnstone -g turnstone /etc/turnstone +cat > /etc/turnstone/config.toml <<'TOML' +[auth] +jwt_secret = "" +[database] +backend = "postgresql" +url = "postgresql+psycopg://turnstone:@:5432/turnstone" +[api] +base_url = "http://localhost:8000/v1" # a real model backend is configured in the console UI +api_key = "dummy" +TOML +chown turnstone:turnstone /etc/turnstone/config.toml +chmod 600 /etc/turnstone/config.toml + +# 4. Units + per-host drop-in. +cp turnstone-server.service turnstone.slice /etc/systemd/system/ +install -d /etc/systemd/system/turnstone-server.service.d +cp turnstone-server.service.d/node.conf.example \ + /etc/systemd/system/turnstone-server.service.d/node.conf +$EDITOR /etc/systemd/system/turnstone-server.service.d/node.conf # set the addresses + +# 5. Go. +systemctl daemon-reload +systemctl enable --now turnstone-server.service +journalctl -u turnstone-server -f # watch it register + (if the cluster runs mTLS) enroll +``` + +`tls.enabled` is **not** set here — a joining node inherits it from the cluster's +shared settings (the database). If the cluster runs mTLS, the node auto-enrolls a +cert from the console's ACME endpoint and re-advertises itself over `https://`. + +> **mTLS + cross-host caveat:** a node on a *different* host than the console +> currently can't complete ACME enrollment — the console advertises an +> unroutable in-container address in its ACME directory +> ([turnstonelabs/lacme#22](https://github.com/turnstonelabs/lacme/issues/22)). +> Same-host bare-metal nodes, and any node in a non-mTLS cluster, are unaffected. diff --git a/deploy/systemd/turnstone-server.service b/deploy/systemd/turnstone-server.service new file mode 100644 index 00000000..779477e5 --- /dev/null +++ b/deploy/systemd/turnstone-server.service @@ -0,0 +1,85 @@ +# Run a bare-metal turnstone-server as a systemd service so it joins a cluster +# (e.g. the docker-compose stack) from outside Docker — typically to use a local +# GPU. Install steps + the cluster-side prerequisites are in deploy/systemd/README.md +# and docs/docker.md ("Join a bare-metal host"). Per-host identity + the cluster +# URLs go in a drop-in (see node.conf.example); secrets go in config.toml. +[Unit] +Description=Turnstone server (chat workstreams + LLM gateway) +Documentation=https://github.com/turnstonelabs/turnstone +# Postgres is required. After= orders against a colocated postgresql.service +# when present and silently no-ops otherwise (the cluster DB is usually remote). +After=network.target postgresql.service +StartLimitIntervalSec=60 +StartLimitBurst=5 + +[Service] +Type=exec +User=turnstone +Group=turnstone + +# Secrets live in config.toml — JWT secret, Postgres URL+password, LLM API key — +# kept out of os.environ so a prompt-injected tool can't dump them via `env`. +Environment=TURNSTONE_CONFIG=/etc/turnstone/config.toml +Environment=TURNSTONE_LOG_LEVEL=info + +Slice=turnstone.slice + +# Per-host node identity + cluster wiring (TURNSTONE_NODE_ID / _ADVERTISE_URL / +# _CONSOLE_URL / _SEARXNG_URL) go in a drop-in, not here — see node.conf.example. + +StateDirectory=turnstone +StateDirectoryMode=0750 +LogsDirectory=turnstone +LogsDirectoryMode=0750 +WorkingDirectory=/var/lib/turnstone + +# --host 0.0.0.0 so the console collector + peer nodes can dial this node back +# at its advertised address. (A single-node, Caddy-fronted install can use +# 127.0.0.1 instead.) Rewrite --port if :8080 is already taken on the host. +ExecStart=/opt/turnstone-venv/bin/turnstone-server --host 0.0.0.0 --port 8080 + +Restart=on-failure +RestartSec=5s +TimeoutStartSec=120 +TimeoutStopSec=30 +KillSignal=SIGTERM +KillMode=mixed + +# --- Resource limits --- +# SSE keeps an fd per active workstream + outbound LLM stream + MCP stdio pipe. +LimitNOFILE=65535 +LimitNPROC=8192 +TasksMax=8192 +LimitCORE=0 + +# --- Hardening --- +NoNewPrivileges=true +CapabilityBoundingSet= +AmbientCapabilities= +UMask=0027 +PrivateTmp=true +# PrivateDevices=true — disabled: GPU access via /sys/class/drm +ProtectSystem=strict +ProtectHome=true +ProtectKernelTunables=true +ProtectKernelModules=true +ProtectKernelLogs=true +ProtectControlGroups=true +ProtectClock=true +ProtectHostname=true +RestrictNamespaces=true +RestrictRealtime=true +RestrictSUIDSGID=true +RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6 +LockPersonality=true +MemoryDenyWriteExecute=true +SystemCallArchitectures=native +SystemCallFilter=@system-service +SystemCallFilter=~@privileged @mount + +StandardOutput=journal +StandardError=journal +SyslogIdentifier=turnstone-server + +[Install] +WantedBy=multi-user.target diff --git a/deploy/systemd/turnstone-server.service.d/node.conf.example b/deploy/systemd/turnstone-server.service.d/node.conf.example new file mode 100644 index 00000000..5170d318 --- /dev/null +++ b/deploy/systemd/turnstone-server.service.d/node.conf.example @@ -0,0 +1,25 @@ +# Per-host node identity + cluster wiring for a bare-metal turnstone-server. +# Copy to /etc/systemd/system/turnstone-server.service.d/node.conf and edit the +# addresses, then `systemctl daemon-reload`. Identity + URLs are NOT secrets, so +# they live here; the JWT secret + DB URL live in /etc/turnstone/config.toml. +# +# Addresses below use RFC 5737 documentation IPs — replace them: +# = the bare-metal host's own LAN IP (what the console dials back) +# = the host running the cluster / docker-compose stack, started +# with TURNSTONE_HOST_IP= so :8090 and :8081 are +# published on its LAN interface (see docs/docker.md). +[Service] +# Unique node id (defaults to the hostname if unset). +Environment=TURNSTONE_NODE_ID=host-1 + +# The address peers + the console collector dial back. Auto-upgrades to https:// +# once the node enrolls its mTLS cert. +Environment=TURNSTONE_ADVERTISE_URL=http://192.0.2.10:8080 + +# The cluster console's reachable plain-HTTP ACME/API endpoint. A bare-metal node +# can't resolve the in-cluster name (console:8090), so point it at the published +# port; turnstone-server honors this for cert enrollment. +Environment=TURNSTONE_CONSOLE_URL=http://192.0.2.1:8090 + +# The cluster's published SearxNG, for the web_search tool. +Environment=TURNSTONE_SEARXNG_URL=http://192.0.2.1:8081 diff --git a/deploy/systemd/turnstone.slice b/deploy/systemd/turnstone.slice new file mode 100644 index 00000000..cc018015 --- /dev/null +++ b/deploy/systemd/turnstone.slice @@ -0,0 +1,15 @@ +# Shared resource budget for the colocated Turnstone units. Without a slice each +# unit's MemoryMax= is enforced independently — three units at 85% each can sum +# to 255% of host RAM before any throttles. Under a shared slice the cap is +# hierarchical: the slice ceiling is the real limit. (A bare-metal node that runs +# only turnstone-server still benefits — and keeps the unit's Slice= reference +# valid.) Adjust if the host runs other meaningful workloads alongside Turnstone. +[Unit] +Description=Turnstone services slice (server + console + channel) +Documentation=https://github.com/turnstonelabs/turnstone +Before=slices.target + +[Slice] +MemoryHigh=70% +MemoryMax=85% +TasksMax=16384 diff --git a/docs/docker.md b/docs/docker.md index fb5fe269..fcf42ac7 100644 --- a/docs/docker.md +++ b/docs/docker.md @@ -110,6 +110,9 @@ box's own IP (the address the console dials back). **Set a strong user account + API-token hash in it), the console API, and the unauthenticated SearxNG to your network. +To run the bare-metal node as a hardened, persistent service instead of by hand, +use the systemd units in [`deploy/systemd/`](../deploy/systemd/). + ## Production stack For a real deployment use the bundled stack, which pulls released images