From 2c3e537cb861ab4a8b1d42008eda47fc8913d50e Mon Sep 17 00:00:00 2001 From: Peter Steinberger Date: Wed, 12 Aug 2026 20:49:54 -0700 Subject: [PATCH] fix(deploy): repair probes that silently pass and a blueprint that cannot boot (#122963) Three defects proven by running the shipped configs: Kubernetes probes checked only the status code against /startupz, but the pinned image predates that route and the Control UI answers unknown paths with a catch-all 200. A wedged pod was therefore marked Ready forever. Probes now assert the JSON probe contract and target routes the pinned image actually serves; verified in a kind cluster where the old command exits 0 on the missing route and the new one exits 1. render.yaml set no dockerCommand, so the image CMD ran without --allow-unconfigured and a fresh Render disk exited 78 with 'Missing config' before binding. Reproduced locally with Render's exact env. The Cloudflare Container readiness poll had the same route mismatch against operator-supplied official image digests; it now polls /healthz, which every published image serves. Also replaces an R2 verification step that could never fail: wrangler cannot list object keys, so the documented command 404'd into || true. --- docs/install/cloudflare.md | 18 +++++++++++------- docs/install/kubernetes.md | 4 +++- docs/install/render.mdx | 3 ++- render.yaml | 4 ++++ scripts/cloudflare/README.md | 7 ++++--- scripts/cloudflare/src/container.ts | 9 ++++++++- scripts/k8s/manifests/deployment.yaml | 13 ++++++++++--- 7 files changed, 42 insertions(+), 16 deletions(-) diff --git a/docs/install/cloudflare.md b/docs/install/cloudflare.md index 406b19ab25ba..855756261d1d 100644 --- a/docs/install/cloudflare.md +++ b/docs/install/cloudflare.md @@ -25,7 +25,7 @@ The template lives in [`scripts/cloudflare`](https://github.com/openclaw/opencla ## How it works -The Worker forwards every HTTP and WebSocket request to one stable Durable Object name. That Durable Object owns one Container instance and is the single-writer fence around the Litestream replica. The Container exposes OpenClaw on port `8080`; `/startupz` is its traffic-readiness check. +The Worker forwards every HTTP and WebSocket request to one stable Durable Object name. That Durable Object owns one Container instance and is the single-writer fence around the Litestream replica. The Container exposes OpenClaw on port `8080`, and the Durable Object polls `/healthz` there before routing to it. ```mermaid flowchart TD @@ -179,21 +179,23 @@ Measured on this template against a real R2 bucket: about 2.4 seconds from write Run these checks after the first bootstrap, before you depend on this deployment. -Confirm the Gateway admits traffic. `/startupz` reports startup completion and ignores channel health, so it stays green when one channel account is broken: +Confirm the Gateway answers. `/healthz` reports that the listener is up. `/startupz` additionally reports that startup work finished while ignoring channel health, so it stays green when one channel account is broken; it is served only by images built from the release that added it: ```bash -curl -sS https://.workers.dev/startupz +curl -sS https://.workers.dev/healthz curl -sS -H "Authorization: Bearer $OPENCLAW_GATEWAY_TOKEN" \ https://.workers.dev/readyz ``` -Confirm replication is actually reaching R2. Objects should appear under `replicas/` within seconds of activity: +Confirm replication is actually reaching R2. Litestream writes keys under `replicas/state///`, so list that prefix with any S3-compatible client using the same R2 credentials you gave Litestream. Wrangler cannot list object keys, only fetch them by exact path: ```bash -npx wrangler r2 object get openclaw-backups/replicas --remote 2>/dev/null || true -npx wrangler r2 bucket list +aws s3 ls "s3://openclaw-backups/replicas/" --recursive \ + --endpoint-url "https://.r2.cloudflarestorage.com" ``` +The Cloudflare dashboard's R2 object browser shows the same tree. An empty prefix after several minutes of activity means replication is not working; fix it before continuing. + Rehearse recovery before you need it. An untested restore path is not a backup: 1. Send one message so the Gateway writes a session row. @@ -267,7 +269,9 @@ Test updates and rollbacks against a separate R2 bucket first. Preserve current **Worker returns 5xx and the Container never becomes ready** -- Cloudflare only runs `linux/amd64` images pulled from a public registry. Rebuild with `--platform linux/amd64`, confirm the derived Docker Hub repository is public, and confirm `containers[].image` uses the pushed digest rather than a moving tag. -**Deployment succeeds but every request times out** -- The Container helper waits for `GET /startupz`. Check that the Gateway inside the Container listens on port `8080` and that no bootstrap step changed the port. +**Deployment succeeds but every request times out** -- The Container helper waits for `GET /healthz`. Check that the Gateway inside the Container listens on port `8080` and that no bootstrap step changed the port. + +**A probe passes but the Gateway is not actually serving** -- The Control UI answers unknown paths with a catch-all `200`, so probing a route your image does not serve looks permanently healthy. Verify the response body is JSON, not HTML, before trusting a probe. **Litestream logs authentication or signature errors** -- Litestream needs R2 S3 API credentials, which are not the same as a Cloudflare API token. Create an R2 API token and use its access key ID and secret access key, and confirm `LITESTREAM_ENDPOINT` contains your account ID. diff --git a/docs/install/kubernetes.md b/docs/install/kubernetes.md index ab9023e33b6e..ca00f182b867 100644 --- a/docs/install/kubernetes.md +++ b/docs/install/kubernetes.md @@ -90,7 +90,9 @@ Namespace: openclaw (configurable via OPENCLAW_NAMESPACE) └── Secret/openclaw-secrets # Gateway token + API keys ``` -The Deployment uses `/startupz` for both startup and traffic-readiness probes, with a five-minute startup budget. Channel failures do not evict a healthy Gateway or Control UI from Service endpoints. `/healthz` remains the liveness probe; use `/readyz` separately when monitoring should include channel-account health. +The Deployment probes `/readyz` for startup and traffic readiness with a five-minute startup budget, and `/healthz` for liveness. Every probe asserts the JSON probe contract rather than the status code alone, because the Control UI answers unknown paths with a catch-all `200`; a status-only check would pass forever against an image whose probe route does not exist yet. + +`/startupz` is the better traffic-admission probe because it ignores channel health, so one failing channel account cannot evict an otherwise healthy Gateway from Service endpoints. It requires an image built from the release that introduced it, which is newer than the tag pinned above. After pinning such an image, switch the startup and readiness probes to `/startupz` and keep `/readyz` for monitoring that should include channel-account health. ## Customization diff --git a/docs/install/render.mdx b/docs/install/render.mdx index be2d8be21695..8e9ddf3ca7b2 100644 --- a/docs/install/render.mdx +++ b/docs/install/render.mdx @@ -27,6 +27,7 @@ services: name: openclaw runtime: docker plan: starter + dockerCommand: node openclaw.mjs gateway --allow-unconfigured healthCheckPath: /startupz envVars: - key: OPENCLAW_GATEWAY_PORT @@ -58,7 +59,7 @@ services: | Starter | Never | 1GB+ | Personal use, small teams | | Standard+ | Never | 1GB+ | Production, multiple channels | -The Blueprint defaults to `starter`. To use the free tier, change `plan: free` in your fork's `render.yaml` — note that with no persistent disk, OpenClaw state resets on each deploy. +The Blueprint defaults to `starter`. To use the free tier, change `plan: free` **and delete the `disk:` block** in your fork's `render.yaml`; Render rejects a Blueprint that attaches a persistent disk to a free instance. Without that disk, OpenClaw state resets on every deploy. ## After deployment diff --git a/render.yaml b/render.yaml index 8b364059c60f..fc44f1a7ee39 100644 --- a/render.yaml +++ b/render.yaml @@ -3,6 +3,10 @@ services: name: openclaw runtime: docker plan: starter + # The image CMD has no --allow-unconfigured, so a fresh disk with no + # openclaw.json exits before binding. Render has no shell before the first + # boot, so the flag has to come from the blueprint. + dockerCommand: node openclaw.mjs gateway --allow-unconfigured healthCheckPath: /startupz envVars: - key: OPENCLAW_GATEWAY_PORT diff --git a/scripts/cloudflare/README.md b/scripts/cloudflare/README.md index 4d4557820486..ae8739a6ebae 100644 --- a/scripts/cloudflare/README.md +++ b/scripts/cloudflare/README.md @@ -21,7 +21,7 @@ OpenClaw + Litestream container :8080 +--> R2 S3 API (SQLite replicas) ``` -Every HTTP and WebSocket request is forwarded to port `8080`. The Container helper checks `GET /startupz` before admitting traffic. `max_instances: 1` and the single Durable Object name are the installation's outer single-writer fence. +Every HTTP and WebSocket request is forwarded to port `8080`. The Container helper polls `GET /healthz`, which every published image serves, before admitting traffic. `max_instances: 1` and the single Durable Object name are the installation's outer single-writer fence. ## Prerequisites @@ -128,7 +128,7 @@ Keep the exact bootstrap recipe in a private, reproducible runbook. Litestream d ## 5. Verify before relying on it ```bash -curl -sS https://.workers.dev/startupz +curl -sS https://.workers.dev/healthz npx wrangler tail ``` @@ -170,7 +170,8 @@ Treat rollbacks like restores: stop traffic where possible, preserve the current ## Troubleshooting - **Container never becomes ready:** the image must be `linux/amd64` and pulled from a public registry, referenced by digest rather than a moving tag. -- **Requests time out after a successful deploy:** the Container helper waits for `GET /startupz` on port `8080`; confirm the Gateway still binds that port. +- **Requests time out after a successful deploy:** the Container helper waits for `GET /healthz` on port `8080`; confirm the Gateway still binds that port. +- **A probe passes but nothing serves:** the Control UI answers unknown paths with a catch-all `200`, so probing a route the image does not serve looks healthy forever; assert the JSON body, not just the status. - **Litestream authentication or signature errors:** Litestream needs R2 _S3 API_ credentials, not a Cloudflare API token, and `LITESTREAM_ENDPOINT` must contain the account ID. - **First boot reports no databases to restore:** expected on an empty bucket; the entrypoint treats that as a fresh installation. - **`/readyz` is 503 while `/startupz` is 200:** by design. Startup finished and a channel account is unhealthy; inspect channel status instead of restarting. diff --git a/scripts/cloudflare/src/container.ts b/scripts/cloudflare/src/container.ts index 834ff722ea4e..3d79b3fca48d 100644 --- a/scripts/cloudflare/src/container.ts +++ b/scripts/cloudflare/src/container.ts @@ -53,7 +53,14 @@ function buildContainerEnv(env: OpenClawContainerEnv): Record { export class OpenClawContainer extends Container { override defaultPort = 8080; - override pingEndpoint = "localhost/startupz"; + // /healthz exists in every published OpenClaw image and answers as soon as the + // Gateway's listener is up, which is exactly what this readiness poll asks. + // Do not point this at a route the pinned image may not serve: the Control UI + // answers unknown paths with a catch-all 200, so a missing route would look + // permanently healthy instead of failing. /startupz additionally waits for + // startup work to finish and is the better signal once the derived image comes + // from a release that serves it. + override pingEndpoint = "localhost/healthz"; override sleepAfter = "10m"; private readonly webhookOnly: boolean; diff --git a/scripts/k8s/manifests/deployment.yaml b/scripts/k8s/manifests/deployment.yaml index 95376373e41a..7969c6a1e26b 100644 --- a/scripts/k8s/manifests/deployment.yaml +++ b/scripts/k8s/manifests/deployment.yaml @@ -107,12 +107,19 @@ spec: limits: memory: 2Gi cpu: "1" + # Probes assert the JSON probe contract, not just the status code. The + # Control UI serves a catch-all 200 for unknown paths, so a status-only + # check would pass forever against an image whose probe route does not + # exist yet. Failing closed makes that mismatch loud instead of silent. + # /startupz (startup admission, ignores channel health) needs an image + # built from the release that added it; switch the startup and + # readiness probes to it after pinning such an image. startupProbe: exec: command: - node - -e - - "require('http').get('http://127.0.0.1:18789/startupz', r => process.exit(r.statusCode < 400 ? 0 : 1)).on('error', () => process.exit(1))" + - "require('http').get('http://127.0.0.1:18789/readyz', r => { let b = ''; r.on('data', d => b += d); r.on('end', () => { try { process.exit(r.statusCode < 400 && JSON.parse(b).ready === true ? 0 : 1); } catch { process.exit(1); } }); }).on('error', () => process.exit(1))" periodSeconds: 10 timeoutSeconds: 5 failureThreshold: 30 @@ -121,7 +128,7 @@ spec: command: - node - -e - - "require('http').get('http://127.0.0.1:18789/healthz', r => process.exit(r.statusCode < 400 ? 0 : 1)).on('error', () => process.exit(1))" + - "require('http').get('http://127.0.0.1:18789/healthz', r => { let b = ''; r.on('data', d => b += d); r.on('end', () => { try { process.exit(r.statusCode < 400 && JSON.parse(b).ok === true ? 0 : 1); } catch { process.exit(1); } }); }).on('error', () => process.exit(1))" initialDelaySeconds: 60 periodSeconds: 30 timeoutSeconds: 10 @@ -130,7 +137,7 @@ spec: command: - node - -e - - "require('http').get('http://127.0.0.1:18789/startupz', r => process.exit(r.statusCode < 400 ? 0 : 1)).on('error', () => process.exit(1))" + - "require('http').get('http://127.0.0.1:18789/readyz', r => { let b = ''; r.on('data', d => b += d); r.on('end', () => { try { process.exit(r.statusCode < 400 && JSON.parse(b).ready === true ? 0 : 1); } catch { process.exit(1); } }); }).on('error', () => process.exit(1))" initialDelaySeconds: 15 periodSeconds: 10 timeoutSeconds: 5