ewilliams28/openclaw
1
0
Fork 0
mirror of https://github.com/openclaw/openclaw.git synced 2026-08-12 21:53:00 -06:00
Code Issues Packages Projects Releases Wiki Activity

fix(deploy): repair probes that silently pass and a blueprint that cannot boot (#122963)

Browse Source 
Three defects proven by running the shipped configs:

Kubernetes probes checked only the status code against /startupz, but the
pinned image predates that route and the Control UI answers unknown paths
with a catch-all 200. A wedged pod was therefore marked Ready forever.
Probes now assert the JSON probe contract and target routes the pinned
image actually serves; verified in a kind cluster where the old command
exits 0 on the missing route and the new one exits 1.

render.yaml set no dockerCommand, so the image CMD ran without
--allow-unconfigured and a fresh Render disk exited 78 with 'Missing
config' before binding. Reproduced locally with Render's exact env.

The Cloudflare Container readiness poll had the same route mismatch
against operator-supplied official image digests; it now polls /healthz,
which every published image serves.

Also replaces an R2 verification step that could never fail: wrangler
cannot list object keys, so the documented command 404'd into || true.
This commit is contained in:
Peter Steinberger
2026-08-12 20:49:54 -07:00
committed by GitHub
parent 89a5507602
commit 2c3e537cb8
7 changed files with 42 additions and 16 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/install/cloudflare.md
+11 -7
View File
@@ -25,7 +25,7 @@ The template lives in [`scripts/cloudflare`](https://github.com/openclaw/opencla
## How it works
The Worker forwards every HTTP and WebSocket request to one stable Durable Object name. That Durable Object owns one Container instance and is the single-writer fence around the Litestream replica. The Container exposes OpenClaw on port `8080`; `/startupz` is its traffic-readiness check.
The Worker forwards every HTTP and WebSocket request to one stable Durable Object name. That Durable Object owns one Container instance and is the single-writer fence around the Litestream replica. The Container exposes OpenClaw on port `8080`, and the Durable Object polls `/healthz` there before routing to it.
```mermaid
flowchart TD
@@ -179,21 +179,23 @@ Measured on this template against a real R2 bucket: about 2.4 seconds from write
Run these checks after the first bootstrap, before you depend on this deployment.
Confirm the Gateway admits traffic. `/startupz` reports startup completion and ignores channel health, so it stays green when one channel account is broken:
Confirm the Gateway answers. `/healthz` reports that the listener is up. `/startupz` additionally reports that startup work finished while ignoring channel health, so it stays green when one channel account is broken; it is served only by images built from the release that added it:
```bash
curl -sS https://<worker-subdomain>.workers.dev/startupz
curl -sS https://<worker-subdomain>.workers.dev/healthz
curl -sS -H "Authorization: Bearer $OPENCLAW_GATEWAY_TOKEN" \
https://<worker-subdomain>.workers.dev/readyz
```
Confirm replication is actually reaching R2. Objects should appear under `replicas/` within seconds of activity:
Confirm replication is actually reaching R2. Litestream writes keys under `replicas/state/<database>/<generation>/`, so list that prefix with any S3-compatible client using the same R2 credentials you gave Litestream. Wrangler cannot list object keys, only fetch them by exact path:
```bash
npx wrangler r2 object get openclaw-backups/replicas --remote 2>/dev/null || true
npx wrangler r2 bucket list
aws s3 ls "s3://openclaw-backups/replicas/" --recursive \
--endpoint-url "https://<account-id>.r2.cloudflarestorage.com"
```
The Cloudflare dashboard's R2 object browser shows the same tree. An empty prefix after several minutes of activity means replication is not working; fix it before continuing.
Rehearse recovery before you need it. An untested restore path is not a backup:
1. Send one message so the Gateway writes a session row.
@@ -267,7 +269,9 @@ Test updates and rollbacks against a separate R2 bucket first. Preserve current
**Worker returns 5xx and the Container never becomes ready** -- Cloudflare only runs `linux/amd64` images pulled from a public registry. Rebuild with `--platform linux/amd64`, confirm the derived Docker Hub repository is public, and confirm `containers[].image` uses the pushed digest rather than a moving tag.
**Deployment succeeds but every request times out** -- The Container helper waits for `GET /startupz`. Check that the Gateway inside the Container listens on port `8080` and that no bootstrap step changed the port.
**Deployment succeeds but every request times out** -- The Container helper waits for `GET /healthz`. Check that the Gateway inside the Container listens on port `8080` and that no bootstrap step changed the port.
**A probe passes but the Gateway is not actually serving** -- The Control UI answers unknown paths with a catch-all `200`, so probing a route your image does not serve looks permanently healthy. Verify the response body is JSON, not HTML, before trusting a probe.
**Litestream logs authentication or signature errors** -- Litestream needs R2 S3 API credentials, which are not the same as a Cloudflare API token. Create an R2 API token and use its access key ID and secret access key, and confirm `LITESTREAM_ENDPOINT` contains your account ID.
docs/install/kubernetes.md
+3 -1
View File
@@ -90,7 +90,9 @@ Namespace: openclaw (configurable via OPENCLAW_NAMESPACE)
└── Secret/openclaw-secrets # Gateway token + API keys
```
The Deployment uses `/startupz` for both startup and traffic-readiness probes, with a five-minute startup budget. Channel failures do not evict a healthy Gateway or Control UI from Service endpoints. `/healthz` remains the liveness probe; use `/readyz` separately when monitoring should include channel-account health.
The Deployment probes `/readyz` for startup and traffic readiness with a five-minute startup budget, and `/healthz` for liveness. Every probe asserts the JSON probe contract rather than the status code alone, because the Control UI answers unknown paths with a catch-all `200`; a status-only check would pass forever against an image whose probe route does not exist yet.
`/startupz` is the better traffic-admission probe because it ignores channel health, so one failing channel account cannot evict an otherwise healthy Gateway from Service endpoints. It requires an image built from the release that introduced it, which is newer than the tag pinned above. After pinning such an image, switch the startup and readiness probes to `/startupz` and keep `/readyz` for monitoring that should include channel-account health.
## Customization
docs/install/render.mdx
+2 -1
View File
@@ -27,6 +27,7 @@ services:
name: openclaw
runtime: docker
plan: starter
dockerCommand: node openclaw.mjs gateway --allow-unconfigured
healthCheckPath: /startupz
envVars:
- key: OPENCLAW_GATEWAY_PORT
@@ -58,7 +59,7 @@ services:
| Starter | Never | 1GB+ | Personal use, small teams |
| Standard+ | Never | 1GB+ | Production, multiple channels |
The Blueprint defaults to `starter`. To use the free tier, change `plan: free` in your fork's `render.yaml` — note that with no persistent disk, OpenClaw state resets on each deploy.
The Blueprint defaults to `starter`. To use the free tier, change `plan: free` **and delete the `disk:` block** in your fork's `render.yaml`; Render rejects a Blueprint that attaches a persistent disk to a free instance. Without that disk, OpenClaw state resets on every deploy.
## After deployment
render.yaml
+4
View File
@@ -3,6 +3,10 @@ services:
name: openclaw
runtime: docker
plan: starter
# The image CMD has no --allow-unconfigured, so a fresh disk with no
# openclaw.json exits before binding. Render has no shell before the first
# boot, so the flag has to come from the blueprint.
dockerCommand: node openclaw.mjs gateway --allow-unconfigured
healthCheckPath: /startupz
envVars:
- key: OPENCLAW_GATEWAY_PORT
scripts/cloudflare/README.md
+4 -3
View File
@@ -21,7 +21,7 @@ OpenClaw + Litestream container :8080
+--> R2 S3 API (SQLite replicas)
```
Every HTTP and WebSocket request is forwarded to port `8080`. The Container helper checks `GET /startupz` before admitting traffic. `max_instances: 1` and the single Durable Object name are the installation's outer single-writer fence.
Every HTTP and WebSocket request is forwarded to port `8080`. The Container helper polls `GET /healthz`, which every published image serves, before admitting traffic. `max_instances: 1` and the single Durable Object name are the installation's outer single-writer fence.
## Prerequisites
@@ -128,7 +128,7 @@ Keep the exact bootstrap recipe in a private, reproducible runbook. Litestream d
## 5. Verify before relying on it
```bash
curl -sS https://<worker-subdomain>.workers.dev/startupz
curl -sS https://<worker-subdomain>.workers.dev/healthz
npx wrangler tail
```
@@ -170,7 +170,8 @@ Treat rollbacks like restores: stop traffic where possible, preserve the current
## Troubleshooting
- **Container never becomes ready:** the image must be `linux/amd64` and pulled from a public registry, referenced by digest rather than a moving tag.
- **Requests time out after a successful deploy:** the Container helper waits for `GET /startupz` on port `8080`; confirm the Gateway still binds that port.
- **Requests time out after a successful deploy:** the Container helper waits for `GET /healthz` on port `8080`; confirm the Gateway still binds that port.
- **A probe passes but nothing serves:** the Control UI answers unknown paths with a catch-all `200`, so probing a route the image does not serve looks healthy forever; assert the JSON body, not just the status.
- **Litestream authentication or signature errors:** Litestream needs R2 _S3 API_ credentials, not a Cloudflare API token, and `LITESTREAM_ENDPOINT` must contain the account ID.
- **First boot reports no databases to restore:** expected on an empty bucket; the entrypoint treats that as a fresh installation.
- **`/readyz` is 503 while `/startupz` is 200:** by design. Startup finished and a channel account is unhealthy; inspect channel status instead of restarting.
scripts/cloudflare/src/container.ts
+8 -1
View File
@@ -53,7 +53,14 @@ function buildContainerEnv(env: OpenClawContainerEnv): Record<string, string> {
export class OpenClawContainer extends Container<OpenClawContainerEnv> {
override defaultPort = 8080;
override pingEndpoint = "localhost/startupz";
// /healthz exists in every published OpenClaw image and answers as soon as the
// Gateway's listener is up, which is exactly what this readiness poll asks.
// Do not point this at a route the pinned image may not serve: the Control UI
// answers unknown paths with a catch-all 200, so a missing route would look
// permanently healthy instead of failing. /startupz additionally waits for
// startup work to finish and is the better signal once the derived image comes
// from a release that serves it.
override pingEndpoint = "localhost/healthz";
override sleepAfter = "10m";
private readonly webhookOnly: boolean;
scripts/k8s/manifests/deployment.yaml
+10 -3
View File
@@ -107,12 +107,19 @@ spec:
limits:
memory: 2Gi
cpu: "1"
# Probes assert the JSON probe contract, not just the status code. The
# Control UI serves a catch-all 200 for unknown paths, so a status-only
# check would pass forever against an image whose probe route does not
# exist yet. Failing closed makes that mismatch loud instead of silent.
# /startupz (startup admission, ignores channel health) needs an image
# built from the release that added it; switch the startup and
# readiness probes to it after pinning such an image.
startupProbe:
exec:
command:
- node
- -e
- "require('http').get('http://127.0.0.1:18789/startupz', r => process.exit(r.statusCode < 400 ? 0 : 1)).on('error', () => process.exit(1))"
- "require('http').get('http://127.0.0.1:18789/readyz', r => { let b = ''; r.on('data', d => b += d); r.on('end', () => { try { process.exit(r.statusCode < 400 && JSON.parse(b).ready === true ? 0 : 1); } catch { process.exit(1); } }); }).on('error', () => process.exit(1))"
periodSeconds: 10
timeoutSeconds: 5
failureThreshold: 30
@@ -121,7 +128,7 @@ spec:
command:
- node
- -e
- "require('http').get('http://127.0.0.1:18789/healthz', r => process.exit(r.statusCode < 400 ? 0 : 1)).on('error', () => process.exit(1))"
- "require('http').get('http://127.0.0.1:18789/healthz', r => { let b = ''; r.on('data', d => b += d); r.on('end', () => { try { process.exit(r.statusCode < 400 && JSON.parse(b).ok === true ? 0 : 1); } catch { process.exit(1); } }); }).on('error', () => process.exit(1))"
initialDelaySeconds: 60
periodSeconds: 30
timeoutSeconds: 10
@@ -130,7 +137,7 @@ spec:
command:
- node
- -e
- "require('http').get('http://127.0.0.1:18789/startupz', r => process.exit(r.statusCode < 400 ? 0 : 1)).on('error', () => process.exit(1))"
- "require('http').get('http://127.0.0.1:18789/readyz', r => { let b = ''; r.on('data', d => b += d); r.on('end', () => { try { process.exit(r.statusCode < 400 && JSON.parse(b).ready === true ? 0 : 1); } catch { process.exit(1); } }); }).on('error', () => process.exit(1))"
initialDelaySeconds: 15
periodSeconds: 10
timeoutSeconds: 5