diff --git a/CHANGELOG.md b/CHANGELOG.md index 8c95a09177fc..6ae1b20c2986 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -41,6 +41,7 @@ Docs: https://docs.openclaw.ai - Agents: retry empty final turns for generic `anthropic-messages` providers instead of limiting non-visible recovery to Kimi, so custom/proxied Anthropic-compatible routes can recover with a visible answer. Addresses #46080. Thanks @wmgx, @w1tv, and @iFwu. - Agents/replies: strip workflow `` scaffolding from user-visible sanitizer paths so raw tool output does not leak into chat history, transcript mirrors, or channel replies. Fixes #47444. Thanks @5toCode. - Agents/media: deliver generated image, music, and video results through structured attachments, keep message-tool-only Codex completions on the message tool, and fail completion handoff when expected media is not actually sent. +- Diagnostics/Codex: recover stalled embedded Codex app-server runs after the shorter default stalled-run window so queued turns resume sooner. - Control UI: rotate browser service-worker caches per build so updated Gateways are less likely to keep serving stale dashboard bundles that trigger protocol mismatch errors. - Gateway/protocol: lazy-compile protocol validators on first use instead of compiling every AJV schema during cold import, reducing startup CPU and RSS. (#82064) Thanks @samzong. - Discord: report unresolved configured bot-token SecretRefs during startup instead of treating the account as unconfigured. (#82009) Thanks @giodl73-repo. diff --git a/docs/concepts/agent-loop.md b/docs/concepts/agent-loop.md index fa8474a4bfc8..90c65945fde1 100644 --- a/docs/concepts/agent-loop.md +++ b/docs/concepts/agent-loop.md @@ -165,7 +165,7 @@ surfaces, while Codex native hooks remain a separate lower-level Codex mechanism - `agent.wait` default: 30s (just the wait). `timeoutMs` param overrides. - Agent runtime: `agents.defaults.timeoutSeconds` default 172800s (48 hours); enforced in `runEmbeddedPiAgent` abort timer. - Cron runtime: isolated agent-turn `timeoutSeconds` is owned by cron. The scheduler starts that timer when execution begins, aborts the underlying run at the configured deadline, then runs bounded cleanup before recording the timeout so a stale child session cannot keep the lane stuck. -- Session liveness diagnostics: with diagnostics enabled, `diagnostics.stuckSessionWarnMs` classifies long `processing` sessions that have no observed reply, tool, status, block, or ACP progress. Active embedded runs, model calls, and tool calls report as `session.long_running`; active work with no recent progress reports as `session.stalled`; `session.stuck` is reserved for stale session bookkeeping with no active work. Stale session bookkeeping releases the affected session lane immediately; stalled embedded runs are abort-drained only after `diagnostics.stuckSessionAbortMs` (default: at least 10 minutes and 5x the warning threshold) so queued work can resume without cutting off merely slow runs. Recovery emits structured requested/completed outcomes, and diagnostic state is marked idle only if the same processing generation is still current. Repeated `session.stuck` diagnostics back off while the session remains unchanged. +- Session liveness diagnostics: with diagnostics enabled, `diagnostics.stuckSessionWarnMs` classifies long `processing` sessions that have no observed reply, tool, status, block, or ACP progress. Active embedded runs, model calls, and tool calls report as `session.long_running`; active work with no recent progress reports as `session.stalled`; `session.stuck` is reserved for stale session bookkeeping with no active work. Stale session bookkeeping releases the affected session lane immediately; stalled embedded runs are abort-drained only after `diagnostics.stuckSessionAbortMs` (default: at least 5 minutes and 3x the warning threshold) so queued work can resume without cutting off merely slow runs. Recovery emits structured requested/completed outcomes, and diagnostic state is marked idle only if the same processing generation is still current. Repeated `session.stuck` diagnostics back off while the session remains unchanged. - Model idle timeout: OpenClaw aborts a model request when no response chunks arrive before the idle window. `models.providers..timeoutSeconds` extends this idle watchdog for slow local/self-hosted providers; otherwise OpenClaw uses `agents.defaults.timeoutSeconds` when configured, capped at 120s by default. Cron-triggered runs with no explicit model or agent timeout disable the idle watchdog and rely on the cron outer timeout. - Provider HTTP request timeout: `models.providers..timeoutSeconds` applies to that provider's model HTTP fetches, including connect, headers, body, SDK request timeout, total guarded-fetch abort handling, and model stream idle watchdog. Use this for slow local/self-hosted providers such as Ollama before raising the whole agent runtime timeout. diff --git a/docs/gateway/configuration-reference.md b/docs/gateway/configuration-reference.md index 56ceb8b19597..180f4af3e20c 100644 --- a/docs/gateway/configuration-reference.md +++ b/docs/gateway/configuration-reference.md @@ -1014,7 +1014,7 @@ Notes: enabled: true, flags: ["telegram.*"], stuckSessionWarnMs: 30000, - stuckSessionAbortMs: 600000, + stuckSessionAbortMs: 300000, otel: { enabled: false, @@ -1054,7 +1054,7 @@ Notes: - `enabled`: master toggle for instrumentation output (default: `true`). - `flags`: array of flag strings enabling targeted log output (supports wildcards like `"telegram.*"` or `"*"`). - `stuckSessionWarnMs`: no-progress age threshold in ms for classifying long-running processing sessions as `session.long_running`, `session.stalled`, or `session.stuck`. Reply, tool, status, block, and ACP progress reset the timer; repeated `session.stuck` diagnostics back off while unchanged. -- `stuckSessionAbortMs`: no-progress age threshold in ms before eligible stalled active work may be abort-drained for recovery. When unset, OpenClaw uses the safer extended embedded-run window of at least 10 minutes and 5x `stuckSessionWarnMs`. +- `stuckSessionAbortMs`: no-progress age threshold in ms before eligible stalled active work may be abort-drained for recovery. When unset, OpenClaw uses the safer extended embedded-run window of at least 5 minutes and 3x `stuckSessionWarnMs`. - `otel.enabled`: enables the OpenTelemetry export pipeline (default: `false`). For the full configuration, signal catalog, and privacy model, see [OpenTelemetry export](/gateway/opentelemetry). - `otel.endpoint`: collector URL for OTel export. - `otel.tracesEndpoint` / `otel.metricsEndpoint` / `otel.logsEndpoint`: optional signal-specific OTLP endpoints. When set, they override `otel.endpoint` for that signal only. diff --git a/docs/gateway/opentelemetry.md b/docs/gateway/opentelemetry.md index 0952269f684f..34bc4f3d1a16 100644 --- a/docs/gateway/opentelemetry.md +++ b/docs/gateway/opentelemetry.md @@ -230,7 +230,7 @@ OpenClaw classifies sessions by the work it can still observe: recent progress. Stalled embedded runs stay observe-only at first, then abort-drain after `diagnostics.stuckSessionAbortMs` with no progress so queued turns behind the lane can resume. When unset, the abort threshold defaults to - the safer extended window of at least 10 minutes and 5x + the safer extended window of at least 5 minutes and 3x `diagnostics.stuckSessionWarnMs`. - `session.stuck`: stale session bookkeeping with no active work. This releases the affected session lane immediately. diff --git a/src/logging/diagnostic.test.ts b/src/logging/diagnostic.test.ts index 8d47c8c0b3b3..cf0795dea150 100644 --- a/src/logging/diagnostic.test.ts +++ b/src/logging/diagnostic.test.ts @@ -1339,7 +1339,7 @@ describe("stuck session diagnostics threshold", () => { 30_000, ), ).toBe(48 * 60 * 60_000); - expect(resolveStuckSessionAbortMs(undefined, 30_000)).toBe(10 * 60_000); + expect(resolveStuckSessionAbortMs(undefined, 30_000)).toBe(5 * 60_000); }); }); diff --git a/src/logging/diagnostic.ts b/src/logging/diagnostic.ts index 58bce7a05175..b844b2698886 100644 --- a/src/logging/diagnostic.ts +++ b/src/logging/diagnostic.ts @@ -73,8 +73,8 @@ const webhookStats = { const DEFAULT_STUCK_SESSION_WARN_MS = 120_000; const MIN_STUCK_SESSION_WARN_MS = 1_000; const MAX_STUCK_SESSION_WARN_MS = 24 * 60 * 60 * 1000; -const MIN_STALLED_EMBEDDED_RUN_ABORT_MS = 10 * 60_000; -const STALLED_EMBEDDED_RUN_ABORT_WARN_MULTIPLIER = 5; +const MIN_STALLED_EMBEDDED_RUN_ABORT_MS = 5 * 60_000; +const STALLED_EMBEDDED_RUN_ABORT_WARN_MULTIPLIER = 3; const RECENT_DIAGNOSTIC_ACTIVITY_MS = 120_000; const DEFAULT_LIVENESS_EVENT_LOOP_DELAY_WARN_MS = 1_000; const DEFAULT_LIVENESS_EVENT_LOOP_UTILIZATION_WARN = 0.95;