Codex native subagent results now survive parent cleanup, app-server persistence races, resumptions, and transient delivery failures. The monitor treats app-server thread/read and thread/turns/list history as canonical recovery state (works on non-stdio transports), retains the shared physical client only while detached children need monitoring, fences stale reads across lifecycle changes, keeps interrupted children resumable, preserves multi-agent V2 subAgentActivity lineage, and bounds terminal delivery retries with a durable failed state. Removes rollout-file discovery, transcript parsing, periodic whole-client rescans, and the deferred one-shot cleanup path.
Adds an opt-in live E2E lane (OPENCLAW_LIVE_CODEX_NATIVE_SUBAGENT=1) proving detached delivery and history-only recovery against a real app-server with an OpenAI key, restores the registered-child user-spoof regression test, and registers both codex opt-in live files in the release shard optional-env map.
Closes#97593
Related: #93313
* refactor(sessions): migrate runtime storage to sqlite
* test(sessions): fix sqlite CI regressions
* test(sessions): align remaining sqlite fixtures
* fix(codex): require sqlite trajectory recorder
* test(sessions): align orphan recovery sqlite fixture
* test(sessions): align sqlite rebase fixtures
* fix(sessions): finish current-main integration of the sqlite flip
Resolve the whole-store SDK removal across its owner boundary: drop the
loadSessionStore re-export and the registry whole-store wrappers, wire
hasTrackedActiveSessionRun into gateway chat, complete the
preserveLockedHarnessIds cleanup contract, flip the codex thread-history
import to storePath targets, and port remaining main-side tests from
file-store helpers to session accessor reads.
* chore: drop committed pebbles log, revert plugin-inspector bump, refresh generated docs
Remove the 1.8k-line .pebbles/events.jsonl work log from the branch, restore
the plugin-inspector advisory lane to main's pinned 0.3.10 so the supply-chain
bump gets its own review, and regenerate docs_map, the plugin SDK API baseline,
and the export-surface ratchet for the merged tree.
* feat(sessions): keep archived transcripts by default with zstd cold storage
Codex-style retention: deleting or resetting a session archives its
transcript as a zstd-compressed JSONL artifact (plain when the runtime
lacks node:zlib zstd) and keeps it until the disk budget evicts oldest
first. resetArchiveRetention now governs both deleted and reset archives
and defaults to keep; maxDiskBytes defaults to 2gb so retention stays
bounded, with archives evicted before live sessions. The cron reaper
follows the same knob instead of deleting archives on its own timer.
* fix(state): converge agent DB migration lineages and bound database growth
Merge coherence: run both structure-gated legacy memory-schema repairs
(flip-lineage drop, main-lineage identity rebuild) before the flip
migration so pre-flip v1/v2 and pre-merge flip v1/v4 databases all
converge, and hoist foreign_keys=OFF outside the schema transaction
where the pragma was silently ignored and the v1 sessions rebuild
cascade-deleted session_entries.
Growth guards: fresh agent DBs enable auto_vacuum=INCREMENTAL, WAL
maintenance releases freed pages in bounded passes (never a blocking
full VACUUM), and doctor reports state/agent DB bloat from freelist
stats.
* fix(codex): resolve the store path for thread-history import via the SDK
The supervision catalog passed the legacy sessionFile locator to the
storePath-targeted transcript mirror; resolve the agent store path with
the session-store SDK helper instead of a runtime-object seam so test
fakes and headless callers need no extra surface. Drop the obsolete
missing-session-id preprocessing case: sessions rows are NOT NULL on
session_id and upsert repairs id-less patches at write time.
* fix(sessions): fail safe on malformed disk-budget config and doctor stat errors
A malformed explicit maxDiskBytes disables the budget instead of
falling back to the destructive 2gb default the user never chose, and
the doctor bloat check skips databases whose paths stat-fail instead of
aborting doctor.
* fix(sessions): complete sqlite conflict translations
* test(sqlite): align hardening checks with maintenance
* test(sessions): inspect compressed transcript archives
* fix(tests): await session seeds and drop unused helpers flagged by CI lint
The five unawaited writeSessionStoreSeed calls raced their SQLite seeds
against the assertions, failing compact shards; the bloat probe drops a
useless initializer and the merged tests drop now-unused helpers.
* test(sessions): type legacy proof events directly
* test(sessions): align hardening contracts
* perf(sessions): read usage transcript sizes from SQL aggregates
Usage/cost scans walked every session and materialized every transcript
event just to re-stringify it for a byte estimate — the #86718 stall
class reborn on the DB. readTranscriptStatsSync sums stored JSON bytes
in SQLite without loading a single row.
* fix(sessions): re-root foreign-root transcript paths onto the current sessions dir
Restored backups, moved OPENCLAW_STATE_DIR, and rehearsal copies carry
absolute sessionFile paths from the old root; the containment fallback
kept those foreign paths, so migration read (and would archive) files in
the original root and reported local copies missing. Re-root the
canonical agents/<id>/sessions suffix onto the current dir when the file
exists there; genuine cross-root layouts still fall through unchanged.
* test(agents): seed harness admission through sqlite
* fix(sqlite): close agent db on pragma setup failure
* fix(doctor): compact and retrofit incremental auto-vacuum after session import
The migration is the sanctioned offline window: post-import compact
reclaims import churn and applies auto_vacuum=INCREMENTAL to databases
created before the fresh-DB pragma existed, so runtime maintenance can
release pages in bounded passes on every install.
---------
Co-authored-by: Peter Steinberger <steipete@gmail.com>
* feat(codex): scope app-server rate limits to the physical client
Replace the process-global rate-limit cache with a WeakMap keyed by the
physical app-server client, tracking per-limitId revisions. Rolling
account/rateLimits/updated notifications merge sparsely per the protocol
contract (credits/individualLimit/planType survive nulls), and
usage-limit errors only trust a snapshot for auth-profile blocking when
the same client observed a primary update during the failing turn's
startup. Fixes cross-client rate-limit bleed in usage-limit error
messages. A new client-runtime module installs the
account/chatgptAuthTokens/refresh handler and the rate-limit observer
once per physical client, replacing per-start inline handlers in
shared-client, run-attempt, and side-question.
* refactor(codex): split thread/resume subscription safety into thread-resume
Move the thread/resume request out of thread-lifecycle into a dedicated
thread-resume module that retires the exact physical client when resume
acceptance is indeterminate: only a structured RPC rejection proves
Codex holds no subscription, so any other failure abandons the client
instead of returning it to the shared pool. Resume responses naming a
different thread now fail closed (assertCodexThreadResumeSubscription),
and the fresh-thread fallback requires a released subscription unless
the resume was a proven RPC rejection.
* refactor(codex): replace client-factory positional DI with shared-client factory
Delete the lazy positional-argument CodexAppServerClientFactory and use
the options-object factory type exported from shared-client. Callers in
run-attempt, compact, bounded-turn, provider-capabilities, and the
web-search provider now default to getLeasedSharedCodexAppServerClient
directly; the lazy indirection was ineffective because those modules
already import shared-client statically.
* feat(codex): route app-server turn traffic through a keyed turn router
Install one turn router per physical app-server client and replace the
broad per-attempt notification/request fanout with explicit per-thread
routes. Attempt startup reserves the thread route (before thread/resume
on the resume path, so early notifications buffer instead of racing),
run-attempt activates it with receive-time, queued, and request
handlers, arms the route before turn/start, binds the accepted turn id
to flush buffered traffic in wire order, and releases the route on
cleanup. Requests for a pending turn wait for binding instead of being
auto-declined, native turn completion waits use route state instead of
scanning buffered notifications, and correlation readers now match the
canonical v2 wire shapes only (top-level threadId, nested turn.id). The
unscoped response-delta lease-count attribution and its client API are
deleted along with the retired correlation predicates.
* test(codex): reset the shared binding store between thread-lifecycle tests
SQLite bindings are keyed by session identity rather than the per-test
temp dir, so earlier tests leaked resumable threads into fresh-start
expectations. The old silent resume-failure fallback masked the leak;
subscription safety surfaces it.
* test(codex): reset the binding store between delivery-hint iterations
The loop reuses one session identity across iterations, so the previous
iteration's thread would resume against a harness that cannot serve it.
* feat: correlate native search outcomes in audit history
Metadata-only audit ledger for agent runs and tool actions in the shared
state DB: stable event identity, closed action/status/error vocabularies,
one-way-hashed tool-call ids, never-inferred terminal outcomes for native
web-search (explicit completed/failed only; otherwise unknown), bounded
retention, audit.list gateway RPC and openclaw audit CLI. Squashed from
the 82-commit audit stack for replay onto current main.
* feat(audit): add audit.enabled config gate (default on)
The metadata-only audit ledger records by default: an audit trail enabled
only after an incident cannot explain the incident, and the rows are
strictly less sensitive than the transcripts every install already
stores. audit.enabled=false stops new writes at the gateway subscription
seam; audit.list and openclaw audit keep serving existing records until
they expire. Documented in the configuration reference, protocol page,
and CLI reference.
* fix: repair full-matrix CI findings after rebase
- break the dynamic-tools/dynamic-tool-execution import cycle by
extracting resolveCodexToolAbortTerminalReason into a leaf module
- restore main's session-worktree protocol exports lost in the
index.ts auto-merge
- register the audit event writer worker as a knip entry point
- docs table formatting; subagent wait-cancellation test scoped to its
audit intent (outcome + timing) and advanced past main's new
lifecycle-timeout retry grace
Approval wait now fits inside the Codex dynamic-tool watchdog (70s +10s
gateway grace under the 90s kill), approval cards carry proposal id,
skill name, description, file count, and body size (spoof-safe
rendering), and timeouts return a structured pending-not-failed outcome
instead of a bare error. Expired requests cannot execute late; no
auto-apply; generic plugin approvals unchanged.
* fix(ios): restore in-flight chat runs from gateway history
Restore active Apple chat ownership across reconnect, foreground, and sequence-gap recovery using the existing chat.history snapshot. Preserve agent/session scoping and Gateway user-turn identity across Codex and Copilot mirrors, including current offline-cache integration.
* fix(ios): restore in-flight chat runs from gateway history
* fix(codex): derive terminal-idle watchdog from effective run timeout
Fixes the early-abort half of #85242. The Codex app-server terminal-idle
watchdog used a hardcoded 30-minute default that was not derived from the
effective run timeout, so a scheduled turn configured with a longer
timeoutSeconds could be aborted early at 30 minutes even with budget left.
resolveCodexTurnTerminalIdleTimeoutMs (now in attempt-timeouts.ts after the
upstream split) accepts the effective run timeout and, with no explicit
override, follows the run budget instead of the 30-minute default:
- explicit override always wins (advanced config / tests)
- otherwise terminal-idle = max(30min floor, run budget), so a longer run is
no longer cut short and existing protection is never shortened
- falls back to the 30min default when no run budget is known
Reuses the existing resolvePositiveIntegerTimeoutMs helper, matching the
neighbouring post-tool resolver. Adds focused unit tests for the derivation.
The diagnostic-wording half of #85242 (naming the terminal-idle watchdog in
the surfaced guidance) is left as a separate follow-up.
* fix(codex): derive terminal-idle watchdog from effective run timeout
* fix(codex): preserve default terminal idle watchdog
---------
Co-authored-by: openclaw-clownfish[bot] <280122609+openclaw-clownfish[bot]@users.noreply.github.com>
Co-authored-by: Vincent Koc <vincentkoc@ieee.org>
Peter Steinberger