Final review findings:
- Wrap the operator-approval copy/drop/rename in a single immediate
transaction so a crash mid-migration cannot strand or lose rows.
- The inference fallback marked a provider attempted before probing, so a
stale requester credential blocked another agent's valid same-provider
route. Dedup by credential owner and only retire a whole provider on
provider-wide failures (auth/billing may differ per credential owner).
Refs #107237
Review P2: the doctor repair only checked operator_approvals column NAMES, so
a look-alike table with different constraints/defaults would get its rows
copied under today's schema, silently discarding those semantics. Gate the
destructive replacement on the exact prior canonical DDL (the two-kind
constraint before system-agent). Test proves it refuses a same-columns table
with a different kind constraint.
Refs #107237
New openclaw delegation tool relays to openclaw.chat in-process; persistent
writes surface through the existing durable operator-approval registry as a
third system-agent kind (no parallel store), armed only by a human operator
in the Control UI — a delegated agent can never self-approve. Setup wizards
(channel/model/open-setup/open-tui) are refused in delegated mode so a
machine agent cannot complete setup or persist credentials unattended.
Vendor-neutral inference fallback ladder (requester route first, then other
authed providers by provider id). update.run removed from the gateway tool.
Caveman system-prompt fragment steers config/channels/plugins/agents/updates
to the openclaw tool. Doctor-owned state migration widens the approval-kind
constraint; runtime stays canonical-only.
Refs #107237
Stops the legacy run-log dual-write, auto-imports remaining rows and drops
the table at state-DB open (table-presence guard: self-heals downgrade
round-trips), keeps the newest 2000 terminal runs per job in the ledger,
retires cron.runLog config via doctor, and moves live-run cancellation to
src/cron behind the shared task control seam.
Part 2 of 2 for #106041.
Cron executions now write full-fidelity task_runs rows (detail_json column,
store-key scoped) and cron.runs/CLI/task-maintenance read from the ledger;
legacy cron_run_logs dual-write stays as a revert safety net. Existing
history auto-imports at first state-DB open (verified against a production
DB copy: 2750/2750 rows, byte-identical parity, idempotent). Startup crash
recovery restores finished runs from finalized ledger rows instead of
reporting synthetic interruptions.
Part 1 of 2 for #106041.
* feat(gateway-protocol): add session placement schema
Closed state discriminator for session execution placement (local/requested/provisioning/syncing/starting/active/draining/reconciling/reclaimed/failed), sessions.dispatch params, and worker-admission transcript/live cursor extensions. Swift protocol models mirror the schema.
* feat(state): add worker session placement table
worker_session_placements rows carry placement state, transition generation, worker ownership metadata, ACK cursors, and the turn claim columns used for atomic admission.
* feat(cloud-workers): add durable placement state machine store
SQLite-backed placement store split by concern: state table (placement-state), discriminated record types + shape invariants (placement-record), row codec + CAS transition values (placement-row-codec), atomic turn-claim admission/release/waiters (placement-turn-claims), and lifecycle CAS transitions (placement-store).
* feat(cloud-workers): sync workspaces and attach sessions to worker environments
Environment service session attachment + turn credentials, tunnel workspace commands over a dedicated SSH runner, and git/plain workspace sync into $HOME/.openclaw-worker/workspaces with an immutable manifest. Symlink escapes are rejected locally before transfer (macOS openrsync stat-fails them opaquely) and again by the remote manifest guard.
* feat(worker): run one-shot embedded turns from launch descriptors
Worker runtime executes a single embedded turn from a stdin launch descriptor and reports completed/failed/fenced on stdout for the gateway launcher. Terminal lifecycle live events are deferred past the final transcript flush; transcript projection helpers are shared via transcript-message instead of duplicated in the runtime.
* feat(cloud-workers): dispatch placements and route worker turns
Dispatch service drives local->requested->provisioning->syncing->starting->active with failure teardown (placement-dispatch-failure) and restart/runtime recovery incl. lost-worker reclaim (placement-dispatch-recovery). Worker turn launcher claims the placement turn atomically, builds a windowed launch descriptor (worker-turn-payload), runs the remote one-shot worker, and reconciles the committed transcript; agent runners route turns through the session placement admission provider.
* feat(gateway): expose session placement RPCs and startup reconciliation
sessions.dispatch RPC with lifecycle admission barriers, operator-facing placement projection on session listings, placement-aware session reset guard, and startup/interval reconciliation wiring for worker placements.
* refactor(deadcode): trim auto-reply and CLI exports
* refactor(deadcode): trim cron and task exports
* refactor(deadcode): trim fleet and process exports
* test(deadcode): exercise live task and process seams
* test(fleet): cover stream redaction through owner module
* refactor(security): trim dead internal exports
* refactor(secrets): trim dead internal exports
* refactor(deadcode): trim remaining src exports
* refactor(deadcode): remove test-only runtime exports
* refactor(deadcode): trim pairing test exports
* refactor(deadcode): reconcile refreshed baseline
* test(auto-reply): deduplicate queue state imports
* fix(gateway): make hot reload transactional
Replace partial reload side effects with a deferred transaction that publishes config, secrets, auth, and subsystem state together, and drains in-flight reload work before shutdown.
Co-authored-by: LZY3538 <293718838+LZY3538@users.noreply.github.com>
* fix(auth): preserve state-only credential ownership
Keep derived runtime snapshots in place for main-store state mutations so order refreshes do not look like credential replacement.
* fix(gateway): close reload transaction gaps
* fix(gateway): close merged reload gaps
* chore: move reload note to PR context
* fix(gateway): exclude restart emission root
---------
Co-authored-by: LZY3538 <293718838+LZY3538@users.noreply.github.com>
Adds a core sessions_search agent tool backed by a SQLite FTS5 index that
lives next to the transcript rows in the per-agent database. Indexing is
transactional: user/assistant text is indexed inside the same write
transaction that persists the event, deletes drop index rows in the same
transaction, and branch rewinds mark the session dirty so the next search
rebuilds it from the same visible-path resolution sessions_history uses
(which also lazily backfills doctor-migrated databases). Search executes
gateway-side behind an additive sessions.search method with a bounded
session-key allowlist; visibility, sandbox restrictions, and snippet
redaction mirror sessions_history.
Closes#100978
* feat(gateway): persist operator approvals
* fix(gateway): preserve approval ids exactly
* fix(gateway): enforce approval reviewer bindings
* fix(gateway): reconcile durable approvals with exec revocation hardening
Map #103515 semantics onto the durable lifecycle: resolutionSource and
one-shot consumeAskFallback stay process-local record facts, trusted
auto-review resolves through the durable CAS as a runtime resolver, and
lost races settle with the winner's operator source. Also: derive the
audience walker cap from the store cap, report APPROVAL_ALREADY_RESOLVED
for a resolve that loses the CAS race, document approval.get/resolve in
the protocol docs, and regenerate Swift models on the new base.
* fix(approvals): validate resolver kind
* docs(approvals): explain malformed verdict denial
* fix(protocol): regenerate approval models after rebase
* chore: leave changelog entry to release generation
* fix(gateway): preserve approval registration boundaries
* test(gateway): prove multi-device approval races
* test(gateway): keep approval order assertion stable
* docs: index operator approval architecture
* fix(protocol): bound approval declaration exports
* feat(ui): detect background tasks eagerly and show live tool activity in the chat rail
Track tool-start counts and the last tool name per task run in the task
registry, persist them on task_runs, and expose them as additive optional
TaskSummary fields. The chat background-tasks rail now loads its snapshot
eagerly so the collapsed toggle badge detects running work immediately,
and rows show a live elapsed timer, tool-use count, and the tool
currently in use (run duration for finished rows).
Issue: #104775
* fix(ui): translate background-task activity strings and refresh generated protocol models
The singular tool-use label is literal ("1 tool use") because several
locales legitimately drop the count placeholder in singular forms, which
fail-closes the placeholder guard in the i18n sync. Locale bundles are
the authenticated ui:i18n:sync output (fallbacks=0); GatewayModels.swift
is protocol:gen:swift output for the new TaskSummary fields.
* chore(ui): refresh i18n raw-copy baseline after rebase
* perf(acp): maintain ledger byte budgets at insert time
estimateSqliteLedgerBytes ran a full SUM(length(...)) scan over every
replay event on every append, and once the 16MB budget was hit the trim
loop rescanned after each single-row delete (#100622). Event rows now
carry estimated_bytes computed at insert, session rows keep a running
footprint aggregate, budget checks sum over at most maxSessions rows,
and eviction deletes bounded batches with RETURNING so aggregates stay
exact. Additive columns backfill once via ensureAdditiveStateColumns.
Fixes#100622
* fix(acp): keep ledger footprint aggregates exact across key changes and conflicts
Session-key/cwd length changes now adjust the aggregate via SET
expressions that read the pre-update row, and the conflict-upsert
recomputes from surviving event rows instead of clobbering to row
overhead. Invariant test now rebinds a provisional key to a longer
canonical key mid-stream.
* refactor(sessions): migrate runtime storage to sqlite
* test(sessions): fix sqlite CI regressions
* test(sessions): align remaining sqlite fixtures
* fix(codex): require sqlite trajectory recorder
* test(sessions): align orphan recovery sqlite fixture
* test(sessions): align sqlite rebase fixtures
* fix(sessions): finish current-main integration of the sqlite flip
Resolve the whole-store SDK removal across its owner boundary: drop the
loadSessionStore re-export and the registry whole-store wrappers, wire
hasTrackedActiveSessionRun into gateway chat, complete the
preserveLockedHarnessIds cleanup contract, flip the codex thread-history
import to storePath targets, and port remaining main-side tests from
file-store helpers to session accessor reads.
* chore: drop committed pebbles log, revert plugin-inspector bump, refresh generated docs
Remove the 1.8k-line .pebbles/events.jsonl work log from the branch, restore
the plugin-inspector advisory lane to main's pinned 0.3.10 so the supply-chain
bump gets its own review, and regenerate docs_map, the plugin SDK API baseline,
and the export-surface ratchet for the merged tree.
* feat(sessions): keep archived transcripts by default with zstd cold storage
Codex-style retention: deleting or resetting a session archives its
transcript as a zstd-compressed JSONL artifact (plain when the runtime
lacks node:zlib zstd) and keeps it until the disk budget evicts oldest
first. resetArchiveRetention now governs both deleted and reset archives
and defaults to keep; maxDiskBytes defaults to 2gb so retention stays
bounded, with archives evicted before live sessions. The cron reaper
follows the same knob instead of deleting archives on its own timer.
* fix(state): converge agent DB migration lineages and bound database growth
Merge coherence: run both structure-gated legacy memory-schema repairs
(flip-lineage drop, main-lineage identity rebuild) before the flip
migration so pre-flip v1/v2 and pre-merge flip v1/v4 databases all
converge, and hoist foreign_keys=OFF outside the schema transaction
where the pragma was silently ignored and the v1 sessions rebuild
cascade-deleted session_entries.
Growth guards: fresh agent DBs enable auto_vacuum=INCREMENTAL, WAL
maintenance releases freed pages in bounded passes (never a blocking
full VACUUM), and doctor reports state/agent DB bloat from freelist
stats.
* fix(codex): resolve the store path for thread-history import via the SDK
The supervision catalog passed the legacy sessionFile locator to the
storePath-targeted transcript mirror; resolve the agent store path with
the session-store SDK helper instead of a runtime-object seam so test
fakes and headless callers need no extra surface. Drop the obsolete
missing-session-id preprocessing case: sessions rows are NOT NULL on
session_id and upsert repairs id-less patches at write time.
* fix(sessions): fail safe on malformed disk-budget config and doctor stat errors
A malformed explicit maxDiskBytes disables the budget instead of
falling back to the destructive 2gb default the user never chose, and
the doctor bloat check skips databases whose paths stat-fail instead of
aborting doctor.
* fix(sessions): complete sqlite conflict translations
* test(sqlite): align hardening checks with maintenance
* test(sessions): inspect compressed transcript archives
* fix(tests): await session seeds and drop unused helpers flagged by CI lint
The five unawaited writeSessionStoreSeed calls raced their SQLite seeds
against the assertions, failing compact shards; the bloat probe drops a
useless initializer and the merged tests drop now-unused helpers.
* test(sessions): type legacy proof events directly
* test(sessions): align hardening contracts
* perf(sessions): read usage transcript sizes from SQL aggregates
Usage/cost scans walked every session and materialized every transcript
event just to re-stringify it for a byte estimate — the #86718 stall
class reborn on the DB. readTranscriptStatsSync sums stored JSON bytes
in SQLite without loading a single row.
* fix(sessions): re-root foreign-root transcript paths onto the current sessions dir
Restored backups, moved OPENCLAW_STATE_DIR, and rehearsal copies carry
absolute sessionFile paths from the old root; the containment fallback
kept those foreign paths, so migration read (and would archive) files in
the original root and reported local copies missing. Re-root the
canonical agents/<id>/sessions suffix onto the current dir when the file
exists there; genuine cross-root layouts still fall through unchanged.
* test(agents): seed harness admission through sqlite
* fix(sqlite): close agent db on pragma setup failure
* fix(doctor): compact and retrofit incremental auto-vacuum after session import
The migration is the sanctioned offline window: post-import compact
reclaims import churn and applies auto_vacuum=INCREMENTAL to databases
created before the fresh-DB pragma existed, so runtime maintenance can
release pages in bounded passes on every install.
---------
Co-authored-by: Peter Steinberger <steipete@gmail.com>
* feat: add metadata-only message audit events
* chore(protocol): restore generated swift models and config baseline after rebase
* fix(state): gate agent-db ownership check before schema version
* fix(cli): sync audit command description into the root-help catalog
* fix(audit): require destination proof before classifying outbound messages as direct
* refactor(audit): drop dead migration guard and add contract comments
* docs(gateway): add dedicated audit history page and cross-links
* test(e2e): enable direct-mode message audit in the telegram proof SUT config
* test(channels): expect declared conversationKind in durable delivery session context
* fix(audit): record the routing channel id for inbound rows from plugin channels
* fix(audit): match channel-prefixed delivery targets in the destination route gate
* fix(audit): validate explicit target kind against the destination route kind
* fix(audit): use the canonical target-prefix grammar in the destination route gate and fail closed on foreign migration tables
* fix(audit): normalize nested provider and kind target prefixes in the destination gate
* fix(audit): strip registered provider aliases and the direct kind prefix in the destination gate
* chore(docs): refresh config baseline after rebase
* feat(sessions): durable session state events with parent invalidation
Parents no longer act on stale child-session state: a durable, typed
signal log (session_state_events + per-agent heads) records direct human
messages to children, child spawn/terminal outcomes, goal changes, and
compaction at the seams where those facts are created. A frozen-watermark
cursor protocol (session_watch_cursors) delivers one coalesced system
notice to the parent through the existing system-event + heartbeat wake
idiom, acknowledged at the shared generic drain, with a deferred startup
sweep for restart recovery. session_status gains stateVersion and
changesSince (with exact pruned-history gap signaling); sessions_list
rows gain stateVersion.
Closes#104565
* chore: regenerate docs map and prompt snapshots for session_status changesSince
* feat(fleet): add openclaw fleet cell supervisor for multi-tenant hosting
* test(fleet): cover cell profile, registry, containers, and service flows
* docs(fleet): document multi-tenant hosting and the fleet CLI
* fix(fleet): verify upgrade replacements and release foreign-collision reservations
* fix(fleet): gate upgrade commit on replacement health
* docs(fleet): mark fleet experimental and pin its single-host scope
* test(fleet): make incomplete-profile casts explicit for test-type lane
* fix(state): regenerate kysely schema artifacts after rebase conflict
* fix(auth): preserve token health after OAuth migration
After a user migrates from OAuth to a token/setup-token credential, the
gateway model-auth rollup reported the provider as missing, producing a
false "model auth expired" warning.
Rename aggregateOAuthStatus → aggregateRefreshableAuthStatus and
extract aggregateProfileStatus helper. OAuth remains authoritative when
present; token credentials are the fallback when no effective OAuth
profile exists. Empty effectiveProfiles stays authoritative (missing).
Token fallback applies regardless of expectsOAuth flag.
Fixes#97996
Co-authored-by: SunnyShu0925 <SunnyShu0925@users.noreply.github.com>
* test(auth): use fake token fixture
* fix(doctor): repair stale auth profile orders
* fix(doctor): inspect retained auth profile stores
* fix(doctor): harden retained auth store proof
---------
Co-authored-by: SunnyShu0925 <SunnyShu0925@users.noreply.github.com>
Co-authored-by: Peter Steinberger <steipete@gmail.com>
* feat(gateway): session worktree targeting, branch listing, and group catalog
- sessions.create accepts worktreeBaseRef/worktreeName (write scope) and
execNode (admin); worktree binding persists on the session entry as
worktree { id, branch, repoRoot } and projects onto session rows with
execNode so UIs can show checkout/branch/node state
- new worktrees.branches RPC lists local/remote branches (no fetch) for
base-ref pickers; worktrees.remove now reports snapshotError
- sessions.delete reports preserved dirty checkouts as worktreePreserved
- gateway-owned session group catalog in the shared state DB with
sessions.groups.list/put/rename/delete; rename/delete update member
categories server-side without bumping recency; sessions.patch absorbs
ad-hoc categories into the catalog
- group session display names prefer the human chat title (subject or
space #channel) over stored compact tokens
Part of #103431
* fix(gateway): route group category updates through the session accessor and regen protocol models
- bulk member-category rename/clear uses applySessionEntryReplacements
instead of legacy updateSessionStore call sites (session accessor
boundary guard)
- regenerate Swift GatewayModels for the worktrees.branches schemas,
snapshotError, and new sessions.create params
Part of #103431
* fix(gateway): resolvable remote branch refs and workspace-scoped branch listing
- worktrees.branches returns remote-only branches remote-qualified
(origin/feature-a) so every advertised name works as a worktree base ref
- write-scoped worktrees.branches callers are limited to configured agent
workspaces; other host paths require operator.admin, matching the
sessions.create cwd bar
Part of #103431
* fix(gateway): guard worktree name reuse by owner and env-scope group transactions
- managedWorktrees.create rejects a caller-chosen name whose live or
restorable record belongs to a different owner, so write-scoped
sessions.create cannot bind a session into another session's or a
manual checkout
- session group catalog writes run their SQLite transaction on the same
env-scoped handle as their statements, keeping OPENCLAW_STATE_DIR
overrides atomic and away from the default state DB
Part of #103431
Peter Steinberger