* feat(hooks): dispatch hook agent runs into a dedicated command lane
Hook agent runs passed lane:"cron", which resolveCronAgentLane remaps to
cron-nested — the same lane cron's own inner agent work uses, capped at the
hardcoded cron budget of 8. Eight busy cron turns therefore starved every hook.
Adds CommandLane.HookDispatch and dispatches hook runs into it. Neither lane
resolver needs changing: resolveCronAgentLane (agents/lanes.ts:15-22) and
resolveGlobalLane (embedded-agent-runner/lanes.ts:11-18) special-case only
"cron" and pass every other lane through.
This is lane identity only. It does NOT yet bound aggregate capacity — that is
the capacity group in the following commits. On its own this widens total
command-lane concurrency by the hook lane's width (1).
Consumers that inferred cron-ness from the lane, both preserved rather than
silently changed:
- heartbeat-runner-execution: HookDispatch added to the busy-lane check so hook
work still suppresses heartbeats; only the lane it occupies changed.
- session-suspension: explicit resume concurrency and gateway-managed-lane
membership, instead of falling through to the custom-lane default.
server-lanes publishes the lane at width 1: the guarantee is that a hook can
always start under cron saturation, not that hooks run concurrently.
Test: server.hooks-lane.test.ts asserts the dispatched lane and that it survives
cron lane resolution unremapped, with a positive control on the inputs that DO
remap. Mutation-verified — reverting the call site to "cron" fails it with
'expected cron to be hook-dispatch'. Nothing else in the suite reads the
dispatched lane, so without this assertion the change regresses silently.
Refs: openclaw#98813, openclaw#43235
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* feat(queue): capacity groups with hard per-member reservations
Adds optional capacity groups to the command queue: lanes in a group share one
hard aggregate budget, and a member may hold a non-borrowable reservation
within it. This is what makes a separate hook lane safe — it bounds hook and
cron-nested work together at the existing cron cap instead of adding a slot
outside it (openclaw#98813 maintainer audit measured cron-nested=8 + hook-a=1 +
hook-b=1 = 10).
Group capacity is always DERIVED from members' activeTaskIds, never a separate
counter. Timeout, abort, clear, reset and stale-generation completion therefore
release capacity for free, because they all remove the task id; the only
remaining obligation is that those paths re-drain the group.
- setCommandLaneGroup / clearCommandLaneGroup / drainCommandLaneGroup
- admission: lane max, then group budget, then sibling reservations. A member
may burst above its own reservation only into unreserved capacity.
- both completion paths (success AND error) wake group siblings; freed capacity
belongs to the group, so a lane-local pump would strand a queued sibling
behind capacity that is already free. resetCommandLane likewise.
- membership lives in the queue singleton keyed by lane name, NOT in LaneState,
so setCommandLaneConcurrency cannot detach a member from its group.
- deadlock guard: rejects cron/main/subagent/nested and session:*/nested:*/
context-engine-turn-maintenance:* — lanes that can be synchronously awaited,
where a group wait would become a deadlock.
- rejects sum(reservations) > budget rather than starving silently.
- snapshot exposes group/groupActive/groupBudget/reservedForLane/blockedBy.
Tests: 9 new, all mutation-verified — dropping group admission fails 5,
removing the sibling wake fails 2, making reservations borrowable fails 2
("expected 4 to be 3": an idle sibling's reserve being borrowed). 57 pass
across all command-queue suites.
Not yet wired: no group is configured by default. That, plus group-wait
visibility and atomic publish, are the following commits.
Refs: openclaw#98813, openclaw#43235
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* test(queue): blockedBy answers hypothetical immediate admission, not queue state
Round-4 review (costaff-lapclaw-001) named this as the precision requirement
that decides whether the wait-visibility fix is vacuous:
noteLaneWaitIfBusy runs BEFORE enqueue, so it snapshots the lane with
queuedCount === 0. If blockedBy were populated only for an already-queued
head entry, that snapshot would read "not blocked", no
onLaneWait(waiting:true) would fire, and agent-watchdog's setup-timeout
suppression would never engage — a run merely waiting on group capacity
would take a false setup timeout.
resolveLaneBlockReason already answers "could this lane start work right now?"
independent of queue contents; these tests pin that contract:
- 7 cron active, hook holding the group's reserved slot: cron reports
sibling-reservation with queuedCount 0 and activeCount < maxConcurrent, while
the hook reports null (so the assertion discriminates rather than being
always-truthy).
- a member lane that was never enqueued or was retired while idle still reports
its group block state, instead of the not-found path returning a bare default
that reads as free.
Mutation-verified: gating blockedBy on queue.length > 0 fails 3 tests with
'expected null to be sibling-reservation' — the exact symptom predicted.
Refs: openclaw#98813
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* feat(queue): atomic lane publication, group-aware wait visibility, opt-in group
Closes the remaining two round-2 blockers and wires the default group.
publishLaneConfiguration({lanes, groups, clearGroups}) applies lane maxima and
group definitions as ONE transaction: install with dispatch suppressed, then a
single commit-time drain. The per-lane setter drains the instant a lane goes
positive and gateway publication was sequential, so a member could be widened
and dispatch BEFORE its group existed — admitting work above the budget the
group was meant to enforce. Validation throws before any drain, so a rejected
configuration cannot strand lanes widened and ungoverned.
lane-controller's noteLaneWaitIfBusy now also emits a wait when
snapshot.blockedBy != null. A group-blocked member has
activeCount < maxConcurrent and can have queuedCount === 0, so both lane-local
terms were false while the task genuinely could not start. This is not just
observability: agent-watchdog.ts suppresses the cron setup timeout only while
waitingForLane is true, so an invisible group wait produced a FALSE setup
timeout for cron-shaped runs.
The group is opt-in on hooks.enabled. The reservation is a real cost — it
withholds a slot from cron inner work even while the hook lane is idle — so it
is only paid where it buys something. This surfaced as a genuine regression:
server-lanes.test.ts asserts cron-nested alone reaches all 8, which an
unconditional group breaks. With hooks off, no group is installed, cron keeps
the entire budget, and such a deployment sees no behaviour change at all.
Turning hooks off on reload tears the group down (clearGroups).
With hooks ON, cron inner work trades one slot for the guarantee that hooks
cannot be starved. Aggregate stays exactly the pre-existing cron cap — no slot
added outside it, which is what openclaw#98813 was held for.
Tests: 6 new (3 publication, 3 opt-in), 135 passing across all affected suites.
Mutation-verified:
- sequential per-lane publication: 'expected 12 to be less than or equal to 8',
the exact 8+4 additive leak, caught at PEAK not post-state as review required
- unconditional group: 'expected cron-hooks to be undefined'
Refs: openclaw#98813, openclaw#43235
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* test(queue): pin group-blocked lane waits to the setup-timeout suppression chain
Round-4 review (fiducian-spencer-001) asked for the 8-cron / hook-holding-
reserve / 8th-cron-waiting regression asserting no false setup timeout.
The chain spans three files:
lane-controller.noteLaneWaitIfBusy -> onLaneWait({waiting:true})
-> timer-job-runner.noteLaneState -> watchdog.noteLaneWait()
-> agent-watchdog:159-164 -> waitingForLane = true, clear timeout
-> agent-watchdog:98 -> setup timeout suppressed
The watchdog end is already covered by agent-watchdog.test.ts. The link this
change introduced is the FIRST one, and it is the one that fails silently: a
group-blocked lane looks idle to a lane-local view, so no wait is reported and
a healthy run queued behind group capacity takes a false setup timeout.
The predicate was an inline closure, so it was untestable without the full
runner harness — and asserting a copy of it in a test would prove nothing.
Extracted as shouldNoteLaneWait(snapshot) and driven with real snapshots from a
real group:
- 7 cron active, hook holding the reserve: the test asserts explicitly that
BOTH lane-local terms are false (activeCount 7 < maxConcurrent 8,
queuedCount 0) and that the predicate still reports a wait.
- a hook blocked by a full group budget reports a wait.
- negative control: lanes that can start immediately report no wait, so a
predicate hardcoded to true would fail.
- ordinary lane-local saturation still reports a wait (pre-existing behaviour).
Mutation-verified: reverting to the lane-local predicate fails 3 tests with
'expected false to be true'. 182 pass across all affected suites.
Refs: openclaw#98813
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(queue): make rejected publication a no-op; always tear down the group on hooks-off
Round-5 implementation review (fiducian-spencer-001, CHANGES REQUESTED) found
one real atomicity bug and two test gaps. Both bugs are fixed and both are now
mutation-guarded.
BLOCKER — rejected publishLaneConfiguration left lane maxima mutated.
Phase 1 widened lanes, then setCommandLaneGroup could throw (e.g.
sum(reservations) > budget) with no rollback. No drain ran, so the old test's
activeCount === 0 assertion passed — but the lane sat at the new width governed
by NO group, and the next unrelated drain trigger would dispatch its preserved
queue ungoverned. The function comment promised exactly what the code did not
do. Validation is now a distinct phase 0 over every group spec before anything
is mutated; validateCommandLaneGroupSpec/installCommandLaneGroup split out so
setCommandLaneGroup and the transaction share one validation path.
BUG — hooks-off skipped group teardown when the grouped lane was suspended.
applyGatewayLaneConcurrency published only when the lane map was non-empty. With
hooks off, cron-nested is the only lane that can enter it, so a suspended
cron-nested left the map empty and clearGroups was never published. A previously
installed cron-hooks group survived, and the member resumed still paying a
reservation for a hook lane receiving no work. Now publishes whenever hooks are
disabled, regardless of the lane map.
Also (review item 4): a lane may now belong to at most one group.
installCommandLaneGroup removes it from any prior owner's members, which
otherwise kept counting its active tasks toward a budget it had left. Not
reachable with the single default group, but this is a public API.
Tests: 3 new. Mutation-verified —
- validating during install instead of before: 'expected 8 to be +0'
- restoring the non-empty-lane-map guard: 'expected cron-hooks to be undefined'
(this one initially SURVIVED; the first version of the teardown test never
simulated suspension, so it could not see the bug. Now seeds a cleared lane
resume and publishes via the gatewayStart path.)
- hooks-off now proves it DRAINS the work its teardown releases, not just that
membership was deleted.
Test teardown fixed: resetAllLanes preserves queued entries by design, so work
on a lane that never opened never settles. clearCommandLane rejects it instead.
Typecheck clean (tsgo core + core test, exit 0).
Refs: openclaw#98813
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(queue): make setCommandLaneGroup self-waking; guard the clearGroups+invalid case
Round-5 implementation review from costaff-lapclaw-001 independently found the
same two bugs fiducian did (rejected-publish partial mutation, and hooks-off
teardown skipped when the grouped lane is suspended) — both already fixed in
0eb96a7. It raised two things fiducian did not:
1. The exported setCommandLaneGroup primitive was not self-waking. Replacing a
group can FREE capacity — wider budget, dropped reservation, removed member —
and queued members must not sit behind capacity that is already available.
publishLaneConfiguration drains at commit, but the bare primitive is exported
and its "replace" semantics silently stranded members until an unrelated
poke. Now drains the union of previous and next members.
2. clearGroups combined with an invalid replacement was the worst case: the old
group could be removed before the new one threw, leaving BOTH lane width and
group membership partially committed. Phase 0 validation already ran before
the clear after 0eb96a7, but nothing pinned it.
Also documents a limitation costaff noted: reservations are not validated
against the member's own maxConcurrent, because lane widths and group
definitions are published together and the width may not be applied yet at
validation time. A too-large reservation is accepted but partly unusable.
Tests: 2 new. Mutation-verified —
- removing the self-wake: 'expected 2 to be 5'
- validating during install instead of before the clear:
'expected undefined to be cron-hooks' (the existing group torn down by a
rejected replacement — costaff's exact worst case)
Reviewer agreement on the rest: admission arithmetic sound for the concrete
config, clearCommandLane correctly not wired (frees no active capacity), the
peak-occupancy publication test is the right shape, the static deadlock deny set
matches the known synchronous-wait lanes, and shouldNoteLaneWait's export is the
right trade over asserting a copied closure.
Typecheck clean (tsgo core + core test, exit 0).
Refs: openclaw#98813
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* style(queue): satisfy oxlint in the capacity-group tests
`check-lint` was failing on 21 errors across the four new suites:
- 17 `curly`: single-statement `for`/`for-of` bodies without braces.
- 4 `no-promise-executor-return`: `new Promise((resolve) => setTimeout(resolve, 0))`
implicitly returns the Timeout handle from the executor. Rewritten to the
braced form already used ~85 times elsewhere in the repo, e.g.
`src/plugins/install-paths.test.ts:41`.
No behaviour change; the suites pass unchanged.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* refactor(queue): split capacity groups and shared state out of command-queue.ts
`check-lint` was failing `max-lines` on src/process/command-queue.ts: 918
counted lines against the repo cap of 700. The file was already only 51 lines
under the cap before this branch, so the new capacity-group code could not fit
in it.
Two pure moves, no logic change:
- `command-queue.state.ts` — the globalThis-backed queue singleton
(`getQueueState`, same `Symbol.for` key), `normalizeLane`, and the
`QueueEntry` / `LaneState` / `ActiveTaskWaiter` / `CommandLaneTaskMarker`
types. Lets the group policy read lane state without importing the queue.
- `command-queue.capacity-groups.ts` — the group registry, eligibility policy,
spec validation, install, and the block-reason computation.
The four near-identical "drain these member lanes" loops collapse into one
`drainMembers` helper. It keeps the load-bearing part of each original: the
lane is looked up rather than created, because `drainLane` goes through
`getLaneState` and would resurrect a scoped lane that
`retireIdleScopedCommandLane` had just removed. The one loop that additionally
tested `maxConcurrent > 0` loses that check, which was an optimisation only —
a zero-width lane's pump admits nothing.
The dependency on `drainLane` is passed in as a parameter rather than imported,
so the new modules stay acyclic; `setCommandLaneGroup`, `clearCommandLaneGroup`
and `drainCommandLaneGroup` remain exported from command-queue.ts as thin
wrappers, and every previously exported symbol is still exported from there.
command-queue.ts is now 676 counted lines; all three modules are under the cap.
Verified: `tsgo:core`, `tsgo:core:test`, `oxfmt --check`, `oxlint` all clean;
1007 tests across the 43 suites that touch command-queue pass.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(queue): allow concurrent hook dispatch within cron budget
* test(gateway): prove hook burst concurrency stays bounded
* refactor(queue): keep capacity groups internal
* test(gateway): isolate steady-state hook admission
* fix(gateway): close hook lane on disable
* fix(gateway): retarget suspended hook resumes
* fix(gateway): restore hook group before lane resume
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Vincent Koc <vincentkoc@ieee.org>
* fix(process): clarify command lane timeout cause
* fix(process): complete timeout cause formatting
---------
Co-authored-by: Peter Steinberger <steipete@gmail.com>
* feat(onboard): guided CLI onboarding with live AI verification and classic fallback
Interactive `openclaw onboard` (and bare `openclaw` on a fresh install) now
runs a guided flow with macOS-app parity: detect existing AI access, live-test
candidates with a real completion before persisting anything, walk down the
ladder on failure with mapped reasons, and offer verified manual API-key entry
from installed provider manifests (masked input). In-flow escapes: classic
wizard, Crestodian chat, skip-AI. Classic wizard gains an optional post-auth
live verification step. `--classic`, `--modern`, and `--non-interactive`
contracts unchanged. Docs corrected for post-#99935 routing.
Closes#101851
* improve(onboard): quiet probe diagnostics in wizard TTY, carry risk ack into classic escape
Candidate live-tests during guided setup are probes: rename their run id and
lane to the existing probe conventions (logging/subsystem.ts console
suppression, command-queue quiet probe lanes) so expected failures stop
leaking raw diagnostics into the Clack UI; file diagnostics unchanged. The
classic-wizard escape now passes the already-collected risk acknowledgement
through instead of re-prompting in the same session.
* fix(onboard): quiet the session-derived setup-inference probe lane too
The live-test run enqueues on two lanes: the explicit probe lane and one
derived from its temp session key. Extend the shared quiet-probe predicate to
cover the derived lane so a failing candidate cannot leak lane-task
diagnostics into the wizard TTY.
* improve(onboard): suppress subsystem console output during wizard live tests
Provider-transport subsystem loggers (model-fetch start/response, transport
errors) carry no run id, so probe suppression cannot catch them and a failing
candidate printed raw log lines into the Clack TTY. Reuse the TUI console
subsystem-filter seam via a finally-safe scoped helper around guided
activation and the classic live-verify; file logging is unchanged and the
gateway (macOS app) surface is unaffected.
* fix(onboard): never auto-replace a configured model when its live check fails
The re-run verification probe executes outside the configured workspace (setup
never runs workspace plugins), so a workspace-backed current model can fail
the check while working fine in the agent. Stop the auto ladder on an
existing-model failure and hand the decision to the manual stage instead of
silently persisting a different candidate as the default. Docs note the
fail-safe and the workspace caveat.
* feat(onboard): two-way switching between Crestodian chat and the menu wizards
From the chat, `open setup wizard`, `open classic wizard`, and `open channel
wizard for <channel>` hand off to the guided flow, the classic wizard, or the
masked `channels add` wizard after the chat TUI tears down (mirrors the
open-tui handoff; gateway surface gets a text pointer instead). The hosted
channel wizard no longer dead-ends at sensitive steps — it offers the switch
and remembers the channel. New read-only `channel info <channel>` operation
and ring-zero action surface label, blurb, configured state, and the real
docs URL from channel-setup discovery so the assistant can explain Slack or
Telegram prerequisites instead of guessing; both prompts instruct it to use
them. `channels add --channel <id>` now preselects the channel. Docs cover
the interchangeable flows.
* fix(onboard): avoid param reassignment in open-setup handoff
* improve(onboard): separate ask-about vs connect intent in channel prompt guidance
Live test showed the agent detouring an explicit connect request through
channel_info because the guidance said to consult it first. Both prompts now
distinguish asking about a channel (channel info + docs link) from asking to
connect (connect right away).
* fix(channels): mark channel token entry as sensitive input
The shared single-token prompt lacked sensitive:true, so terminal wizards
echoed pasted channel tokens and the Crestodian chat bridge (which refuses
plain-text secrets based on this flag) hosted the Telegram token step in
visible chat. Found live-testing the chat-to-wizard switch; pre-existing on
main but load-bearing for the masked-wizard contract this PR documents.
* fix(onboard): restore terminal state around the guided flow's TUI launch
Mirror the classic finalize handoff so the chat TUI never inherits the wizard
prompter's raw/paused terminal state on the default first-run path.
* fix(channels): type the token prompter mock for the sensitive-flag assertion
* fix(gateway): map the TUI-only open-setup action to none for app clients
Engine-side surface gating already prevents open-setup replies on the gateway
surface; this keeps the client-visible action enum stable even if that gate
ever regresses. (Reviewed with the switching round; missed in its commit.)
* docs: regenerate docs map for onboarding page changes
Ensure deferred context-engine maintenance rejects cleanly when the gateway command queue is draining, including coalesced active-run requests. This prevents budget compaction from treating an unscheduled deferred maintenance run as successful and leaving the context engine alive.
Verification:
- pnpm exec oxfmt --check --threads=1 src/process/command-queue.ts src/agents/pi-embedded-runner/compact.queued.ts src/agents/pi-embedded-runner/context-engine-maintenance.ts src/agents/pi-embedded-runner/context-engine-maintenance.test.ts
- pnpm test src/auto-reply/reply/agent-runner-memory.test.ts src/agents/pi-embedded-runner/compact.hooks.test.ts src/agents/pi-embedded-runner/context-engine-maintenance.test.ts src/tasks/task-flow-registry.store.test.ts src/auto-reply/reply/commands-compact.test.ts src/agents/pi-embedded-runner/compact-reasons.test.ts
- .agents/skills/autoreview/scripts/autoreview --mode branch --base origin/main
- GitHub Actions CI run 26475226442: relevant Node/Linux, lint, type, security, CodeQL, OpenGrep, Socket, Real behavior proof, and build jobs passed; Windows job failed before tests due current runner image Node 22.19.0 vs required 24.x, matching current main infra failure.
After a SIGUSR1 in-process restart following an npm upgrade from v2026.4.2
to v2026.4.5, the globalThis singleton created by the old code version
lacks the activeTaskWaiters field added in v2026.4.5. resolveGlobalSingleton
returns the stale object as-is, causing notifyActiveTaskWaiters() to call
Array.from(undefined) and crash the gateway in a loop.
Add a schema migration step in getQueueState() that patches the missing
field on legacy singleton objects. Add a regression test that plants a
v2026.4.2-shaped state object and verifies resetAllLanes() and
waitForActiveTasks() succeed without throwing.
Fixes#61905
- reject new lane enqueues once gateway drain begins
- always reset lane draining state and isolate onWait callback failures
- persist per-session abort cutoff and skip stale queued messages
- avoid false 600s agentTurn timeout in isolated cron jobs
Fixes#27407Fixes#27332Fixes#27427
Co-authored-by: Kevin Shenghui <shenghuikevin@github.com>
Co-authored-by: zjmy <zhangjunmengyang@gmail.com>
Co-authored-by: suko <miha.sukic@gmail.com>
Replace bare `new Error("Command lane cleared")` with a dedicated
`CommandLaneClearedError` class so callers that fire-and-forget
enqueued tasks can catch this specific type and avoid surfacing
unhandled rejection warnings.
clearCommandLane() was truncating the queue array without calling
resolve/reject on pending entries, causing never-settling promises
and memory leaks when upstream callers await enqueueCommandInLane().
Splice entries and reject each before clearing so callers can handle
the cancellation gracefully.
* fix(gateway): drain active turns before restart to prevent message loss
On SIGUSR1 restart, the gateway now waits up to 30s for in-flight agent
turns to complete before tearing down the server. This prevents buffered
messages from being dropped when config.patch or update triggers a restart
while agents are mid-turn.
Changes:
- command-queue.ts: add getActiveTaskCount() and waitForActiveTasks()
helpers to track and wait on active lane tasks
- run-loop.ts: on restart signal, drain active tasks before server.close()
with a 30s timeout; extend force-exit timer accordingly
- command-queue.test.ts: update imports for new exports
Fixes#13883
* fix(queue): snapshot active tasks for restart drain
---------
Co-authored-by: Elonito <0xRaini@users.noreply.github.com>
Co-authored-by: Tak Hoffman <781889+Takhoffman@users.noreply.github.com>
Peter Steinberger