* [AI] fix(diagnostics-otel): keep telemetry exporting across in-process restarts
Owned diagnostics-otel generations previously registered their OpenTelemetry
providers globally and only shut the providers down on stop. The pinned
sdk-node registers globals without override and never unregisters them, so a
second in-process generation (config-watcher reload, OPENCLAW_NO_RESPAWN=1,
containers) kept exporting through the first, already-shutdown providers and
telemetry silently stopped.
Switch owned mode to private BasicTracerProvider + MeterProvider instances
whose handles are injected directly into the existing recorder runtime, and
keep the preloaded (OPENCLAW_OTEL_PRELOADED=1) mode on the host's global
providers without ever registering or replacing globals. Resource detection
honors the pinned NodeSDK OTEL_NODE_RESOURCE_DETECTORS contract (unset
defaults to env+process+host; none/subset/all are respected), and the unused
@opentelemetry/sdk-node dependency is removed with a minimal lockfile update.
Adds a real-SDK two-generation restart regression, real-SDK resource-detector
selection coverage, migrates the unit mocks to the provider lifecycle, and
updates the OpenTelemetry docs.
Fixes#119997
Co-Authored-By: glm-5.2 <noreply@anthropic.com>
* [AI] test(diagnostics-otel): restore OpenTelemetry globals after restart test
The restart regression disables global context, metrics, propagation, and
trace APIs and forces OPENCLAW_OTEL_PRELOADED=0, but its cleanup only reset
diagnostic events, so a later test in the same worker could inherit no-op
providers. Snapshot the prior global registrations and preloaded env at
module load, and re-register/restore them in afterEach, matching the
existing exporter-health integration test pattern.
Co-Authored-By: glm-5.2 <noreply@anthropic.com>
* [AI] fix(diagnostics-otel): reconcile private providers with OTEL_SDK_DISABLED
Rebase onto main after #119961 (honor OTEL_SDK_DISABLED) changed the
disabled admission path to return before SDK construction. Restore the
removed getBooleanFromEnv import, drop the stale ownedNodeSdkDisabled
guard, and update disabled-mode tests to the new semantics (all routes
off, disabled runtime registered) plus the private-provider signal
path for the integration suite.
Related to #119997
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(diagnostics-otel): isolate restartable provider generations
* test(diagnostics-otel): prove same-pid generation routing
* test(diagnostics-otel): use supported watcher reload mode
* test(diagnostics-otel): satisfy ownership proof gates
* test(diagnostics-otel): use managed proof cleanup
* test(diagnostics-otel): verify injected trace ancestry
---------
Co-authored-by: glm-5.2 <noreply@anthropic.com>
Co-authored-by: Vincent Koc <vincentkoc@ieee.org>
Emit usage once at the shared plugin completion boundary and expose host-derived plugin identity only on OTel usage spans. Preserve aggregate metric labels and cover the redacted Gateway path end to end.
Refs #98968
Co-authored-by: Vincent Koc <vincentkoc@ieee.org>
Co-authored-by: ZengWen-DT <ceng.wen@xydigit.com>
* fix(diagnostics-otel): harness.run root span adopts diagnostic trace context
Fall back to internalOrTrustedParentContext when there is no active
tracked parent, so the openclaw.harness.run root span (and its whole
OTel subtree) adopts the diagnostic traceId instead of minting a new
OTel root. This lets spans propagated via the diagnostic traceparent
(e.g. a2a-gateway a2a.call / downstream a2a.task) share one trace with
the run internals.
* fix(diagnostics-otel): propagate exported span context
Co-authored-by: HughTang <44694613+HughTang@users.noreply.github.com>
* fix(diagnostics): break propagation import cycle
* fix(diagnostics): preserve trace propagation contracts
---------
Co-authored-by: Vincent Koc <vincentkoc@ieee.org>
Keep shared endpoint routing signal-aware while preserving explicit signal-specific URLs exactly. Cover the resolver through mocked constructors and real OTLP/HTTP trace, metric, and log exporters.
Co-authored-by: Vincent Koc <vincentkoc@ieee.org>
Diagnostics exporters are plugin services that only ever start inside the
Gateway process, so embedded `openclaw agent --local` turns emitted
diagnostic events in the CLI process with no subscriber: zero spans,
metrics, or logs reached the OTLP collector while identical Gateway-
dispatched turns exported fine.
The CLI now starts the `diagnostics-otel` service around embedded agent
runs via a scoped, non-activating plugin load that honors the same
plugin-enablement config as the Gateway, and flushes before exit: drain
the async diagnostic-event queue, then shut the SDK down (force flush).
Each step is bounded separately (5s drain, 10s flush) so an unreachable
collector cannot hold the CLI open, and a stalled drain cannot consume
the flush window and discard telemetry that was already buffered.
When OTel is configured but the plugin is absent, the run now says so
instead of exporting nothing with nothing explaining why.
`diagnostics-prometheus` stays Gateway-only: it is a pull-based scrape
server that is useless in a short-lived process and would race the
Gateway for its port. No new config surface; configs without OTel enabled
short-circuit before any plugin load.
Co-authored-by: Alex Knight <15041791+amknight@users.noreply.github.com>
* fix(diagnostics-otel): keep parent span links across long turns
Retained trusted span contexts translate a completed lifecycle span's
diagnostic span id to the real OTel span context. That table was evicted
5s after run.completed, so children of a longer turn missed their parent
and OTel minted a fresh trace id for each one, silently splitting a turn
into single-span traces.
Retention is now identity-based and lives for the service lifetime,
bounded by MAX_RETAINED_TRUSTED_SPAN_CONTEXTS, which removes the timing
race and the drain/timeout cleanup subsystem.
Also nests openclaw.exec under its run: the exec event now carries a
trusted trace context and the recorder resolves the ambient run scope.
* fix(diagnostics-otel): key retained span contexts by diagnostic trace id
Retained lifecycle span contexts were stored under the span's OTel trace id
but looked up with the event's diagnostic trace id. Those are different id
spaces, so every post-completion parent lookup missed and each straggler
span started a brand-new trace.
The mocked suite could not catch this: its tracer reports the same trace id
the test feeds in, collapsing both id spaces into one value. Adds a boundary
test that runs the real OTel SDK and OTLP/protobuf exporter against a local
receiver and asserts on exported span bytes.
* fix(diagnostics-otel): harden span-linking contract after review
Review findings on the parent-span linking fix:
- completeTrackedLifecycleSpan now takes the DiagnosticTraceContext instead of a
bare trace id beside a span id. Two adjacent hex strings let a transposition
compile clean and silently reinstate split traces; DiagnosticTraceContext is
the only shape an OTel SpanContext cannot satisfy (traceFlags string vs number).
- internalOrTrustedTraceContext composes normalizedTrustedTraceContext rather
than spelling the trust predicate a third time.
- Corrected comments that claimed OTel ids never equal diagnostic ids. Spans
parented from an upstream traceparent do adopt the diagnostic trace id; the
removed guard only ever misfired on root lifecycle spans.
- Exec comment now states the real parent: the openclaw harness opens no run
scope, so exec nests under openclaw.harness.run.
Test gaps closed:
- The core emitter switch was untested; reverting it kept every test green.
The pty-fallback test now asserts the emitted metadata.
- The boundary test covered only run.completed of three lifecycle owners; it now
covers harness.run.completed and message.processed too, each with a straggler.
- Dropped the service-context cast for the typed helper, which caught a bogus
context.assembled field, and moved teardown into afterEach so a failed
assertion cannot leak the real NodeSDK across files.
- Scoped the one-trace assertion to this turn's spans; faked Date so a lazy
Date.now expiry cannot pass.
Docs: exec and tool spans now listed as staying on an upstream request trace.
* fix(diagnostics-otel): keep late children on the trace on every terminator
Native review found three correctness gaps in the previous commit:
- Exec spans could be exported naming a parent span id nothing ever emits. The
ambient lookup fell through to a remote-parent fallback built from diagnostic
ids, which is a regression from the previous clean-root behavior and breaks
waterfalls and parent-id-keyed backends. Exec now resolves only spans this
process exported and stays a root on a miss; message spans keep the remote
fallback because their context can come from an inbound traceparent.
- harness.run.error never retained its span context, so an aborted turn still
split. For the openclaw harness that span is the only ancestor a late child
has, and aborted turns emit no run.completed.
- completeTrackedLifecycleSpan no longer takes a redundant spanId.
The boundary test now runs through the OPENCLAW_OTEL_PRELOADED seam with an
in-memory exporter and releases the global tracer provider in teardown. A
NodeSDK cannot be unregistered, so the old version would have made any later
real-SDK test silently export nothing. That also deletes the hand-rolled OTLP
protobuf decoder. Added guards for the aborted-turn path and for exec staying
parentless; both are mutation-verified.
* fix(diagnostics-otel): age retained parent contexts out after 10 minutes
Retention had no time bound at all, so a command backgrounded by a turn could
finish hours later and still attach to a parent that ended long before. Backends
derive trace latency from the span envelope, so a 30s turn rendered as hours with
a child extending past its parent.
The horizon is checked lazily on lookup rather than on a timer: a timer firing
mid-turn is exactly what silently split traces before. Past the horizon a
straggler starts its own trace, which only loses causality for a span that would
have skewed its parent's duration anyway.
The two bounds now pin each other - shrinking the horizon to the old 5s window
fails the original regression tests, and removing it fails the new horizon test.
* fix(diagnostics-otel): retain parent contexts until capacity
* docs: correct retired cron/audit config keys, cron failure-alert default, memory recall default, and tool-search telemetry claims
- configuration-reference: cron block documented cron.webhook and cron.failureDestination, both retired by the config-surface reduction tranches (58452de711, edecdbd05e); the cron schema is strict so a copied snippet is rejected. Document only the live keys and note the doctor --fix migrations.
- configuration-reference: root-level audit block is retired; canonical path is logging.audit (src/config/zod-schema.root-shape.ts).
- configuration-reference: cron.failureAlert.after default is 2, not 3 (src/cron/service/failure-alerts.ts).
- memory-config: rememberAcrossConversations defaults on for personal installs (packages/memory-host-sdk/src/host/config-utils.ts), matching the canonical table earlier in the page.
- tool-search: telemetry records catalogSize, per-source counts, and search/describe/call counts, and only on tool_search_code results. No byte accounting exists in the runtime.
* docs: retire remaining references to removed cron, audit, and logging config keys
Sweep follow-up to the previous commit, covering the same bug class in the pages that still contradicted it.
- cron-jobs/cli-cron: global cron.failureDestination is retired; the destination fields now live on cron.failureAlert (src/config/zod-schema.root-shape.ts, merged by legacy-config-migrations.runtime.retired.ts:379). Per-job delivery.failureDestination bullets left intact.
- gateway/audit, cli/audit, gateway/protocol: root-level audit.* is retired; canonical path is logging.audit.*.
- logging: logging.redactSensitive is retired (dead-config-keys.test.ts:198; removed by legacy-config-migrations.runtime.tier-eval.ts:12). resolveConfigRedaction hardcodes DEFAULT_REDACT_MODE = tools, so redaction is unconditional. Also documented that redactPatterns replaces the defaults on the log path (redact.ts:419) while tool payloads always merge them.
- logging: consoleStyle accepts only pretty|json (zod-schema.root-shape.ts:106); compact remains the automatic non-TTY rendering style (logging/console.ts:40) but is no longer settable, and doctor maps a stored one to pretty.
- security: security --fix no longer touches redaction and the logging.redact_off audit check is retired (src/security/audit-loopback-logging.test.ts asserts it never fires).
* chore(docs): regenerate docs map after retired-key cleanup
* improve(diagnostics): surface run startup phases as session-correlated events
The embedded runner reports ordered startup milestones (workspace, auth,
context_engine, ..., model_call_started) through the private
onExecutionPhase callback, so they reach typing indicators and cron
watchdogs but never the diagnostic bus. External status surfaces that
already consume run.attempt / run.progress / model.call.* cannot see
where a turn is during startup — exactly the window that matters when a
run hangs before its first model call.
Wrap params.onExecutionPhase once at the runner entry so every phase
transition also emits a session-correlated run.execution_phase event
(runId, sessionId, sessionKey, closed EmbeddedAgentExecutionPhase +
provider/model/tool context). The event rides the async diagnostic lane
like the other high-frequency lifecycle events, so ordering against
model.call.* is preserved and no subscriber work runs on the hot path.
The wrapper tracks session rotation via onSessionIdChanged so
post-compaction phases carry the current session id. Stability records
project the milestone into their dedicated phase/provider/model fields.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(diagnostics): preserve trusted phase observability
* chore(plugin-sdk): refresh diagnostic API baseline
* fix(diagnostics): publish execution phases to plugins
* refactor(logging): keep public diagnostic recorder path
* fix(plugin-sdk): keep deprecated budget exact
* fix(plugin-sdk): refresh diagnostic surface budget
* fix(diagnostics): preserve execution phase coverage
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Peter Steinberger <steipete@gmail.com>
* fix: show exported tool results in trace viewers
* fix(diagnostics-otel): emit semconv response key and execute_tool identity so trace viewers show tool results
---------
Co-authored-by: Alex Knight <15041791+amknight@users.noreply.github.com>
Adds stdout and both-mode diagnostics OTEL log export, with focused QA Lab smoke coverage and docs/config updates.
Prepared head SHA: efa2ef07ab
Verification: CI 27808480969 passed for the prepared head.
Reviewed-by: @jesse-merhi
Classify owned silent model calls as long-running until the abort threshold while preserving stalled handling for ownerless stale activity, with diagnostics tests and docs.
diagnostics.otel.captureContent.{toolInputs,toolOutputs} were documented
and config-wired but never produced any span content. Emit tool args and
results over the trusted private-data diagnostic channel (mirroring the
model-content path), and have the OTel exporter bound/redact/truncate them
before span export. Raw tool content never rides the public event bus.
Scope: core embedded-runner tool path (canonical producer). Codex
(async-batched) and Claude CLI remain follow-ups tracked by the issue.
Refs #77391
Count streamed text/thinking/tool-call deltas incrementally in model diagnostics instead of repeatedly estimating full event payloads. Updates diagnostics docs and OTEL wording for the new response byte baseline.\n\nVerification: node scripts/run-vitest.mjs run src/agents/embedded-agent-runner/run/attempt.model-diagnostic-events.test.ts; GitHub Actions CI run 27064304709; CodeQL run 27064304710; OpenGrep PR Diff run 27064304716.
Count model stream diagnostic response bytes from snapshotless stream chunks, excluding accumulated partial snapshots on delta events. This avoids repeatedly serializing answer-so-far snapshots during streamed model calls and updates OTEL/docs wording for the new metric baseline.
Refs #86599.
Co-authored-by: Vincent Koc <vincentkoc@ieee.org>
Recover idle queued sessions whose diagnostic activity retained stale ownerless model or tool calls by classifying them as recoverable session.stuck after the usual recovery gates. Yield the event loop before stale session-lock process inspection so sync process lookup cannot monopolize lock contention paths.
Docs now describe the widened session.stuck telemetry contract for recoverable stale bookkeeping, including ownerless activity. Thanks @samuelsoaress.
Refs #84903.
Co-authored-by: samuelsoaress <samuelsoares177778@gmail.com>
Summary:
- This replacement PR adds inbound delivery diagnostic events, gateway status counters and warnings, transport ... ut, Prometheus/OpenTelemetry metrics, docs, changelog, and regression coverage for gateway delivery health.
- Reproducibility: no. high-confidence live reproduction of the original Feishu failure was run here. Source i ... ch/turn telemetry, and the source PR supplies after-fix live output for the connected WebChat gateway path.
Automerge notes:
- PR branch already contained follow-up commit before automerge: fix(types): restore PR conflict resolution type checks
Validation:
- ClawSweeper review passed for head 6ffe08a9c7.
- Required merge gates passed before the squash merge.
Prepared head SHA: 6ffe08a9c7
Review: https://github.com/openclaw/openclaw/pull/85016#issuecomment-4510224436
Co-authored-by: Andi Liao <liaoandi95@gmail.com>
Co-authored-by: clawsweeper[bot] <274271284+clawsweeper[bot]@users.noreply.github.com>
Approved-by: takhoffman
Co-authored-by: takhoffman <781889+takhoffman@users.noreply.github.com>
Adds bounded Talk lifecycle/audio diagnostics and session recovery metrics for OTEL, Prometheus, and stability snapshots after the Talk SDK/session refactor. Includes changelog/docs updates and Testbox/live proof.
SunnyShu