Emit usage once at the shared plugin completion boundary and expose host-derived plugin identity only on OTel usage spans. Preserve aggregate metric labels and cover the redacted Gateway path end to end.
Refs #98968
Co-authored-by: Vincent Koc <vincentkoc@ieee.org>
Co-authored-by: ZengWen-DT <ceng.wen@xydigit.com>
* fix(diagnostics-otel): harness.run root span adopts diagnostic trace context
Fall back to internalOrTrustedParentContext when there is no active
tracked parent, so the openclaw.harness.run root span (and its whole
OTel subtree) adopts the diagnostic traceId instead of minting a new
OTel root. This lets spans propagated via the diagnostic traceparent
(e.g. a2a-gateway a2a.call / downstream a2a.task) share one trace with
the run internals.
* fix(diagnostics-otel): propagate exported span context
Co-authored-by: HughTang <44694613+HughTang@users.noreply.github.com>
* fix(diagnostics): break propagation import cycle
* fix(diagnostics): preserve trace propagation contracts
---------
Co-authored-by: Vincent Koc <vincentkoc@ieee.org>
Keep shared endpoint routing signal-aware while preserving explicit signal-specific URLs exactly. Cover the resolver through mocked constructors and real OTLP/HTTP trace, metric, and log exporters.
Co-authored-by: Vincent Koc <vincentkoc@ieee.org>
Diagnostics exporters are plugin services that only ever start inside the
Gateway process, so embedded `openclaw agent --local` turns emitted
diagnostic events in the CLI process with no subscriber: zero spans,
metrics, or logs reached the OTLP collector while identical Gateway-
dispatched turns exported fine.
The CLI now starts the `diagnostics-otel` service around embedded agent
runs via a scoped, non-activating plugin load that honors the same
plugin-enablement config as the Gateway, and flushes before exit: drain
the async diagnostic-event queue, then shut the SDK down (force flush).
Each step is bounded separately (5s drain, 10s flush) so an unreachable
collector cannot hold the CLI open, and a stalled drain cannot consume
the flush window and discard telemetry that was already buffered.
When OTel is configured but the plugin is absent, the run now says so
instead of exporting nothing with nothing explaining why.
`diagnostics-prometheus` stays Gateway-only: it is a pull-based scrape
server that is useless in a short-lived process and would race the
Gateway for its port. No new config surface; configs without OTel enabled
short-circuit before any plugin load.
Co-authored-by: Alex Knight <15041791+amknight@users.noreply.github.com>
* feat(gateway): advertise chat attachment limits on hello-ok
Clients had no way to learn the gateway attachment ceilings, so external
clients hardcoded guesses that drifted from server enforcement. Publish the
two unconditional decoded-size ceilings on hello-ok policy.attachments from
one shared resolver so advertised values cannot drift from the parser.
MIME acceptance and per-message counts stay server-side: they depend on the
entrypoint, the resolved model, and payload sniffing, so they cannot be stated
once per connection.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 17d6c355-8948-4b48-a936-e08b1c8806ef
* feat(gateway): advertise chat attachment limits on hello-ok
---------
Co-authored-by: Omar Shahine <10343873+omarshahine@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: clawsweeper <274271284+clawsweeper[bot]@users.noreply.github.com>
Copilot-Session: 17d6c355-8948-4b48-a936-e08b1c8806ef
* docs(gateway): document llamacpp toolSchemaProfile for custom llama.cpp endpoints
Custom openai-completions providers pointed at a llama.cpp / llama-server
endpoint bypass the GBNF-safe schema normalization that built-in llama-cpp,
ollama, and lmstudio providers apply automatically. The toolSchemaProfile:
"llamacpp" switch activates it, but the supported value was never named in
the docs, so users hit llama.cpp 400 'failed to parse grammar' on tools with
large minLength/maxLength (e.g. cron trigger.script, maxLength 65536).
- Expand the toolSchemaProfile row to name the values recognized on a model's
compat (llamacpp, gemini) and when llamacpp must be set explicitly;
cross-reference the unsupportedToolSchemaKeywords escape hatch.
- Add a 'Local models (llama.cpp / llama-server)' provider example showing
the required compat.toolSchemaProfile: "llamacpp" on a custom endpoint.
Related to #117070
Co-Authored-By: iCodeMate <noreply@anthropic.com>
* docs(gateway): describe llamacpp profile as stripping, not clamping
Address review feedback on #117120: the fallback sentence claimed the
llamacpp profile "also clamps" constraints, but cleanSchemaForLlamacppGbnf
only removes `pattern` outright and drops `maxLength` at or above the
2000-repetition threshold (LLAMACPP_GBNF_MAX_REPETITION_THRESHOLD). It
never clamps a value, and small bounds like `minLength: 1` are preserved
(confirmed by the projection test at src/plugin-sdk/provider-tools.test.ts).
Reword the escape-hatch sentence to describe the implemented stripping
behavior and contrast it with the unconditional keyword-list drops, so
operators do not expect a preserved bounded constraint.
No code or behavior changes; documentation only.
* docs(gateway): scope llamacpp profile description to its actual transformations
Address follow-up review on #117120: the toolSchemaProfile table row
described `llamacpp` as "rewriting tool schemas into the JSON Schema
subset llama.cpp can compile to GBNF," which overstates what
cleanSchemaForLlamacppGbnf does. The cleaner only removes `pattern`
outright and drops `maxLength` at or above the 2000-repetition threshold
(LLAMACPP_GBNF_MAX_REPETITION_THRESHOLD); it leaves every other keyword
and `minLength` untouched, and those are not guaranteed GBNF-safe.
Reword the row to state the exact transformation contract and explicitly
note it is a targeted compatibility cleaner, not a blanket GBNF-safety
guarantee. No code or behavior changes; documentation only.
Co-Authored-By: iCodeMate <noreply@anthropic.com>
* docs(gateway): tighten llama.cpp profile contract
* docs(gateway): order local model examples
---------
Co-authored-by: iCodeMate <noreply@anthropic.com>
Co-authored-by: Vincent Koc <vincentkoc@ieee.org>
* docs(config-channels): replace legacy allow:true with enabled:true for Discord, Slack, and Google Chat config examples
* docs(config-channels): repair channel configuration examples
Make the complete Discord, Slack, and Google Chat setup examples valid and keep Slack channel routing on stable IDs.
Co-authored-by: 赵旺0668001248 <zhao.wang1@xydigit.com>
---------
Co-authored-by: Peter Steinberger <steipete@gmail.com>
The gateway.reload documentation advertised four settings the shipped
build rejects. gateway.reload is a strictObject accepting only `mode`,
and `mode` is a union of "off" | "hybrid", so a config written from the
docs fails validation and the gateway fails closed on the unknown keys.
Retired but still documented:
- `debounceMs` and `deferralTimeoutMs` (RETIRED_TUNING_PATHS; stripped by
`openclaw doctor --fix` via stripRetiredTuningKnobs)
- `mode: "restart"` and `mode: "hot"` (mapped to "hybrid" by
`openclaw doctor --fix`)
Correct all four docs pages that still describe them, and say what
replaced them: debounce and active-work deferral now run behind built-in
defaults rather than config keys.
Refs #116973
* docs(automation): rename scheduled-tasks feature wording to Automations
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WhJ8EiMXue6ADLmHfb7FL6
* docs: regenerate docs map and add Automations glossary entries
* docs(templates): follow renamed automations-vs-heartbeat anchor
* docs(automation): fix markdown formatting drift
* docs(automation): teach the canonical automations tool and sync the copied heartbeat default
Review follow-ups: normal instructions use the automations tool with cron as
an explicit compatibility alias; every verbatim copy of the default heartbeat
prompt matches the new shipped text from the strings PR.
---------
Co-authored-by: Omar Shahine <10343873+omarshahine@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* fix(diagnostics-otel): keep parent span links across long turns
Retained trusted span contexts translate a completed lifecycle span's
diagnostic span id to the real OTel span context. That table was evicted
5s after run.completed, so children of a longer turn missed their parent
and OTel minted a fresh trace id for each one, silently splitting a turn
into single-span traces.
Retention is now identity-based and lives for the service lifetime,
bounded by MAX_RETAINED_TRUSTED_SPAN_CONTEXTS, which removes the timing
race and the drain/timeout cleanup subsystem.
Also nests openclaw.exec under its run: the exec event now carries a
trusted trace context and the recorder resolves the ambient run scope.
* fix(diagnostics-otel): key retained span contexts by diagnostic trace id
Retained lifecycle span contexts were stored under the span's OTel trace id
but looked up with the event's diagnostic trace id. Those are different id
spaces, so every post-completion parent lookup missed and each straggler
span started a brand-new trace.
The mocked suite could not catch this: its tracer reports the same trace id
the test feeds in, collapsing both id spaces into one value. Adds a boundary
test that runs the real OTel SDK and OTLP/protobuf exporter against a local
receiver and asserts on exported span bytes.
* fix(diagnostics-otel): harden span-linking contract after review
Review findings on the parent-span linking fix:
- completeTrackedLifecycleSpan now takes the DiagnosticTraceContext instead of a
bare trace id beside a span id. Two adjacent hex strings let a transposition
compile clean and silently reinstate split traces; DiagnosticTraceContext is
the only shape an OTel SpanContext cannot satisfy (traceFlags string vs number).
- internalOrTrustedTraceContext composes normalizedTrustedTraceContext rather
than spelling the trust predicate a third time.
- Corrected comments that claimed OTel ids never equal diagnostic ids. Spans
parented from an upstream traceparent do adopt the diagnostic trace id; the
removed guard only ever misfired on root lifecycle spans.
- Exec comment now states the real parent: the openclaw harness opens no run
scope, so exec nests under openclaw.harness.run.
Test gaps closed:
- The core emitter switch was untested; reverting it kept every test green.
The pty-fallback test now asserts the emitted metadata.
- The boundary test covered only run.completed of three lifecycle owners; it now
covers harness.run.completed and message.processed too, each with a straggler.
- Dropped the service-context cast for the typed helper, which caught a bogus
context.assembled field, and moved teardown into afterEach so a failed
assertion cannot leak the real NodeSDK across files.
- Scoped the one-trace assertion to this turn's spans; faked Date so a lazy
Date.now expiry cannot pass.
Docs: exec and tool spans now listed as staying on an upstream request trace.
* fix(diagnostics-otel): keep late children on the trace on every terminator
Native review found three correctness gaps in the previous commit:
- Exec spans could be exported naming a parent span id nothing ever emits. The
ambient lookup fell through to a remote-parent fallback built from diagnostic
ids, which is a regression from the previous clean-root behavior and breaks
waterfalls and parent-id-keyed backends. Exec now resolves only spans this
process exported and stays a root on a miss; message spans keep the remote
fallback because their context can come from an inbound traceparent.
- harness.run.error never retained its span context, so an aborted turn still
split. For the openclaw harness that span is the only ancestor a late child
has, and aborted turns emit no run.completed.
- completeTrackedLifecycleSpan no longer takes a redundant spanId.
The boundary test now runs through the OPENCLAW_OTEL_PRELOADED seam with an
in-memory exporter and releases the global tracer provider in teardown. A
NodeSDK cannot be unregistered, so the old version would have made any later
real-SDK test silently export nothing. That also deletes the hand-rolled OTLP
protobuf decoder. Added guards for the aborted-turn path and for exec staying
parentless; both are mutation-verified.
* fix(diagnostics-otel): age retained parent contexts out after 10 minutes
Retention had no time bound at all, so a command backgrounded by a turn could
finish hours later and still attach to a parent that ended long before. Backends
derive trace latency from the span envelope, so a 30s turn rendered as hours with
a child extending past its parent.
The horizon is checked lazily on lookup rather than on a timer: a timer firing
mid-turn is exactly what silently split traces before. Past the horizon a
straggler starts its own trace, which only loses causality for a span that would
have skewed its parent's duration anyway.
The two bounds now pin each other - shrinking the horizon to the old 5s window
fails the original regression tests, and removing it fails the new horizon test.
* fix(diagnostics-otel): retain parent contexts until capacity
The group-mentions ack gate returned false whenever a group did not require mentions, conflating group policy with whether this message mentioned the agent. Under the default scope, mentioning the agent in a group configured to answer everything produced no ack at all, so the user had no signal the turn was picked up until the reply arrived.
Behavior change on a default path: those groups now show the 👀 ack (and the lifecycle status reactions when messages.statusReactions.enabled is true) on mentions. Mention-required groups, group-all, unmentioned messages, and off/none are unchanged. The gate no longer reads requireMention, so the parameter is gone from the shared type and all channel call sites.
Proven live on Telegram across three configurations, including a no-mention control confirming group-mentions did not become group-all.
Vincent Koc