ewilliams28/openclaw
1
0
Fork 0
mirror of https://github.com/openclaw/openclaw.git synced 2026-08-12 21:53:00 -06:00
Code Issues Packages Projects Releases Wiki Activity

fix(diagnostics): honor signal-specific OTLP protocols (#119708)

Browse Source 
* fix(diagnostics): honor signal-specific OTLP protocols

* fix(docker): forward signal-specific OTLP protocols
This commit is contained in:
Vincent Koc
2026-08-06 05:39:26 +08:00
committed by GitHub
parent 7b8bba26c9
commit 92b4757227
10 changed files with 434 additions and 75 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docker-compose.yml
+3
View File
@@ -31,6 +31,9 @@ services:
OTEL_EXPORTER_OTLP_METRICS_ENDPOINT: ${OTEL_EXPORTER_OTLP_METRICS_ENDPOINT:-}
OTEL_EXPORTER_OTLP_LOGS_ENDPOINT: ${OTEL_EXPORTER_OTLP_LOGS_ENDPOINT:-}
OTEL_EXPORTER_OTLP_PROTOCOL: ${OTEL_EXPORTER_OTLP_PROTOCOL:-http/protobuf}
OTEL_EXPORTER_OTLP_TRACES_PROTOCOL: ${OTEL_EXPORTER_OTLP_TRACES_PROTOCOL:-}
OTEL_EXPORTER_OTLP_METRICS_PROTOCOL: ${OTEL_EXPORTER_OTLP_METRICS_PROTOCOL:-}
OTEL_EXPORTER_OTLP_LOGS_PROTOCOL: ${OTEL_EXPORTER_OTLP_LOGS_PROTOCOL:-}
OTEL_SERVICE_NAME: ${OTEL_SERVICE_NAME:-}
OTEL_SEMCONV_STABILITY_OPT_IN: ${OTEL_SEMCONV_STABILITY_OPT_IN:-}
OPENCLAW_OTEL_PRELOADED: ${OPENCLAW_OTEL_PRELOADED:-}
docs/gateway/configuration-reference.md
+2 -1
View File
@@ -1340,7 +1340,8 @@ writer is best-effort, not a lossless compliance archive.
- `OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental`: environment toggle for latest experimental GenAI inference span shape, including `{gen_ai.operation.name} {gen_ai.request.model}` span names, `CLIENT` span kind, and `gen_ai.provider.name` instead of legacy `gen_ai.system`. By default spans keep `openclaw.model.call` and `gen_ai.system` for compatibility; GenAI metrics use bounded semantic attributes.
- `OPENCLAW_OTEL_PRELOADED=1`: environment toggle for hosts that already registered a global OpenTelemetry SDK. OpenClaw then skips plugin-owned SDK startup/shutdown while keeping diagnostic listeners active.
- `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT`, `OTEL_EXPORTER_OTLP_METRICS_ENDPOINT`, and `OTEL_EXPORTER_OTLP_LOGS_ENDPOINT`: signal-specific endpoint env vars used when the matching config key is unset.
- `OTEL_EXPORTER_OTLP_PROTOCOL`: protocol fallback used only when `otel.protocol` is unset. Set it to `http/protobuf` or leave it unset; unsupported values are rejected when an OTLP signal is enabled and are not rewritten by Doctor.
- `OTEL_EXPORTER_OTLP_TRACES_PROTOCOL`, `OTEL_EXPORTER_OTLP_METRICS_PROTOCOL`, and `OTEL_EXPORTER_OTLP_LOGS_PROTOCOL`: signal-specific protocol fallbacks used when `otel.protocol` is unset. Each overrides `OTEL_EXPORTER_OTLP_PROTOCOL` for its signal.
- `OTEL_EXPORTER_OTLP_PROTOCOL`: shared protocol fallback used when neither `otel.protocol` nor the matching signal-specific variable is set. Only `http/protobuf` is supported. Protocol validation is isolated per signal, so an unsupported resolved value disables that signal's OTLP exporter without blocking supported sibling signals. Doctor does not rewrite environment variables.
- `cacheTrace.enabled`: log cache trace snapshots for embedded runs (default: `false`).
---
docs/gateway/opentelemetry.md
+9 -6
View File
@@ -68,11 +68,13 @@ section. For root or array includes, nested include chains, sibling overrides,
external include targets, or another ambiguous source, Doctor leaves the files
unchanged and lists the candidate source file or files to edit manually.
`OTEL_EXPORTER_OTLP_PROTOCOL` is a process-environment fallback used only when
`diagnostics.otel.protocol` is unset. Doctor does not rewrite process
environment variables. An unsupported fallback is rejected at runtime when an
OTLP signal is enabled; set it to `http/protobuf` or unset it. A stdout-only log
configuration does not use the OTLP transport and continues to work.
When `diagnostics.otel.protocol` is unset, each plugin-owned OTLP signal first
checks its nonblank `OTEL_EXPORTER_OTLP_*_PROTOCOL` value, then
`OTEL_EXPORTER_OTLP_PROTOCOL`, then defaults to `http/protobuf`. Doctor does not
rewrite process environment variables. An unsupported value disables only that
plugin-owned OTLP signal; supported sibling signals continue, as does the stdout
branch of `logsExporter: "both"`. Preloaded trace and metric SDKs own their own
transport selection and are not rejected by this plugin.
</Note>
## Signals exported
@@ -170,7 +172,8 @@ dashboards, alerts, and recording rules that query the old names.
| `OTEL_EXPORTER_OTLP_ENDPOINT` | Fallback for `diagnostics.otel.endpoint` when the config key is unset. |
| `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` / `OTEL_EXPORTER_OTLP_METRICS_ENDPOINT` / `OTEL_EXPORTER_OTLP_LOGS_ENDPOINT` | Signal-specific endpoint fallbacks used when the matching `diagnostics.otel.*Endpoint` config key is unset. Signal-specific config wins over signal-specific env, which wins over the shared endpoint. |
| `OTEL_SERVICE_NAME` | Fallback for `diagnostics.otel.serviceName` when the config key is unset. Default service name is `openclaw`. |
| `OTEL_EXPORTER_OTLP_PROTOCOL` | Process-environment fallback used only when `diagnostics.otel.protocol` is unset. Only `http/protobuf` enables OTLP export; unsupported values are rejected when an OTLP signal is enabled and are not rewritten by Doctor. |
| `OTEL_EXPORTER_OTLP_PROTOCOL` | Shared process-environment fallback used when `diagnostics.otel.protocol` and the signal-specific protocol variable are unset. Only `http/protobuf` enables a plugin-owned OTLP exporter. |
| `OTEL_EXPORTER_OTLP_TRACES_PROTOCOL` / `OTEL_EXPORTER_OTLP_METRICS_PROTOCOL` / `OTEL_EXPORTER_OTLP_LOGS_PROTOCOL` | Signal-specific protocol fallbacks used when `diagnostics.otel.protocol` is unset. A nonblank signal-specific value wins over the shared protocol value. Unsupported values disable only that plugin-owned OTLP signal. |
| `OTEL_SEMCONV_STABILITY_OPT_IN` | Set to `gen_ai_latest_experimental` to emit the latest GenAI inference span shape: `{gen_ai.operation.name} {gen_ai.request.model}` span names, `CLIENT` span kind, and `gen_ai.provider.name` instead of the legacy `gen_ai.system`. GenAI metrics always use bounded, low-cardinality attributes regardless. |
| `OPENCLAW_OTEL_PRELOADED` | Set to `1` when another preload or host process already registered the global OpenTelemetry SDK. The plugin then skips its own NodeSDK lifecycle but still wires diagnostic listeners and honors `traces`/`metrics`/`logs`. |
docs/install/docker.md
+2 -1
View File
@@ -173,7 +173,8 @@ Optional variables accepted by `scripts/docker/setup.sh` (and, for the gateway c
| `OPENCLAW_DISABLE_BUNDLED_SOURCE_OVERLAYS` | Disable bundled plugin source bind-mount overlays |
| `OTEL_EXPORTER_OTLP_ENDPOINT` | Shared OTLP/HTTP collector endpoint for OpenTelemetry export |
| `OTEL_EXPORTER_OTLP_*_ENDPOINT` | Signal-specific OTLP endpoints for traces, metrics, or logs |
| `OTEL_EXPORTER_OTLP_PROTOCOL` | OTLP protocol override. Only `http/protobuf` is supported today |
| `OTEL_EXPORTER_OTLP_PROTOCOL` | Shared OTLP protocol fallback. Only `http/protobuf` is supported today |
| `OTEL_EXPORTER_OTLP_*_PROTOCOL` | Signal-specific protocol fallback for traces, metrics, or logs; wins over the shared fallback |
| `OTEL_SERVICE_NAME` | Service name used for OpenTelemetry resources |
| `OTEL_SEMCONV_STABILITY_OPT_IN` | Opt in to latest experimental GenAI semantic attributes |
| `OPENCLAW_OTEL_PRELOADED` | Skip starting a second OpenTelemetry SDK when one is preloaded |
extensions/diagnostics-otel/src/service-constants.ts
+4
View File
@@ -33,6 +33,10 @@ export const OTEL_EXPORTER_OTLP_ENDPOINT_ENV = "OTEL_EXPORTER_OTLP_ENDPOINT";
export const OTEL_EXPORTER_OTLP_TRACES_ENDPOINT_ENV = "OTEL_EXPORTER_OTLP_TRACES_ENDPOINT";
export const OTEL_EXPORTER_OTLP_METRICS_ENDPOINT_ENV = "OTEL_EXPORTER_OTLP_METRICS_ENDPOINT";
export const OTEL_EXPORTER_OTLP_LOGS_ENDPOINT_ENV = "OTEL_EXPORTER_OTLP_LOGS_ENDPOINT";
export const OTEL_EXPORTER_OTLP_PROTOCOL_ENV = "OTEL_EXPORTER_OTLP_PROTOCOL";
export const OTEL_EXPORTER_OTLP_TRACES_PROTOCOL_ENV = "OTEL_EXPORTER_OTLP_TRACES_PROTOCOL";
export const OTEL_EXPORTER_OTLP_METRICS_PROTOCOL_ENV = "OTEL_EXPORTER_OTLP_METRICS_PROTOCOL";
export const OTEL_EXPORTER_OTLP_LOGS_PROTOCOL_ENV = "OTEL_EXPORTER_OTLP_LOGS_PROTOCOL";
export const OTEL_EXPORTER_OTLP_CERTIFICATE_ENV = "OTEL_EXPORTER_OTLP_CERTIFICATE";
export const OTEL_EXPORTER_OTLP_CLIENT_CERTIFICATE_ENV = "OTEL_EXPORTER_OTLP_CLIENT_CERTIFICATE";
export const OTEL_EXPORTER_OTLP_CLIENT_KEY_ENV = "OTEL_EXPORTER_OTLP_CLIENT_KEY";
extensions/diagnostics-otel/src/service.otlp-export.test.ts
+74
View File
@@ -46,6 +46,10 @@ const ENDPOINT_ENV_KEYS = [
"OTEL_EXPORTER_OTLP_TRACES_ENDPOINT",
"OTEL_EXPORTER_OTLP_METRICS_ENDPOINT",
"OTEL_EXPORTER_OTLP_LOGS_ENDPOINT",
"OTEL_EXPORTER_OTLP_PROTOCOL",
"OTEL_EXPORTER_OTLP_TRACES_PROTOCOL",
"OTEL_EXPORTER_OTLP_METRICS_PROTOCOL",
"OTEL_EXPORTER_OTLP_LOGS_PROTOCOL",
"OTEL_EXPORTER_OTLP_CERTIFICATE",
"OTEL_EXPORTER_OTLP_CLIENT_CERTIFICATE",
"OTEL_EXPORTER_OTLP_CLIENT_KEY",
@@ -333,6 +337,76 @@ test("uses real signal-specific exporter endpoints verbatim", async () => {
}
}, 30_000);
test("exports only signals whose resolved protocol is supported", async () => {
const receiver = await startOtlpReceiver();
releasePreloadedOtelGlobals();
process.env.OTEL_EXPORTER_OTLP_PROTOCOL = "grpc";
process.env.OTEL_EXPORTER_OTLP_TRACES_PROTOCOL = "http/protobuf";
process.env.OTEL_EXPORTER_OTLP_METRICS_PROTOCOL = "http/json";
process.env.OTEL_EXPORTER_OTLP_LOGS_PROTOCOL = "http/protobuf";
const { service, ctx } = await startOtelService({
endpoint: receiver.endpoint,
traces: true,
metrics: true,
logs: true,
configure: (serviceContext) => {
delete serviceContext.config.diagnostics!.otel!.protocol;
},
});
try {
await emitRealSdkSignals();
await service.stop?.(ctx);
expect(new Set(receiver.requests.map((request) => request.url))).toEqual(
new Set(["/v1/traces", "/v1/logs"]),
);
expect(
receiver.requests.every(
(request) => request.method === "POST" && request.contentType === "application/x-protobuf",
),
).toBe(true);
expect(ctx.logger.warn).toHaveBeenCalledWith(
"diagnostics-otel: unsupported metrics protocol http/json; OTLP export disabled",
);
} finally {
await service.stop?.(ctx);
await receiver.close();
}
}, 30_000);
test("does not auto-enable rejected traces when metrics start the real SDK", async () => {
const receiver = await startOtlpReceiver();
releasePreloadedOtelGlobals();
process.env.OTEL_EXPORTER_OTLP_PROTOCOL = "http/protobuf";
process.env.OTEL_EXPORTER_OTLP_TRACES_PROTOCOL = "grpc";
process.env.OTEL_EXPORTER_OTLP_METRICS_PROTOCOL = "http/protobuf";
const { service, ctx } = await startOtelService({
endpoint: receiver.endpoint,
traces: true,
metrics: true,
logs: false,
configure: (serviceContext) => {
delete serviceContext.config.diagnostics!.otel!.protocol;
},
});
try {
await emitRealSdkSignals();
await service.stop?.(ctx);
expect(new Set(receiver.requests.map((request) => request.url))).toEqual(
new Set(["/v1/metrics"]),
);
expect(ctx.logger.warn).toHaveBeenCalledWith(
"diagnostics-otel: unsupported traces protocol grpc; OTLP export disabled",
);
} finally {
await service.stop?.(ctx);
await receiver.close();
}
}, 30_000);
// Covers all three completeTrackedLifecycleSpan owners: run.completed,
// harness.run.completed, and message.processed. The mocked suite cannot tell the two id
// spaces apart, so a regression at any one of them is only visible here.
extensions/diagnostics-otel/src/service.test.ts
+215 -12
View File
@@ -229,12 +229,20 @@ const OTEL_TRUNCATED_SUFFIX_MAX_CHARS = 20;
const OTEL_TEST_USERINFO = ["operator", "example-fixture"].join(":");
const ORIGINAL_OPENCLAW_OTEL_PRELOADED = process.env.OPENCLAW_OTEL_PRELOADED;
const ORIGINAL_OTEL_EXPORTER_OTLP_ENDPOINT = process.env.OTEL_EXPORTER_OTLP_ENDPOINT;
const ORIGINAL_OTEL_EXPORTER_OTLP_PROTOCOL = process.env.OTEL_EXPORTER_OTLP_PROTOCOL;
const ORIGINAL_OTEL_EXPORTER_OTLP_TRACES_ENDPOINT = process.env.OTEL_EXPORTER_OTLP_TRACES_ENDPOINT;
const ORIGINAL_OTEL_EXPORTER_OTLP_METRICS_ENDPOINT =
process.env.OTEL_EXPORTER_OTLP_METRICS_ENDPOINT;
const ORIGINAL_OTEL_EXPORTER_OTLP_LOGS_ENDPOINT = process.env.OTEL_EXPORTER_OTLP_LOGS_ENDPOINT;
const ORIGINAL_OTEL_SEMCONV_STABILITY_OPT_IN = process.env.OTEL_SEMCONV_STABILITY_OPT_IN;
const OTEL_PROTOCOL_ENV_KEYS = [
"OTEL_EXPORTER_OTLP_PROTOCOL",
"OTEL_EXPORTER_OTLP_TRACES_PROTOCOL",
"OTEL_EXPORTER_OTLP_METRICS_PROTOCOL",
"OTEL_EXPORTER_OTLP_LOGS_PROTOCOL",
] as const;
const ORIGINAL_OTEL_PROTOCOL_ENV = Object.fromEntries(
OTEL_PROTOCOL_ENV_KEYS.map((key) => [key, process.env[key]]),
) as Record<(typeof OTEL_PROTOCOL_ENV_KEYS)[number], string | undefined>;
const OTEL_CERT_ENV_KEYS = [
"OTEL_EXPORTER_OTLP_CERTIFICATE",
"OTEL_EXPORTER_OTLP_CLIENT_CERTIFICATE",
@@ -583,7 +591,9 @@ describe("diagnostics-otel service", () => {
beforeEach(() => {
resetDiagnosticEventsForTest();
delete process.env.OPENCLAW_OTEL_PRELOADED;
delete process.env.OTEL_EXPORTER_OTLP_PROTOCOL;
for (const key of OTEL_PROTOCOL_ENV_KEYS) {
delete process.env[key];
}
delete process.env.OTEL_SEMCONV_STABILITY_OPT_IN;
telemetryState.counters.clear();
telemetryState.histograms.clear();
@@ -628,10 +638,13 @@ describe("diagnostics-otel service", () => {
} else {
process.env.OTEL_EXPORTER_OTLP_ENDPOINT = ORIGINAL_OTEL_EXPORTER_OTLP_ENDPOINT;
}
if (ORIGINAL_OTEL_EXPORTER_OTLP_PROTOCOL === undefined) {
delete process.env.OTEL_EXPORTER_OTLP_PROTOCOL;
} else {
process.env.OTEL_EXPORTER_OTLP_PROTOCOL = ORIGINAL_OTEL_EXPORTER_OTLP_PROTOCOL;
for (const key of OTEL_PROTOCOL_ENV_KEYS) {
const value = ORIGINAL_OTEL_PROTOCOL_ENV[key];
if (value === undefined) {
delete process.env[key];
} else {
process.env[key] = value;
}
}
if (ORIGINAL_OTEL_SEMCONV_STABILITY_OPT_IN === undefined) {
delete process.env.OTEL_SEMCONV_STABILITY_OPT_IN;
@@ -1292,6 +1305,9 @@ describe("diagnostics-otel service", () => {
test("keeps explicit HTTP exporters canonical when ambient protocol is gRPC", async () => {
process.env.OTEL_EXPORTER_OTLP_PROTOCOL = "grpc";
process.env.OTEL_EXPORTER_OTLP_TRACES_PROTOCOL = "grpc";
process.env.OTEL_EXPORTER_OTLP_METRICS_PROTOCOL = "http/json";
process.env.OTEL_EXPORTER_OTLP_LOGS_PROTOCOL = "grpc";
const { ctx } = await startOtelService({
protocol: "http/protobuf",
traces: true,
@@ -1352,7 +1368,11 @@ describe("diagnostics-otel service", () => {
{ signal: "metrics", status: "failure", reason: "unsupported_protocol" },
{ signal: "logs", status: "failure", reason: "unsupported_protocol" },
]);
expect(ctx.logger.warn).toHaveBeenCalledWith("diagnostics-otel: unsupported protocol grpc");
expect(vi.mocked(ctx.logger.warn).mock.calls).toEqual(
["traces", "metrics", "logs"].map((signal) => [
`diagnostics-otel: unsupported ${signal} protocol grpc; OTLP export disabled`,
]),
);
expect(traceExporterCtor).not.toHaveBeenCalled();
expect(metricExporterCtor).not.toHaveBeenCalled();
expect(logExporterCtor).not.toHaveBeenCalled();
@@ -1361,6 +1381,187 @@ describe("diagnostics-otel service", () => {
unsubscribe();
});
test("uses signal protocol overrides without disabling supported siblings", async () => {
const events: TelemetryExporterEvent[] = [];
const unsubscribe = onInternalDiagnosticEvent((event) => {
if (event.type === "telemetry.exporter") {
events.push(event);
}
});
process.env.OTEL_EXPORTER_OTLP_PROTOCOL = "grpc";
process.env.OTEL_EXPORTER_OTLP_TRACES_PROTOCOL = "http/protobuf";
process.env.OTEL_EXPORTER_OTLP_METRICS_PROTOCOL = "http/json";
process.env.OTEL_EXPORTER_OTLP_LOGS_PROTOCOL = "http/protobuf";
const { ctx } = await startOtelService({
traces: true,
metrics: true,
logs: true,
configure: (context) => {
delete context.config.diagnostics?.otel?.protocol;
},
});
expect(traceExporterCtor).toHaveBeenCalledTimes(1);
expect(metricExporterCtor).not.toHaveBeenCalled();
expect(logExporterCtor).toHaveBeenCalledTimes(1);
expect(sdkStart).toHaveBeenCalledTimes(1);
expect(ctx.logger.warn).toHaveBeenCalledWith(
"diagnostics-otel: unsupported metrics protocol http/json; OTLP export disabled",
);
expect(
events.map((event) => ({
signal: event.signal,
status: event.status,
reason: event.reason,
})),
).toEqual([
{ signal: "metrics", status: "failure", reason: "unsupported_protocol" },
{ signal: "traces", status: "started", reason: "configured" },
{ signal: "logs", status: "started", reason: "configured" },
]);
unsubscribe();
});
test("keeps rejected traces disabled when metrics still start NodeSDK", async () => {
process.env.OTEL_EXPORTER_OTLP_PROTOCOL = "http/protobuf";
process.env.OTEL_EXPORTER_OTLP_TRACES_PROTOCOL = "grpc";
process.env.OTEL_EXPORTER_OTLP_METRICS_PROTOCOL = "http/protobuf";
const { ctx } = await startOtelService({
traces: true,
metrics: true,
logs: false,
configure: (context) => {
delete context.config.diagnostics?.otel?.protocol;
},
});
const options = mockCallArg(sdkCtor, 0) as {
metricReaders?: unknown[];
spanProcessors?: unknown[];
};
expect(traceExporterCtor).not.toHaveBeenCalled();
expect(metricExporterCtor).toHaveBeenCalledTimes(1);
expect(options.spanProcessors).toEqual([]);
expect(options.metricReaders).toHaveLength(1);
expect(ctx.logger.warn).toHaveBeenCalledWith(
"diagnostics-otel: unsupported traces protocol grpc; OTLP export disabled",
);
});
test("keeps stdout logs active when the OTLP branch of both is unsupported", async () => {
const events: TelemetryExporterEvent[] = [];
const unsubscribe = onInternalDiagnosticEvent((event) => {
if (event.type === "telemetry.exporter") {
events.push(event);
}
});
process.env.OTEL_EXPORTER_OTLP_PROTOCOL = "http/protobuf";
process.env.OTEL_EXPORTER_OTLP_LOGS_PROTOCOL = "grpc";
const capture = captureStdoutWrites();
try {
const { ctx } = await startOtelService({
traces: false,
metrics: false,
logs: true,
logsExporter: "both",
configure: (context) => {
delete context.config.diagnostics?.otel?.protocol;
},
});
emitDiagnosticEvent({
type: "log.record",
level: "INFO",
message: "stdout fallback log",
});
await flushDiagnosticEvents();
expect(logExporterCtor).not.toHaveBeenCalled();
expect(parseSingleStdoutDiagnosticLogLine(capture.writes).body).toBe("log");
expect(ctx.logger.warn).toHaveBeenCalledWith(
"diagnostics-otel: unsupported logs protocol grpc; OTLP export disabled",
);
expect(ctx.logger.info).toHaveBeenCalledWith(
"diagnostics-otel: logs exporter enabled (stdout JSONL)",
);
expect(
events.map((event) => ({
signal: event.signal,
status: event.status,
reason: event.reason,
})),
).toEqual([
{ signal: "logs", status: "failure", reason: "unsupported_protocol" },
{ signal: "logs", status: "started", reason: "configured" },
]);
} finally {
capture.spy.mockRestore();
unsubscribe();
}
});
test("does not validate externally owned trace and metric protocols", async () => {
const events: TelemetryExporterEvent[] = [];
const unsubscribe = onInternalDiagnosticEvent((event) => {
if (event.type === "telemetry.exporter") {
events.push(event);
}
});
process.env.OPENCLAW_OTEL_PRELOADED = "1";
process.env.OTEL_EXPORTER_OTLP_PROTOCOL = "grpc";
process.env.OTEL_EXPORTER_OTLP_TRACES_PROTOCOL = "http/json";
process.env.OTEL_EXPORTER_OTLP_METRICS_PROTOCOL = "grpc";
const { ctx } = await startOtelService({
traces: true,
metrics: true,
logs: false,
configure: (context) => {
delete context.config.diagnostics?.otel?.protocol;
},
});
expect(sdkCtor).not.toHaveBeenCalled();
expect(traceExporterCtor).not.toHaveBeenCalled();
expect(metricExporterCtor).not.toHaveBeenCalled();
expect(ctx.logger.warn).not.toHaveBeenCalledWith(expect.stringContaining("unsupported"));
expect(
events.map((event) => ({
signal: event.signal,
status: event.status,
reason: event.reason,
})),
).toEqual([
{ signal: "traces", status: "started", reason: "configured" },
{ signal: "metrics", status: "started", reason: "configured" },
]);
unsubscribe();
});
test("ignores blank signal protocol overrides in favor of the shared fallback", async () => {
process.env.OTEL_EXPORTER_OTLP_PROTOCOL = "http/protobuf";
process.env.OTEL_EXPORTER_OTLP_TRACES_PROTOCOL = " \t ";
process.env.OTEL_EXPORTER_OTLP_METRICS_PROTOCOL = "\u2000";
process.env.OTEL_EXPORTER_OTLP_LOGS_PROTOCOL = "\ufeff";
await startOtelService({
traces: true,
metrics: true,
logs: true,
configure: (context) => {
delete context.config.diagnostics?.otel?.protocol;
},
});
expect(traceExporterCtor).toHaveBeenCalledTimes(1);
expect(metricExporterCtor).toHaveBeenCalledTimes(1);
expect(logExporterCtor).toHaveBeenCalledTimes(1);
});
test("starts stdout-only logs when OTLP protocol env override is unsupported", async () => {
process.env.OTEL_EXPORTER_OTLP_PROTOCOL = "grpc";
const { ctx } = await startOtelService({
@@ -1404,7 +1605,7 @@ describe("diagnostics-otel service", () => {
name: "preserves nonblank OTLP protocol env overrides",
value: " http/protobuf ",
exporterCalls: 0,
warning: "diagnostics-otel: unsupported protocol http/protobuf ",
warning: " http/protobuf ",
},
])("$name", async ({ value, exporterCalls, warning }) => {
process.env.OTEL_EXPORTER_OTLP_PROTOCOL = value;
@@ -1419,11 +1620,13 @@ describe("diagnostics-otel service", () => {
expect(traceExporterCtor).toHaveBeenCalledTimes(exporterCalls);
expect(metricExporterCtor).toHaveBeenCalledTimes(exporterCalls);
if (warning) {
expect(ctx.logger.warn).toHaveBeenCalledWith(warning);
} else {
expect(ctx.logger.warn).not.toHaveBeenCalledWith(
"diagnostics-otel: unsupported protocol ",
expect(vi.mocked(ctx.logger.warn).mock.calls).toEqual(
["traces", "metrics"].map((signal) => [
`diagnostics-otel: unsupported ${signal} protocol ${warning}; OTLP export disabled`,
]),
);
} else {
expect(ctx.logger.warn).not.toHaveBeenCalled();
}
});
extensions/diagnostics-otel/src/service.ts
+100 -55
View File
@@ -18,8 +18,12 @@ import {
DEFAULT_SERVICE_NAME,
OTEL_EXPORTER_OTLP_ENDPOINT_ENV,
OTEL_EXPORTER_OTLP_LOGS_ENDPOINT_ENV,
OTEL_EXPORTER_OTLP_LOGS_PROTOCOL_ENV,
OTEL_EXPORTER_OTLP_METRICS_ENDPOINT_ENV,
OTEL_EXPORTER_OTLP_METRICS_PROTOCOL_ENV,
OTEL_EXPORTER_OTLP_PROTOCOL_ENV,
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT_ENV,
OTEL_EXPORTER_OTLP_TRACES_PROTOCOL_ENV,
} from "./service-constants.js";
import {
hasPreloadedOtelSdk,
@@ -47,6 +51,30 @@ import { createUsageRecorders } from "./service-recorders-usage.js";
import { createDiagnosticsTraceRuntime } from "./service-traces.js";
import type { OtelLogsExporter, TelemetryExporterDiagnosticEvent } from "./service-types.js";
const OTLP_HTTP_PROTOBUF_PROTOCOL = "http/protobuf";
const OTEL_SIGNAL_PROTOCOL_ENV = {
traces: OTEL_EXPORTER_OTLP_TRACES_PROTOCOL_ENV,
metrics: OTEL_EXPORTER_OTLP_METRICS_PROTOCOL_ENV,
logs: OTEL_EXPORTER_OTLP_LOGS_PROTOCOL_ENV,
} satisfies Record<TelemetryExporterDiagnosticEvent["signal"], string>;
function readNonblankOtelEnv(name: string): string | undefined {
const value = process.env[name];
return value?.trim() ? value : undefined;
}
function resolveSignalProtocol(
signal: TelemetryExporterDiagnosticEvent["signal"],
configuredProtocol: string | undefined,
): string {
return (
configuredProtocol ??
readNonblankOtelEnv(OTEL_SIGNAL_PROTOCOL_ENV[signal]) ??
readNonblankOtelEnv(OTEL_EXPORTER_OTLP_PROTOCOL_ENV) ??
OTLP_HTTP_PROTOBUF_PROTOCOL
);
}
export function createDiagnosticsOtelService(): OpenClawPluginService {
let sdk: NodeSDK | null = null;
let logProvider: LoggerProvider | null = null;
@@ -127,19 +155,15 @@ export function createDiagnosticsOtelService(): OpenClawPluginService {
const metricsEnabled = otel.metrics !== false;
const logsEnabled = otel.logs === true;
const logsExporter: OtelLogsExporter = otel.logsExporter ?? "otlp";
const logsToOtlp = logsEnabled && (logsExporter === "otlp" || logsExporter === "both");
const logsToOtlpRequested =
logsEnabled && (logsExporter === "otlp" || logsExporter === "both");
const logsToStdout = logsEnabled && (logsExporter === "stdout" || logsExporter === "both");
const otlpSignals: TelemetryExporterDiagnosticEvent["signal"][] = [
...(tracesEnabled ? (["traces"] as const) : []),
...(metricsEnabled ? (["metrics"] as const) : []),
...(logsToOtlp ? (["logs"] as const) : []),
];
const enabledSignals: TelemetryExporterDiagnosticEvent["signal"][] = [
const configuredSignals: TelemetryExporterDiagnosticEvent["signal"][] = [
...(tracesEnabled ? (["traces"] as const) : []),
...(metricsEnabled ? (["metrics"] as const) : []),
...(logsEnabled ? (["logs"] as const) : []),
];
if (enabledSignals.length === 0) {
if (configuredSignals.length === 0) {
return;
}
// This capability is the admission gate; no exporter may outlive a denied start.
@@ -149,15 +173,41 @@ export function createDiagnosticsOtelService(): OpenClawPluginService {
return;
}
const envProtocol = process.env.OTEL_EXPORTER_OTLP_PROTOCOL;
const protocol = otel.protocol ?? (envProtocol?.trim() ? envProtocol : "http/protobuf");
if (otlpSignals.length > 0 && protocol !== "http/protobuf") {
emitForSignals(otlpSignals, {
const sdkPreloaded = hasPreloadedOtelSdk();
const ownedOtlpSignals: TelemetryExporterDiagnosticEvent["signal"][] = [
...(!sdkPreloaded && tracesEnabled ? (["traces"] as const) : []),
...(!sdkPreloaded && metricsEnabled ? (["metrics"] as const) : []),
...(logsToOtlpRequested ? (["logs"] as const) : []),
];
const supportedOtlpSignals = new Set<TelemetryExporterDiagnosticEvent["signal"]>();
for (const signal of ownedOtlpSignals) {
const protocol = resolveSignalProtocol(signal, otel.protocol);
if (protocol === OTLP_HTTP_PROTOBUF_PROTOCOL) {
supportedOtlpSignals.add(signal);
continue;
}
emitExporterEvent({
signal,
exporter: "diagnostics-otel",
status: "failure",
reason: "unsupported_protocol",
});
ctx.logger.warn(`diagnostics-otel: unsupported protocol ${protocol}`);
ctx.logger.warn(
`diagnostics-otel: unsupported ${signal} protocol ${protocol}; OTLP export disabled`,
);
}
const tracesToOtlp = !sdkPreloaded && tracesEnabled && supportedOtlpSignals.has("traces");
const metricsToOtlp = !sdkPreloaded && metricsEnabled && supportedOtlpSignals.has("metrics");
const logsToOtlp = logsToOtlpRequested && supportedOtlpSignals.has("logs");
const tracesActive = sdkPreloaded ? tracesEnabled : tracesToOtlp;
const metricsActive = sdkPreloaded ? metricsEnabled : metricsToOtlp;
const logsActive = logsToStdout || logsToOtlp;
const startedSignals: TelemetryExporterDiagnosticEvent["signal"][] = [
...(tracesActive ? (["traces"] as const) : []),
...(metricsActive ? (["metrics"] as const) : []),
...(logsActive ? (["logs"] as const) : []),
];
if (startedSignals.length === 0) {
return;
}
@@ -168,7 +218,6 @@ export function createDiagnosticsOtelService(): OpenClawPluginService {
otel.serviceName?.trim() || process.env.OTEL_SERVICE_NAME || DEFAULT_SERVICE_NAME;
const sampleRate = resolveSampleRate(otel.sampleRate);
const contentCapturePolicy = resolveContentCapturePolicy(otel.captureContent);
const sdkPreloaded = hasPreloadedOtelSdk();
const resource = resourceFromAttributes({
[ATTR_SERVICE_NAME]: serviceName,
@@ -183,40 +232,36 @@ export function createDiagnosticsOtelService(): OpenClawPluginService {
path: "v1/logs",
})
: undefined;
const traceUrl =
!sdkPreloaded && tracesEnabled
? resolveSignalOtelUrl({
signalEndpoint: otel.tracesEndpoint,
signalEnvEndpoint: process.env[OTEL_EXPORTER_OTLP_TRACES_ENDPOINT_ENV],
sharedEnvEndpoint,
endpoint,
path: "v1/traces",
})
: undefined;
const metricUrl =
!sdkPreloaded && metricsEnabled
? resolveSignalOtelUrl({
signalEndpoint: otel.metricsEndpoint,
signalEnvEndpoint: process.env[OTEL_EXPORTER_OTLP_METRICS_ENDPOINT_ENV],
sharedEnvEndpoint,
endpoint,
path: "v1/metrics",
})
: undefined;
const traceUrl = tracesToOtlp
? resolveSignalOtelUrl({
signalEndpoint: otel.tracesEndpoint,
signalEnvEndpoint: process.env[OTEL_EXPORTER_OTLP_TRACES_ENDPOINT_ENV],
sharedEnvEndpoint,
endpoint,
path: "v1/traces",
})
: undefined;
const metricUrl = metricsToOtlp
? resolveSignalOtelUrl({
signalEndpoint: otel.metricsEndpoint,
signalEnvEndpoint: process.env[OTEL_EXPORTER_OTLP_METRICS_ENDPOINT_ENV],
sharedEnvEndpoint,
endpoint,
path: "v1/metrics",
})
: undefined;
// Validate every owned signal before any SDK can export with downgraded TLS trust.
const logHttpAgentOptions = logsToOtlp
? resolveOtelHttpAgentOptions({ url: logUrl, signalIdentifier: "LOGS" })
: undefined;
const traceHttpAgentOptions =
!sdkPreloaded && tracesEnabled
? resolveOtelHttpAgentOptions({ url: traceUrl, signalIdentifier: "TRACES" })
: undefined;
const metricHttpAgentOptions =
!sdkPreloaded && metricsEnabled
? resolveOtelHttpAgentOptions({ url: metricUrl, signalIdentifier: "METRICS" })
: undefined;
if (!sdkPreloaded && (tracesEnabled || metricsEnabled)) {
const traceExporter = tracesEnabled
const traceHttpAgentOptions = tracesToOtlp
? resolveOtelHttpAgentOptions({ url: traceUrl, signalIdentifier: "TRACES" })
: undefined;
const metricHttpAgentOptions = metricsToOtlp
? resolveOtelHttpAgentOptions({ url: metricUrl, signalIdentifier: "METRICS" })
: undefined;
if (tracesToOtlp || metricsToOtlp) {
const traceExporter = tracesToOtlp
? new OTLPTraceExporter({
...(traceUrl ? { url: traceUrl } : {}),
...(headers ? { headers } : {}),
@@ -232,7 +277,7 @@ export function createDiagnosticsOtelService(): OpenClawPluginService {
]
: undefined;
const metricExporter = metricsEnabled
const metricExporter = metricsToOtlp
? new OTLPMetricExporter({
...(metricUrl ? { url: metricUrl } : {}),
...(headers ? { headers } : {}),
@@ -251,8 +296,8 @@ export function createDiagnosticsOtelService(): OpenClawPluginService {
sdk = new NodeSDK({
resource,
// Explicit empty arrays keep NodeSDK from restoring disabled exporters
// from ambient OTEL_* settings; OpenClaw owns every signal exporter.
// Empty arrays are required in mixed-signal cases too; omission lets NodeSDK
// restore a protocol-rejected exporter from ambient OTEL_* settings.
...(spanProcessors
? { spanProcessors }
: traceExporter
@@ -274,8 +319,8 @@ export function createDiagnosticsOtelService(): OpenClawPluginService {
} catch (err) {
emitForSignals(
[
...(tracesEnabled ? (["traces"] as const) : []),
...(metricsEnabled ? (["metrics"] as const) : []),
...(tracesToOtlp ? (["traces"] as const) : []),
...(metricsToOtlp ? (["metrics"] as const) : []),
],
{
exporter: "diagnostics-otel",
@@ -304,7 +349,7 @@ export function createDiagnosticsOtelService(): OpenClawPluginService {
flushIntervalMs: otel.flushIntervalMs,
headers,
logger: ctx.logger,
logsEnabled,
logsEnabled: logsActive,
logsToOtlp,
logsToStdout,
logHttpAgentOptions,
@@ -319,7 +364,7 @@ export function createDiagnosticsOtelService(): OpenClawPluginService {
contentCapturePolicy,
metrics: diagnosticMetrics,
traces: diagnosticsTrace,
tracesEnabled,
tracesEnabled: tracesActive,
});
const recorders = {
...createUsageRecorders(recorderRuntime),
@@ -350,17 +395,17 @@ export function createDiagnosticsOtelService(): OpenClawPluginService {
return true;
});
emitForSignals(enabledSignals, {
emitForSignals(startedSignals, {
exporter: "diagnostics-otel",
status: "started",
reason: "configured",
});
if (logsEnabled) {
if (logsActive) {
const label =
logsExporter === "both"
logsToOtlp && logsToStdout
? "OTLP/Protobuf + stdout JSONL"
: logsExporter === "stdout"
: logsToStdout
? "stdout JSONL"
: "OTLP/Protobuf";
ctx.logger.info(`diagnostics-otel: logs exporter enabled (${label})`);
scripts/docker/setup.sh
+6
View File
@@ -542,6 +542,9 @@ export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT="${OTEL_EXPORTER_OTLP_TRACES_ENDPOINT:
export OTEL_EXPORTER_OTLP_METRICS_ENDPOINT="${OTEL_EXPORTER_OTLP_METRICS_ENDPOINT:-}"
export OTEL_EXPORTER_OTLP_LOGS_ENDPOINT="${OTEL_EXPORTER_OTLP_LOGS_ENDPOINT:-}"
export OTEL_EXPORTER_OTLP_PROTOCOL="${OTEL_EXPORTER_OTLP_PROTOCOL:-}"
export OTEL_EXPORTER_OTLP_TRACES_PROTOCOL="${OTEL_EXPORTER_OTLP_TRACES_PROTOCOL:-}"
export OTEL_EXPORTER_OTLP_METRICS_PROTOCOL="${OTEL_EXPORTER_OTLP_METRICS_PROTOCOL:-}"
export OTEL_EXPORTER_OTLP_LOGS_PROTOCOL="${OTEL_EXPORTER_OTLP_LOGS_PROTOCOL:-}"
export OTEL_SERVICE_NAME="${OTEL_SERVICE_NAME:-}"
export OTEL_SEMCONV_STABILITY_OPT_IN="${OTEL_SEMCONV_STABILITY_OPT_IN:-}"
export OPENCLAW_OTEL_PRELOADED="${OPENCLAW_OTEL_PRELOADED:-}"
@@ -749,6 +752,9 @@ upsert_env "$ENV_FILE" \
OTEL_EXPORTER_OTLP_METRICS_ENDPOINT \
OTEL_EXPORTER_OTLP_LOGS_ENDPOINT \
OTEL_EXPORTER_OTLP_PROTOCOL \
OTEL_EXPORTER_OTLP_TRACES_PROTOCOL \
OTEL_EXPORTER_OTLP_METRICS_PROTOCOL \
OTEL_EXPORTER_OTLP_LOGS_PROTOCOL \
OTEL_SERVICE_NAME \
OTEL_SEMCONV_STABILITY_OPT_IN \
OPENCLAW_OTEL_PRELOADED \
src/docker-setup.e2e.test.ts
+19
View File
@@ -214,6 +214,25 @@ describe("scripts/docker/setup.sh", () => {
expect(envFile).toContain("OPENCLAW_DISABLE_BONJOUR=0");
});
it("persists and forwards signal-specific OTLP protocol overrides", async () => {
const activeSandbox = requireSandbox(sandbox);
const protocolEnv = {
OTEL_EXPORTER_OTLP_TRACES_PROTOCOL: "http/protobuf",
OTEL_EXPORTER_OTLP_METRICS_PROTOCOL: "http/protobuf",
OTEL_EXPORTER_OTLP_LOGS_PROTOCOL: "http/protobuf",
};
const result = runDockerSetup(activeSandbox, protocolEnv);
expect(result.status).toBe(0);
const envFile = await readFile(join(activeSandbox.rootDir, ".env"), "utf8");
const compose = await readFile(join(repoRoot, "docker-compose.yml"), "utf8");
for (const [key, value] of Object.entries(protocolEnv)) {
expect(envFile).toContain(`${key}=${value}`);
expect(compose).toContain(`${key}: \${${key}:-}`);
}
});
it("normalizes legacy OPENCLAW_DOCKER_APT_PACKAGES into OPENCLAW_IMAGE_APT_PACKAGES", async () => {
const activeSandbox = requireSandbox(sandbox);
await resetDockerLog(activeSandbox);