diff --git a/docs/concepts/qa-e2e-automation.md b/docs/concepts/qa-e2e-automation.md index 82c5e452e609..8b7ae7823185 100644 --- a/docs/concepts/qa-e2e-automation.md +++ b/docs/concepts/qa-e2e-automation.md @@ -48,7 +48,7 @@ script aliases; both forms work. | `qa mock-openai` | Start only the scenario-aware `mock-openai` provider server. | | `qa credentials doctor` / `add` / `list` / `remove` | Manage the shared Convex credential pool. | | `qa buzz` | Live transport lane against a real Buzz relay room with dedicated driver and SUT identities. | -| `qa discord` | Live transport lane against a real private Discord guild channel. | +| `qa discord` | Discord transport lane through the real bundled plugin: live Discord by default, or Crabline's deterministic local provider server with `--channel-driver crabline`. | | `qa matrix` | QA Lab Matrix catalog scenarios against a disposable Tuwunel homeserver. See [Matrix live lane](#matrix-live-lane). | | `qa slack` | Live transport lane against a real private Slack channel. | | `qa telegram` | Live transport lane against a real private Telegram group. | @@ -458,8 +458,9 @@ guest can write back through the mounted workspace. ## Buzz, Discord, Slack, Telegram, and WhatsApp QA reference The Matrix adapter uses the disposable Docker-backed lane documented above. -Buzz, Discord, Slack, Telegram, and WhatsApp run against pre-existing real -transports, so their reference lives here. +Buzz, Slack, Telegram, and WhatsApp run against pre-existing real transports. +Discord can run against either live Discord or Crabline's deterministic local +provider server. Their reference lives here. ### Shared CLI flags @@ -481,10 +482,15 @@ accept the same flags: | `--credential-file ` | - | Buzz-only JSON credential file for local runs. | | `--allow-failures` | off | Write artifacts without returning a failing exit code when scenarios fail. | +Discord also accepts `--channel-driver ` (default `live`) and +`--list-scenarios`. Crabline mode is local and rejects `--credential-source` +and `--credential-role`; provider mode and model selection remain independent +of the channel driver. + Each lane exits non-zero on any failed scenario. `--allow-failures` writes -artifacts without setting a failing exit code. Telegram also accepts -`--list-scenarios` to print available scenario ids and exit; the other lanes -do not expose that flag. +artifacts without setting a failing exit code. Telegram and Discord also accept +`--list-scenarios` to print the scenario ids selected by the same profile, +provider, model, execution-kind, and channel-driver rules used for execution. ### Buzz QA @@ -581,13 +587,46 @@ creating a separate RTT command or Telegram-specific summary format. ```bash pnpm openclaw qa discord +pnpm openclaw qa discord --channel-driver live ``` -Targets one real private Discord guild channel with two bots: a driver bot +Both commands select the live driver. They target one real private Discord +guild channel with two bots: a driver bot controlled by the harness and a SUT bot started by the child OpenClaw gateway through the bundled Discord plugin. Verifies channel mention handling, that the SUT bot has registered the native `/help` command with Discord, and -opt-in Mantis evidence scenarios. +opt-in Mantis evidence scenarios. Live Discord remains the release/provider +coverage lane; catalog and profile membership determine its current scenario +inventory. + +For deterministic local transport proof without Discord credentials or a +credential lease, run: + +```bash +pnpm openclaw qa discord \ + --channel-driver crabline \ + --provider-mode mock-openai + +pnpm openclaw qa suite \ + --channel-driver crabline \ + --channel discord \ + --provider-mode mock-openai +``` + +Crabline starts a separate Discord-shaped local provider server, then the child +Gateway runs the real bundled OpenClaw Discord plugin against that server's REST +and Gateway/WebSocket boundaries. The harness injects provider-native inbound +events through Crabline's authenticated admin ingress and records the plugin's +outbound REST traffic. This is not Crabline's fixture-level Discord local mock +provider, which fixture commands call directly without running the OpenClaw +plugin. + +The local server covers Gateway handshake and message events, REST message +delivery and reply metadata, mention handling, and native command registration. +Use live Discord for behavior the local server cannot faithfully exercise, +including voice, attachment/CDN and multipart behavior, interactions and +activities, webhooks, Gateway compression or multi-shard behavior, replay, and +Discord's distributed permissions and rate limits. Required env when `--credential-source env`: diff --git a/extensions/qa-lab/index.ts b/extensions/qa-lab/index.ts index 27bf9122d780..d70044862b64 100644 --- a/extensions/qa-lab/index.ts +++ b/extensions/qa-lab/index.ts @@ -4,6 +4,7 @@ import { setTimeout as sleep } from "node:timers/promises"; import { definePluginEntry } from "openclaw/plugin-sdk/plugin-entry"; import { jsonResult } from "openclaw/plugin-sdk/tool-results"; import { registerQaLabCli } from "./src/cli.js"; +import { registerCrablineDiscordProviderEndpoint } from "./src/crabline-discord-provider-endpoint.js"; import { createQaLabWebSearchProvider } from "./src/qa-web-search-provider.js"; import { createStaticSshWorkerProvider } from "./src/static-ssh-worker-provider.js"; @@ -18,6 +19,7 @@ export default definePluginEntry({ name: "QA Lab", description: "Private QA automation harness and debugger UI", register(api) { + registerCrablineDiscordProviderEndpoint(api); api.registerTool( { name: "qa_restart_wait", diff --git a/extensions/qa-lab/openclaw.plugin.json b/extensions/qa-lab/openclaw.plugin.json index 475eb0d8a0f5..7d6aafa61b0f 100644 --- a/extensions/qa-lab/openclaw.plugin.json +++ b/extensions/qa-lab/openclaw.plugin.json @@ -9,7 +9,8 @@ } ], "activation": { - "onStartup": false + "onStartup": false, + "onConfigPaths": ["plugins.entries.qa-lab"] }, "contracts": { "webSearchProviders": ["qa-lab-search"], diff --git a/extensions/qa-lab/src/cli.runtime.test.ts b/extensions/qa-lab/src/cli.runtime.test.ts index ac3115664ecb..d2f881a1345f 100644 --- a/extensions/qa-lab/src/cli.runtime.test.ts +++ b/extensions/qa-lab/src/cli.runtime.test.ts @@ -1210,6 +1210,34 @@ describe("qa cli runtime", () => { }); }); + it("dispatches generic suite Discord selection through Crabline", async () => { + await runQaSuiteCommand({ + channelDriver: "crabline", + channel: "discord", + providerMode: "mock-openai", + scenarioIds: ["channel-canary"], + }); + + expect(runQaSuite).toHaveBeenCalledWith( + expect.objectContaining({ + channelDriver: "crabline", + channelDriverSelection: expect.objectContaining({ channel: "discord" }), + scenarioIds: ["channel-canary"], + }), + ); + }); + + it("rejects channels outside Crabline's canonical server contract", async () => { + await expect( + runQaSuiteCommand({ + channelDriver: "crabline", + channel: "imessage", + providerMode: "mock-openai", + }), + ).rejects.toThrow("--channel must be one of"); + expect(runQaSuite).not.toHaveBeenCalled(); + }); + it("keeps implicit channel membership identical for live and Crabline drivers", async () => { await runQaSuiteCommand({ channelDriver: "live", diff --git a/extensions/qa-lab/src/crabline-discord-provider-endpoint-artifact.ts b/extensions/qa-lab/src/crabline-discord-provider-endpoint-artifact.ts new file mode 100644 index 000000000000..bcdff79b12a8 --- /dev/null +++ b/extensions/qa-lab/src/crabline-discord-provider-endpoint-artifact.ts @@ -0,0 +1,2 @@ +export const CRABLINE_DISCORD_PROVIDER_ENDPOINT_ARTIFACT = + "crabline-discord-provider-endpoint.json"; diff --git a/extensions/qa-lab/src/crabline-discord-provider-endpoint.test.ts b/extensions/qa-lab/src/crabline-discord-provider-endpoint.test.ts new file mode 100644 index 000000000000..b9d10ab43ea1 --- /dev/null +++ b/extensions/qa-lab/src/crabline-discord-provider-endpoint.test.ts @@ -0,0 +1,89 @@ +import fs from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import { createTestPluginApi } from "openclaw/plugin-sdk/plugin-test-api"; +import { afterEach, describe, expect, it, vi } from "vitest"; + +const setDiscordProviderEndpointDescriptor = vi.hoisted(() => vi.fn()); + +vi.mock("@openclaw/discord/provider-endpoint-api.js", () => ({ + setDiscordProviderEndpointDescriptor, +})); + +import { CRABLINE_DISCORD_PROVIDER_ENDPOINT_ARTIFACT } from "./crabline-discord-provider-endpoint-artifact.js"; +import { registerCrablineDiscordProviderEndpoint } from "./crabline-discord-provider-endpoint.js"; + +const QA_TEMP_ROOT_ENV = "OPENCLAW_QA_TEMP_ROOT"; +const originalTempRoot = process.env[QA_TEMP_ROOT_ENV]; + +afterEach(() => { + if (originalTempRoot === undefined) { + delete process.env[QA_TEMP_ROOT_ENV]; + } else { + process.env[QA_TEMP_ROOT_ENV] = originalTempRoot; + } + setDiscordProviderEndpointDescriptor.mockReset(); +}); + +describe("Crabline Discord child provider endpoint", () => { + it("loads the QA-owned artifact before startup and clears it during teardown", async () => { + const tempRoot = fs.mkdtempSync(path.join(os.tmpdir(), "openclaw-qa-discord-endpoint-")); + process.env[QA_TEMP_ROOT_ENV] = tempRoot; + const descriptor = { + restApiBaseUrl: "http://127.0.0.1:43123/api/v10", + gatewayBotUrl: "http://127.0.0.1:43123/api/v10/gateway/bot", + gatewayOrigin: "ws://127.0.0.1:43123", + }; + fs.writeFileSync( + path.join(tempRoot, CRABLINE_DISCORD_PROVIDER_ENDPOINT_ARTIFACT), + `${JSON.stringify(descriptor)}\n`, + { mode: 0o600 }, + ); + const registerRuntimeLifecycle = vi.fn(); + + try { + registerCrablineDiscordProviderEndpoint( + createTestPluginApi({ registrationMode: "full", registerRuntimeLifecycle }), + ); + + expect(setDiscordProviderEndpointDescriptor).toHaveBeenCalledWith(descriptor); + expect(registerRuntimeLifecycle).toHaveBeenCalledOnce(); + expect(setDiscordProviderEndpointDescriptor.mock.invocationCallOrder[0]).toBeLessThan( + registerRuntimeLifecycle.mock.invocationCallOrder[0] ?? Number.POSITIVE_INFINITY, + ); + + const lifecycle = registerRuntimeLifecycle.mock.calls[0]?.[0]; + await lifecycle?.cleanup({ reason: "shutdown" }); + expect(setDiscordProviderEndpointDescriptor).toHaveBeenLastCalledWith(undefined); + } finally { + fs.rmSync(tempRoot, { recursive: true, force: true }); + } + }); + + it("does not load the artifact during non-runtime registration", () => { + process.env[QA_TEMP_ROOT_ENV] = "/does/not/exist"; + + registerCrablineDiscordProviderEndpoint(createTestPluginApi({ registrationMode: "discovery" })); + + expect(setDiscordProviderEndpointDescriptor).not.toHaveBeenCalled(); + }); + + it("rejects an artifact with fields outside the endpoint contract", () => { + const tempRoot = fs.mkdtempSync(path.join(os.tmpdir(), "openclaw-qa-discord-endpoint-")); + process.env[QA_TEMP_ROOT_ENV] = tempRoot; + fs.writeFileSync( + path.join(tempRoot, CRABLINE_DISCORD_PROVIDER_ENDPOINT_ARTIFACT), + JSON.stringify({ apiRoot: "http://127.0.0.1:43123/api", version: 1 }), + { mode: 0o600 }, + ); + + try { + expect(() => + registerCrablineDiscordProviderEndpoint(createTestPluginApi({ registrationMode: "full" })), + ).toThrow("Crabline Discord provider endpoint artifact is invalid"); + expect(setDiscordProviderEndpointDescriptor).not.toHaveBeenCalled(); + } finally { + fs.rmSync(tempRoot, { recursive: true, force: true }); + } + }); +}); diff --git a/extensions/qa-lab/src/crabline-discord-provider-endpoint.ts b/extensions/qa-lab/src/crabline-discord-provider-endpoint.ts new file mode 100644 index 000000000000..5d25343ac69e --- /dev/null +++ b/extensions/qa-lab/src/crabline-discord-provider-endpoint.ts @@ -0,0 +1,57 @@ +import fs from "node:fs"; +import path from "node:path"; +import { + setDiscordProviderEndpointDescriptor, + type DiscordProviderEndpointDescriptor, +} from "@openclaw/discord/provider-endpoint-api.js"; +import type { OpenClawPluginApi } from "openclaw/plugin-sdk/plugin-entry"; +import { isRecord } from "openclaw/plugin-sdk/string-coerce-runtime"; +import { CRABLINE_DISCORD_PROVIDER_ENDPOINT_ARTIFACT } from "./crabline-discord-provider-endpoint-artifact.js"; + +const QA_TEMP_ROOT_ENV = "OPENCLAW_QA_TEMP_ROOT"; +const DESCRIPTOR_KEYS = ["gatewayBotUrl", "gatewayOrigin", "restApiBaseUrl"]; + +function readDescriptor(value: unknown): DiscordProviderEndpointDescriptor { + if ( + !isRecord(value) || + Object.keys(value).toSorted().join("\0") !== DESCRIPTOR_KEYS.join("\0") || + typeof value.restApiBaseUrl !== "string" || + typeof value.gatewayBotUrl !== "string" || + typeof value.gatewayOrigin !== "string" + ) { + throw new Error("Crabline Discord provider endpoint artifact is invalid"); + } + return { + restApiBaseUrl: value.restApiBaseUrl, + gatewayBotUrl: value.gatewayBotUrl, + gatewayOrigin: value.gatewayOrigin, + }; +} + +export function registerCrablineDiscordProviderEndpoint(api: OpenClawPluginApi): void { + if (api.registrationMode !== "full") { + return; + } + const tempRoot = process.env[QA_TEMP_ROOT_ENV]?.trim(); + if (!tempRoot) { + return; + } + const artifactPath = path.join(tempRoot, CRABLINE_DISCORD_PROVIDER_ENDPOINT_ARTIFACT); + let serialized: string; + try { + serialized = fs.readFileSync(artifactPath, "utf8"); + } catch (error) { + if ((error as NodeJS.ErrnoException).code === "ENOENT") { + return; + } + throw error; + } + const descriptor = readDescriptor(JSON.parse(serialized)); + // Install before channel startup; the child Gateway owns clearing this process-local seam. + setDiscordProviderEndpointDescriptor(descriptor); + api.lifecycle.registerRuntimeLifecycle({ + id: "qa-crabline-discord-provider-endpoint", + description: "Clears the QA-owned Discord provider endpoint when the child Gateway stops.", + cleanup: () => setDiscordProviderEndpointDescriptor(undefined), + }); +} diff --git a/extensions/qa-lab/src/crabline-discord-transport.test.ts b/extensions/qa-lab/src/crabline-discord-transport.test.ts new file mode 100644 index 000000000000..9114298a2f65 --- /dev/null +++ b/extensions/qa-lab/src/crabline-discord-transport.test.ts @@ -0,0 +1,156 @@ +// QA Lab tests cover the Discord-specific Crabline provider lifecycle. +import fs from "node:fs/promises"; +import path from "node:path"; +import type { OpenClawCrablineChannelDriverSelection } from "@openclaw/crabline"; +import { withTempDir } from "openclaw/plugin-sdk/test-env"; +import { describe, expect, it } from "vitest"; +import { createQaBusState } from "./bus-state.js"; +import { CRABLINE_DISCORD_PROVIDER_ENDPOINT_ARTIFACT } from "./crabline-discord-provider-endpoint-artifact.js"; +import { createQaCrablineTransportAdapter } from "./crabline-transport.js"; + +const DISCORD_SELECTION = { + capabilityMatrixPath: "crabline-channel-driver-capabilities.json", + channel: "discord", + channelDriver: "crabline", + providerReadinessArtifactPath: "crabline-provider-readiness.json", +} as const satisfies OpenClawCrablineChannelDriverSelection; + +describe("Crabline Discord transport", () => { + it("stages the private endpoint and closes the provider after the Gateway phase", async () => { + await withTempDir("qa-crabline-transport-", async (outputDir) => { + const transport = await createQaCrablineTransportAdapter({ + outputDir, + transportPolicy: { requireGroupMention: true, senderAllowlist: ["driver"] }, + selection: DISCORD_SELECTION, + state: createQaBusState(), + }); + const gatewayTempRoot = path.join(outputDir, "gateway-temp"); + await fs.mkdir(gatewayTempRoot); + + try { + expect(transport.requiredPluginIds).toEqual(["qa-lab", "discord"]); + const config = transport.createGatewayConfig({ baseUrl: "http://127.0.0.1:1" }); + const discord = config.channels?.discord as + | { + applicationId?: string; + guilds?: Record; users?: string[] }>; + token?: string; + } + | undefined; + expect(discord).toMatchObject({ + allowFrom: [expect.stringMatching(/^\d{17,20}$/u)], + applicationId: expect.stringMatching(/^\d{17,20}$/u), + dmPolicy: "allowlist", + guilds: { + "*": { + channels: { "*": { enabled: true, requireMention: true } }, + users: [expect.stringMatching(/^\d{17,20}$/u)], + }, + }, + token: expect.any(String), + }); + + await transport.stageGatewayRuntime?.({ tempRoot: gatewayTempRoot }); + const descriptorPath = path.join( + gatewayTempRoot, + CRABLINE_DISCORD_PROVIDER_ENDPOINT_ARTIFACT, + ); + const descriptor = JSON.parse(await fs.readFile(descriptorPath, "utf8")) as { + gatewayBotUrl: string; + gatewayOrigin: string; + restApiBaseUrl: string; + }; + expect(descriptor).toEqual({ + gatewayBotUrl: expect.stringMatching( + /^http:\/\/127\.0\.0\.1:\d+\/api\/v10\/gateway\/bot$/u, + ), + gatewayOrigin: expect.stringMatching(/^ws:\/\/127\.0\.0\.1:\d+$/u), + restApiBaseUrl: expect.stringMatching(/^http:\/\/127\.0\.0\.1:\d+\/api\/v10$/u), + }); + if (process.platform !== "win32") { + expect((await fs.stat(descriptorPath)).mode & 0o077).toBe(0); + } + + await expect( + transport.sendInbound({ + conversation: { id: "discord-crabline-primary", kind: "group" }, + senderId: "driver", + senderName: "QA Driver", + text: "@openclaw Discord provider marker.", + threadId: "discord-crabline-thread", + }), + ).resolves.toMatchObject({ + conversation: { id: "discord-crabline-primary", kind: "group" }, + text: "@openclaw Discord provider marker.", + threadId: "discord-crabline-thread", + }); + const delivery = transport.buildAgentDelivery({ + target: "thread:discord-crabline-primary/discord-crabline-thread", + }); + expect(delivery).toEqual({ + channel: "discord", + replyChannel: "discord", + replyTo: expect.stringMatching(/^channel:\d{17,20}$/u), + to: expect.stringMatching(/^channel:\d{17,20}$/u), + }); + + const probeUrl = `${descriptor.restApiBaseUrl}/users/@me`; + const headers = { authorization: `Bot ${discord?.token}` }; + const channelId = delivery.to?.replace(/^channel:/u, ""); + const outboundResponse = await fetch( + `${descriptor.restApiBaseUrl}/channels/${channelId}/messages`, + { + body: JSON.stringify({ content: "Discord provider outbound marker." }), + headers: { ...headers, "content-type": "application/json" }, + method: "POST", + }, + ); + expect(outboundResponse.ok).toBe(true); + await outboundResponse.body?.cancel(); + await expect( + transport.waitForOutbound({ + conversation: { id: "discord-crabline-primary", kind: "group" }, + textIncludes: "Discord provider outbound marker.", + threadId: "discord-crabline-thread", + timeoutMs: 1_000, + }), + ).resolves.toMatchObject({ + conversation: { id: "discord-crabline-primary", kind: "group" }, + text: "Discord provider outbound marker.", + threadId: "discord-crabline-thread", + }); + const beforeCleanup = await fetch(probeUrl, { headers }); + expect(beforeCleanup.ok).toBe(true); + await beforeCleanup.body?.cancel(); + + await transport.cleanup?.(); + const beforeGatewayStop = await fetch(probeUrl, { headers }); + expect(beforeGatewayStop.ok).toBe(true); + await beforeGatewayStop.body?.cancel(); + + await transport.cleanupAfterGatewayStop?.(); + await expect(fetch(probeUrl, { headers })).rejects.toThrow(); + } finally { + await transport.cleanupAfterGatewayStop?.(); + } + }); + }); + + it("completes Crabline's open Discord DM binding without a sender restriction", async () => { + await withTempDir("qa-crabline-discord-open-dm-", async (outputDir) => { + const transport = await createQaCrablineTransportAdapter({ + outputDir, + selection: DISCORD_SELECTION, + state: createQaBusState(), + }); + + try { + expect(transport.createGatewayConfig({ baseUrl: "http://127.0.0.1:1" })).toMatchObject({ + channels: { discord: { allowFrom: ["*"], dmPolicy: "open" } }, + }); + } finally { + await transport.cleanupAfterGatewayStop?.(); + } + }); + }); +}); diff --git a/extensions/qa-lab/src/crabline-provider-targets.ts b/extensions/qa-lab/src/crabline-provider-targets.ts index c4dbe1e17427..66ad86663c4c 100644 --- a/extensions/qa-lab/src/crabline-provider-targets.ts +++ b/extensions/qa-lab/src/crabline-provider-targets.ts @@ -13,6 +13,17 @@ const TELEGRAM_QA_MAX_NATIVE_ID = (1n << 52n) - 1n; const MATTERMOST_ID_PATTERN = /^[a-z0-9]{26}$/u; const MATRIX_QA_SERVER_NAME = "matrix-qa.test"; const MATRIX_QA_DRIVER_ID = `@driver:${MATRIX_QA_SERVER_NAME}`; +const DISCORD_ID_PATTERN = /^\d{17,20}$/u; +const DISCORD_ID_FLOOR = 100_000_000_000_000_000n; + +export function resolveDiscordQaId(value: string) { + const trimmed = value.trim(); + if (DISCORD_ID_PATTERN.test(trimmed)) { + return trimmed; + } + const digest = BigInt(`0x${createHash("sha256").update(trimmed).digest("hex").slice(0, 16)}`); + return String(DISCORD_ID_FLOOR + (digest % DISCORD_ID_FLOOR)); +} function resolveQaNumericId(value: string, range: bigint) { const digest = BigInt(`0x${createHash("sha256").update(value).digest("hex").slice(0, 16)}`); @@ -135,6 +146,30 @@ function resolveMatrixQaText(text: string, botUserId: string) { ); } +function resolveDiscordQaText(text: string, botUserId: string) { + return text.replace( + /(^|[\s([{])@openclaw(?=$|[\s.,!?;)\]}])/gu, + (_match, prefix: string) => `${prefix}<@${botUserId}>`, + ); +} + +function resolveDiscordQaTarget(target: string) { + const normalized = target.trim(); + if (normalized.startsWith("thread:")) { + const threadTarget = normalized.slice("thread:".length); + const separator = threadTarget.indexOf("/"); + if (separator > 0) { + return `thread:${resolveDiscordQaId(threadTarget.slice(0, separator))}/${resolveDiscordQaId(threadTarget.slice(separator + 1))}`; + } + } + for (const prefix of ["channel:", "group:", "dm:", "user:"]) { + if (normalized.startsWith(prefix)) { + return `${prefix}${resolveDiscordQaId(normalized.slice(prefix.length))}`; + } + } + return resolveDiscordQaId(normalized); +} + function resolveTelegramQaTarget(target: string) { const normalized = target.trim(); if (normalized.startsWith("thread:")) { @@ -190,7 +225,9 @@ export function createCrablineProviderInboundInput( ? resolveMatrixQaConversationId(input.conversation.id) : adapter.channel === "mattermost" ? resolveMattermostQaId(input.conversation.id) - : input.conversation.id, + : adapter.channel === "discord" + ? resolveDiscordQaId(input.conversation.id) + : input.conversation.id, kind, }, senderId: @@ -200,11 +237,18 @@ export function createCrablineProviderInboundInput( ? resolveMatrixQaSenderId(input.senderId) : adapter.channel === "mattermost" ? resolveMattermostQaId(input.senderId) - : input.senderId, + : adapter.channel === "discord" + ? resolveDiscordQaId(input.senderId) + : input.senderId, text: adapter.channel === "matrix" && adapter.manifest.provider === "matrix" ? resolveMatrixQaText(input.text, adapter.manifest.botUserId) - : input.text, + : adapter.channel === "discord" && adapter.manifest.provider === "discord" + ? resolveDiscordQaText(input.text, adapter.manifest.botUserId) + : input.text, + ...(input.threadId && adapter.channel === "discord" + ? { threadId: resolveDiscordQaId(input.threadId) } + : {}), }; } @@ -213,7 +257,7 @@ export function resolveCrablineStateConversation(params: { input: QaBusInboundMessageInput; providerInbound: OpenClawCrablineInbound; }) { - return ["mattermost", "matrix", "telegram"].includes(params.adapter.channel) + return ["discord", "mattermost", "matrix", "telegram"].includes(params.adapter.channel) ? params.input.conversation : params.providerInbound.stateConversation; } @@ -230,11 +274,17 @@ export function createCrablineProviderDelivery( ? resolveMatrixQaTarget(target) : adapter.channel === "mattermost" ? resolveMattermostQaTarget(target) - : target, + : adapter.channel === "discord" + ? resolveDiscordQaTarget(target) + : target, }); return { delivery, providerTargetKey: - adapter.channel === "matrix" ? delivery.to.replace(/^room:/u, "") : delivery.to, + adapter.channel === "matrix" + ? delivery.to.replace(/^room:/u, "") + : adapter.channel === "discord" + ? delivery.to.replace(/^(?:channel|user):/u, "") + : delivery.to, }; } diff --git a/extensions/qa-lab/src/crabline-transport.ts b/extensions/qa-lab/src/crabline-transport.ts index 8dbde783aed1..06113ffb0adf 100644 --- a/extensions/qa-lab/src/crabline-transport.ts +++ b/extensions/qa-lab/src/crabline-transport.ts @@ -15,10 +15,12 @@ import { readStringValue, } from "openclaw/plugin-sdk/string-coerce-runtime"; import { createQaBusState, type QaBusState } from "./bus-state.js"; +import { CRABLINE_DISCORD_PROVIDER_ENDPOINT_ARTIFACT } from "./crabline-discord-provider-endpoint-artifact.js"; import { createCrablineProviderDelivery, createCrablineProviderInboundInput, resolveCrablineStateConversation, + resolveDiscordQaId, resolveTelegramQaSenderId, } from "./crabline-provider-targets.js"; import { discardIgnoredResponseBody } from "./ignored-response-body.js"; @@ -88,6 +90,11 @@ function formatLogicalQaTarget({ conversation, threadId }: QaBusInboundMessageIn return threadId ? `thread:${conversation.id}/${threadId}` : `${prefix}:${conversation.id}`; } +function formatLogicalQaConversationTarget({ conversation }: QaBusInboundMessageInput) { + const prefix = conversation.kind === "direct" ? "dm" : conversation.kind; + return `${prefix}:${conversation.id}`; +} + const TELEGRAM_LIFECYCLE_METHOD_RE = /\/(sendMessage|editMessageText|deleteMessage)$/u; function readTelegramLifecycleEvent(params: { @@ -212,6 +219,7 @@ function createCrablineState(params: { }): QaCrablineTransportState { const baseState = params.state; const targetByProviderTarget = new Map(); + const logicalRouteByTarget = new Map(); const telegramMessageByProviderId = new Map(); const pendingTelegramMessagesByChat = new Map(); const outboundEvents: QaTransportOutboundEvent[] = []; @@ -220,6 +228,7 @@ function createCrablineState(params: { reset() { baseState.reset(); targetByProviderTarget.clear(); + logicalRouteByTarget.clear(); telegramMessageByProviderId.clear(); pendingTelegramMessagesByChat.clear(); outboundEvents.length = 0; @@ -258,7 +267,16 @@ function createCrablineState(params: { targetByProviderTarget, }) as QaBusOutboundMessageInput | null; if (outbound) { - baseState.addOutboundMessage(outbound); + const logicalRoute = logicalRouteByTarget.get(outbound.to); + baseState.addOutboundMessage( + logicalRoute + ? { + ...outbound, + to: logicalRoute.target, + ...(logicalRoute.threadId ? { threadId: logicalRoute.threadId } : {}), + } + : outbound, + ); } }, async addInboundMessage(input: QaBusInboundMessageInput) { @@ -284,7 +302,12 @@ function createCrablineState(params: { } // Providers may coerce channel conversations to groups; preserve the scenario's logical // target so outbound waits and assertions still match the original input. - targetByProviderTarget.set(providerInbound.providerTargetKey, formatLogicalQaTarget(input)); + const logicalTarget = formatLogicalQaTarget(input); + targetByProviderTarget.set(providerInbound.providerTargetKey, logicalTarget); + logicalRouteByTarget.set(logicalTarget, { + target: formatLogicalQaConversationTarget(input), + ...(input.threadId ? { threadId: input.threadId } : {}), + }); const providerMessageId = await postCrablineInbound({ adapter: params.adapter, providerInbound, @@ -297,7 +320,11 @@ function createCrablineState(params: { input, providerInbound, }), - ...(providerInbound.threadId ? { threadId: providerInbound.threadId } : {}), + ...(input.threadId + ? { threadId: input.threadId } + : providerInbound.threadId + ? { threadId: providerInbound.threadId } + : {}), }, providerMessageId, ); @@ -320,6 +347,7 @@ class QaCrablineTransport extends QaStateBackedTransportAdapter { readonly #selection: OpenClawCrablineChannelDriverSelection; readonly #transportPolicy?: QaTransportPolicy; readonly #state: QaCrablineTransportState; + #cleanupPromise?: Promise; readonly sendNativeCommand?: (input: QaTransportNativeCommandInput) => Promise; readonly waitForOutboundSequence?: (input: QaTransportOutboundSequenceMatch) => Promise<{ events: QaTransportOutboundEvent[]; @@ -336,7 +364,10 @@ class QaCrablineTransport extends QaStateBackedTransportAdapter { id: CRABLINE_TRANSPORT_ID, label: `crabline local ${params.selection.channel}`, accountId: params.adapter.accountId, - requiredPluginIds: params.adapter.requiredPluginIds, + requiredPluginIds: + params.selection.channel === "discord" + ? ["qa-lab", ...params.adapter.requiredPluginIds] + : params.adapter.requiredPluginIds, state: params.state, }); this.#adapter = params.adapter; @@ -367,6 +398,50 @@ class QaCrablineTransport extends QaStateBackedTransportAdapter { this.#selection.channel === "signal" ? normalizeCrablineSignalGatewayConfig(rawConfig) : rawConfig; + if (this.#selection.channel === "discord") { + const discord = config.channels?.discord; + const senderAllowlist = this.#transportPolicy?.senderAllowlist?.map(resolveDiscordQaId); + // Crabline's local binding opens DMs, so the real Discord plugin also requires the matching + // allowlist. Keep an explicit sender restriction; otherwise its open policy uses a wildcard. + const dmAllowlist = senderAllowlist ?? discord?.allowFrom ?? ["*"]; + const discordWithOpenDms = { + ...discord, + allowFrom: [...dmAllowlist], + ...(dmAllowlist.includes("*") ? {} : { dmPolicy: "allowlist" as const }), + }; + const wildcardGuild = discord?.guilds?.["*"]; + const wildcardChannel = wildcardGuild?.channels?.["*"]; + if (!this.#transportPolicy?.requireGroupMention && !senderAllowlist) { + return { + ...config, + channels: { ...config.channels, discord: discordWithOpenDms }, + } as QaTransportGatewayConfig; + } + return { + ...config, + channels: { + ...config.channels, + discord: { + ...discordWithOpenDms, + ...(senderAllowlist ? { groupPolicy: "allowlist" as const } : {}), + guilds: { + ...discord?.guilds, + "*": { + ...wildcardGuild, + ...(senderAllowlist ? { users: [...senderAllowlist] } : {}), + channels: { + ...wildcardGuild?.channels, + "*": { + ...wildcardChannel, + ...(this.#transportPolicy?.requireGroupMention ? { requireMention: true } : {}), + }, + }, + }, + }, + }, + }, + } as QaTransportGatewayConfig; + } if (this.#selection.channel !== "telegram") { return config as QaTransportGatewayConfig; } @@ -414,6 +489,26 @@ class QaCrablineTransport extends QaStateBackedTransportAdapter { createRuntimeEnvPatch = () => this.#adapter.createProviderReadinessEnv({}); + stageGatewayRuntime = async ({ tempRoot }: { tempRoot: string }) => { + if (this.#adapter.manifest.provider !== "discord") { + return; + } + const gatewayUrl = new URL(this.#adapter.manifest.endpoints.gatewayUrl); + await fs.writeFile( + path.join(tempRoot, CRABLINE_DISCORD_PROVIDER_ENDPOINT_ARTIFACT), + `${JSON.stringify( + { + restApiBaseUrl: `${this.#adapter.manifest.endpoints.apiRoot}/v10`, + gatewayBotUrl: this.#adapter.manifest.endpoints.gatewayBotUrl, + gatewayOrigin: gatewayUrl.origin, + }, + null, + 2, + )}\n`, + { encoding: "utf8", flag: "wx", mode: 0o600 }, + ); + }; + handleAction = async (_params: { action: QaTransportActionName; args: Record; @@ -428,8 +523,24 @@ class QaCrablineTransport extends QaStateBackedTransportAdapter { "No live channel service or external credential lease is required.", ]; + #cleanupTransport() { + this.#cleanupPromise ??= this.#state.cleanup(); + void this.#cleanupPromise.catch(() => { + this.#cleanupPromise = undefined; + }); + return this.#cleanupPromise; + } + async cleanup() { - await this.#state.cleanup(); + if (this.#selection.channel !== "discord") { + await this.#cleanupTransport(); + } + } + + async cleanupAfterGatewayStop() { + if (this.#selection.channel === "discord") { + await this.#cleanupTransport(); + } } } @@ -439,12 +550,16 @@ export async function createQaCrablineTransportAdapter(params: { selection: OpenClawCrablineChannelDriverSelection; state?: QaBusState; }) { - const requiresTelegramPolicy = + const requiresGroupPolicy = params.transportPolicy?.requireGroupMention === true || params.transportPolicy?.senderAllowlist !== undefined; - if (params.selection.channel !== "telegram" && requiresTelegramPolicy) { + if ( + params.selection.channel !== "telegram" && + params.selection.channel !== "discord" && + requiresGroupPolicy + ) { throw new Error( - `Crabline ${params.selection.channel} does not support the requested group transport policy; use the Crabline Telegram bridge or a live channel adapter`, + `Crabline ${params.selection.channel} does not support the requested group transport policy; use the Crabline Telegram or Discord bridge, or a live channel adapter`, ); } const recorderPath = path.join( diff --git a/extensions/qa-lab/src/discord-crabline-roundtrip.e2e.test.ts b/extensions/qa-lab/src/discord-crabline-roundtrip.e2e.test.ts new file mode 100644 index 000000000000..670db08c3c54 --- /dev/null +++ b/extensions/qa-lab/src/discord-crabline-roundtrip.e2e.test.ts @@ -0,0 +1,148 @@ +import fs from "node:fs/promises"; +import path from "node:path"; +import { resolveOpenClawCrablineChannelDriverSelection } from "@openclaw/crabline"; +import { describe, expect, it } from "vitest"; +import { CRABLINE_DISCORD_PROVIDER_ENDPOINT_ARTIFACT } from "./crabline-discord-provider-endpoint-artifact.js"; +import { runQaSuite } from "./suite-launch.runtime.js"; + +const RUN_DISCORD_CRABLINE_E2E = process.env.OPENCLAW_QA_DISCORD_CRABLINE_E2E === "1"; +const SCENARIO_ID = "discord-crabline-roundtrip"; +const EXPECTED_MARKER = "DISCORD-CRABLINE-ROUNDTRIP-OK"; + +type RecorderEvent = { + accepted?: boolean; + body?: Record; + method?: string; + path?: string; + type?: string; +}; + +function readObject(value: unknown): Record | undefined { + return value && typeof value === "object" && !Array.isArray(value) + ? (value as Record) + : undefined; +} + +function readString(value: unknown): string { + return typeof value === "string" ? value : ""; +} + +async function readRecorderEvents(recorderPath: string): Promise { + const raw = await fs.readFile(recorderPath, "utf8"); + if (raw.includes("discord.com") || raw.includes("discordapp.com")) { + throw new Error("Discord Crabline recorder contains a public Discord service target."); + } + return raw + .split("\n") + .filter(Boolean) + .map((line) => JSON.parse(line) as RecorderEvent); +} + +describe("Discord Crabline real-plugin roundtrip", () => { + it.runIf(RUN_DISCORD_CRABLINE_E2E)( + "crosses the real Discord REST and Gateway boundaries and flushes cleanup evidence", + async () => { + const repoRoot = process.cwd(); + const outputDir = path.join( + repoRoot, + ".artifacts", + "qa-e2e", + `discord-crabline-roundtrip-${process.pid}-${Date.now()}`, + ); + const selection = resolveOpenClawCrablineChannelDriverSelection({ channel: "discord" }); + + const suite = await runQaSuite({ + channelDriver: "crabline", + channelDriverSelection: selection, + channelId: "discord", + controlUiEnabled: false, + outputDir, + primaryModel: "mock-openai/gpt-5.6-luna", + providerMode: "mock-openai", + repoRoot, + scenarioIds: [SCENARIO_ID], + }); + + expect(suite.executionKind).toBe("flow"); + expect(suite.result.scenarios).toEqual([ + expect.objectContaining({ name: expect.any(String), status: "pass" }), + ]); + + const recorderPath = path.join( + suite.result.outputDir, + "artifacts", + "crabline", + "discord-provider-server.jsonl", + ); + const events = await readRecorderEvents(recorderPath); + const gatewayMetadata = events.find( + (event) => + event.type === "api" && + event.method === "GET" && + event.path === "/api/v10/gateway/bot" && + event.accepted === true, + ); + const gatewayIdentify = events.find( + (event) => + event.type === "api" && + event.method === "WS" && + event.path === "/gateway" && + readObject(event.body)?.op === 2 && + event.accepted === true, + ); + expect(gatewayMetadata).toBeDefined(); + expect(gatewayIdentify).toBeDefined(); + + const inbound = events.find( + (event) => + event.type === "admin" && + event.method === "POST" && + event.path === "/crabline/discord/inbound" && + event.accepted === true, + ); + const inboundBody = readObject(inbound?.body); + const inboundChannelId = readString(inboundBody?.channelId); + const parentChannelId = readString(inboundBody?.parentChannelId); + expect(inboundChannelId).toMatch(/^\d{17,20}$/u); + expect(parentChannelId).toMatch(/^\d{17,20}$/u); + expect(inboundChannelId).not.toBe(parentChannelId); + expect(readString(inboundBody?.content)).toMatch(/<@\d{17,20}>/u); + + const outbound = events.find( + (event) => + event.type === "api" && + event.method === "POST" && + event.path === `/api/v10/channels/${inboundChannelId}/messages` && + readString(readObject(event.body)?.content).includes(EXPECTED_MARKER) && + event.accepted === true, + ); + const outboundBody = readObject(outbound?.body); + const messageReference = readObject(outboundBody?.message_reference); + expect(outbound).toBeDefined(); + expect(readString(outboundBody?.content)).toContain(EXPECTED_MARKER); + expect(readString(outboundBody?.content)).not.toMatch(/<@\d{17,20}>/u); + expect(messageReference).toMatchObject({ + message_id: expect.stringMatching(/^\d{17,20}$/u), + }); + + expect( + events.some( + (event) => + event.type === "api" && + event.accepted === true && + (event.method === "PUT" || event.method === "POST") && + /^\/api\/v10\/applications\/\d{17,20}\/commands$/u.test(event.path ?? ""), + ), + ).toBe(true); + + // Suite return occurs only after the child Gateway and its WebSocket are stopped, then the + // Discord HTTP server and recorder are closed. The QA-owned descriptor lives in the removed + // child temp root and must never escape into durable suite artifacts. + await expect( + fs.access(path.join(suite.result.outputDir, CRABLINE_DISCORD_PROVIDER_ENDPOINT_ARTIFACT)), + ).rejects.toMatchObject({ code: "ENOENT" }); + await expect(fs.readFile(recorderPath, "utf8")).resolves.toContain(EXPECTED_MARKER); + }, + 180_000, + ); +}); diff --git a/extensions/qa-lab/src/gateway-child.ts b/extensions/qa-lab/src/gateway-child.ts index f11f375a3e62..063070c8308f 100644 --- a/extensions/qa-lab/src/gateway-child.ts +++ b/extensions/qa-lab/src/gateway-child.ts @@ -100,11 +100,14 @@ export type QaGatewayChildListeningContext = { runtimeEnv: NodeJS.ProcessEnv; }; -function createQaGatewayEmptyTransport() { +function createQaGatewayEmptyTransport(): Pick< + QaTransportAdapter, + "requiredPluginIds" | "createGatewayConfig" | "stageGatewayRuntime" +> { return { requiredPluginIds: [] as const, createGatewayConfig: () => ({}), - } satisfies Pick; + }; } function appendQaGatewayTempRoot(details: string, tempRoot: string) { @@ -222,7 +225,10 @@ export async function startQaGatewayChild(params: { command?: QaGatewayChildCommand; useRepoCli?: boolean; providerBaseUrl?: string; - transport?: Pick; + transport?: Pick< + QaTransportAdapter, + "requiredPluginIds" | "createGatewayConfig" | "stageGatewayRuntime" + >; transportBaseUrl: string; controlUiAllowedOrigins?: string[]; providerMode?: QaProviderMode; @@ -278,6 +284,7 @@ export async function startQaGatewayChild(params: { const packagedAuthConfigPath = path.join(stateDir, "qa-auth-bootstrap", "openclaw.json"); const gatewayToken = `qa-suite-${randomUUID()}`; const transport = params.transport ?? createQaGatewayEmptyTransport(); + await transport.stageGatewayRuntime?.({ tempRoot }); await seedQaAgentWorkspace({ workspaceDir, repoRoot: params.repoRoot, @@ -389,7 +396,6 @@ export async function startQaGatewayChild(params: { let packagedMockAuthStaged = false; let migrationConvergenceRestartUsed = false; let reuseStartupLaunchState = false; - const nodeExecPath = gatewayExecutablePath ?? (await resolveQaNodeExecPath()); const cliArgsPrefix = gatewayExecutablePath ? gatewayArgsPrefix diff --git a/extensions/qa-lab/src/live-transports/cli.test.ts b/extensions/qa-lab/src/live-transports/cli.test.ts index 65f6e3c38ca3..be9825bdd48e 100644 --- a/extensions/qa-lab/src/live-transports/cli.test.ts +++ b/extensions/qa-lab/src/live-transports/cli.test.ts @@ -66,4 +66,31 @@ describe("live transport QA contributions", () => { expect.objectContaining({ scenarioIds: ["telegram-canary"] }), ); }); + + it("maps the dedicated Discord driver and listing options", async () => { + const registration = listLiveTransportQaCliRegistrations().find( + (candidate) => candidate.commandName === "discord", + ); + const qa = new Command(); + registration?.register(qa); + + await qa.parseAsync([ + "node", + "openclaw", + "discord", + "--channel-driver", + "crabline", + "--list-scenarios", + ]); + + expect(runLiveTransportQaSuiteCommand).toHaveBeenCalledWith( + expect.objectContaining({ + channelId: "discord", + options: expect.objectContaining({ + channelDriver: "crabline", + listScenarios: true, + }), + }), + ); + }); }); diff --git a/extensions/qa-lab/src/live-transports/discord/cli.ts b/extensions/qa-lab/src/live-transports/discord/cli.ts index 3865d1a7aa83..e886543f0d10 100644 --- a/extensions/qa-lab/src/live-transports/discord/cli.ts +++ b/extensions/qa-lab/src/live-transports/discord/cli.ts @@ -38,7 +38,10 @@ export const discordQaCliRegistration: LiveTransportQaCliRegistration = roleDescription: "Credential role for convex auth: maintainer or ci (default: ci in CI, maintainer otherwise)", }, - description: "Run the Discord live QA lane against a private guild bot-to-bot harness", + channelDriverHelp: + "Discord transport boundary: live (default) or Crabline local provider server", + description: "Run Discord QA through the live service or Crabline local provider server", + listScenariosHelp: "Print the selected Discord scenario ids and exit", outputDirHelp: "Discord QA artifact directory", scenarioHelp: "Run only the named Discord QA scenario (repeatable)", sutAccountHelp: "Temporary Discord account id inside the QA gateway config", diff --git a/extensions/qa-lab/src/live-transports/discord/scenario-selection.ts b/extensions/qa-lab/src/live-transports/discord/scenario-selection.ts index dcf0395c4799..5d966ea2fbe8 100644 --- a/extensions/qa-lab/src/live-transports/discord/scenario-selection.ts +++ b/extensions/qa-lab/src/live-transports/discord/scenario-selection.ts @@ -1,16 +1,19 @@ import type { QaProviderModeInput } from "../../model-selection.js"; -import { resolveLiveTransportQaScenarioIds } from "../shared/scenario-selection.js"; +import type { QaScorecardChannelDriver } from "../../scorecard-taxonomy.js"; +import { resolveTransportQaScenarioIds } from "../shared/scenario-selection.js"; export function resolveDiscordQaScenarioIds(params: { profile?: string; + channelDriver?: QaScorecardChannelDriver; primaryModel?: string; providerMode?: QaProviderModeInput; scenarioIds?: readonly string[]; }) { - return resolveLiveTransportQaScenarioIds({ + return resolveTransportQaScenarioIds({ channelId: "discord", - supportsModuleFlows: true, ...params, + channelDriver: params.channelDriver ?? "live", providerMode: params.providerMode ?? "live-frontier", + supportsModuleFlows: true, }); } diff --git a/extensions/qa-lab/src/live-transports/shared/live-transport-cli.ts b/extensions/qa-lab/src/live-transports/shared/live-transport-cli.ts index 29db66c87287..8c42606143ea 100644 --- a/extensions/qa-lab/src/live-transports/shared/live-transport-cli.ts +++ b/extensions/qa-lab/src/live-transports/shared/live-transport-cli.ts @@ -18,6 +18,7 @@ export type LiveTransportQaCommandOptions = { sutAccountId?: string; credentialSource?: string; credentialRole?: string; + channelDriver?: string; }; type LiveTransportQaCommanderOptions = { @@ -35,6 +36,7 @@ type LiveTransportQaCommanderOptions = { sutAccount?: string; credentialSource?: string; credentialRole?: string; + channelDriver?: string; }; export type LiveTransportQaCliRegistration = QaRunnerCliRegistration; @@ -45,6 +47,7 @@ type LiveTransportQaCliRegistrationOptions = { sourceDescription?: string; roleDescription?: string; }; + channelDriverHelp?: string; defaultProviderMode: string; description: string; providerModeHelp: string; @@ -88,6 +91,7 @@ function mapCommanderOptions(opts: LiveTransportQaCommanderOptions): LiveTranspo sutAccountId: opts.sutAccount, credentialSource: opts.credentialSource, credentialRole: opts.credentialRole, + channelDriver: opts.channelDriver, }; } @@ -132,6 +136,9 @@ function createSharedLiveTransportQaCliRegistration( command.option("--credential-role ", params.credentialOptions.roleDescription); } } + if (params.channelDriverHelp) { + command.option("--channel-driver ", params.channelDriverHelp); + } command.action(async (opts: LiveTransportQaCommanderOptions) => { await params.run(mapCommanderOptions(opts)); }); diff --git a/extensions/qa-lab/src/live-transports/shared/live-transport-suite.runtime.test.ts b/extensions/qa-lab/src/live-transports/shared/live-transport-suite.runtime.test.ts index 3c498bc724cf..7a27d8bad0b1 100644 --- a/extensions/qa-lab/src/live-transports/shared/live-transport-suite.runtime.test.ts +++ b/extensions/qa-lab/src/live-transports/shared/live-transport-suite.runtime.test.ts @@ -93,6 +93,114 @@ describe("live transport suite runtime", () => { ); }); + it.each([undefined, "live"] as const)( + "keeps the Discord live driver identical when selected as %s", + async (channelDriver) => { + const selectScenarioIds = vi.fn(() => ["discord-canary"]); + + await runLiveTransportQaSuiteCommand({ + channelId: "discord", + defaultProviderMode: "live-frontier", + options: { channelDriver }, + selectScenarioIds, + }); + + expect(selectScenarioIds).toHaveBeenCalledWith( + expect.objectContaining({ channelDriver: "live" }), + ); + expect(runQaSuiteCommand).toHaveBeenCalledWith( + expect.objectContaining({ + channel: "discord", + channelDriver: "live", + scenarioIds: ["discord-canary"], + }), + ); + }, + ); + + it("selects Discord Crabline without forwarding credential lease options", async () => { + const selectScenarioIds = vi.fn(() => ["discord-crabline-roundtrip"]); + + await runLiveTransportQaSuiteCommand({ + channelId: "discord", + defaultProviderMode: "live-frontier", + options: { + channelDriver: "crabline", + providerMode: "mock-openai", + primaryModel: "mock-openai/custom", + }, + selectScenarioIds, + }); + + expect(selectScenarioIds).toHaveBeenCalledWith({ + channelDriver: "crabline", + profile: undefined, + primaryModel: "mock-openai/custom", + providerMode: "mock-openai", + scenarioIds: undefined, + }); + expect(runQaSuiteCommand).toHaveBeenCalledWith( + expect.objectContaining({ + channel: "discord", + channelDriver: "crabline", + providerMode: "mock-openai", + primaryModel: "mock-openai/custom", + scenarioIds: ["discord-crabline-roundtrip"], + }), + ); + expect(runQaSuiteCommand.mock.calls[0]?.[0]).not.toHaveProperty("credentialSource"); + expect(runQaSuiteCommand.mock.calls[0]?.[0]).not.toHaveProperty("credentialRole"); + }); + + it("uses the same Discord selection for listing and execution", async () => { + const stdoutWrite = vi.spyOn(process.stdout, "write").mockImplementation(() => true); + const selectScenarioIds = vi.fn((_selection: unknown) => [ + "channel-canary", + "thread-follow-up", + ]); + const base = { + channelId: "discord", + defaultProviderMode: "mock-openai" as const, + selectScenarioIds, + }; + + try { + await runLiveTransportQaSuiteCommand({ + ...base, + options: { channelDriver: "crabline", listScenarios: true }, + }); + await runLiveTransportQaSuiteCommand({ + ...base, + options: { channelDriver: "crabline" }, + }); + } finally { + stdoutWrite.mockRestore(); + } + + expect(selectScenarioIds).toHaveBeenCalledTimes(2); + expect(selectScenarioIds.mock.calls[0]?.[0]).toEqual(selectScenarioIds.mock.calls[1]?.[0]); + expect(runQaSuiteCommand).toHaveBeenCalledWith( + expect.objectContaining({ scenarioIds: ["channel-canary", "thread-follow-up"] }), + ); + }); + + it.each(["credentialSource", "credentialRole"] as const)( + "rejects Discord Crabline with %s", + async (option) => { + await expect( + runLiveTransportQaSuiteCommand({ + channelId: "discord", + defaultProviderMode: "mock-openai", + options: { channelDriver: "crabline", [option]: "ci" }, + selectScenarioIds: () => ["channel-canary"], + }), + ).rejects.toThrow( + `do not use --${option === "credentialSource" ? "credential-source" : "credential-role"}`, + ); + expect(runQaSuiteCommand).not.toHaveBeenCalled(); + }, + ); + it("rejects shared credentials for disposable transports", async () => { await expect( runLiveTransportQaSuiteCommand({ diff --git a/extensions/qa-lab/src/live-transports/shared/live-transport-suite.runtime.ts b/extensions/qa-lab/src/live-transports/shared/live-transport-suite.runtime.ts index 14dff12084b6..21c6bc8c6df2 100644 --- a/extensions/qa-lab/src/live-transports/shared/live-transport-suite.runtime.ts +++ b/extensions/qa-lab/src/live-transports/shared/live-transport-suite.runtime.ts @@ -3,26 +3,47 @@ import { runQaSuiteCommand } from "../../cli.runtime.js"; import type { QaProviderMode } from "../../providers/index.js"; import { defaultQaModelForMode, normalizeQaProviderMode } from "../../run-config.js"; +type DedicatedLiveTransportQaCommandOptions = LiveTransportQaCommandOptions & { + channelDriver?: string; +}; + type LiveTransportScenarioSelection = (params: { + channelDriver: "live" | "crabline"; profile?: string; primaryModel: string; providerMode: QaProviderMode; scenarioIds?: readonly string[]; }) => string[]; +function resolveDedicatedChannelDriver(value: string | undefined): "live" | "crabline" { + const normalized = value?.trim().toLowerCase() || "live"; + if (normalized !== "live" && normalized !== "crabline") { + throw new Error(`channel driver must be live or crabline, got "${value}".`); + } + return normalized; +} + export async function runLiveTransportQaSuiteCommand(params: { channelId: string; credentialMode?: "env-only" | "shared-lease"; defaultProviderMode: QaProviderMode; envCredentialReason?: string; laneLabel?: string; - options: LiveTransportQaCommandOptions; + options: DedicatedLiveTransportQaCommandOptions; selectScenarioIds: LiveTransportScenarioSelection; }) { const options = params.options; const credentialSource = options.credentialSource?.trim() || process.env.OPENCLAW_QA_CREDENTIAL_SOURCE?.trim(); - if (params.credentialMode === "env-only") { + const channelDriver = resolveDedicatedChannelDriver(options.channelDriver); + if (channelDriver === "crabline") { + if (options.credentialSource?.trim()) { + throw new Error("QA Lab Crabline channel drivers do not use --credential-source."); + } + if (options.credentialRole?.trim()) { + throw new Error("QA Lab Crabline channel drivers do not use --credential-role."); + } + } else if (params.credentialMode === "env-only") { const laneLabel = params.laneLabel ?? params.channelId; if (credentialSource && credentialSource.toLowerCase() !== "env") { throw new Error( @@ -40,11 +61,18 @@ export async function runLiveTransportQaSuiteCommand(params: { : normalizeQaProviderMode(options.providerMode); const primaryModel = options.primaryModel?.trim() || defaultQaModelForMode(providerMode); const selectedScenarioIds = params.selectScenarioIds({ + channelDriver, profile: options.profile, primaryModel, providerMode, scenarioIds: options.scenarioIds, }); + if (options.listScenarios) { + for (const scenarioId of selectedScenarioIds) { + process.stdout.write(`${scenarioId}\n`); + } + return undefined; + } return runQaSuiteCommand({ repoRoot: options.repoRoot, outputDir: options.outputDir, @@ -54,13 +82,13 @@ export async function runLiveTransportQaSuiteCommand(params: { fastMode: options.fastMode, allowFailures: options.allowFailures, failFast: options.failFast, - channelDriver: "live", + channelDriver, channel: params.channelId, concurrency: 1, scenarioIds: selectedScenarioIds, sutAccountId: options.sutAccountId, ...(options.credentialFile ? { credentialFile: options.credentialFile } : {}), - ...(params.credentialMode === "env-only" + ...(channelDriver === "crabline" || params.credentialMode === "env-only" ? {} : { credentialSource, diff --git a/extensions/qa-lab/src/live-transports/shared/scenario-selection.test.ts b/extensions/qa-lab/src/live-transports/shared/scenario-selection.test.ts index 58db609027a3..de4061339c88 100644 --- a/extensions/qa-lab/src/live-transports/shared/scenario-selection.test.ts +++ b/extensions/qa-lab/src/live-transports/shared/scenario-selection.test.ts @@ -140,6 +140,7 @@ describe("live transport QA scenario selection", () => { { channelId: "matrix", scenarioId: "thread-follow-up" }, { channelId: "telegram", scenarioId: "channel-canary" }, { channelId: "telegram", scenarioId: "channel-message-flows" }, + { channelId: "discord", scenarioId: "channel-canary" }, ] as const)( "keeps $scenarioId eligible through both $channelId drivers", ({ channelId, scenarioId }) => { @@ -185,4 +186,22 @@ describe("live transport QA scenario selection", () => { "selected QA scenario(s) do not match the current QA lane: channel-canary (channel=qa-channel|telegram|buzz|msteams)", ); }); + + it("defaults dedicated Discord selection to live and excludes live-only modules from Crabline", () => { + const implicitLive = resolveDiscordQaScenarioIds(MOCK_LANE); + const explicitLive = resolveDiscordQaScenarioIds({ ...MOCK_LANE, channelDriver: "live" }); + const crabline = resolveDiscordQaScenarioIds({ ...MOCK_LANE, channelDriver: "crabline" }); + + expect(explicitLive).toEqual(implicitLive); + expect(implicitLive).toContain("discord-canary"); + expect(crabline).toContain("discord-crabline-roundtrip"); + expect(crabline).not.toContain("discord-canary"); + expect(() => + resolveDiscordQaScenarioIds({ + ...MOCK_LANE, + channelDriver: "crabline", + scenarioIds: ["discord-canary"], + }), + ).toThrow("channelDriver=live"); + }); }); diff --git a/extensions/qa-lab/src/live-transports/shared/scenario-selection.ts b/extensions/qa-lab/src/live-transports/shared/scenario-selection.ts index 9003fbc16c19..8e7292801e9f 100644 --- a/extensions/qa-lab/src/live-transports/shared/scenario-selection.ts +++ b/extensions/qa-lab/src/live-transports/shared/scenario-selection.ts @@ -41,19 +41,29 @@ export function resolveCatalogLiveTransportQaScenarioIds(params: { return selectedScenarios.map((scenario) => scenario.id); } -export function resolveLiveTransportQaScenarioIds(params: { +export function resolveTransportQaScenarioIds(params: { channelId: string; + channelDriver: QaScorecardChannelDriver; profile?: string; primaryModel?: string; providerMode: QaProviderModeInput; scenarioIds?: readonly string[]; supportsModuleFlows?: boolean; }) { + if (!params.profile?.trim() && params.channelDriver === "crabline") { + return resolveCatalogLiveTransportQaScenarioIds({ + channelId: params.channelId, + channelDriver: params.channelDriver, + primaryModel: params.primaryModel, + providerMode: params.providerMode, + scenarioIds: params.scenarioIds, + }); + } return resolveQaProfileScenarios({ profile: params.profile?.trim() || "release", providerMode: params.providerMode, primaryModel: params.primaryModel, - channelDriver: "live", + channelDriver: params.channelDriver, channel: params.channelId, executionKind: "flow", requireDeclaredChannel: true, @@ -62,6 +72,12 @@ export function resolveLiveTransportQaScenarioIds(params: { }).scenarios.map((scenario) => scenario.id); } +export function resolveLiveTransportQaScenarioIds( + params: Omit[0], "channelDriver">, +) { + return resolveTransportQaScenarioIds({ ...params, channelDriver: "live" }); +} + export function listLiveTransportQaScenarios(params: { channelId: string; primaryModel?: string; diff --git a/extensions/qa-lab/src/qa-transport.ts b/extensions/qa-lab/src/qa-transport.ts index 3aba2e13995f..bdeffa038a13 100644 --- a/extensions/qa-lab/src/qa-transport.ts +++ b/extensions/qa-lab/src/qa-transport.ts @@ -315,6 +315,7 @@ export type QaTransportAdapter = Omit< timeoutMs?: number, intervalMs?: number, ) => Promise; + stageGatewayRuntime?: (params: { tempRoot: string }) => Promise; }; export abstract class QaStateBackedTransportAdapter implements QaTransportAdapter { diff --git a/extensions/qa-lab/src/run-config.test.ts b/extensions/qa-lab/src/run-config.test.ts index dc05c91f2386..67a0c73808a4 100644 --- a/extensions/qa-lab/src/run-config.test.ts +++ b/extensions/qa-lab/src/run-config.test.ts @@ -624,6 +624,71 @@ describe("qa run config", () => { ); }); + it("selects only scenarios that declare an explicit external channel", () => { + const catalog = readQaScenarioPack(); + const scorecardReport = readQaScorecardTaxonomyReport(catalog.scenarios); + const selection = normalizeQaRunSelection( + { + profile: "all", + channel: "discord", + channelDriver: "crabline", + providerMode: "mock-openai", + }, + catalog.scenarios, + scorecardReport.profiles, + ); + + const plan = resolveQaLabRunPlan({ + selection, + scenarios: catalog.scenarios, + scorecardReport, + defaultChannel: "discord", + supportsChannel: (channel) => channel === "discord", + }); + + expect(plan.status).toBe("ready"); + expect(plan.selectedScenarios.map((scenario) => scenario.id)).toEqual([ + "discord-crabline-roundtrip", + ]); + expect(plan.exclusions).toEqual( + expect.arrayContaining([ + expect.objectContaining({ + scenarioId: "compaction-retry-mutating-tool", + reasons: ["does not declare channel discord"], + }), + ]), + ); + }); + + it("rejects an explicit scenario that does not declare the external channel", () => { + const catalog = readQaScenarioPack(); + const scorecardReport = readQaScorecardTaxonomyReport(catalog.scenarios); + const selection = normalizeQaRunSelection( + { + profile: "all", + channel: "discord", + channelDriver: "crabline", + providerMode: "mock-openai", + scenarioIds: ["compaction-retry-mutating-tool"], + }, + catalog.scenarios, + scorecardReport.profiles, + ); + + const plan = resolveQaLabRunPlan({ + selection, + scenarios: catalog.scenarios, + scorecardReport, + defaultChannel: "discord", + supportsChannel: (channel) => channel === "discord", + }); + + expect(plan.status).toBe("invalid"); + expect(plan.errors).toContain( + "Explicit QA scenario selection is not runnable: compaction-retry-mutating-tool (does not declare channel discord).", + ); + }); + it("fails closed when an explicit scenario conflicts with execution.channel", () => { const catalog = readQaScenarioPack(); const scorecardReport = readQaScorecardTaxonomyReport(catalog.scenarios); diff --git a/extensions/qa-lab/src/run-config.ts b/extensions/qa-lab/src/run-config.ts index d07dd30e1f26..08c8e6e2a0b5 100644 --- a/extensions/qa-lab/src/run-config.ts +++ b/extensions/qa-lab/src/run-config.ts @@ -16,6 +16,7 @@ import { import { resolveQaRunProfileExecutionSelection, resolveQaRunProfileMembership, + scenarioDeclaresQaChannel, } from "./profile-planning.js"; import { DEFAULT_QA_LIVE_PROVIDER_MODE, @@ -360,8 +361,27 @@ export function resolveQaLabRunPlan(params: { const scenario = scenarioById.get(scenarioId); return scenario ? [scenario] : []; }); + const requiresDeclaredChannel = Boolean( + selection.channel && + selection.channel !== "qa-channel" && + selection.channelDriver !== "qa-channel", + ); + const channelScopedScenarios = requiresDeclaredChannel + ? laneScenarios.filter((scenario) => scenarioDeclaresQaChannel(scenario, selection.channel!)) + : laneScenarios; + if (requiresDeclaredChannel) { + exclusions.push( + ...laneScenarios + .filter((scenario) => !channelScopedScenarios.includes(scenario)) + .map((scenario) => ({ + scenarioId: scenario.id, + executionKind: scenario.execution.kind, + reasons: [`does not declare channel ${selection.channel}`], + })), + ); + } const profileExecution = resolveQaRunProfileExecutionSelection({ - scenarios: laneScenarios, + scenarios: channelScopedScenarios, providerMode: selection.providerMode, primaryModel: selection.primaryModel, channelDriver: selection.channelDriver, diff --git a/extensions/qa-lab/src/suite-planning.test.ts b/extensions/qa-lab/src/suite-planning.test.ts index 7f2cd629aa58..3cce9ca6b48f 100644 --- a/extensions/qa-lab/src/suite-planning.test.ts +++ b/extensions/qa-lab/src/suite-planning.test.ts @@ -990,8 +990,8 @@ describe("qa suite planning helpers", () => { channel: "telegram", }).map((scenario) => scenario.id); - expect(selectForDriver("crabline")).toEqual(["generic", "telegram"]); - expect(selectForDriver("live")).toEqual(["generic", "live-only", "telegram"]); + expect(selectForDriver("crabline")).toEqual(["telegram"]); + expect(selectForDriver("live")).toEqual(["live-only", "telegram"]); }); it("rejects explicitly requested scenarios that do not match the current lane", () => { diff --git a/extensions/qa-lab/src/suite-planning.ts b/extensions/qa-lab/src/suite-planning.ts index 718dd070a4a5..e7e32ca5685f 100644 --- a/extensions/qa-lab/src/suite-planning.ts +++ b/extensions/qa-lab/src/suite-planning.ts @@ -7,6 +7,7 @@ import { createQaArtifactRunId } from "./artifact-run-id.js"; import { ensureRepoBoundDirectory, resolveRepoRelativeOutputDir } from "./cli-paths.js"; import type { QaCliBackendAuthMode } from "./gateway-child.js"; import { splitQaModelRef as splitModelRef, type QaProviderMode } from "./model-selection.js"; +import { scenarioDeclaresQaChannel } from "./profile-planning.js"; import { readQaBootstrapScenarioCatalog } from "./scenario-catalog.js"; import { describeQaProviderLaneMismatches, diff --git a/qa/scenarios/channels/discord-canary.yaml b/qa/scenarios/channels/discord-canary.yaml index 0719f5a3c757..664325ee1c44 100644 --- a/qa/scenarios/channels/discord-canary.yaml +++ b/qa/scenarios/channels/discord-canary.yaml @@ -8,6 +8,8 @@ scenario: execution: kind: flow channel: discord + config: + requiredChannelDriver: live timeoutMs: 45000 retryCount: 1 suiteIsolation: isolated diff --git a/qa/scenarios/channels/discord-crabline-roundtrip.yaml b/qa/scenarios/channels/discord-crabline-roundtrip.yaml new file mode 100644 index 000000000000..770bacc8e431 --- /dev/null +++ b/qa/scenarios/channels/discord-crabline-roundtrip.yaml @@ -0,0 +1,78 @@ +title: Discord Crabline real-plugin roundtrip + +scenario: + id: discord-crabline-roundtrip + surface: channels + coverage: + primary: + - discord.configured-and-runtime-routing + secondary: + - discord.mention-gating + - discord.native-slash-command-registration + objective: Verify a provider-native Discord event traverses the real bundled plugin and produces one provider REST reply. + successCriteria: + - The real Discord plugin completes its Gateway handshake against Crabline. + - One mentioned guild message produces one visible reply containing the exact marker. + - The reply stays scoped to the originating Discord channel. + docsRefs: + - docs/concepts/qa-e2e-automation.md + - docs/channels/discord.md + codeRefs: + - extensions/discord/src/monitor/provider.ts + - extensions/qa-lab/src/crabline-transport.ts + execution: + kind: flow + channel: discord + config: + requiredChannelDriver: crabline + conversationId: discord-crabline-primary + threadId: discord-crabline-thread + expectedMarker: DISCORD-CRABLINE-ROUNDTRIP-OK + transportPolicy: + requireGroupMention: true + timeoutMs: 60000 + suiteIsolation: isolated + +flow: + steps: + - name: completes Discord Gateway readiness + actions: + - call: waitForGatewayHealthy + args: [{ ref: env }, 60000] + - call: waitForTransportReady + args: [{ ref: env }, 60000] + - resetTransport: true + - name: injects an authenticated provider-native Discord event + actions: + - sendInbound: + conversation: + id: { ref: config.conversationId } + kind: group + senderId: driver + senderName: QA Driver + threadId: { ref: config.threadId } + text: + expr: "`@openclaw reply exactly: ${config.expectedMarker}`" + saveAs: inbound + - name: observes the real Discord plugin outbound REST request + actions: + - waitForOutbound: + conversation: + id: { ref: config.conversationId } + kind: group + threadId: { ref: config.threadId } + textIncludes: { ref: config.expectedMarker } + timeoutMs: + expr: liveTurnTimeoutMs(env, 60000) + saveAs: outbound + - set: matchingOutbound + value: + expr: "state.getSnapshot().messages.filter((message) => message.direction === 'outbound' && message.conversation.id === config.conversationId && String(message.text ?? '').includes(config.expectedMarker))" + - assert: + expr: matchingOutbound.length === 1 + message: + expr: "`expected one Discord Crabline reply, saw ${matchingOutbound.length}; transcript=${formatTransportTranscript(state, { conversationId: config.conversationId })}`" + - assert: + expr: Boolean(inbound.id) && Boolean(outbound.id) + message: Discord Crabline roundtrip did not preserve provider message identity + detailsExpr: outbound.text diff --git a/qa/scenarios/channels/discord-mention-gating.yaml b/qa/scenarios/channels/discord-mention-gating.yaml index e0df973743ae..14303be56dfa 100644 --- a/qa/scenarios/channels/discord-mention-gating.yaml +++ b/qa/scenarios/channels/discord-mention-gating.yaml @@ -8,6 +8,8 @@ scenario: execution: kind: flow channel: discord + config: + requiredChannelDriver: live timeoutMs: 8000 retryCount: 1 suiteIsolation: isolated diff --git a/qa/scenarios/channels/discord-native-help-command-registration.yaml b/qa/scenarios/channels/discord-native-help-command-registration.yaml index 1c9c5b3a81bb..4361cb3c7378 100644 --- a/qa/scenarios/channels/discord-native-help-command-registration.yaml +++ b/qa/scenarios/channels/discord-native-help-command-registration.yaml @@ -7,6 +7,8 @@ scenario: execution: kind: flow channel: discord + config: + requiredChannelDriver: live timeoutMs: 45000 retryCount: 1 suiteIsolation: isolated diff --git a/qa/scenarios/channels/discord-status-reactions-tool-only.yaml b/qa/scenarios/channels/discord-status-reactions-tool-only.yaml index 4c50d09c3054..fb2d3c108602 100644 --- a/qa/scenarios/channels/discord-status-reactions-tool-only.yaml +++ b/qa/scenarios/channels/discord-status-reactions-tool-only.yaml @@ -8,6 +8,8 @@ scenario: execution: kind: flow channel: discord + config: + requiredChannelDriver: live timeoutMs: 75000 retryCount: 1 suiteIsolation: isolated diff --git a/qa/scenarios/channels/discord-thread-reply-filepath-attachment.yaml b/qa/scenarios/channels/discord-thread-reply-filepath-attachment.yaml index 07109ad0837d..f53f211aa5f0 100644 --- a/qa/scenarios/channels/discord-thread-reply-filepath-attachment.yaml +++ b/qa/scenarios/channels/discord-thread-reply-filepath-attachment.yaml @@ -9,6 +9,8 @@ scenario: execution: kind: flow channel: discord + config: + requiredChannelDriver: live timeoutMs: 45000 retryCount: 1 suiteIsolation: isolated diff --git a/qa/scenarios/channels/discord-voice-autojoin.yaml b/qa/scenarios/channels/discord-voice-autojoin.yaml index 7c6e354a4ddf..22efbdbf658f 100644 --- a/qa/scenarios/channels/discord-voice-autojoin.yaml +++ b/qa/scenarios/channels/discord-voice-autojoin.yaml @@ -7,6 +7,8 @@ scenario: execution: kind: flow channel: discord + config: + requiredChannelDriver: live timeoutMs: 60000 retryCount: 1 suiteIsolation: isolated