mirror of
https://github.com/openclaw/openclaw.git
synced 2026-08-25 11:55:47 -06:00
feat(qa): add Crabline Discord channel driver
This commit is contained in:
Dallin Romney
@@ -48,7 +48,7 @@ script aliases; both forms work.
|
||||
| `qa mock-openai` | Start only the scenario-aware `mock-openai` provider server. |
|
||||
| `qa credentials doctor` / `add` / `list` / `remove` | Manage the shared Convex credential pool. |
|
||||
| `qa buzz` | Live transport lane against a real Buzz relay room with dedicated driver and SUT identities. |
|
||||
| `qa discord` | Live transport lane against a real private Discord guild channel. |
|
||||
| `qa discord` | Discord transport lane through the real bundled plugin: live Discord by default, or Crabline's deterministic local provider server with `--channel-driver crabline`. |
|
||||
| `qa matrix` | QA Lab Matrix catalog scenarios against a disposable Tuwunel homeserver. See [Matrix live lane](#matrix-live-lane). |
|
||||
| `qa slack` | Live transport lane against a real private Slack channel. |
|
||||
| `qa telegram` | Live transport lane against a real private Telegram group. |
|
||||
@@ -458,8 +458,9 @@ guest can write back through the mounted workspace.
|
||||
## Buzz, Discord, Slack, Telegram, and WhatsApp QA reference
|
||||
|
||||
The Matrix adapter uses the disposable Docker-backed lane documented above.
|
||||
Buzz, Discord, Slack, Telegram, and WhatsApp run against pre-existing real
|
||||
transports, so their reference lives here.
|
||||
Buzz, Slack, Telegram, and WhatsApp run against pre-existing real transports.
|
||||
Discord can run against either live Discord or Crabline's deterministic local
|
||||
provider server. Their reference lives here.
|
||||
|
||||
### Shared CLI flags
|
||||
|
||||
@@ -481,10 +482,15 @@ accept the same flags:
|
||||
| `--credential-file <path>` | - | Buzz-only JSON credential file for local runs. |
|
||||
| `--allow-failures` | off | Write artifacts without returning a failing exit code when scenarios fail. |
|
||||
|
||||
Discord also accepts `--channel-driver <live|crabline>` (default `live`) and
|
||||
`--list-scenarios`. Crabline mode is local and rejects `--credential-source`
|
||||
and `--credential-role`; provider mode and model selection remain independent
|
||||
of the channel driver.
|
||||
|
||||
Each lane exits non-zero on any failed scenario. `--allow-failures` writes
|
||||
artifacts without setting a failing exit code. Telegram also accepts
|
||||
`--list-scenarios` to print available scenario ids and exit; the other lanes
|
||||
do not expose that flag.
|
||||
artifacts without setting a failing exit code. Telegram and Discord also accept
|
||||
`--list-scenarios` to print the scenario ids selected by the same profile,
|
||||
provider, model, execution-kind, and channel-driver rules used for execution.
|
||||
|
||||
### Buzz QA
|
||||
|
||||
@@ -581,13 +587,46 @@ creating a separate RTT command or Telegram-specific summary format.
|
||||
|
||||
```bash
|
||||
pnpm openclaw qa discord
|
||||
pnpm openclaw qa discord --channel-driver live
|
||||
```
|
||||
|
||||
Targets one real private Discord guild channel with two bots: a driver bot
|
||||
Both commands select the live driver. They target one real private Discord
|
||||
guild channel with two bots: a driver bot
|
||||
controlled by the harness and a SUT bot started by the child OpenClaw gateway
|
||||
through the bundled Discord plugin. Verifies channel mention handling, that
|
||||
the SUT bot has registered the native `/help` command with Discord, and
|
||||
opt-in Mantis evidence scenarios.
|
||||
opt-in Mantis evidence scenarios. Live Discord remains the release/provider
|
||||
coverage lane; catalog and profile membership determine its current scenario
|
||||
inventory.
|
||||
|
||||
For deterministic local transport proof without Discord credentials or a
|
||||
credential lease, run:
|
||||
|
||||
```bash
|
||||
pnpm openclaw qa discord \
|
||||
--channel-driver crabline \
|
||||
--provider-mode mock-openai
|
||||
|
||||
pnpm openclaw qa suite \
|
||||
--channel-driver crabline \
|
||||
--channel discord \
|
||||
--provider-mode mock-openai
|
||||
```
|
||||
|
||||
Crabline starts a separate Discord-shaped local provider server, then the child
|
||||
Gateway runs the real bundled OpenClaw Discord plugin against that server's REST
|
||||
and Gateway/WebSocket boundaries. The harness injects provider-native inbound
|
||||
events through Crabline's authenticated admin ingress and records the plugin's
|
||||
outbound REST traffic. This is not Crabline's fixture-level Discord local mock
|
||||
provider, which fixture commands call directly without running the OpenClaw
|
||||
plugin.
|
||||
|
||||
The local server covers Gateway handshake and message events, REST message
|
||||
delivery and reply metadata, mention handling, and native command registration.
|
||||
Use live Discord for behavior the local server cannot faithfully exercise,
|
||||
including voice, attachment/CDN and multipart behavior, interactions and
|
||||
activities, webhooks, Gateway compression or multi-shard behavior, replay, and
|
||||
Discord's distributed permissions and rate limits.
|
||||
|
||||
Required env when `--credential-source env`:
|
||||
|
||||
|
||||
@@ -4,6 +4,7 @@ import { setTimeout as sleep } from "node:timers/promises";
|
||||
import { definePluginEntry } from "openclaw/plugin-sdk/plugin-entry";
|
||||
import { jsonResult } from "openclaw/plugin-sdk/tool-results";
|
||||
import { registerQaLabCli } from "./src/cli.js";
|
||||
import { registerCrablineDiscordProviderEndpoint } from "./src/crabline-discord-provider-endpoint.js";
|
||||
import { createQaLabWebSearchProvider } from "./src/qa-web-search-provider.js";
|
||||
import { createStaticSshWorkerProvider } from "./src/static-ssh-worker-provider.js";
|
||||
|
||||
@@ -18,6 +19,7 @@ export default definePluginEntry({
|
||||
name: "QA Lab",
|
||||
description: "Private QA automation harness and debugger UI",
|
||||
register(api) {
|
||||
registerCrablineDiscordProviderEndpoint(api);
|
||||
api.registerTool(
|
||||
{
|
||||
name: "qa_restart_wait",
|
||||
|
||||
@@ -9,7 +9,8 @@
|
||||
}
|
||||
],
|
||||
"activation": {
|
||||
"onStartup": false
|
||||
"onStartup": false,
|
||||
"onConfigPaths": ["plugins.entries.qa-lab"]
|
||||
},
|
||||
"contracts": {
|
||||
"webSearchProviders": ["qa-lab-search"],
|
||||
|
||||
@@ -1210,6 +1210,34 @@ describe("qa cli runtime", () => {
|
||||
});
|
||||
});
|
||||
|
||||
it("dispatches generic suite Discord selection through Crabline", async () => {
|
||||
await runQaSuiteCommand({
|
||||
channelDriver: "crabline",
|
||||
channel: "discord",
|
||||
providerMode: "mock-openai",
|
||||
scenarioIds: ["channel-canary"],
|
||||
});
|
||||
|
||||
expect(runQaSuite).toHaveBeenCalledWith(
|
||||
expect.objectContaining({
|
||||
channelDriver: "crabline",
|
||||
channelDriverSelection: expect.objectContaining({ channel: "discord" }),
|
||||
scenarioIds: ["channel-canary"],
|
||||
}),
|
||||
);
|
||||
});
|
||||
|
||||
it("rejects channels outside Crabline's canonical server contract", async () => {
|
||||
await expect(
|
||||
runQaSuiteCommand({
|
||||
channelDriver: "crabline",
|
||||
channel: "imessage",
|
||||
providerMode: "mock-openai",
|
||||
}),
|
||||
).rejects.toThrow("--channel must be one of");
|
||||
expect(runQaSuite).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("keeps implicit channel membership identical for live and Crabline drivers", async () => {
|
||||
await runQaSuiteCommand({
|
||||
channelDriver: "live",
|
||||
|
||||
@@ -0,0 +1,2 @@
|
||||
export const CRABLINE_DISCORD_PROVIDER_ENDPOINT_ARTIFACT =
|
||||
"crabline-discord-provider-endpoint.json";
|
||||
@@ -0,0 +1,89 @@
|
||||
import fs from "node:fs";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
import { createTestPluginApi } from "openclaw/plugin-sdk/plugin-test-api";
|
||||
import { afterEach, describe, expect, it, vi } from "vitest";
|
||||
|
||||
const setDiscordProviderEndpointDescriptor = vi.hoisted(() => vi.fn());
|
||||
|
||||
vi.mock("@openclaw/discord/provider-endpoint-api.js", () => ({
|
||||
setDiscordProviderEndpointDescriptor,
|
||||
}));
|
||||
|
||||
import { CRABLINE_DISCORD_PROVIDER_ENDPOINT_ARTIFACT } from "./crabline-discord-provider-endpoint-artifact.js";
|
||||
import { registerCrablineDiscordProviderEndpoint } from "./crabline-discord-provider-endpoint.js";
|
||||
|
||||
const QA_TEMP_ROOT_ENV = "OPENCLAW_QA_TEMP_ROOT";
|
||||
const originalTempRoot = process.env[QA_TEMP_ROOT_ENV];
|
||||
|
||||
afterEach(() => {
|
||||
if (originalTempRoot === undefined) {
|
||||
delete process.env[QA_TEMP_ROOT_ENV];
|
||||
} else {
|
||||
process.env[QA_TEMP_ROOT_ENV] = originalTempRoot;
|
||||
}
|
||||
setDiscordProviderEndpointDescriptor.mockReset();
|
||||
});
|
||||
|
||||
describe("Crabline Discord child provider endpoint", () => {
|
||||
it("loads the QA-owned artifact before startup and clears it during teardown", async () => {
|
||||
const tempRoot = fs.mkdtempSync(path.join(os.tmpdir(), "openclaw-qa-discord-endpoint-"));
|
||||
process.env[QA_TEMP_ROOT_ENV] = tempRoot;
|
||||
const descriptor = {
|
||||
restApiBaseUrl: "http://127.0.0.1:43123/api/v10",
|
||||
gatewayBotUrl: "http://127.0.0.1:43123/api/v10/gateway/bot",
|
||||
gatewayOrigin: "ws://127.0.0.1:43123",
|
||||
};
|
||||
fs.writeFileSync(
|
||||
path.join(tempRoot, CRABLINE_DISCORD_PROVIDER_ENDPOINT_ARTIFACT),
|
||||
`${JSON.stringify(descriptor)}\n`,
|
||||
{ mode: 0o600 },
|
||||
);
|
||||
const registerRuntimeLifecycle = vi.fn();
|
||||
|
||||
try {
|
||||
registerCrablineDiscordProviderEndpoint(
|
||||
createTestPluginApi({ registrationMode: "full", registerRuntimeLifecycle }),
|
||||
);
|
||||
|
||||
expect(setDiscordProviderEndpointDescriptor).toHaveBeenCalledWith(descriptor);
|
||||
expect(registerRuntimeLifecycle).toHaveBeenCalledOnce();
|
||||
expect(setDiscordProviderEndpointDescriptor.mock.invocationCallOrder[0]).toBeLessThan(
|
||||
registerRuntimeLifecycle.mock.invocationCallOrder[0] ?? Number.POSITIVE_INFINITY,
|
||||
);
|
||||
|
||||
const lifecycle = registerRuntimeLifecycle.mock.calls[0]?.[0];
|
||||
await lifecycle?.cleanup({ reason: "shutdown" });
|
||||
expect(setDiscordProviderEndpointDescriptor).toHaveBeenLastCalledWith(undefined);
|
||||
} finally {
|
||||
fs.rmSync(tempRoot, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
it("does not load the artifact during non-runtime registration", () => {
|
||||
process.env[QA_TEMP_ROOT_ENV] = "/does/not/exist";
|
||||
|
||||
registerCrablineDiscordProviderEndpoint(createTestPluginApi({ registrationMode: "discovery" }));
|
||||
|
||||
expect(setDiscordProviderEndpointDescriptor).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("rejects an artifact with fields outside the endpoint contract", () => {
|
||||
const tempRoot = fs.mkdtempSync(path.join(os.tmpdir(), "openclaw-qa-discord-endpoint-"));
|
||||
process.env[QA_TEMP_ROOT_ENV] = tempRoot;
|
||||
fs.writeFileSync(
|
||||
path.join(tempRoot, CRABLINE_DISCORD_PROVIDER_ENDPOINT_ARTIFACT),
|
||||
JSON.stringify({ apiRoot: "http://127.0.0.1:43123/api", version: 1 }),
|
||||
{ mode: 0o600 },
|
||||
);
|
||||
|
||||
try {
|
||||
expect(() =>
|
||||
registerCrablineDiscordProviderEndpoint(createTestPluginApi({ registrationMode: "full" })),
|
||||
).toThrow("Crabline Discord provider endpoint artifact is invalid");
|
||||
expect(setDiscordProviderEndpointDescriptor).not.toHaveBeenCalled();
|
||||
} finally {
|
||||
fs.rmSync(tempRoot, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,57 @@
|
||||
import fs from "node:fs";
|
||||
import path from "node:path";
|
||||
import {
|
||||
setDiscordProviderEndpointDescriptor,
|
||||
type DiscordProviderEndpointDescriptor,
|
||||
} from "@openclaw/discord/provider-endpoint-api.js";
|
||||
import type { OpenClawPluginApi } from "openclaw/plugin-sdk/plugin-entry";
|
||||
import { isRecord } from "openclaw/plugin-sdk/string-coerce-runtime";
|
||||
import { CRABLINE_DISCORD_PROVIDER_ENDPOINT_ARTIFACT } from "./crabline-discord-provider-endpoint-artifact.js";
|
||||
|
||||
const QA_TEMP_ROOT_ENV = "OPENCLAW_QA_TEMP_ROOT";
|
||||
const DESCRIPTOR_KEYS = ["gatewayBotUrl", "gatewayOrigin", "restApiBaseUrl"];
|
||||
|
||||
function readDescriptor(value: unknown): DiscordProviderEndpointDescriptor {
|
||||
if (
|
||||
!isRecord(value) ||
|
||||
Object.keys(value).toSorted().join("\0") !== DESCRIPTOR_KEYS.join("\0") ||
|
||||
typeof value.restApiBaseUrl !== "string" ||
|
||||
typeof value.gatewayBotUrl !== "string" ||
|
||||
typeof value.gatewayOrigin !== "string"
|
||||
) {
|
||||
throw new Error("Crabline Discord provider endpoint artifact is invalid");
|
||||
}
|
||||
return {
|
||||
restApiBaseUrl: value.restApiBaseUrl,
|
||||
gatewayBotUrl: value.gatewayBotUrl,
|
||||
gatewayOrigin: value.gatewayOrigin,
|
||||
};
|
||||
}
|
||||
|
||||
export function registerCrablineDiscordProviderEndpoint(api: OpenClawPluginApi): void {
|
||||
if (api.registrationMode !== "full") {
|
||||
return;
|
||||
}
|
||||
const tempRoot = process.env[QA_TEMP_ROOT_ENV]?.trim();
|
||||
if (!tempRoot) {
|
||||
return;
|
||||
}
|
||||
const artifactPath = path.join(tempRoot, CRABLINE_DISCORD_PROVIDER_ENDPOINT_ARTIFACT);
|
||||
let serialized: string;
|
||||
try {
|
||||
serialized = fs.readFileSync(artifactPath, "utf8");
|
||||
} catch (error) {
|
||||
if ((error as NodeJS.ErrnoException).code === "ENOENT") {
|
||||
return;
|
||||
}
|
||||
throw error;
|
||||
}
|
||||
const descriptor = readDescriptor(JSON.parse(serialized));
|
||||
// Install before channel startup; the child Gateway owns clearing this process-local seam.
|
||||
setDiscordProviderEndpointDescriptor(descriptor);
|
||||
api.lifecycle.registerRuntimeLifecycle({
|
||||
id: "qa-crabline-discord-provider-endpoint",
|
||||
description: "Clears the QA-owned Discord provider endpoint when the child Gateway stops.",
|
||||
cleanup: () => setDiscordProviderEndpointDescriptor(undefined),
|
||||
});
|
||||
}
|
||||
@@ -0,0 +1,156 @@
|
||||
// QA Lab tests cover the Discord-specific Crabline provider lifecycle.
|
||||
import fs from "node:fs/promises";
|
||||
import path from "node:path";
|
||||
import type { OpenClawCrablineChannelDriverSelection } from "@openclaw/crabline";
|
||||
import { withTempDir } from "openclaw/plugin-sdk/test-env";
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { createQaBusState } from "./bus-state.js";
|
||||
import { CRABLINE_DISCORD_PROVIDER_ENDPOINT_ARTIFACT } from "./crabline-discord-provider-endpoint-artifact.js";
|
||||
import { createQaCrablineTransportAdapter } from "./crabline-transport.js";
|
||||
|
||||
const DISCORD_SELECTION = {
|
||||
capabilityMatrixPath: "crabline-channel-driver-capabilities.json",
|
||||
channel: "discord",
|
||||
channelDriver: "crabline",
|
||||
providerReadinessArtifactPath: "crabline-provider-readiness.json",
|
||||
} as const satisfies OpenClawCrablineChannelDriverSelection;
|
||||
|
||||
describe("Crabline Discord transport", () => {
|
||||
it("stages the private endpoint and closes the provider after the Gateway phase", async () => {
|
||||
await withTempDir("qa-crabline-transport-", async (outputDir) => {
|
||||
const transport = await createQaCrablineTransportAdapter({
|
||||
outputDir,
|
||||
transportPolicy: { requireGroupMention: true, senderAllowlist: ["driver"] },
|
||||
selection: DISCORD_SELECTION,
|
||||
state: createQaBusState(),
|
||||
});
|
||||
const gatewayTempRoot = path.join(outputDir, "gateway-temp");
|
||||
await fs.mkdir(gatewayTempRoot);
|
||||
|
||||
try {
|
||||
expect(transport.requiredPluginIds).toEqual(["qa-lab", "discord"]);
|
||||
const config = transport.createGatewayConfig({ baseUrl: "http://127.0.0.1:1" });
|
||||
const discord = config.channels?.discord as
|
||||
| {
|
||||
applicationId?: string;
|
||||
guilds?: Record<string, { channels?: Record<string, unknown>; users?: string[] }>;
|
||||
token?: string;
|
||||
}
|
||||
| undefined;
|
||||
expect(discord).toMatchObject({
|
||||
allowFrom: [expect.stringMatching(/^\d{17,20}$/u)],
|
||||
applicationId: expect.stringMatching(/^\d{17,20}$/u),
|
||||
dmPolicy: "allowlist",
|
||||
guilds: {
|
||||
"*": {
|
||||
channels: { "*": { enabled: true, requireMention: true } },
|
||||
users: [expect.stringMatching(/^\d{17,20}$/u)],
|
||||
},
|
||||
},
|
||||
token: expect.any(String),
|
||||
});
|
||||
|
||||
await transport.stageGatewayRuntime?.({ tempRoot: gatewayTempRoot });
|
||||
const descriptorPath = path.join(
|
||||
gatewayTempRoot,
|
||||
CRABLINE_DISCORD_PROVIDER_ENDPOINT_ARTIFACT,
|
||||
);
|
||||
const descriptor = JSON.parse(await fs.readFile(descriptorPath, "utf8")) as {
|
||||
gatewayBotUrl: string;
|
||||
gatewayOrigin: string;
|
||||
restApiBaseUrl: string;
|
||||
};
|
||||
expect(descriptor).toEqual({
|
||||
gatewayBotUrl: expect.stringMatching(
|
||||
/^http:\/\/127\.0\.0\.1:\d+\/api\/v10\/gateway\/bot$/u,
|
||||
),
|
||||
gatewayOrigin: expect.stringMatching(/^ws:\/\/127\.0\.0\.1:\d+$/u),
|
||||
restApiBaseUrl: expect.stringMatching(/^http:\/\/127\.0\.0\.1:\d+\/api\/v10$/u),
|
||||
});
|
||||
if (process.platform !== "win32") {
|
||||
expect((await fs.stat(descriptorPath)).mode & 0o077).toBe(0);
|
||||
}
|
||||
|
||||
await expect(
|
||||
transport.sendInbound({
|
||||
conversation: { id: "discord-crabline-primary", kind: "group" },
|
||||
senderId: "driver",
|
||||
senderName: "QA Driver",
|
||||
text: "@openclaw Discord provider marker.",
|
||||
threadId: "discord-crabline-thread",
|
||||
}),
|
||||
).resolves.toMatchObject({
|
||||
conversation: { id: "discord-crabline-primary", kind: "group" },
|
||||
text: "@openclaw Discord provider marker.",
|
||||
threadId: "discord-crabline-thread",
|
||||
});
|
||||
const delivery = transport.buildAgentDelivery({
|
||||
target: "thread:discord-crabline-primary/discord-crabline-thread",
|
||||
});
|
||||
expect(delivery).toEqual({
|
||||
channel: "discord",
|
||||
replyChannel: "discord",
|
||||
replyTo: expect.stringMatching(/^channel:\d{17,20}$/u),
|
||||
to: expect.stringMatching(/^channel:\d{17,20}$/u),
|
||||
});
|
||||
|
||||
const probeUrl = `${descriptor.restApiBaseUrl}/users/@me`;
|
||||
const headers = { authorization: `Bot ${discord?.token}` };
|
||||
const channelId = delivery.to?.replace(/^channel:/u, "");
|
||||
const outboundResponse = await fetch(
|
||||
`${descriptor.restApiBaseUrl}/channels/${channelId}/messages`,
|
||||
{
|
||||
body: JSON.stringify({ content: "Discord provider outbound marker." }),
|
||||
headers: { ...headers, "content-type": "application/json" },
|
||||
method: "POST",
|
||||
},
|
||||
);
|
||||
expect(outboundResponse.ok).toBe(true);
|
||||
await outboundResponse.body?.cancel();
|
||||
await expect(
|
||||
transport.waitForOutbound({
|
||||
conversation: { id: "discord-crabline-primary", kind: "group" },
|
||||
textIncludes: "Discord provider outbound marker.",
|
||||
threadId: "discord-crabline-thread",
|
||||
timeoutMs: 1_000,
|
||||
}),
|
||||
).resolves.toMatchObject({
|
||||
conversation: { id: "discord-crabline-primary", kind: "group" },
|
||||
text: "Discord provider outbound marker.",
|
||||
threadId: "discord-crabline-thread",
|
||||
});
|
||||
const beforeCleanup = await fetch(probeUrl, { headers });
|
||||
expect(beforeCleanup.ok).toBe(true);
|
||||
await beforeCleanup.body?.cancel();
|
||||
|
||||
await transport.cleanup?.();
|
||||
const beforeGatewayStop = await fetch(probeUrl, { headers });
|
||||
expect(beforeGatewayStop.ok).toBe(true);
|
||||
await beforeGatewayStop.body?.cancel();
|
||||
|
||||
await transport.cleanupAfterGatewayStop?.();
|
||||
await expect(fetch(probeUrl, { headers })).rejects.toThrow();
|
||||
} finally {
|
||||
await transport.cleanupAfterGatewayStop?.();
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
it("completes Crabline's open Discord DM binding without a sender restriction", async () => {
|
||||
await withTempDir("qa-crabline-discord-open-dm-", async (outputDir) => {
|
||||
const transport = await createQaCrablineTransportAdapter({
|
||||
outputDir,
|
||||
selection: DISCORD_SELECTION,
|
||||
state: createQaBusState(),
|
||||
});
|
||||
|
||||
try {
|
||||
expect(transport.createGatewayConfig({ baseUrl: "http://127.0.0.1:1" })).toMatchObject({
|
||||
channels: { discord: { allowFrom: ["*"], dmPolicy: "open" } },
|
||||
});
|
||||
} finally {
|
||||
await transport.cleanupAfterGatewayStop?.();
|
||||
}
|
||||
});
|
||||
});
|
||||
});
|
||||
@@ -13,6 +13,17 @@ const TELEGRAM_QA_MAX_NATIVE_ID = (1n << 52n) - 1n;
|
||||
const MATTERMOST_ID_PATTERN = /^[a-z0-9]{26}$/u;
|
||||
const MATRIX_QA_SERVER_NAME = "matrix-qa.test";
|
||||
const MATRIX_QA_DRIVER_ID = `@driver:${MATRIX_QA_SERVER_NAME}`;
|
||||
const DISCORD_ID_PATTERN = /^\d{17,20}$/u;
|
||||
const DISCORD_ID_FLOOR = 100_000_000_000_000_000n;
|
||||
|
||||
export function resolveDiscordQaId(value: string) {
|
||||
const trimmed = value.trim();
|
||||
if (DISCORD_ID_PATTERN.test(trimmed)) {
|
||||
return trimmed;
|
||||
}
|
||||
const digest = BigInt(`0x${createHash("sha256").update(trimmed).digest("hex").slice(0, 16)}`);
|
||||
return String(DISCORD_ID_FLOOR + (digest % DISCORD_ID_FLOOR));
|
||||
}
|
||||
|
||||
function resolveQaNumericId(value: string, range: bigint) {
|
||||
const digest = BigInt(`0x${createHash("sha256").update(value).digest("hex").slice(0, 16)}`);
|
||||
@@ -135,6 +146,30 @@ function resolveMatrixQaText(text: string, botUserId: string) {
|
||||
);
|
||||
}
|
||||
|
||||
function resolveDiscordQaText(text: string, botUserId: string) {
|
||||
return text.replace(
|
||||
/(^|[\s([{])@openclaw(?=$|[\s.,!?;)\]}])/gu,
|
||||
(_match, prefix: string) => `${prefix}<@${botUserId}>`,
|
||||
);
|
||||
}
|
||||
|
||||
function resolveDiscordQaTarget(target: string) {
|
||||
const normalized = target.trim();
|
||||
if (normalized.startsWith("thread:")) {
|
||||
const threadTarget = normalized.slice("thread:".length);
|
||||
const separator = threadTarget.indexOf("/");
|
||||
if (separator > 0) {
|
||||
return `thread:${resolveDiscordQaId(threadTarget.slice(0, separator))}/${resolveDiscordQaId(threadTarget.slice(separator + 1))}`;
|
||||
}
|
||||
}
|
||||
for (const prefix of ["channel:", "group:", "dm:", "user:"]) {
|
||||
if (normalized.startsWith(prefix)) {
|
||||
return `${prefix}${resolveDiscordQaId(normalized.slice(prefix.length))}`;
|
||||
}
|
||||
}
|
||||
return resolveDiscordQaId(normalized);
|
||||
}
|
||||
|
||||
function resolveTelegramQaTarget(target: string) {
|
||||
const normalized = target.trim();
|
||||
if (normalized.startsWith("thread:")) {
|
||||
@@ -190,7 +225,9 @@ export function createCrablineProviderInboundInput(
|
||||
? resolveMatrixQaConversationId(input.conversation.id)
|
||||
: adapter.channel === "mattermost"
|
||||
? resolveMattermostQaId(input.conversation.id)
|
||||
: input.conversation.id,
|
||||
: adapter.channel === "discord"
|
||||
? resolveDiscordQaId(input.conversation.id)
|
||||
: input.conversation.id,
|
||||
kind,
|
||||
},
|
||||
senderId:
|
||||
@@ -200,11 +237,18 @@ export function createCrablineProviderInboundInput(
|
||||
? resolveMatrixQaSenderId(input.senderId)
|
||||
: adapter.channel === "mattermost"
|
||||
? resolveMattermostQaId(input.senderId)
|
||||
: input.senderId,
|
||||
: adapter.channel === "discord"
|
||||
? resolveDiscordQaId(input.senderId)
|
||||
: input.senderId,
|
||||
text:
|
||||
adapter.channel === "matrix" && adapter.manifest.provider === "matrix"
|
||||
? resolveMatrixQaText(input.text, adapter.manifest.botUserId)
|
||||
: input.text,
|
||||
: adapter.channel === "discord" && adapter.manifest.provider === "discord"
|
||||
? resolveDiscordQaText(input.text, adapter.manifest.botUserId)
|
||||
: input.text,
|
||||
...(input.threadId && adapter.channel === "discord"
|
||||
? { threadId: resolveDiscordQaId(input.threadId) }
|
||||
: {}),
|
||||
};
|
||||
}
|
||||
|
||||
@@ -213,7 +257,7 @@ export function resolveCrablineStateConversation(params: {
|
||||
input: QaBusInboundMessageInput;
|
||||
providerInbound: OpenClawCrablineInbound;
|
||||
}) {
|
||||
return ["mattermost", "matrix", "telegram"].includes(params.adapter.channel)
|
||||
return ["discord", "mattermost", "matrix", "telegram"].includes(params.adapter.channel)
|
||||
? params.input.conversation
|
||||
: params.providerInbound.stateConversation;
|
||||
}
|
||||
@@ -230,11 +274,17 @@ export function createCrablineProviderDelivery(
|
||||
? resolveMatrixQaTarget(target)
|
||||
: adapter.channel === "mattermost"
|
||||
? resolveMattermostQaTarget(target)
|
||||
: target,
|
||||
: adapter.channel === "discord"
|
||||
? resolveDiscordQaTarget(target)
|
||||
: target,
|
||||
});
|
||||
return {
|
||||
delivery,
|
||||
providerTargetKey:
|
||||
adapter.channel === "matrix" ? delivery.to.replace(/^room:/u, "") : delivery.to,
|
||||
adapter.channel === "matrix"
|
||||
? delivery.to.replace(/^room:/u, "")
|
||||
: adapter.channel === "discord"
|
||||
? delivery.to.replace(/^(?:channel|user):/u, "")
|
||||
: delivery.to,
|
||||
};
|
||||
}
|
||||
|
||||
@@ -15,10 +15,12 @@ import {
|
||||
readStringValue,
|
||||
} from "openclaw/plugin-sdk/string-coerce-runtime";
|
||||
import { createQaBusState, type QaBusState } from "./bus-state.js";
|
||||
import { CRABLINE_DISCORD_PROVIDER_ENDPOINT_ARTIFACT } from "./crabline-discord-provider-endpoint-artifact.js";
|
||||
import {
|
||||
createCrablineProviderDelivery,
|
||||
createCrablineProviderInboundInput,
|
||||
resolveCrablineStateConversation,
|
||||
resolveDiscordQaId,
|
||||
resolveTelegramQaSenderId,
|
||||
} from "./crabline-provider-targets.js";
|
||||
import { discardIgnoredResponseBody } from "./ignored-response-body.js";
|
||||
@@ -88,6 +90,11 @@ function formatLogicalQaTarget({ conversation, threadId }: QaBusInboundMessageIn
|
||||
return threadId ? `thread:${conversation.id}/${threadId}` : `${prefix}:${conversation.id}`;
|
||||
}
|
||||
|
||||
function formatLogicalQaConversationTarget({ conversation }: QaBusInboundMessageInput) {
|
||||
const prefix = conversation.kind === "direct" ? "dm" : conversation.kind;
|
||||
return `${prefix}:${conversation.id}`;
|
||||
}
|
||||
|
||||
const TELEGRAM_LIFECYCLE_METHOD_RE = /\/(sendMessage|editMessageText|deleteMessage)$/u;
|
||||
|
||||
function readTelegramLifecycleEvent(params: {
|
||||
@@ -212,6 +219,7 @@ function createCrablineState(params: {
|
||||
}): QaCrablineTransportState {
|
||||
const baseState = params.state;
|
||||
const targetByProviderTarget = new Map<string, string>();
|
||||
const logicalRouteByTarget = new Map<string, { target: string; threadId?: string }>();
|
||||
const telegramMessageByProviderId = new Map<string, QaBusMessage>();
|
||||
const pendingTelegramMessagesByChat = new Map<string, QaBusMessage[]>();
|
||||
const outboundEvents: QaTransportOutboundEvent[] = [];
|
||||
@@ -220,6 +228,7 @@ function createCrablineState(params: {
|
||||
reset() {
|
||||
baseState.reset();
|
||||
targetByProviderTarget.clear();
|
||||
logicalRouteByTarget.clear();
|
||||
telegramMessageByProviderId.clear();
|
||||
pendingTelegramMessagesByChat.clear();
|
||||
outboundEvents.length = 0;
|
||||
@@ -258,7 +267,16 @@ function createCrablineState(params: {
|
||||
targetByProviderTarget,
|
||||
}) as QaBusOutboundMessageInput | null;
|
||||
if (outbound) {
|
||||
baseState.addOutboundMessage(outbound);
|
||||
const logicalRoute = logicalRouteByTarget.get(outbound.to);
|
||||
baseState.addOutboundMessage(
|
||||
logicalRoute
|
||||
? {
|
||||
...outbound,
|
||||
to: logicalRoute.target,
|
||||
...(logicalRoute.threadId ? { threadId: logicalRoute.threadId } : {}),
|
||||
}
|
||||
: outbound,
|
||||
);
|
||||
}
|
||||
},
|
||||
async addInboundMessage(input: QaBusInboundMessageInput) {
|
||||
@@ -284,7 +302,12 @@ function createCrablineState(params: {
|
||||
}
|
||||
// Providers may coerce channel conversations to groups; preserve the scenario's logical
|
||||
// target so outbound waits and assertions still match the original input.
|
||||
targetByProviderTarget.set(providerInbound.providerTargetKey, formatLogicalQaTarget(input));
|
||||
const logicalTarget = formatLogicalQaTarget(input);
|
||||
targetByProviderTarget.set(providerInbound.providerTargetKey, logicalTarget);
|
||||
logicalRouteByTarget.set(logicalTarget, {
|
||||
target: formatLogicalQaConversationTarget(input),
|
||||
...(input.threadId ? { threadId: input.threadId } : {}),
|
||||
});
|
||||
const providerMessageId = await postCrablineInbound({
|
||||
adapter: params.adapter,
|
||||
providerInbound,
|
||||
@@ -297,7 +320,11 @@ function createCrablineState(params: {
|
||||
input,
|
||||
providerInbound,
|
||||
}),
|
||||
...(providerInbound.threadId ? { threadId: providerInbound.threadId } : {}),
|
||||
...(input.threadId
|
||||
? { threadId: input.threadId }
|
||||
: providerInbound.threadId
|
||||
? { threadId: providerInbound.threadId }
|
||||
: {}),
|
||||
},
|
||||
providerMessageId,
|
||||
);
|
||||
@@ -320,6 +347,7 @@ class QaCrablineTransport extends QaStateBackedTransportAdapter {
|
||||
readonly #selection: OpenClawCrablineChannelDriverSelection;
|
||||
readonly #transportPolicy?: QaTransportPolicy;
|
||||
readonly #state: QaCrablineTransportState;
|
||||
#cleanupPromise?: Promise<void>;
|
||||
readonly sendNativeCommand?: (input: QaTransportNativeCommandInput) => Promise<void>;
|
||||
readonly waitForOutboundSequence?: (input: QaTransportOutboundSequenceMatch) => Promise<{
|
||||
events: QaTransportOutboundEvent[];
|
||||
@@ -336,7 +364,10 @@ class QaCrablineTransport extends QaStateBackedTransportAdapter {
|
||||
id: CRABLINE_TRANSPORT_ID,
|
||||
label: `crabline local ${params.selection.channel}`,
|
||||
accountId: params.adapter.accountId,
|
||||
requiredPluginIds: params.adapter.requiredPluginIds,
|
||||
requiredPluginIds:
|
||||
params.selection.channel === "discord"
|
||||
? ["qa-lab", ...params.adapter.requiredPluginIds]
|
||||
: params.adapter.requiredPluginIds,
|
||||
state: params.state,
|
||||
});
|
||||
this.#adapter = params.adapter;
|
||||
@@ -367,6 +398,50 @@ class QaCrablineTransport extends QaStateBackedTransportAdapter {
|
||||
this.#selection.channel === "signal"
|
||||
? normalizeCrablineSignalGatewayConfig(rawConfig)
|
||||
: rawConfig;
|
||||
if (this.#selection.channel === "discord") {
|
||||
const discord = config.channels?.discord;
|
||||
const senderAllowlist = this.#transportPolicy?.senderAllowlist?.map(resolveDiscordQaId);
|
||||
// Crabline's local binding opens DMs, so the real Discord plugin also requires the matching
|
||||
// allowlist. Keep an explicit sender restriction; otherwise its open policy uses a wildcard.
|
||||
const dmAllowlist = senderAllowlist ?? discord?.allowFrom ?? ["*"];
|
||||
const discordWithOpenDms = {
|
||||
...discord,
|
||||
allowFrom: [...dmAllowlist],
|
||||
...(dmAllowlist.includes("*") ? {} : { dmPolicy: "allowlist" as const }),
|
||||
};
|
||||
const wildcardGuild = discord?.guilds?.["*"];
|
||||
const wildcardChannel = wildcardGuild?.channels?.["*"];
|
||||
if (!this.#transportPolicy?.requireGroupMention && !senderAllowlist) {
|
||||
return {
|
||||
...config,
|
||||
channels: { ...config.channels, discord: discordWithOpenDms },
|
||||
} as QaTransportGatewayConfig;
|
||||
}
|
||||
return {
|
||||
...config,
|
||||
channels: {
|
||||
...config.channels,
|
||||
discord: {
|
||||
...discordWithOpenDms,
|
||||
...(senderAllowlist ? { groupPolicy: "allowlist" as const } : {}),
|
||||
guilds: {
|
||||
...discord?.guilds,
|
||||
"*": {
|
||||
...wildcardGuild,
|
||||
...(senderAllowlist ? { users: [...senderAllowlist] } : {}),
|
||||
channels: {
|
||||
...wildcardGuild?.channels,
|
||||
"*": {
|
||||
...wildcardChannel,
|
||||
...(this.#transportPolicy?.requireGroupMention ? { requireMention: true } : {}),
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
} as QaTransportGatewayConfig;
|
||||
}
|
||||
if (this.#selection.channel !== "telegram") {
|
||||
return config as QaTransportGatewayConfig;
|
||||
}
|
||||
@@ -414,6 +489,26 @@ class QaCrablineTransport extends QaStateBackedTransportAdapter {
|
||||
|
||||
createRuntimeEnvPatch = () => this.#adapter.createProviderReadinessEnv({});
|
||||
|
||||
stageGatewayRuntime = async ({ tempRoot }: { tempRoot: string }) => {
|
||||
if (this.#adapter.manifest.provider !== "discord") {
|
||||
return;
|
||||
}
|
||||
const gatewayUrl = new URL(this.#adapter.manifest.endpoints.gatewayUrl);
|
||||
await fs.writeFile(
|
||||
path.join(tempRoot, CRABLINE_DISCORD_PROVIDER_ENDPOINT_ARTIFACT),
|
||||
`${JSON.stringify(
|
||||
{
|
||||
restApiBaseUrl: `${this.#adapter.manifest.endpoints.apiRoot}/v10`,
|
||||
gatewayBotUrl: this.#adapter.manifest.endpoints.gatewayBotUrl,
|
||||
gatewayOrigin: gatewayUrl.origin,
|
||||
},
|
||||
null,
|
||||
2,
|
||||
)}\n`,
|
||||
{ encoding: "utf8", flag: "wx", mode: 0o600 },
|
||||
);
|
||||
};
|
||||
|
||||
handleAction = async (_params: {
|
||||
action: QaTransportActionName;
|
||||
args: Record<string, unknown>;
|
||||
@@ -428,8 +523,24 @@ class QaCrablineTransport extends QaStateBackedTransportAdapter {
|
||||
"No live channel service or external credential lease is required.",
|
||||
];
|
||||
|
||||
#cleanupTransport() {
|
||||
this.#cleanupPromise ??= this.#state.cleanup();
|
||||
void this.#cleanupPromise.catch(() => {
|
||||
this.#cleanupPromise = undefined;
|
||||
});
|
||||
return this.#cleanupPromise;
|
||||
}
|
||||
|
||||
async cleanup() {
|
||||
await this.#state.cleanup();
|
||||
if (this.#selection.channel !== "discord") {
|
||||
await this.#cleanupTransport();
|
||||
}
|
||||
}
|
||||
|
||||
async cleanupAfterGatewayStop() {
|
||||
if (this.#selection.channel === "discord") {
|
||||
await this.#cleanupTransport();
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -439,12 +550,16 @@ export async function createQaCrablineTransportAdapter(params: {
|
||||
selection: OpenClawCrablineChannelDriverSelection;
|
||||
state?: QaBusState;
|
||||
}) {
|
||||
const requiresTelegramPolicy =
|
||||
const requiresGroupPolicy =
|
||||
params.transportPolicy?.requireGroupMention === true ||
|
||||
params.transportPolicy?.senderAllowlist !== undefined;
|
||||
if (params.selection.channel !== "telegram" && requiresTelegramPolicy) {
|
||||
if (
|
||||
params.selection.channel !== "telegram" &&
|
||||
params.selection.channel !== "discord" &&
|
||||
requiresGroupPolicy
|
||||
) {
|
||||
throw new Error(
|
||||
`Crabline ${params.selection.channel} does not support the requested group transport policy; use the Crabline Telegram bridge or a live channel adapter`,
|
||||
`Crabline ${params.selection.channel} does not support the requested group transport policy; use the Crabline Telegram or Discord bridge, or a live channel adapter`,
|
||||
);
|
||||
}
|
||||
const recorderPath = path.join(
|
||||
|
||||
@@ -0,0 +1,148 @@
|
||||
import fs from "node:fs/promises";
|
||||
import path from "node:path";
|
||||
import { resolveOpenClawCrablineChannelDriverSelection } from "@openclaw/crabline";
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { CRABLINE_DISCORD_PROVIDER_ENDPOINT_ARTIFACT } from "./crabline-discord-provider-endpoint-artifact.js";
|
||||
import { runQaSuite } from "./suite-launch.runtime.js";
|
||||
|
||||
const RUN_DISCORD_CRABLINE_E2E = process.env.OPENCLAW_QA_DISCORD_CRABLINE_E2E === "1";
|
||||
const SCENARIO_ID = "discord-crabline-roundtrip";
|
||||
const EXPECTED_MARKER = "DISCORD-CRABLINE-ROUNDTRIP-OK";
|
||||
|
||||
type RecorderEvent = {
|
||||
accepted?: boolean;
|
||||
body?: Record<string, unknown>;
|
||||
method?: string;
|
||||
path?: string;
|
||||
type?: string;
|
||||
};
|
||||
|
||||
function readObject(value: unknown): Record<string, unknown> | undefined {
|
||||
return value && typeof value === "object" && !Array.isArray(value)
|
||||
? (value as Record<string, unknown>)
|
||||
: undefined;
|
||||
}
|
||||
|
||||
function readString(value: unknown): string {
|
||||
return typeof value === "string" ? value : "";
|
||||
}
|
||||
|
||||
async function readRecorderEvents(recorderPath: string): Promise<RecorderEvent[]> {
|
||||
const raw = await fs.readFile(recorderPath, "utf8");
|
||||
if (raw.includes("discord.com") || raw.includes("discordapp.com")) {
|
||||
throw new Error("Discord Crabline recorder contains a public Discord service target.");
|
||||
}
|
||||
return raw
|
||||
.split("\n")
|
||||
.filter(Boolean)
|
||||
.map((line) => JSON.parse(line) as RecorderEvent);
|
||||
}
|
||||
|
||||
describe("Discord Crabline real-plugin roundtrip", () => {
|
||||
it.runIf(RUN_DISCORD_CRABLINE_E2E)(
|
||||
"crosses the real Discord REST and Gateway boundaries and flushes cleanup evidence",
|
||||
async () => {
|
||||
const repoRoot = process.cwd();
|
||||
const outputDir = path.join(
|
||||
repoRoot,
|
||||
".artifacts",
|
||||
"qa-e2e",
|
||||
`discord-crabline-roundtrip-${process.pid}-${Date.now()}`,
|
||||
);
|
||||
const selection = resolveOpenClawCrablineChannelDriverSelection({ channel: "discord" });
|
||||
|
||||
const suite = await runQaSuite({
|
||||
channelDriver: "crabline",
|
||||
channelDriverSelection: selection,
|
||||
channelId: "discord",
|
||||
controlUiEnabled: false,
|
||||
outputDir,
|
||||
primaryModel: "mock-openai/gpt-5.6-luna",
|
||||
providerMode: "mock-openai",
|
||||
repoRoot,
|
||||
scenarioIds: [SCENARIO_ID],
|
||||
});
|
||||
|
||||
expect(suite.executionKind).toBe("flow");
|
||||
expect(suite.result.scenarios).toEqual([
|
||||
expect.objectContaining({ name: expect.any(String), status: "pass" }),
|
||||
]);
|
||||
|
||||
const recorderPath = path.join(
|
||||
suite.result.outputDir,
|
||||
"artifacts",
|
||||
"crabline",
|
||||
"discord-provider-server.jsonl",
|
||||
);
|
||||
const events = await readRecorderEvents(recorderPath);
|
||||
const gatewayMetadata = events.find(
|
||||
(event) =>
|
||||
event.type === "api" &&
|
||||
event.method === "GET" &&
|
||||
event.path === "/api/v10/gateway/bot" &&
|
||||
event.accepted === true,
|
||||
);
|
||||
const gatewayIdentify = events.find(
|
||||
(event) =>
|
||||
event.type === "api" &&
|
||||
event.method === "WS" &&
|
||||
event.path === "/gateway" &&
|
||||
readObject(event.body)?.op === 2 &&
|
||||
event.accepted === true,
|
||||
);
|
||||
expect(gatewayMetadata).toBeDefined();
|
||||
expect(gatewayIdentify).toBeDefined();
|
||||
|
||||
const inbound = events.find(
|
||||
(event) =>
|
||||
event.type === "admin" &&
|
||||
event.method === "POST" &&
|
||||
event.path === "/crabline/discord/inbound" &&
|
||||
event.accepted === true,
|
||||
);
|
||||
const inboundBody = readObject(inbound?.body);
|
||||
const inboundChannelId = readString(inboundBody?.channelId);
|
||||
const parentChannelId = readString(inboundBody?.parentChannelId);
|
||||
expect(inboundChannelId).toMatch(/^\d{17,20}$/u);
|
||||
expect(parentChannelId).toMatch(/^\d{17,20}$/u);
|
||||
expect(inboundChannelId).not.toBe(parentChannelId);
|
||||
expect(readString(inboundBody?.content)).toMatch(/<@\d{17,20}>/u);
|
||||
|
||||
const outbound = events.find(
|
||||
(event) =>
|
||||
event.type === "api" &&
|
||||
event.method === "POST" &&
|
||||
event.path === `/api/v10/channels/${inboundChannelId}/messages` &&
|
||||
readString(readObject(event.body)?.content).includes(EXPECTED_MARKER) &&
|
||||
event.accepted === true,
|
||||
);
|
||||
const outboundBody = readObject(outbound?.body);
|
||||
const messageReference = readObject(outboundBody?.message_reference);
|
||||
expect(outbound).toBeDefined();
|
||||
expect(readString(outboundBody?.content)).toContain(EXPECTED_MARKER);
|
||||
expect(readString(outboundBody?.content)).not.toMatch(/<@\d{17,20}>/u);
|
||||
expect(messageReference).toMatchObject({
|
||||
message_id: expect.stringMatching(/^\d{17,20}$/u),
|
||||
});
|
||||
|
||||
expect(
|
||||
events.some(
|
||||
(event) =>
|
||||
event.type === "api" &&
|
||||
event.accepted === true &&
|
||||
(event.method === "PUT" || event.method === "POST") &&
|
||||
/^\/api\/v10\/applications\/\d{17,20}\/commands$/u.test(event.path ?? ""),
|
||||
),
|
||||
).toBe(true);
|
||||
|
||||
// Suite return occurs only after the child Gateway and its WebSocket are stopped, then the
|
||||
// Discord HTTP server and recorder are closed. The QA-owned descriptor lives in the removed
|
||||
// child temp root and must never escape into durable suite artifacts.
|
||||
await expect(
|
||||
fs.access(path.join(suite.result.outputDir, CRABLINE_DISCORD_PROVIDER_ENDPOINT_ARTIFACT)),
|
||||
).rejects.toMatchObject({ code: "ENOENT" });
|
||||
await expect(fs.readFile(recorderPath, "utf8")).resolves.toContain(EXPECTED_MARKER);
|
||||
},
|
||||
180_000,
|
||||
);
|
||||
});
|
||||
@@ -100,11 +100,14 @@ export type QaGatewayChildListeningContext = {
|
||||
runtimeEnv: NodeJS.ProcessEnv;
|
||||
};
|
||||
|
||||
function createQaGatewayEmptyTransport() {
|
||||
function createQaGatewayEmptyTransport(): Pick<
|
||||
QaTransportAdapter,
|
||||
"requiredPluginIds" | "createGatewayConfig" | "stageGatewayRuntime"
|
||||
> {
|
||||
return {
|
||||
requiredPluginIds: [] as const,
|
||||
createGatewayConfig: () => ({}),
|
||||
} satisfies Pick<QaTransportAdapter, "requiredPluginIds" | "createGatewayConfig">;
|
||||
};
|
||||
}
|
||||
|
||||
function appendQaGatewayTempRoot(details: string, tempRoot: string) {
|
||||
@@ -222,7 +225,10 @@ export async function startQaGatewayChild(params: {
|
||||
command?: QaGatewayChildCommand;
|
||||
useRepoCli?: boolean;
|
||||
providerBaseUrl?: string;
|
||||
transport?: Pick<QaTransportAdapter, "requiredPluginIds" | "createGatewayConfig">;
|
||||
transport?: Pick<
|
||||
QaTransportAdapter,
|
||||
"requiredPluginIds" | "createGatewayConfig" | "stageGatewayRuntime"
|
||||
>;
|
||||
transportBaseUrl: string;
|
||||
controlUiAllowedOrigins?: string[];
|
||||
providerMode?: QaProviderMode;
|
||||
@@ -278,6 +284,7 @@ export async function startQaGatewayChild(params: {
|
||||
const packagedAuthConfigPath = path.join(stateDir, "qa-auth-bootstrap", "openclaw.json");
|
||||
const gatewayToken = `qa-suite-${randomUUID()}`;
|
||||
const transport = params.transport ?? createQaGatewayEmptyTransport();
|
||||
await transport.stageGatewayRuntime?.({ tempRoot });
|
||||
await seedQaAgentWorkspace({
|
||||
workspaceDir,
|
||||
repoRoot: params.repoRoot,
|
||||
@@ -389,7 +396,6 @@ export async function startQaGatewayChild(params: {
|
||||
let packagedMockAuthStaged = false;
|
||||
let migrationConvergenceRestartUsed = false;
|
||||
let reuseStartupLaunchState = false;
|
||||
|
||||
const nodeExecPath = gatewayExecutablePath ?? (await resolveQaNodeExecPath());
|
||||
const cliArgsPrefix = gatewayExecutablePath
|
||||
? gatewayArgsPrefix
|
||||
|
||||
@@ -66,4 +66,31 @@ describe("live transport QA contributions", () => {
|
||||
expect.objectContaining({ scenarioIds: ["telegram-canary"] }),
|
||||
);
|
||||
});
|
||||
|
||||
it("maps the dedicated Discord driver and listing options", async () => {
|
||||
const registration = listLiveTransportQaCliRegistrations().find(
|
||||
(candidate) => candidate.commandName === "discord",
|
||||
);
|
||||
const qa = new Command();
|
||||
registration?.register(qa);
|
||||
|
||||
await qa.parseAsync([
|
||||
"node",
|
||||
"openclaw",
|
||||
"discord",
|
||||
"--channel-driver",
|
||||
"crabline",
|
||||
"--list-scenarios",
|
||||
]);
|
||||
|
||||
expect(runLiveTransportQaSuiteCommand).toHaveBeenCalledWith(
|
||||
expect.objectContaining({
|
||||
channelId: "discord",
|
||||
options: expect.objectContaining({
|
||||
channelDriver: "crabline",
|
||||
listScenarios: true,
|
||||
}),
|
||||
}),
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -38,7 +38,10 @@ export const discordQaCliRegistration: LiveTransportQaCliRegistration =
|
||||
roleDescription:
|
||||
"Credential role for convex auth: maintainer or ci (default: ci in CI, maintainer otherwise)",
|
||||
},
|
||||
description: "Run the Discord live QA lane against a private guild bot-to-bot harness",
|
||||
channelDriverHelp:
|
||||
"Discord transport boundary: live (default) or Crabline local provider server",
|
||||
description: "Run Discord QA through the live service or Crabline local provider server",
|
||||
listScenariosHelp: "Print the selected Discord scenario ids and exit",
|
||||
outputDirHelp: "Discord QA artifact directory",
|
||||
scenarioHelp: "Run only the named Discord QA scenario (repeatable)",
|
||||
sutAccountHelp: "Temporary Discord account id inside the QA gateway config",
|
||||
|
||||
@@ -1,16 +1,19 @@
|
||||
import type { QaProviderModeInput } from "../../model-selection.js";
|
||||
import { resolveLiveTransportQaScenarioIds } from "../shared/scenario-selection.js";
|
||||
import type { QaScorecardChannelDriver } from "../../scorecard-taxonomy.js";
|
||||
import { resolveTransportQaScenarioIds } from "../shared/scenario-selection.js";
|
||||
|
||||
export function resolveDiscordQaScenarioIds(params: {
|
||||
profile?: string;
|
||||
channelDriver?: QaScorecardChannelDriver;
|
||||
primaryModel?: string;
|
||||
providerMode?: QaProviderModeInput;
|
||||
scenarioIds?: readonly string[];
|
||||
}) {
|
||||
return resolveLiveTransportQaScenarioIds({
|
||||
return resolveTransportQaScenarioIds({
|
||||
channelId: "discord",
|
||||
supportsModuleFlows: true,
|
||||
...params,
|
||||
channelDriver: params.channelDriver ?? "live",
|
||||
providerMode: params.providerMode ?? "live-frontier",
|
||||
supportsModuleFlows: true,
|
||||
});
|
||||
}
|
||||
|
||||
@@ -18,6 +18,7 @@ export type LiveTransportQaCommandOptions = {
|
||||
sutAccountId?: string;
|
||||
credentialSource?: string;
|
||||
credentialRole?: string;
|
||||
channelDriver?: string;
|
||||
};
|
||||
|
||||
type LiveTransportQaCommanderOptions = {
|
||||
@@ -35,6 +36,7 @@ type LiveTransportQaCommanderOptions = {
|
||||
sutAccount?: string;
|
||||
credentialSource?: string;
|
||||
credentialRole?: string;
|
||||
channelDriver?: string;
|
||||
};
|
||||
|
||||
export type LiveTransportQaCliRegistration = QaRunnerCliRegistration;
|
||||
@@ -45,6 +47,7 @@ type LiveTransportQaCliRegistrationOptions = {
|
||||
sourceDescription?: string;
|
||||
roleDescription?: string;
|
||||
};
|
||||
channelDriverHelp?: string;
|
||||
defaultProviderMode: string;
|
||||
description: string;
|
||||
providerModeHelp: string;
|
||||
@@ -88,6 +91,7 @@ function mapCommanderOptions(opts: LiveTransportQaCommanderOptions): LiveTranspo
|
||||
sutAccountId: opts.sutAccount,
|
||||
credentialSource: opts.credentialSource,
|
||||
credentialRole: opts.credentialRole,
|
||||
channelDriver: opts.channelDriver,
|
||||
};
|
||||
}
|
||||
|
||||
@@ -132,6 +136,9 @@ function createSharedLiveTransportQaCliRegistration(
|
||||
command.option("--credential-role <role>", params.credentialOptions.roleDescription);
|
||||
}
|
||||
}
|
||||
if (params.channelDriverHelp) {
|
||||
command.option("--channel-driver <live|crabline>", params.channelDriverHelp);
|
||||
}
|
||||
command.action(async (opts: LiveTransportQaCommanderOptions) => {
|
||||
await params.run(mapCommanderOptions(opts));
|
||||
});
|
||||
|
||||
@@ -93,6 +93,114 @@ describe("live transport suite runtime", () => {
|
||||
);
|
||||
});
|
||||
|
||||
it.each([undefined, "live"] as const)(
|
||||
"keeps the Discord live driver identical when selected as %s",
|
||||
async (channelDriver) => {
|
||||
const selectScenarioIds = vi.fn(() => ["discord-canary"]);
|
||||
|
||||
await runLiveTransportQaSuiteCommand({
|
||||
channelId: "discord",
|
||||
defaultProviderMode: "live-frontier",
|
||||
options: { channelDriver },
|
||||
selectScenarioIds,
|
||||
});
|
||||
|
||||
expect(selectScenarioIds).toHaveBeenCalledWith(
|
||||
expect.objectContaining({ channelDriver: "live" }),
|
||||
);
|
||||
expect(runQaSuiteCommand).toHaveBeenCalledWith(
|
||||
expect.objectContaining({
|
||||
channel: "discord",
|
||||
channelDriver: "live",
|
||||
scenarioIds: ["discord-canary"],
|
||||
}),
|
||||
);
|
||||
},
|
||||
);
|
||||
|
||||
it("selects Discord Crabline without forwarding credential lease options", async () => {
|
||||
const selectScenarioIds = vi.fn(() => ["discord-crabline-roundtrip"]);
|
||||
|
||||
await runLiveTransportQaSuiteCommand({
|
||||
channelId: "discord",
|
||||
defaultProviderMode: "live-frontier",
|
||||
options: {
|
||||
channelDriver: "crabline",
|
||||
providerMode: "mock-openai",
|
||||
primaryModel: "mock-openai/custom",
|
||||
},
|
||||
selectScenarioIds,
|
||||
});
|
||||
|
||||
expect(selectScenarioIds).toHaveBeenCalledWith({
|
||||
channelDriver: "crabline",
|
||||
profile: undefined,
|
||||
primaryModel: "mock-openai/custom",
|
||||
providerMode: "mock-openai",
|
||||
scenarioIds: undefined,
|
||||
});
|
||||
expect(runQaSuiteCommand).toHaveBeenCalledWith(
|
||||
expect.objectContaining({
|
||||
channel: "discord",
|
||||
channelDriver: "crabline",
|
||||
providerMode: "mock-openai",
|
||||
primaryModel: "mock-openai/custom",
|
||||
scenarioIds: ["discord-crabline-roundtrip"],
|
||||
}),
|
||||
);
|
||||
expect(runQaSuiteCommand.mock.calls[0]?.[0]).not.toHaveProperty("credentialSource");
|
||||
expect(runQaSuiteCommand.mock.calls[0]?.[0]).not.toHaveProperty("credentialRole");
|
||||
});
|
||||
|
||||
it("uses the same Discord selection for listing and execution", async () => {
|
||||
const stdoutWrite = vi.spyOn(process.stdout, "write").mockImplementation(() => true);
|
||||
const selectScenarioIds = vi.fn((_selection: unknown) => [
|
||||
"channel-canary",
|
||||
"thread-follow-up",
|
||||
]);
|
||||
const base = {
|
||||
channelId: "discord",
|
||||
defaultProviderMode: "mock-openai" as const,
|
||||
selectScenarioIds,
|
||||
};
|
||||
|
||||
try {
|
||||
await runLiveTransportQaSuiteCommand({
|
||||
...base,
|
||||
options: { channelDriver: "crabline", listScenarios: true },
|
||||
});
|
||||
await runLiveTransportQaSuiteCommand({
|
||||
...base,
|
||||
options: { channelDriver: "crabline" },
|
||||
});
|
||||
} finally {
|
||||
stdoutWrite.mockRestore();
|
||||
}
|
||||
|
||||
expect(selectScenarioIds).toHaveBeenCalledTimes(2);
|
||||
expect(selectScenarioIds.mock.calls[0]?.[0]).toEqual(selectScenarioIds.mock.calls[1]?.[0]);
|
||||
expect(runQaSuiteCommand).toHaveBeenCalledWith(
|
||||
expect.objectContaining({ scenarioIds: ["channel-canary", "thread-follow-up"] }),
|
||||
);
|
||||
});
|
||||
|
||||
it.each(["credentialSource", "credentialRole"] as const)(
|
||||
"rejects Discord Crabline with %s",
|
||||
async (option) => {
|
||||
await expect(
|
||||
runLiveTransportQaSuiteCommand({
|
||||
channelId: "discord",
|
||||
defaultProviderMode: "mock-openai",
|
||||
options: { channelDriver: "crabline", [option]: "ci" },
|
||||
selectScenarioIds: () => ["channel-canary"],
|
||||
}),
|
||||
).rejects.toThrow(
|
||||
`do not use --${option === "credentialSource" ? "credential-source" : "credential-role"}`,
|
||||
);
|
||||
expect(runQaSuiteCommand).not.toHaveBeenCalled();
|
||||
},
|
||||
);
|
||||
|
||||
it("rejects shared credentials for disposable transports", async () => {
|
||||
await expect(
|
||||
runLiveTransportQaSuiteCommand({
|
||||
|
||||
@@ -3,26 +3,47 @@ import { runQaSuiteCommand } from "../../cli.runtime.js";
|
||||
import type { QaProviderMode } from "../../providers/index.js";
|
||||
import { defaultQaModelForMode, normalizeQaProviderMode } from "../../run-config.js";
|
||||
|
||||
type DedicatedLiveTransportQaCommandOptions = LiveTransportQaCommandOptions & {
|
||||
channelDriver?: string;
|
||||
};
|
||||
|
||||
type LiveTransportScenarioSelection = (params: {
|
||||
channelDriver: "live" | "crabline";
|
||||
profile?: string;
|
||||
primaryModel: string;
|
||||
providerMode: QaProviderMode;
|
||||
scenarioIds?: readonly string[];
|
||||
}) => string[];
|
||||
|
||||
function resolveDedicatedChannelDriver(value: string | undefined): "live" | "crabline" {
|
||||
const normalized = value?.trim().toLowerCase() || "live";
|
||||
if (normalized !== "live" && normalized !== "crabline") {
|
||||
throw new Error(`channel driver must be live or crabline, got "${value}".`);
|
||||
}
|
||||
return normalized;
|
||||
}
|
||||
|
||||
export async function runLiveTransportQaSuiteCommand(params: {
|
||||
channelId: string;
|
||||
credentialMode?: "env-only" | "shared-lease";
|
||||
defaultProviderMode: QaProviderMode;
|
||||
envCredentialReason?: string;
|
||||
laneLabel?: string;
|
||||
options: LiveTransportQaCommandOptions;
|
||||
options: DedicatedLiveTransportQaCommandOptions;
|
||||
selectScenarioIds: LiveTransportScenarioSelection;
|
||||
}) {
|
||||
const options = params.options;
|
||||
const credentialSource =
|
||||
options.credentialSource?.trim() || process.env.OPENCLAW_QA_CREDENTIAL_SOURCE?.trim();
|
||||
if (params.credentialMode === "env-only") {
|
||||
const channelDriver = resolveDedicatedChannelDriver(options.channelDriver);
|
||||
if (channelDriver === "crabline") {
|
||||
if (options.credentialSource?.trim()) {
|
||||
throw new Error("QA Lab Crabline channel drivers do not use --credential-source.");
|
||||
}
|
||||
if (options.credentialRole?.trim()) {
|
||||
throw new Error("QA Lab Crabline channel drivers do not use --credential-role.");
|
||||
}
|
||||
} else if (params.credentialMode === "env-only") {
|
||||
const laneLabel = params.laneLabel ?? params.channelId;
|
||||
if (credentialSource && credentialSource.toLowerCase() !== "env") {
|
||||
throw new Error(
|
||||
@@ -40,11 +61,18 @@ export async function runLiveTransportQaSuiteCommand(params: {
|
||||
: normalizeQaProviderMode(options.providerMode);
|
||||
const primaryModel = options.primaryModel?.trim() || defaultQaModelForMode(providerMode);
|
||||
const selectedScenarioIds = params.selectScenarioIds({
|
||||
channelDriver,
|
||||
profile: options.profile,
|
||||
primaryModel,
|
||||
providerMode,
|
||||
scenarioIds: options.scenarioIds,
|
||||
});
|
||||
if (options.listScenarios) {
|
||||
for (const scenarioId of selectedScenarioIds) {
|
||||
process.stdout.write(`${scenarioId}\n`);
|
||||
}
|
||||
return undefined;
|
||||
}
|
||||
return runQaSuiteCommand({
|
||||
repoRoot: options.repoRoot,
|
||||
outputDir: options.outputDir,
|
||||
@@ -54,13 +82,13 @@ export async function runLiveTransportQaSuiteCommand(params: {
|
||||
fastMode: options.fastMode,
|
||||
allowFailures: options.allowFailures,
|
||||
failFast: options.failFast,
|
||||
channelDriver: "live",
|
||||
channelDriver,
|
||||
channel: params.channelId,
|
||||
concurrency: 1,
|
||||
scenarioIds: selectedScenarioIds,
|
||||
sutAccountId: options.sutAccountId,
|
||||
...(options.credentialFile ? { credentialFile: options.credentialFile } : {}),
|
||||
...(params.credentialMode === "env-only"
|
||||
...(channelDriver === "crabline" || params.credentialMode === "env-only"
|
||||
? {}
|
||||
: {
|
||||
credentialSource,
|
||||
|
||||
@@ -140,6 +140,7 @@ describe("live transport QA scenario selection", () => {
|
||||
{ channelId: "matrix", scenarioId: "thread-follow-up" },
|
||||
{ channelId: "telegram", scenarioId: "channel-canary" },
|
||||
{ channelId: "telegram", scenarioId: "channel-message-flows" },
|
||||
{ channelId: "discord", scenarioId: "channel-canary" },
|
||||
] as const)(
|
||||
"keeps $scenarioId eligible through both $channelId drivers",
|
||||
({ channelId, scenarioId }) => {
|
||||
@@ -185,4 +186,22 @@ describe("live transport QA scenario selection", () => {
|
||||
"selected QA scenario(s) do not match the current QA lane: channel-canary (channel=qa-channel|telegram|buzz|msteams)",
|
||||
);
|
||||
});
|
||||
|
||||
it("defaults dedicated Discord selection to live and excludes live-only modules from Crabline", () => {
|
||||
const implicitLive = resolveDiscordQaScenarioIds(MOCK_LANE);
|
||||
const explicitLive = resolveDiscordQaScenarioIds({ ...MOCK_LANE, channelDriver: "live" });
|
||||
const crabline = resolveDiscordQaScenarioIds({ ...MOCK_LANE, channelDriver: "crabline" });
|
||||
|
||||
expect(explicitLive).toEqual(implicitLive);
|
||||
expect(implicitLive).toContain("discord-canary");
|
||||
expect(crabline).toContain("discord-crabline-roundtrip");
|
||||
expect(crabline).not.toContain("discord-canary");
|
||||
expect(() =>
|
||||
resolveDiscordQaScenarioIds({
|
||||
...MOCK_LANE,
|
||||
channelDriver: "crabline",
|
||||
scenarioIds: ["discord-canary"],
|
||||
}),
|
||||
).toThrow("channelDriver=live");
|
||||
});
|
||||
});
|
||||
|
||||
@@ -41,19 +41,29 @@ export function resolveCatalogLiveTransportQaScenarioIds(params: {
|
||||
return selectedScenarios.map((scenario) => scenario.id);
|
||||
}
|
||||
|
||||
export function resolveLiveTransportQaScenarioIds(params: {
|
||||
export function resolveTransportQaScenarioIds(params: {
|
||||
channelId: string;
|
||||
channelDriver: QaScorecardChannelDriver;
|
||||
profile?: string;
|
||||
primaryModel?: string;
|
||||
providerMode: QaProviderModeInput;
|
||||
scenarioIds?: readonly string[];
|
||||
supportsModuleFlows?: boolean;
|
||||
}) {
|
||||
if (!params.profile?.trim() && params.channelDriver === "crabline") {
|
||||
return resolveCatalogLiveTransportQaScenarioIds({
|
||||
channelId: params.channelId,
|
||||
channelDriver: params.channelDriver,
|
||||
primaryModel: params.primaryModel,
|
||||
providerMode: params.providerMode,
|
||||
scenarioIds: params.scenarioIds,
|
||||
});
|
||||
}
|
||||
return resolveQaProfileScenarios({
|
||||
profile: params.profile?.trim() || "release",
|
||||
providerMode: params.providerMode,
|
||||
primaryModel: params.primaryModel,
|
||||
channelDriver: "live",
|
||||
channelDriver: params.channelDriver,
|
||||
channel: params.channelId,
|
||||
executionKind: "flow",
|
||||
requireDeclaredChannel: true,
|
||||
@@ -62,6 +72,12 @@ export function resolveLiveTransportQaScenarioIds(params: {
|
||||
}).scenarios.map((scenario) => scenario.id);
|
||||
}
|
||||
|
||||
export function resolveLiveTransportQaScenarioIds(
|
||||
params: Omit<Parameters<typeof resolveTransportQaScenarioIds>[0], "channelDriver">,
|
||||
) {
|
||||
return resolveTransportQaScenarioIds({ ...params, channelDriver: "live" });
|
||||
}
|
||||
|
||||
export function listLiveTransportQaScenarios(params: {
|
||||
channelId: string;
|
||||
primaryModel?: string;
|
||||
|
||||
@@ -315,6 +315,7 @@ export type QaTransportAdapter = Omit<
|
||||
timeoutMs?: number,
|
||||
intervalMs?: number,
|
||||
) => Promise<T>;
|
||||
stageGatewayRuntime?: (params: { tempRoot: string }) => Promise<void>;
|
||||
};
|
||||
|
||||
export abstract class QaStateBackedTransportAdapter implements QaTransportAdapter {
|
||||
|
||||
@@ -624,6 +624,71 @@ describe("qa run config", () => {
|
||||
);
|
||||
});
|
||||
|
||||
it("selects only scenarios that declare an explicit external channel", () => {
|
||||
const catalog = readQaScenarioPack();
|
||||
const scorecardReport = readQaScorecardTaxonomyReport(catalog.scenarios);
|
||||
const selection = normalizeQaRunSelection(
|
||||
{
|
||||
profile: "all",
|
||||
channel: "discord",
|
||||
channelDriver: "crabline",
|
||||
providerMode: "mock-openai",
|
||||
},
|
||||
catalog.scenarios,
|
||||
scorecardReport.profiles,
|
||||
);
|
||||
|
||||
const plan = resolveQaLabRunPlan({
|
||||
selection,
|
||||
scenarios: catalog.scenarios,
|
||||
scorecardReport,
|
||||
defaultChannel: "discord",
|
||||
supportsChannel: (channel) => channel === "discord",
|
||||
});
|
||||
|
||||
expect(plan.status).toBe("ready");
|
||||
expect(plan.selectedScenarios.map((scenario) => scenario.id)).toEqual([
|
||||
"discord-crabline-roundtrip",
|
||||
]);
|
||||
expect(plan.exclusions).toEqual(
|
||||
expect.arrayContaining([
|
||||
expect.objectContaining({
|
||||
scenarioId: "compaction-retry-mutating-tool",
|
||||
reasons: ["does not declare channel discord"],
|
||||
}),
|
||||
]),
|
||||
);
|
||||
});
|
||||
|
||||
it("rejects an explicit scenario that does not declare the external channel", () => {
|
||||
const catalog = readQaScenarioPack();
|
||||
const scorecardReport = readQaScorecardTaxonomyReport(catalog.scenarios);
|
||||
const selection = normalizeQaRunSelection(
|
||||
{
|
||||
profile: "all",
|
||||
channel: "discord",
|
||||
channelDriver: "crabline",
|
||||
providerMode: "mock-openai",
|
||||
scenarioIds: ["compaction-retry-mutating-tool"],
|
||||
},
|
||||
catalog.scenarios,
|
||||
scorecardReport.profiles,
|
||||
);
|
||||
|
||||
const plan = resolveQaLabRunPlan({
|
||||
selection,
|
||||
scenarios: catalog.scenarios,
|
||||
scorecardReport,
|
||||
defaultChannel: "discord",
|
||||
supportsChannel: (channel) => channel === "discord",
|
||||
});
|
||||
|
||||
expect(plan.status).toBe("invalid");
|
||||
expect(plan.errors).toContain(
|
||||
"Explicit QA scenario selection is not runnable: compaction-retry-mutating-tool (does not declare channel discord).",
|
||||
);
|
||||
});
|
||||
|
||||
it("fails closed when an explicit scenario conflicts with execution.channel", () => {
|
||||
const catalog = readQaScenarioPack();
|
||||
const scorecardReport = readQaScorecardTaxonomyReport(catalog.scenarios);
|
||||
|
||||
@@ -16,6 +16,7 @@ import {
|
||||
import {
|
||||
resolveQaRunProfileExecutionSelection,
|
||||
resolveQaRunProfileMembership,
|
||||
scenarioDeclaresQaChannel,
|
||||
} from "./profile-planning.js";
|
||||
import {
|
||||
DEFAULT_QA_LIVE_PROVIDER_MODE,
|
||||
@@ -360,8 +361,27 @@ export function resolveQaLabRunPlan(params: {
|
||||
const scenario = scenarioById.get(scenarioId);
|
||||
return scenario ? [scenario] : [];
|
||||
});
|
||||
const requiresDeclaredChannel = Boolean(
|
||||
selection.channel &&
|
||||
selection.channel !== "qa-channel" &&
|
||||
selection.channelDriver !== "qa-channel",
|
||||
);
|
||||
const channelScopedScenarios = requiresDeclaredChannel
|
||||
? laneScenarios.filter((scenario) => scenarioDeclaresQaChannel(scenario, selection.channel!))
|
||||
: laneScenarios;
|
||||
if (requiresDeclaredChannel) {
|
||||
exclusions.push(
|
||||
...laneScenarios
|
||||
.filter((scenario) => !channelScopedScenarios.includes(scenario))
|
||||
.map((scenario) => ({
|
||||
scenarioId: scenario.id,
|
||||
executionKind: scenario.execution.kind,
|
||||
reasons: [`does not declare channel ${selection.channel}`],
|
||||
})),
|
||||
);
|
||||
}
|
||||
const profileExecution = resolveQaRunProfileExecutionSelection({
|
||||
scenarios: laneScenarios,
|
||||
scenarios: channelScopedScenarios,
|
||||
providerMode: selection.providerMode,
|
||||
primaryModel: selection.primaryModel,
|
||||
channelDriver: selection.channelDriver,
|
||||
|
||||
@@ -990,8 +990,8 @@ describe("qa suite planning helpers", () => {
|
||||
channel: "telegram",
|
||||
}).map((scenario) => scenario.id);
|
||||
|
||||
expect(selectForDriver("crabline")).toEqual(["generic", "telegram"]);
|
||||
expect(selectForDriver("live")).toEqual(["generic", "live-only", "telegram"]);
|
||||
expect(selectForDriver("crabline")).toEqual(["telegram"]);
|
||||
expect(selectForDriver("live")).toEqual(["live-only", "telegram"]);
|
||||
});
|
||||
|
||||
it("rejects explicitly requested scenarios that do not match the current lane", () => {
|
||||
|
||||
@@ -7,6 +7,7 @@ import { createQaArtifactRunId } from "./artifact-run-id.js";
|
||||
import { ensureRepoBoundDirectory, resolveRepoRelativeOutputDir } from "./cli-paths.js";
|
||||
import type { QaCliBackendAuthMode } from "./gateway-child.js";
|
||||
import { splitQaModelRef as splitModelRef, type QaProviderMode } from "./model-selection.js";
|
||||
import { scenarioDeclaresQaChannel } from "./profile-planning.js";
|
||||
import { readQaBootstrapScenarioCatalog } from "./scenario-catalog.js";
|
||||
import {
|
||||
describeQaProviderLaneMismatches,
|
||||
|
||||
@@ -8,6 +8,8 @@ scenario:
|
||||
execution:
|
||||
kind: flow
|
||||
channel: discord
|
||||
config:
|
||||
requiredChannelDriver: live
|
||||
timeoutMs: 45000
|
||||
retryCount: 1
|
||||
suiteIsolation: isolated
|
||||
|
||||
@@ -0,0 +1,78 @@
|
||||
title: Discord Crabline real-plugin roundtrip
|
||||
|
||||
scenario:
|
||||
id: discord-crabline-roundtrip
|
||||
surface: channels
|
||||
coverage:
|
||||
primary:
|
||||
- discord.configured-and-runtime-routing
|
||||
secondary:
|
||||
- discord.mention-gating
|
||||
- discord.native-slash-command-registration
|
||||
objective: Verify a provider-native Discord event traverses the real bundled plugin and produces one provider REST reply.
|
||||
successCriteria:
|
||||
- The real Discord plugin completes its Gateway handshake against Crabline.
|
||||
- One mentioned guild message produces one visible reply containing the exact marker.
|
||||
- The reply stays scoped to the originating Discord channel.
|
||||
docsRefs:
|
||||
- docs/concepts/qa-e2e-automation.md
|
||||
- docs/channels/discord.md
|
||||
codeRefs:
|
||||
- extensions/discord/src/monitor/provider.ts
|
||||
- extensions/qa-lab/src/crabline-transport.ts
|
||||
execution:
|
||||
kind: flow
|
||||
channel: discord
|
||||
config:
|
||||
requiredChannelDriver: crabline
|
||||
conversationId: discord-crabline-primary
|
||||
threadId: discord-crabline-thread
|
||||
expectedMarker: DISCORD-CRABLINE-ROUNDTRIP-OK
|
||||
transportPolicy:
|
||||
requireGroupMention: true
|
||||
timeoutMs: 60000
|
||||
suiteIsolation: isolated
|
||||
|
||||
flow:
|
||||
steps:
|
||||
- name: completes Discord Gateway readiness
|
||||
actions:
|
||||
- call: waitForGatewayHealthy
|
||||
args: [{ ref: env }, 60000]
|
||||
- call: waitForTransportReady
|
||||
args: [{ ref: env }, 60000]
|
||||
- resetTransport: true
|
||||
- name: injects an authenticated provider-native Discord event
|
||||
actions:
|
||||
- sendInbound:
|
||||
conversation:
|
||||
id: { ref: config.conversationId }
|
||||
kind: group
|
||||
senderId: driver
|
||||
senderName: QA Driver
|
||||
threadId: { ref: config.threadId }
|
||||
text:
|
||||
expr: "`@openclaw reply exactly: ${config.expectedMarker}`"
|
||||
saveAs: inbound
|
||||
- name: observes the real Discord plugin outbound REST request
|
||||
actions:
|
||||
- waitForOutbound:
|
||||
conversation:
|
||||
id: { ref: config.conversationId }
|
||||
kind: group
|
||||
threadId: { ref: config.threadId }
|
||||
textIncludes: { ref: config.expectedMarker }
|
||||
timeoutMs:
|
||||
expr: liveTurnTimeoutMs(env, 60000)
|
||||
saveAs: outbound
|
||||
- set: matchingOutbound
|
||||
value:
|
||||
expr: "state.getSnapshot().messages.filter((message) => message.direction === 'outbound' && message.conversation.id === config.conversationId && String(message.text ?? '').includes(config.expectedMarker))"
|
||||
- assert:
|
||||
expr: matchingOutbound.length === 1
|
||||
message:
|
||||
expr: "`expected one Discord Crabline reply, saw ${matchingOutbound.length}; transcript=${formatTransportTranscript(state, { conversationId: config.conversationId })}`"
|
||||
- assert:
|
||||
expr: Boolean(inbound.id) && Boolean(outbound.id)
|
||||
message: Discord Crabline roundtrip did not preserve provider message identity
|
||||
detailsExpr: outbound.text
|
||||
@@ -8,6 +8,8 @@ scenario:
|
||||
execution:
|
||||
kind: flow
|
||||
channel: discord
|
||||
config:
|
||||
requiredChannelDriver: live
|
||||
timeoutMs: 8000
|
||||
retryCount: 1
|
||||
suiteIsolation: isolated
|
||||
|
||||
@@ -7,6 +7,8 @@ scenario:
|
||||
execution:
|
||||
kind: flow
|
||||
channel: discord
|
||||
config:
|
||||
requiredChannelDriver: live
|
||||
timeoutMs: 45000
|
||||
retryCount: 1
|
||||
suiteIsolation: isolated
|
||||
|
||||
@@ -8,6 +8,8 @@ scenario:
|
||||
execution:
|
||||
kind: flow
|
||||
channel: discord
|
||||
config:
|
||||
requiredChannelDriver: live
|
||||
timeoutMs: 75000
|
||||
retryCount: 1
|
||||
suiteIsolation: isolated
|
||||
|
||||
@@ -9,6 +9,8 @@ scenario:
|
||||
execution:
|
||||
kind: flow
|
||||
channel: discord
|
||||
config:
|
||||
requiredChannelDriver: live
|
||||
timeoutMs: 45000
|
||||
retryCount: 1
|
||||
suiteIsolation: isolated
|
||||
|
||||
@@ -7,6 +7,8 @@ scenario:
|
||||
execution:
|
||||
kind: flow
|
||||
channel: discord
|
||||
config:
|
||||
requiredChannelDriver: live
|
||||
timeoutMs: 60000
|
||||
retryCount: 1
|
||||
suiteIsolation: isolated
|
||||
|
||||
Reference in New Issue
Block a user