test: cover dynamic live model refs

docs/help/testing.md

@@ -738,13 +738,13 @@ plugin validation checklist, see
 These Docker runners split into two buckets:
 
 - Live-model runners: `test:docker:live-models` and `test:docker:live-gateway` run only their matching profile-key live file inside the repo Docker image (`src/agents/models.profiles.live.test.ts` and `src/gateway/gateway-models.profiles.live.test.ts`), mounting your local config dir, workspace, and optional profile env file. The matching local entrypoints are `test:live:models-profiles` and `test:live:gateway-profiles`.
-- Docker live runners default to a smaller smoke cap so a full Docker sweep stays practical:
-  `test:docker:live-models` defaults to `OPENCLAW_LIVE_MAX_MODELS=12`, and
+- Docker live runners keep their own practical caps where needed:
+  `test:docker:live-models` defaults to the curated supported high-signal set, and
   `test:docker:live-gateway` defaults to `OPENCLAW_LIVE_GATEWAY_SMOKE=1`,
   `OPENCLAW_LIVE_GATEWAY_MAX_MODELS=8`,
   `OPENCLAW_LIVE_GATEWAY_STEP_TIMEOUT_MS=45000`, and
-  `OPENCLAW_LIVE_GATEWAY_MODEL_TIMEOUT_MS=90000`. Override those env vars when you
-  explicitly want the larger exhaustive scan.
+  `OPENCLAW_LIVE_GATEWAY_MODEL_TIMEOUT_MS=90000`. Set `OPENCLAW_LIVE_MAX_MODELS`
+  or the gateway env vars when you explicitly want a smaller cap or larger scan.
 - `test:docker:all` builds the live Docker image once via `test:docker:live-build`, packs OpenClaw once as an npm tarball through `scripts/package-openclaw-for-docker.mjs`, then builds/reuses two `scripts/e2e/Dockerfile` images. The bare image is only the Node/Git runner for install/update/plugin-dependency lanes; those lanes mount the prebuilt tarball. The functional image installs the same tarball into `/app` for built-app functionality lanes. Docker lane definitions live in `scripts/lib/docker-e2e-scenarios.mjs`; planner logic lives in `scripts/lib/docker-e2e-plan.mjs`; `scripts/test-docker-all.mjs` executes the selected plan. The aggregate uses a weighted local scheduler: `OPENCLAW_DOCKER_ALL_PARALLELISM` controls process slots, while resource caps keep heavy live, npm-install, and multi-service lanes from all starting at once. If a single lane is heavier than the active caps, the scheduler can still start it when the pool is empty and then keeps it running alone until capacity is available again. Defaults are 10 slots, `OPENCLAW_DOCKER_ALL_LIVE_LIMIT=9`, `OPENCLAW_DOCKER_ALL_NPM_LIMIT=10`, and `OPENCLAW_DOCKER_ALL_SERVICE_LIMIT=7`; tune `OPENCLAW_DOCKER_ALL_WEIGHT_LIMIT` or `OPENCLAW_DOCKER_ALL_DOCKER_LIMIT` only when the Docker host has more headroom. The runner performs a Docker preflight by default, removes stale OpenClaw E2E containers, prints status every 30 seconds, stores successful lane timings in `.artifacts/docker-tests/lane-timings.json`, and uses those timings to start longer lanes first on later runs. Use `OPENCLAW_DOCKER_ALL_DRY_RUN=1` to print the weighted lane manifest without building or running Docker, or `node scripts/test-docker-all.mjs --plan-json` to print the CI plan for selected lanes, package/image needs, and credentials.
 - `Package Acceptance` is the GitHub-native package gate for "does this installable tarball work as a product?" It resolves one candidate package from `source=npm`, `source=ref`, `source=url`, or `source=artifact`, uploads it as `package-under-test`, then runs the reusable Docker E2E lanes against that exact tarball instead of repacking the selected ref. Profiles are ordered by breadth: `smoke`, `package`, `product`, and `full`. See [Testing updates and plugins](/help/testing-updates-plugins) for the package/update/plugin contract, published-upgrade survivor matrix, release defaults, and failure triage.
 - Build and release checks run `scripts/check-cli-bootstrap-imports.mjs` after tsdown. The guard walks the static built graph from `dist/entry.js` and `dist/cli/run-main.js` and fails if pre-dispatch startup imports package dependencies such as Commander, prompt UI, undici, or logging before command dispatch; it also keeps the bundled gateway run chunk under budget and rejects static imports of known cold gateway paths. Packaged CLI smoke also covers root help, onboard help, doctor help, status, config schema, and a model-list command.

docs/providers/fireworks.md

@@ -82,7 +82,7 @@ openclaw onboard --non-interactive \
 
 ## Custom Fireworks model ids
 
-OpenClaw accepts any Fireworks model or router id at runtime. Use the exact id shown by Fireworks and prefix it with `fireworks/`. Dynamic resolution clones the Fire Pass template (text + image input, OpenAI-compatible API, default cost zero) and disables thinking automatically when the id matches the Kimi pattern.
+OpenClaw accepts any Fireworks model or router id at runtime. Use the exact id shown by Fireworks and prefix it with `fireworks/`. Dynamic resolution clones the Fire Pass template (text + image input, OpenAI-compatible API, default cost zero) and disables thinking automatically when the id matches the Kimi pattern. GLM dynamic ids are marked text-only unless you configure a custom model entry with image input.
 
 ```json5
 {

extensions/fireworks/index.test.ts

@@ -94,6 +94,7 @@ describe("fireworks provider plugin", () => {
     expect(resolved?.api).toBe("openai-completions");
     expect(resolved?.baseUrl).toBe(FIREWORKS_BASE_URL);
     expect(resolved?.reasoning).toBe(true);
+    expect(resolved?.input).toEqual(["text", "image"]);
   });
 
   it("disables reasoning metadata for Fireworks Kimi dynamic models", async () => {
@@ -109,6 +110,22 @@ describe("fireworks provider plugin", () => {
     expect(resolved?.provider).toBe("fireworks");
     expect(resolved?.id).toBe("accounts/fireworks/models/kimi-k2p5");
     expect(resolved?.reasoning).toBe(false);
+    expect(resolved?.input).toEqual(["text", "image"]);
+  });
+
+  it("keeps Fireworks GLM dynamic models text-only", async () => {
+    const provider = await registerSingleProviderPlugin(fireworksPlugin);
+    const resolved = provider.resolveDynamicModel?.(
+      createProviderDynamicModelContext({
+        provider: "fireworks",
+        modelId: "accounts/fireworks/models/glm-5p1",
+        models: [createFireworksDefaultRuntimeModel({ reasoning: false })],
+      }),
+    );
+
+    expect(resolved?.provider).toBe("fireworks");
+    expect(resolved?.id).toBe("accounts/fireworks/models/glm-5p1");
+    expect(resolved?.input).toEqual(["text"]);
   });
 
   it("disables reasoning metadata for Fireworks Kimi k2.5 aliases", async () => {

extensions/fireworks/index.ts

@@ -19,11 +19,23 @@ import { wrapFireworksProviderStream } from "./stream.js";
 import { resolveFireworksThinkingProfile } from "./thinking-policy.js";
 
 const PROVIDER_ID = "fireworks";
+function isFireworksGlmModelId(modelId: string): boolean {
+  const normalized = modelId.trim().toLowerCase();
+  const lastSegment = normalized.split("/").pop() ?? normalized;
+  return /^glm[-_.]/.test(lastSegment);
+}
+
+function resolveFireworksDynamicInput(modelId: string): Array<"text" | "image"> {
+  return isFireworksGlmModelId(modelId) ? ["text"] : ["text", "image"];
+}
+
 function resolveFireworksDynamicModel(ctx: ProviderResolveDynamicModelContext) {
   const modelId = ctx.modelId.trim();
   if (!modelId) {
     return undefined;
   }
+  const isKimiModel = isFireworksKimiModelId(modelId);
+  const input = resolveFireworksDynamicInput(modelId);
 
   return (
     cloneFirstTemplateModel({
@@ -33,7 +45,8 @@ function resolveFireworksDynamicModel(ctx: ProviderResolveDynamicModelContext) {
       ctx,
       patch: {
         provider: PROVIDER_ID,
-        reasoning: !isFireworksKimiModelId(modelId),
+        reasoning: !isKimiModel,
+        input,
       },
     }) ??
     normalizeModelCompat({
@@ -42,8 +55,8 @@ function resolveFireworksDynamicModel(ctx: ProviderResolveDynamicModelContext) {
       provider: PROVIDER_ID,
       api: "openai-completions",
       baseUrl: FIREWORKS_BASE_URL,
-      reasoning: !isFireworksKimiModelId(modelId),
-      input: ["text", "image"],
+      reasoning: !isKimiModel,
+      input,
       cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 },
       contextWindow: FIREWORKS_DEFAULT_CONTEXT_WINDOW,
       maxTokens: FIREWORKS_DEFAULT_MAX_TOKENS || DEFAULT_CONTEXT_TOKENS,

scripts/test-live-models-docker.sh

@@ -216,7 +216,7 @@ DOCKER_RUN_ARGS+=(--rm -t \
   -e OPENCLAW_LIVE_TEST=1 \
   -e OPENCLAW_LIVE_MODELS="${OPENCLAW_LIVE_MODELS:-modern}" \
   -e OPENCLAW_LIVE_PROVIDERS="${OPENCLAW_LIVE_PROVIDERS:-}" \
-  -e OPENCLAW_LIVE_MAX_MODELS="${OPENCLAW_LIVE_MAX_MODELS:-12}" \
+  -e OPENCLAW_LIVE_MAX_MODELS="${OPENCLAW_LIVE_MAX_MODELS:-}" \
   -e OPENCLAW_LIVE_MODEL_TIMEOUT_MS="${OPENCLAW_LIVE_MODEL_TIMEOUT_MS:-}" \
   -e OPENCLAW_LIVE_REQUIRE_PROFILE_KEYS="${OPENCLAW_LIVE_REQUIRE_PROFILE_KEYS:-}" \
   -e OPENCLAW_LIVE_GATEWAY_MODELS="${OPENCLAW_LIVE_GATEWAY_MODELS:-}" \

src/agents/live-model-dynamic-candidates.test.ts

@@ -0,0 +1,115 @@
+import { describe, expect, it, vi } from "vitest";
+import type { OpenClawConfig } from "../config/types.openclaw.js";
+import type { Model } from "../llm/types.js";
+import { appendPrioritizedDynamicLiveModels } from "./live-model-dynamic-candidates.js";
+
+const REGISTRY = { find: () => undefined } as never;
+type DynamicModelResolver = NonNullable<
+  Parameters<typeof appendPrioritizedDynamicLiveModels>[0]["resolveDynamicModel"]
+>;
+type DynamicModelPreparer = NonNullable<
+  Parameters<typeof appendPrioritizedDynamicLiveModels>[0]["prepareDynamicModel"]
+>;
+
+function model(provider: string, id: string): Model {
+  return {
+    id,
+    name: id,
+    provider,
+    api: "openai-completions",
+    baseUrl: "https://example.test/v1",
+    reasoning: false,
+    input: ["text"],
+    cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 },
+    contextWindow: 128_000,
+    maxTokens: 4_096,
+  };
+}
+
+describe("appendPrioritizedDynamicLiveModels", () => {
+  it("materializes prioritized refs from provider dynamic model hooks", async () => {
+    const resolveDynamicModel: DynamicModelResolver = vi.fn((params) =>
+      params.context.provider === "opencode-go" && params.context.modelId === "glm-5"
+        ? model("opencode-go", "glm-5")
+        : undefined,
+    );
+    const prepareDynamicModel: DynamicModelPreparer = vi.fn(async () => undefined);
+    const config = {
+      models: {
+        providers: {
+          "opencode-go": {
+            api: "openai-completions",
+            baseUrl: "https://configured.example/v1",
+            models: [],
+          },
+        },
+      },
+    } as OpenClawConfig;
+
+    const result = await appendPrioritizedDynamicLiveModels({
+      models: [model("anthropic", "claude-sonnet-4-6")],
+      config,
+      agentDir: "/tmp/openclaw-agent",
+      modelRegistry: REGISTRY,
+      resolveDynamicModel,
+      prepareDynamicModel,
+      refs: [
+        { provider: "anthropic", id: "claude-sonnet-4-6" },
+        { provider: "opencode-go", id: "glm-5" },
+      ],
+    });
+
+    expect(result.added.map((entry) => `${entry.provider}/${entry.id}`)).toEqual([
+      "opencode-go/glm-5",
+    ]);
+    expect(result.models.map((entry) => `${entry.provider}/${entry.id}`)).toEqual([
+      "anthropic/claude-sonnet-4-6",
+      "opencode-go/glm-5",
+    ]);
+    expect(prepareDynamicModel).toHaveBeenCalledTimes(1);
+    expect(prepareDynamicModel).toHaveBeenCalledWith(
+      expect.objectContaining({
+        provider: "opencode-go",
+        context: expect.objectContaining({
+          agentDir: "/tmp/openclaw-agent",
+          modelId: "glm-5",
+          modelRegistry: REGISTRY,
+          provider: "opencode-go",
+          providerConfig: config.models?.providers?.["opencode-go"],
+        }),
+      }),
+    );
+    expect(resolveDynamicModel).toHaveBeenCalledTimes(1);
+    expect(resolveDynamicModel).toHaveBeenCalledWith(
+      expect.objectContaining({
+        provider: "opencode-go",
+        context: expect.objectContaining({
+          agentDir: "/tmp/openclaw-agent",
+          modelId: "glm-5",
+          modelRegistry: REGISTRY,
+          provider: "opencode-go",
+          providerConfig: config.models?.providers?.["opencode-go"],
+        }),
+      }),
+    );
+  });
+
+  it("does not duplicate refs already present in the generated registry", async () => {
+    const resolveDynamicModel: DynamicModelResolver = vi.fn(() => model("opencode-go", "glm-5"));
+    const prepareDynamicModel: DynamicModelPreparer = vi.fn(async () => undefined);
+
+    const result = await appendPrioritizedDynamicLiveModels({
+      models: [model("opencode-go", "glm-5")],
+      agentDir: "/tmp/openclaw-agent",
+      modelRegistry: REGISTRY,
+      resolveDynamicModel,
+      prepareDynamicModel,
+      refs: [{ provider: "opencode-go", id: "glm-5" }],
+    });
+
+    expect(result.added).toEqual([]);
+    expect(result.models).toHaveLength(1);
+    expect(prepareDynamicModel).not.toHaveBeenCalled();
+    expect(resolveDynamicModel).not.toHaveBeenCalled();
+  });
+});

src/agents/live-model-dynamic-candidates.ts

@@ -0,0 +1,91 @@
+import type { OpenClawConfig } from "../config/types.openclaw.js";
+import type { Model } from "../llm/types.js";
+import {
+  prepareProviderDynamicModel,
+  runProviderDynamicModel,
+} from "../plugins/provider-runtime.js";
+import type { ProviderResolveDynamicModelContext } from "../plugins/types.js";
+import { normalizeLowercaseStringOrEmpty } from "../shared/string-coerce.js";
+import { normalizeDiscoveredAgentModel } from "./agent-model-discovery.js";
+import { listPrioritizedHighSignalLiveModelRefs } from "./live-model-filter.js";
+import { findNormalizedProviderValue, normalizeProviderId } from "./provider-id.js";
+
+type DynamicModelResolver = typeof runProviderDynamicModel;
+type DynamicModelPreparer = typeof prepareProviderDynamicModel;
+
+function liveModelKey(provider: string, id: string): string | null {
+  const normalizedProvider = normalizeProviderId(provider);
+  const normalizedId = normalizeLowercaseStringOrEmpty(id);
+  return normalizedProvider && normalizedId ? `${normalizedProvider}/${normalizedId}` : null;
+}
+
+export async function appendPrioritizedDynamicLiveModels(params: {
+  models: Model[];
+  config?: OpenClawConfig;
+  agentDir: string;
+  workspaceDir?: string;
+  env?: NodeJS.ProcessEnv;
+  modelRegistry: ProviderResolveDynamicModelContext["modelRegistry"];
+  resolveDynamicModel?: DynamicModelResolver;
+  prepareDynamicModel?: DynamicModelPreparer;
+  refs?: Array<{ provider: string; id: string }>;
+}): Promise<{ models: Model[]; added: Model[] }> {
+  const resolveDynamicModel = params.resolveDynamicModel ?? runProviderDynamicModel;
+  const prepareDynamicModel = params.prepareDynamicModel ?? prepareProviderDynamicModel;
+  const refs = params.refs ?? listPrioritizedHighSignalLiveModelRefs();
+  const seen = new Set<string>();
+  for (const model of params.models) {
+    const key = liveModelKey(model.provider, model.id);
+    if (key) {
+      seen.add(key);
+    }
+  }
+
+  const models = [...params.models];
+  const added: Model[] = [];
+  for (const ref of refs) {
+    const requestedKey = liveModelKey(ref.provider, ref.id);
+    if (!requestedKey || seen.has(requestedKey)) {
+      continue;
+    }
+    const providerConfig = findNormalizedProviderValue(
+      params.config?.models?.providers,
+      ref.provider,
+    );
+    const context = {
+      config: params.config,
+      agentDir: params.agentDir,
+      workspaceDir: params.workspaceDir,
+      provider: ref.provider,
+      modelId: ref.id,
+      modelRegistry: params.modelRegistry,
+      providerConfig,
+    };
+    await prepareDynamicModel({
+      provider: ref.provider,
+      config: params.config,
+      workspaceDir: params.workspaceDir,
+      env: params.env,
+      context,
+    });
+    const resolved = resolveDynamicModel({
+      provider: ref.provider,
+      config: params.config,
+      workspaceDir: params.workspaceDir,
+      env: params.env,
+      context,
+    });
+    if (!resolved) {
+      continue;
+    }
+    const model = normalizeDiscoveredAgentModel(resolved as Model, params.agentDir);
+    const resolvedKey = liveModelKey(model.provider, model.id);
+    if (!resolvedKey || seen.has(resolvedKey)) {
+      continue;
+    }
+    seen.add(resolvedKey);
+    models.push(model);
+    added.push(model);
+  }
+  return { models, added };
+}

src/agents/live-model-filter.ts

@@ -26,7 +26,6 @@ const HIGH_SIGNAL_LIVE_MODEL_PRIORITY = [
   "openrouter/ai21/jamba-large-1.7",
   "xai/grok-4.3",
   "zai/glm-5.1",
-  "fireworks/accounts/fireworks/models/glm-5",
   "fireworks/accounts/fireworks/models/glm-5p1",
   "minimax-portal/minimax-m2.7",
 ] as const;

src/agents/model-compat.test.ts

@@ -577,7 +577,7 @@ describe("isHighSignalLiveModelRef", () => {
     expect(isHighSignalLiveModelRef({ provider: "zai", id: "glm-5.1" })).toBe(true);
     expect(
       isHighSignalLiveModelRef({ provider: "fireworks", id: "accounts/fireworks/models/glm-5" }),
-    ).toBe(true);
+    ).toBe(false);
     expect(
       isHighSignalLiveModelRef({ provider: "fireworks", id: "accounts/fireworks/models/glm-5p1" }),
     ).toBe(true);
@@ -672,7 +672,6 @@ describe("isPrioritizedHighSignalLiveModelRef", () => {
       { provider: "openrouter", id: "ai21/jamba-large-1.7" },
       { provider: "xai", id: "grok-4.3" },
       { provider: "zai", id: "glm-5.1" },
-      { provider: "fireworks", id: "accounts/fireworks/models/glm-5" },
       { provider: "fireworks", id: "accounts/fireworks/models/glm-5p1" },
       { provider: "minimax-portal", id: "minimax-m2.7" },
     ]);
@@ -729,13 +728,14 @@ describe("selectHighSignalLiveItems", () => {
     ]);
   });
 
-  it("prioritizes Fireworks GLM 5 models over GLM 4.x fallback entries", () => {
+  it("prioritizes supported Fireworks GLM 5 models over GLM 4.x fallback entries", () => {
+    providerRuntimeMocks.resolveProviderModernModelRef.mockReturnValue(true);
     const items = [
       { provider: "fireworks", id: "accounts/fireworks/models/glm-4p7" },
       { provider: "fireworks", id: "accounts/fireworks/models/glm-5" },
       { provider: "fireworks", id: "accounts/fireworks/models/glm-5p1" },
       { provider: "fireworks", id: "accounts/fireworks/models/gpt-oss-120b" },
-    ];
+    ].filter(isHighSignalLiveModelRef);
 
     expect(
       selectHighSignalLiveItems(
@@ -744,10 +744,7 @@ describe("selectHighSignalLiveItems", () => {
         (item) => item,
         (item) => item.provider,
       ),
-    ).toEqual([
-      { provider: "fireworks", id: "accounts/fireworks/models/glm-5" },
-      { provider: "fireworks", id: "accounts/fireworks/models/glm-5p1" },
-    ]);
+    ).toEqual([{ provider: "fireworks", id: "accounts/fireworks/models/glm-5p1" }]);
   });
 });
 

src/agents/models.profiles.live.test.ts

@@ -14,6 +14,7 @@ import { resolveDefaultAgentDir } from "./agent-scope.js";
 import { externalCliDiscoveryForProviders } from "./auth-profiles/external-cli-discovery.js";
 import { isRateLimitErrorMessage } from "./embedded-agent-helpers/errors.js";
 import { collectAnthropicApiKeys } from "./live-auth-keys.js";
+import { appendPrioritizedDynamicLiveModels } from "./live-model-dynamic-candidates.js";
 import { isModelNotFoundErrorMessage } from "./live-model-errors.js";
 import {
   isHighSignalLiveModelRef,
@@ -159,6 +160,16 @@ function formatFailurePreview(
   return lines.join("\n");
 }
 
+function formatSkippedPreview(
+  skipped: Array<{ model: string; reason: string }>,
+  maxItems: number,
+): string {
+  return formatFailurePreview(
+    skipped.map((entry) => ({ model: entry.model, error: entry.reason })),
+    maxItems,
+  );
+}
+
 function isGoogleModelNotFoundError(err: unknown): boolean {
   const msg = String(err);
   if (!/not found/i.test(msg)) {
@@ -755,12 +766,26 @@ describeLive("live models (profile keys)", () => {
           "[live-models] load auth storage",
         );
         logProgress("[live-models] loading model registry");
-        return withLiveStageTimeout(
+        const modelRegistry = await withLiveStageTimeout(
           Promise.resolve().then(() =>
-            discoverModels(authStorage, agentDir, { normalizeModels: false }).getAll(),
+            discoverModels(authStorage, agentDir, { normalizeModels: false }),
           ),
           "[live-models] load model registry",
         );
+        const configuredModels = modelRegistry.getAll();
+        const augmented = await appendPrioritizedDynamicLiveModels({
+          models: configuredModels,
+          config: cfg,
+          agentDir,
+          env: process.env,
+          modelRegistry,
+        });
+        if (augmented.added.length > 0) {
+          logProgress(
+            `[live-models] loaded ${augmented.added.length} prioritized dynamic model refs`,
+          );
+        }
+        return augmented.models;
       })();
       const perModelTimeoutMs = toInt(process.env.OPENCLAW_LIVE_MODEL_TIMEOUT_MS, 30_000);
       const maxModels = resolveHighSignalLiveModelLimit({
@@ -843,6 +868,13 @@ describeLive("live models (profile keys)", () => {
       }
 
       if (candidates.length === 0) {
+        if (useExplicit) {
+          const skippedPreview =
+            skipped.length > 0 ? `\nSkipped candidates:\n${formatSkippedPreview(skipped, 8)}` : "";
+          throw new Error(
+            `[live-models] explicit model selection matched no runnable models.${skippedPreview}`,
+          );
+        }
         logProgress("[live-models] no API keys found; skipping");
         return;
       }