feat(plugins): catch code-mode drift between catalogs shipping the same model (#115183)

* feat(plugins): catch code-mode drift between catalogs sharing one model

Adds a contract test that groups bundled catalog rows by shared upstream
model and requires every row in a group to declare compat.codeMode once any
sibling does. Rows sharing a model id group automatically; rows under
different ids opt in with the new manifest-only `upstreamModel` marker.

Moves the kimi catalog into its manifest so the scan can see it, and records
the tier reseller catalogs were silently missing as explicit "capable".

* docs: regenerate docs map for the shared-model code-mode section
This commit is contained in:
Peter Steinberger
2026-07-28 08:52:21 -04:00
committed by GitHub
parent 4eab19a25d
commit 99ec30b44b
19 changed files with 393 additions and 128 deletions
+1
View File
@@ -9967,6 +9967,7 @@ Do not edit it by hand; run `pnpm docs:map:gen`.
- H2: Automatic per-model activation
- H3: The compat.codeMode catalog flag
- H3: Shipped preferred models
- H3: Models shipped by more than one provider
- H3: Choosing when to enable
- H2: Activation
- H2: Model-visible tools
+24 -21
View File
@@ -981,27 +981,28 @@ Provider fields:
Model fields:
| Field | Type | What it means |
| ------------------ | -------------------------------------------------------------- | --------------------------------------------------------------------------- |
| `id` | `string` | Provider-local model id, without the `provider/` prefix. |
| `name` | `string` | Optional display name. |
| `api` | `ModelApi` | Optional per-model API override. |
| `baseUrl` | `string` | Optional per-model base URL override. |
| `headers` | `Record<string, string>` | Optional per-model static headers. |
| `input` | `Array<"text" \| "image" \| "document">` | Modalities the model accepts. Other values are silently dropped. |
| `reasoning` | `boolean` | Whether the model exposes reasoning behavior. |
| `contextWindow` | `number` | Native provider context window. |
| `contextTokens` | `number` | Optional effective runtime context cap when different from `contextWindow`. |
| `maxTokens` | `number` | Maximum output tokens when known. |
| `thinkingLevelMap` | `Record<string, string \| null>` | Optional per-thinking-level model-id or param overrides. |
| `cost` | `object` | Optional USD per million token pricing, including optional `tieredPricing`. |
| `compat` | `object` | Optional compatibility flags matching OpenClaw model config compatibility. |
| `mediaInput` | `object` | Optional per-modality input config, currently image-only. |
| `status` | `"available"` \| `"preview"` \| `"deprecated"` \| `"disabled"` | Listing status. Suppress only when the row must not appear at all. |
| `statusReason` | `string` | Optional reason shown with non-available status. |
| `replaces` | `string[]` | Older provider-local model ids this model supersedes. |
| `replacedBy` | `string` | Replacement provider-local model id for deprecated rows. |
| `tags` | `string[]` | Stable tags used by pickers and filters. |
| Field | Type | What it means |
| ------------------ | -------------------------------------------------------------- | ------------------------------------------------------------------------------------ |
| `id` | `string` | Provider-local model id, without the `provider/` prefix. |
| `name` | `string` | Optional display name. |
| `api` | `ModelApi` | Optional per-model API override. |
| `baseUrl` | `string` | Optional per-model base URL override. |
| `headers` | `Record<string, string>` | Optional per-model static headers. |
| `input` | `Array<"text" \| "image" \| "document">` | Modalities the model accepts. Other values are silently dropped. |
| `reasoning` | `boolean` | Whether the model exposes reasoning behavior. |
| `contextWindow` | `number` | Native provider context window. |
| `contextTokens` | `number` | Optional effective runtime context cap when different from `contextWindow`. |
| `maxTokens` | `number` | Maximum output tokens when known. |
| `thinkingLevelMap` | `Record<string, string \| null>` | Optional per-thinking-level model-id or param overrides. |
| `cost` | `object` | Optional USD per million token pricing, including optional `tieredPricing`. |
| `compat` | `object` | Optional compatibility flags matching OpenClaw model config compatibility. |
| `upstreamModel` | `string` | Optional `provider/model` ref of the same upstream model in another bundled catalog. |
| `mediaInput` | `object` | Optional per-modality input config, currently image-only. |
| `status` | `"available"` \| `"preview"` \| `"deprecated"` \| `"disabled"` | Listing status. Suppress only when the row must not appear at all. |
| `statusReason` | `string` | Optional reason shown with non-available status. |
| `replaces` | `string[]` | Older provider-local model ids this model supersedes. |
| `replacedBy` | `string` | Replacement provider-local model id for deprecated rows. |
| `tags` | `string[]` | Stable tags used by pickers and filters. |
Suppression fields:
@@ -1013,6 +1014,8 @@ Suppression fields:
| `when.baseUrlHosts` | `string[]` | Optional list of effective provider base URL hosts required before the suppression applies. |
| `when.providerConfigApiIn` | `string[]` | Optional list of exact provider-config `api` values required before the suppression applies. |
`upstreamModel` marks a row that serves the same upstream model as a row in another bundled catalog under a different id, for example a subscription endpoint next to the vendor's API endpoint. It is authoring metadata: normalization drops it, and a contract test uses it to keep capability flags such as `compat.codeMode` from drifting between catalogs that ship the same model. Rows that already share the same model id need no marker. See [Code mode](/tools/code-mode#models-shipped-by-more-than-one-provider).
Do not put runtime-only data in `modelCatalog`. Use `static` only when manifest rows are complete enough for provider-filtered list and picker surfaces to skip registry/runtime discovery. Use `refreshable` when manifest rows are useful listable seeds or supplements but a refresh/cache can add more rows later; refreshable rows are not authoritative by themselves. Use `runtime` when OpenClaw must load provider runtime to know the list.
## modelIdNormalization reference
+18
View File
@@ -310,6 +310,24 @@ Bundled provider catalogs currently flag these models as `"preferred"`:
Everything else, including all Ollama-served local models, stays unflagged and
keeps normal tool exposure under `"auto"`.
### Models shipped by more than one provider
Several vendors are reachable through more than one provider id: a subscription
endpoint next to an API endpoint, or a gateway that resells another vendor's
model. Because `"auto"` resolves the tier from whichever catalog served the run,
two catalogs describing the same upstream model must not disagree by accident.
Every catalog row for a shared model therefore states its tier explicitly once
any sibling row states one. When the shared model is listed under different ids,
the non-canonical row points at the canonical one with `upstreamModel`, as the
`kimi` catalog does for `moonshot/kimi-k3`. Reseller catalogs such as
`github-copilot`, `opencode`, `opencode-go`, `ollama-cloud`, `qianfan`,
`venice`, and `volcengine-plan` currently declare `"capable"` for the models
first-party catalogs flag `"preferred"`: the preferred tier came from
evaluations on the first-party endpoints, and those runs have not been repeated
per reseller. Promoting one of those rows is a deliberate, evidence-backed
change rather than an oversight.
For OpenAI models, the flag matters only when the run resolves to the OpenClaw
embedded agent runtime. Default OpenAI routing uses the Codex-style harness
surface, where OpenClaw code mode does not apply; the catalog flag never
+14 -7
View File
@@ -32,7 +32,8 @@
"input": ["text", "image"],
"contextWindow": 1000000,
"maxTokens": 128000,
"cost": { "input": 10, "output": 50, "cacheRead": 1, "cacheWrite": 12.5 }
"cost": { "input": 10, "output": 50, "cacheRead": 1, "cacheWrite": 12.5 },
"compat": { "codeMode": "capable" }
},
{
"id": "claude-opus-5",
@@ -42,7 +43,8 @@
"input": ["text", "image"],
"contextWindow": 1000000,
"maxTokens": 128000,
"cost": { "input": 5, "output": 25, "cacheRead": 0.5, "cacheWrite": 6.25 }
"cost": { "input": 5, "output": 25, "cacheRead": 0.5, "cacheWrite": 6.25 },
"compat": { "codeMode": "capable" }
},
{
"id": "claude-sonnet-5",
@@ -52,7 +54,8 @@
"input": ["text", "image"],
"contextWindow": 1000000,
"maxTokens": 128000,
"cost": { "input": 2, "output": 10, "cacheRead": 0.2, "cacheWrite": 2.5 }
"cost": { "input": 2, "output": 10, "cacheRead": 0.2, "cacheWrite": 2.5 },
"compat": { "codeMode": "capable" }
},
{
"id": "claude-haiku-4.5",
@@ -131,7 +134,8 @@
"contextWindow": 1050000,
"contextTokens": 922000,
"maxTokens": 128000,
"cost": { "input": 5, "output": 30, "cacheRead": 0.5, "cacheWrite": 0 }
"cost": { "input": 5, "output": 30, "cacheRead": 0.5, "cacheWrite": 0 },
"compat": { "codeMode": "capable" }
},
{
"id": "gpt-5.6-terra",
@@ -141,7 +145,8 @@
"contextWindow": 1050000,
"contextTokens": 922000,
"maxTokens": 128000,
"cost": { "input": 2.5, "output": 15, "cacheRead": 0.25, "cacheWrite": 0 }
"cost": { "input": 2.5, "output": 15, "cacheRead": 0.25, "cacheWrite": 0 },
"compat": { "codeMode": "capable" }
},
{
"id": "gpt-5.6-luna",
@@ -151,7 +156,8 @@
"contextWindow": 1050000,
"contextTokens": 922000,
"maxTokens": 128000,
"cost": { "input": 1, "output": 6, "cacheRead": 0.1, "cacheWrite": 0 }
"cost": { "input": 1, "output": 6, "cacheRead": 0.1, "cacheWrite": 0 },
"compat": { "codeMode": "capable" }
},
{
"id": "gpt-5.3-codex",
@@ -184,7 +190,8 @@
"contextWindow": 1050000,
"contextTokens": 272000,
"maxTokens": 128000,
"cost": { "input": 5, "output": 30, "cacheRead": 0.5, "cacheWrite": 0 }
"cost": { "input": 5, "output": 30, "cacheRead": 0.5, "cacheWrite": 0 },
"compat": { "codeMode": "capable" }
},
{
"id": "gpt-5.4-mini",
+13
View File
@@ -10,3 +10,16 @@ openclaw gateway restart
```
See <https://docs.openclaw.ai/providers/moonshot> for setup and configuration.
## Catalog notes
Model rows live in `openclaw.plugin.json` under `modelCatalog.providers.kimi`.
- `k3` serves up to 1M context, tier-gated server-side; `k3-256k` is the cheaper
256K variant of the same weights. Both point at `moonshot/kimi-k3` through
`upstreamModel`, which keeps their `compat` capability tiers aligned with the
`moonshot` catalog for the same model.
- Legacy `k3[1m]` was retired upstream and normalizes to `k3` for shipped
configurations.
- `KIMI_K3_MODEL_IDS` in `provider-policy-api.ts` must cover exactly the catalog
rows that carry a K3 `thinkingLevelMap`; `provider-catalog.test.ts` asserts it.
@@ -87,6 +87,7 @@ describe("Kimi implicit provider (#22409)", () => {
max: "max",
},
input: ["text", "image"],
compat: { codeMode: "preferred" },
cost: { input: 3, output: 15, cacheRead: 0.3, cacheWrite: 0 },
contextWindow: 1048576,
maxTokens: 131072,
@@ -105,6 +106,7 @@ describe("Kimi implicit provider (#22409)", () => {
max: "max",
},
input: ["text", "image"],
compat: { codeMode: "preferred" },
cost: { input: 3, output: 15, cacheRead: 0.3, cacheWrite: 0 },
contextWindow: 262144,
maxTokens: 131072,
+3 -1
View File
@@ -13,7 +13,9 @@ export const KIMI_MODEL_REF = `kimi/${KIMI_CODING_DEFAULT_MODEL_ID}`;
export const KIMI_CODING_MODEL_REF = KIMI_MODEL_REF;
function resolveKimiCodingDefaultModel() {
return buildKimiCodingProvider().models[0];
return buildKimiCodingProvider().models.find(
(model) => model.id === KIMI_CODING_DEFAULT_MODEL_ID,
);
}
const kimiCodingPresetAppliers = createDefaultModelPresetAppliers({
@@ -17,6 +17,105 @@
}
}
},
"modelCatalog": {
"providers": {
"kimi": {
"baseUrl": "https://api.kimi.com/coding/",
"api": "anthropic-messages",
"headers": {
"User-Agent": "claude-code/0.1.0"
},
"defaultModel": "kimi-for-coding",
"models": [
{
"id": "kimi-for-coding",
"name": "Kimi Code",
"reasoning": true,
"input": ["text", "image"],
"contextWindow": 262144,
"maxTokens": 32768,
"cost": {
"input": 0,
"output": 0,
"cacheRead": 0,
"cacheWrite": 0
}
},
{
"id": "kimi-for-coding-highspeed",
"name": "Kimi K2.7 Code HighSpeed",
"reasoning": true,
"input": ["text", "image"],
"contextWindow": 262144,
"maxTokens": 32768,
"cost": {
"input": 0,
"output": 0,
"cacheRead": 0,
"cacheWrite": 0
}
},
{
"id": "k3",
"name": "Kimi K3",
"reasoning": true,
"upstreamModel": "moonshot/kimi-k3",
"thinkingLevelMap": {
"off": null,
"minimal": "low",
"low": "low",
"medium": "high",
"high": "high",
"xhigh": "max",
"max": "max"
},
"input": ["text", "image"],
"contextWindow": 1048576,
"maxTokens": 131072,
"cost": {
"input": 3,
"output": 15,
"cacheRead": 0.3,
"cacheWrite": 0
},
"compat": {
"codeMode": "preferred"
}
},
{
"id": "k3-256k",
"name": "Kimi K3 (256k)",
"reasoning": true,
"upstreamModel": "moonshot/kimi-k3",
"thinkingLevelMap": {
"off": null,
"minimal": "low",
"low": "low",
"medium": "high",
"high": "high",
"xhigh": "max",
"max": "max"
},
"input": ["text", "image"],
"contextWindow": 262144,
"maxTokens": 131072,
"cost": {
"input": 3,
"output": 15,
"cacheRead": 0.3,
"cacheWrite": 0
},
"compat": {
"codeMode": "preferred"
}
}
]
}
},
"discovery": {
"kimi": "static"
}
},
"modelPricing": {
"providers": {
"kimi": {
@@ -1,7 +1,7 @@
// Kimi Coding tests cover provider catalog plugin behavior.
import { describe, expect, it } from "vitest";
import { buildKimiCodingProvider, normalizeKimiCodingModelId } from "./provider-catalog.js";
import { isKimiK3ModelId } from "./provider-policy-api.js";
import { isKimiK3ModelId, KIMI_K3_MODEL_IDS } from "./provider-policy-api.js";
describe("kimi provider catalog", () => {
it("builds the bundled Kimi coding defaults", () => {
@@ -58,12 +58,22 @@ describe("kimi provider catalog", () => {
maxTokens: 32_768,
},
);
// K2.7 stays unflagged, matching the sibling `moonshot` catalog where only K3 is preferred.
// K2.7 stays unflagged here and in the sibling `moonshot` catalog; only K3 is preferred.
for (const id of ["kimi-for-coding", "kimi-for-coding-highspeed"]) {
expect(provider.models.find((model) => model.id === id)?.compat?.codeMode).toBeUndefined();
}
});
it("covers every K3 catalog row with the K3 thinking policy", () => {
// The manifest owns the rows and `provider-policy-api` owns the thinking
// profile, so a new K3 variant must land in both or lose its levels.
const thinkingRows = buildKimiCodingProvider()
.models.filter((model) => model.thinkingLevelMap)
.map((model) => model.id);
expect(thinkingRows).toEqual([...KIMI_K3_MODEL_IDS]);
});
it("normalizes legacy Kimi coding model ids to the stable API model id", () => {
expect(normalizeKimiCodingModelId("kimi-code")).toBe("kimi-for-coding");
expect(normalizeKimiCodingModelId("k2p5")).toBe("kimi-for-coding");
+15 -86
View File
@@ -1,100 +1,29 @@
// Kimi Coding provider module implements model/runtime integration.
import type {
ModelDefinitionConfig,
ModelProviderConfig,
} from "openclaw/plugin-sdk/provider-model-shared";
import { KIMI_K3_MODEL_IDS } from "./provider-policy-api.js";
import { buildManifestModelProviderConfig } from "openclaw/plugin-sdk/provider-catalog-shared";
import type { ModelProviderConfig } from "openclaw/plugin-sdk/provider-model-shared";
import manifest from "./openclaw.plugin.json" with { type: "json" };
const KIMI_BASE_URL = "https://api.kimi.com/coding/";
const KIMI_CODING_USER_AGENT = "claude-code/0.1.0";
const KIMI_DEFAULT_MODEL_ID = "kimi-for-coding";
const KIMI_HIGHSPEED_MODEL_ID = "kimi-for-coding-highspeed";
const KIMI_PROVIDER_ID = "kimi";
const KIMI_CODING_CATALOG = manifest.modelCatalog.providers.kimi;
const KIMI_LEGACY_MODEL_IDS = ["kimi-code", "k2p5"] as const;
const KIMI_CODING_DEFAULT_CONTEXT_WINDOW = 262144;
const KIMI_CODING_DEFAULT_MAX_TOKENS = 32768;
const KIMI_CODING_DEFAULT_COST = {
input: 0,
output: 0,
cacheRead: 0,
cacheWrite: 0,
};
const KIMI_K3_COST = {
input: 3,
output: 15,
cacheRead: 0.3,
cacheWrite: 0,
};
// k3 serves up to 1M context, tier-gated server-side; k3-256k is the cheaper 256K variant.
// Legacy k3[1m] was retired upstream and normalizes to k3 for shipped configurations.
const KIMI_K3_CONTEXT_WINDOW = 1_048_576;
const KIMI_K3_MAX_TOKENS = 131_072;
const KIMI_K3_THINKING_LEVEL_MAP = {
off: null,
minimal: "low",
low: "low",
medium: "high",
high: "high",
xhigh: "max",
max: "max",
} satisfies NonNullable<ModelDefinitionConfig["thinkingLevelMap"]>;
const KIMI_CODING_INPUT = ["text", "image"] satisfies NonNullable<ModelDefinitionConfig["input"]>;
// K3 is the same model the `moonshot` provider ships as `kimi-k3`, which declares
// `codeMode: "preferred"`. Without this the subscription surface silently drops out of
// `tools.codeMode: "auto"` while the API surface engages it.
const KIMI_K3_COMPAT = {
codeMode: "preferred",
} satisfies NonNullable<ModelDefinitionConfig["compat"]>;
export const KIMI_CODING_BASE_URL = KIMI_CODING_CATALOG.baseUrl;
export const KIMI_CODING_DEFAULT_MODEL_ID = KIMI_CODING_CATALOG.defaultModel;
export const KIMI_CODING_LEGACY_MODEL_IDS = KIMI_LEGACY_MODEL_IDS;
export function buildKimiCodingProvider(): ModelProviderConfig {
return {
baseUrl: KIMI_BASE_URL,
api: "anthropic-messages",
headers: {
"User-Agent": KIMI_CODING_USER_AGENT,
},
models: [
{
id: KIMI_DEFAULT_MODEL_ID,
name: "Kimi Code",
reasoning: true,
input: [...KIMI_CODING_INPUT],
cost: KIMI_CODING_DEFAULT_COST,
contextWindow: KIMI_CODING_DEFAULT_CONTEXT_WINDOW,
maxTokens: KIMI_CODING_DEFAULT_MAX_TOKENS,
},
{
id: KIMI_HIGHSPEED_MODEL_ID,
name: "Kimi K2.7 Code HighSpeed",
reasoning: true,
input: [...KIMI_CODING_INPUT],
cost: KIMI_CODING_DEFAULT_COST,
contextWindow: KIMI_CODING_DEFAULT_CONTEXT_WINDOW,
maxTokens: KIMI_CODING_DEFAULT_MAX_TOKENS,
},
...KIMI_K3_MODEL_IDS.map((id) => ({
id,
name: id === "k3" ? "Kimi K3" : "Kimi K3 (256k)",
reasoning: true,
thinkingLevelMap: { ...KIMI_K3_THINKING_LEVEL_MAP },
compat: { ...KIMI_K3_COMPAT },
input: [...KIMI_CODING_INPUT],
cost: KIMI_K3_COST,
contextWindow: id === "k3" ? KIMI_K3_CONTEXT_WINDOW : KIMI_CODING_DEFAULT_CONTEXT_WINDOW,
maxTokens: KIMI_K3_MAX_TOKENS,
})),
],
};
return buildManifestModelProviderConfig({
providerId: KIMI_PROVIDER_ID,
catalog: KIMI_CODING_CATALOG,
});
}
export function normalizeKimiCodingModelId(modelId: string): string {
// Legacy k3[1m] was retired upstream and normalizes to k3 for shipped configurations.
if (modelId === "k3[1m]") {
return "k3";
}
return KIMI_LEGACY_MODEL_IDS.includes(modelId as (typeof KIMI_LEGACY_MODEL_IDS)[number])
? KIMI_DEFAULT_MODEL_ID
? KIMI_CODING_DEFAULT_MODEL_ID
: modelId;
}
export const KIMI_CODING_BASE_URL = KIMI_BASE_URL;
export const KIMI_CODING_DEFAULT_MODEL_ID = KIMI_DEFAULT_MODEL_ID;
export const KIMI_CODING_LEGACY_MODEL_IDS = KIMI_LEGACY_MODEL_IDS;
+4 -2
View File
@@ -150,7 +150,8 @@
"maxTokens": 8192,
"compat": {
"supportsTools": true,
"supportsUsageInStreaming": true
"supportsUsageInStreaming": true,
"codeMode": "capable"
}
},
{
@@ -170,7 +171,8 @@
"maxTokens": 8192,
"compat": {
"supportsTools": true,
"supportsUsageInStreaming": true
"supportsUsageInStreaming": true,
"codeMode": "capable"
}
}
]
+4 -2
View File
@@ -50,7 +50,8 @@
"compat": {
"supportsUsageInStreaming": true,
"supportsReasoningEffort": true,
"maxTokensField": "max_tokens"
"maxTokensField": "max_tokens",
"codeMode": "capable"
}
},
{
@@ -77,7 +78,8 @@
"compat": {
"supportsUsageInStreaming": true,
"supportsReasoningEffort": true,
"maxTokensField": "max_tokens"
"maxTokensField": "max_tokens",
"codeMode": "capable"
}
}
]
+8 -4
View File
@@ -52,7 +52,8 @@
"compat": {
"supportsUsageInStreaming": true,
"supportsReasoningEffort": true,
"maxTokensField": "max_tokens"
"maxTokensField": "max_tokens",
"codeMode": "capable"
}
},
{
@@ -77,7 +78,8 @@
"compat": {
"supportsUsageInStreaming": true,
"supportsReasoningEffort": true,
"maxTokensField": "max_tokens"
"maxTokensField": "max_tokens",
"codeMode": "capable"
},
"status": "deprecated",
"replacedBy": "claude-opus-5"
@@ -133,7 +135,8 @@
"xhigh",
"max"
],
"maxTokensField": "max_tokens"
"maxTokensField": "max_tokens",
"codeMode": "capable"
}
},
{
@@ -179,7 +182,8 @@
"compat": {
"supportsUsageInStreaming": true,
"supportsReasoningEffort": true,
"maxTokensField": "max_tokens"
"maxTokensField": "max_tokens",
"codeMode": "capable"
},
"status": "deprecated",
"replacedBy": "gpt-5.6-sol"
+3
View File
@@ -33,6 +33,9 @@
"output": 3.543915,
"cacheRead": 0.147663,
"cacheWrite": 0
},
"compat": {
"codeMode": "capable"
}
},
{
+4 -2
View File
@@ -198,7 +198,8 @@
"maxTokens": 128000,
"cost": { "input": 12, "output": 60, "cacheRead": 1.2, "cacheWrite": 15 },
"compat": {
"supportsUsageInStreaming": false
"supportsUsageInStreaming": false,
"codeMode": "capable"
}
},
{
@@ -210,7 +211,8 @@
"maxTokens": 128000,
"cost": { "input": 6, "output": 30, "cacheRead": 0.6, "cacheWrite": 7.5 },
"compat": {
"supportsUsageInStreaming": false
"supportsUsageInStreaming": false,
"codeMode": "capable"
}
},
{
@@ -278,6 +278,9 @@
"output": 0,
"cacheRead": 0,
"cacheWrite": 0
},
"compat": {
"codeMode": "capable"
}
},
{
@@ -294,6 +297,9 @@
"output": 0,
"cacheRead": 0,
"cacheWrite": 0
},
"compat": {
"codeMode": "capable"
}
},
{
@@ -310,6 +316,9 @@
"output": 0,
"cacheRead": 0,
"cacheWrite": 0
},
"compat": {
"codeMode": "capable"
}
}
]
@@ -234,6 +234,13 @@ export type ModelCatalogModel = {
thinkingLevelMap?: ModelCatalogThinkingLevelMap;
cost?: ModelCatalogCost;
compat?: ModelCatalogCompatConfig;
/**
* Provider/model ref of the same upstream model in another bundled catalog,
* for vendors reachable through several provider ids under different model
* ids. Authoring metadata only: normalization drops it, and the shared-model
* contract test uses it to keep `compat` capability tiers from drifting apart.
*/
upstreamModel?: string;
mediaInput?: ModelCatalogMediaInputConfig;
status?: ModelCatalogStatus;
statusReason?: string;
@@ -0,0 +1,152 @@
// Shared upstream model contract tests keep capability flags aligned across bundled catalogs.
import fs from "node:fs";
import path from "node:path";
import { describe, expect, it } from "vitest";
import { listGitTrackedFiles } from "../../test-utils/repo-files.js";
const repoRoot = path.resolve(import.meta.dirname, "../../..");
const MANIFEST_BASENAME = "openclaw.plugin.json";
const CODE_MODE_TIER_LITERAL = /codeMode:\s*"(?:preferred|capable)"/;
// Catalogs still built in plugin source instead of `modelCatalog` manifest rows.
// The shared-model checks below read manifests, so these two stay invisible to
// them; moving their rows into the manifest closes the gap.
const UNCONVERTED_SOURCE_CATALOG_PLUGINS = ["google", "minimax"];
type CatalogEntry = {
/** `provider/model` ref used in failure output. */
ref: string;
/** Shared-model group: the declared upstream model id, else the row's own id. */
groupKey: string;
/** Declared upstream ref, present only on non-canonical rows. */
upstreamModel?: string;
/** Declared code-mode tier; absent rows fall back to the implicit `capable` tier. */
codeMode?: string;
};
function listBundledPluginFiles(): string[] {
const files = listGitTrackedFiles({ repoRoot, pathspecs: "extensions" });
if (!files) {
throw new Error("unable to list bundled plugin files from git");
}
return files;
}
function readBundledManifests(): Array<Record<string, unknown>> {
return listBundledPluginFiles()
.filter((file) => {
const segments = file.split("/");
return segments.length === 3 && segments[2] === MANIFEST_BASENAME;
})
.map(
(file) =>
JSON.parse(fs.readFileSync(path.join(repoRoot, file), "utf8")) as Record<string, unknown>,
);
}
function readRecord(value: unknown): Record<string, unknown> | undefined {
return value && typeof value === "object" && !Array.isArray(value)
? (value as Record<string, unknown>)
: undefined;
}
function collectCatalogEntries(): CatalogEntry[] {
const entries: CatalogEntry[] = [];
for (const manifest of readBundledManifests()) {
const providers = readRecord(readRecord(manifest.modelCatalog)?.providers) ?? {};
// CLI backends run their own harness, so catalog code-mode tiers never reach them.
const cliBackends = new Set(
Array.isArray(manifest.cliBackends) ? (manifest.cliBackends as string[]) : [],
);
for (const [providerId, provider] of Object.entries(providers)) {
if (cliBackends.has(providerId)) {
continue;
}
const models = readRecord(provider)?.models;
if (!Array.isArray(models)) {
continue;
}
for (const rawModel of models) {
const model = readRecord(rawModel);
const id = typeof model?.id === "string" ? model.id : undefined;
if (!id) {
continue;
}
const upstreamModel =
typeof model?.upstreamModel === "string" ? model.upstreamModel : undefined;
const codeMode = readRecord(model?.compat)?.codeMode;
entries.push({
ref: `${providerId}/${id}`,
groupKey: upstreamModel ? upstreamModel.slice(upstreamModel.indexOf("/") + 1) : id,
...(upstreamModel ? { upstreamModel } : {}),
...(typeof codeMode === "string" ? { codeMode } : {}),
});
}
}
}
return entries;
}
function groupBySharedModel(entries: CatalogEntry[]): Map<string, CatalogEntry[]> {
const groups = new Map<string, CatalogEntry[]>();
for (const entry of entries) {
const group = groups.get(entry.groupKey);
if (group) {
group.push(entry);
continue;
}
groups.set(entry.groupKey, [entry]);
}
return groups;
}
describe("bundled shared upstream model catalogs", () => {
// Read lazily: the non-isolated contract runner shares one process, so manifest
// IO belongs inside the tests rather than in collection.
let cached: CatalogEntry[] | undefined;
const readEntries = (): CatalogEntry[] => (cached ??= collectCatalogEntries());
it("resolves every declared upstream model ref to another bundled catalog row", () => {
const entries = readEntries();
const refs = new Set(entries.map((entry) => entry.ref));
const unresolved = entries
.filter((entry) => entry.upstreamModel)
.filter((entry) => entry.upstreamModel === entry.ref || !refs.has(entry.upstreamModel ?? ""))
.map((entry) => `${entry.ref} -> ${entry.upstreamModel}`)
.toSorted();
expect(unresolved).toEqual([]);
});
it("declares the code-mode tier on every catalog row that ships a shared model", () => {
// `compat.codeMode` gates `tools.codeMode: "auto"` per model, so one catalog
// flagging a model `preferred` while a sibling catalog for the same upstream
// model stays silent makes the identical model behave differently by provider.
const silent: string[] = [];
for (const [groupKey, group] of groupBySharedModel(readEntries())) {
if (group.length < 2 || !group.some((entry) => entry.codeMode)) {
continue;
}
const declared = group
.filter((entry) => entry.codeMode)
.map((entry) => `${entry.ref}=${entry.codeMode}`)
.toSorted()
.join(", ");
for (const entry of group.filter((candidate) => !candidate.codeMode)) {
silent.push(`${entry.ref} (shared model ${groupKey}; siblings declare ${declared})`);
}
}
expect(silent.toSorted()).toEqual([]);
});
it("keeps code-mode tiers out of plugin source so shared-model checks can see them", () => {
const plugins = listBundledPluginFiles()
.filter((file) => file.endsWith(".ts") && !file.includes(".test."))
.filter((file) =>
CODE_MODE_TIER_LITERAL.test(fs.readFileSync(path.join(repoRoot, file), "utf8")),
)
.map((file) => file.split("/")[1] as string);
expect([...new Set(plugins)].toSorted()).toEqual(UNCONVERTED_SOURCE_CATALOG_PLUGINS);
});
});
+1 -1
View File
@@ -123,7 +123,7 @@ describe("publish model catalog", () => {
{ cwd: root, encoding: "utf8" },
);
expect(result.status, result.stderr).toBe(0);
const stats = /dry-run schemaVersion=1 providers=39 models=(\d+)/u.exec(result.stdout);
const stats = /dry-run schemaVersion=1 providers=40 models=(\d+)/u.exec(result.stdout);
expect(stats).not.toBeNull();
expect(Number(stats?.[1])).toBeGreaterThanOrEqual(MODEL_CATALOG_MIN_MODELS);
expect(fs.existsSync(out)).toBe(false);