Files
openclaw/src/agents/model-auth-runtime.ts
Mert Başar 196d81195b fix(auth): cooldown inline api key billing failures (#88709)
* fix(auth): cooldown inline api key billing failures

* fix: mark inline api key prompt failures

* fix(auth): propagate inline api key cooldown to provider visibility

* fix(types): fix test type errors in agents and ui

* fix(auth): correct null check for cooldown gate and fix test case

* fix(ui): remove unrelated test drift and fix expired toast expectation

* fix(auth): preserve inline cooldown usageStats in warm worker snapshots

* test(auth): update warm snapshot expectation with usageStats

* test(auth): format inline cooldown worker proof

* fix(auth): narrow inline cooldown classifier

* fix(auth): mark managed file/exec SecretRef inline keys on auth/billing failure

The inline-api-key cooldown marker only fired for literal, env-marker, and
env SecretRef sources, while the resolution gate already covered managed
file/exec SecretRef provider keys via coerceSecretRef. As a result a
file/exec SecretRef models.providers.<id>.apiKey could never record a 402
billing cooldown, so the gate that would honor it never triggered.

Align the runner failure-marker trigger with the gate predicate
(isConfigBackedInlineProviderApiKey) so managed non-env SecretRef inline keys
are marked on auth/billing failure. Adds focused regression coverage.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* refactor(auth): drop now-unused isInlineProviderApiKeyAuth export

The runner failure-marker now gates on isConfigBackedInlineProviderApiKey, so
isInlineProviderApiKeyAuth has no remaining callers. Removing the dead export
also trims one public callable export back under the plugin-SDK surface budget.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(auth): enforce inline cooldown for managed file/exec SecretRef keys

The inline-api-key cooldown was recorded for managed (file/exec) SecretRef
provider keys but never enforced: these keys resolve through the synthetic
runtime path (resolveSyntheticLocalProviderAuth) and the explicit api-key
override path, neither of which consulted the inline-key cooldown gate. So an
exhausted file/exec SecretRef provider kept resolving and reporting available
during an active inline-api-key:<provider> cooldown.

Gate both resolution exits (and the provider-availability check) on
isConfigBackedInlineProviderApiKey + assertInlineProviderApiKeyUsable, matching
the literal/env paths. Local no-auth markers are not config-backed inline keys,
so they stay untouched. Adds regression coverage for both resolution paths.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(auth): gate runtime SecretRef cooldown checks

* fix(auth): honor inline key cooldowns in model-list and tool auth checks

Direct hasRuntimeAvailableProviderAuth callers omitted the auth store, so
inline provider keys in billing cooldown were still advertised as available
in model browsing (models.list) and model-backed tool auth checks. Pass the
scoped auth store at each call site and cover the cooled-key paths with
regression tests, including profile fallback and cooldown expiry.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(auth): resolve merge CI failures (format + browse cooldown mock independence)

- failover-retry-controller.ts: apply oxfmt formatting missed after the port.
- model-auth-availability.ts: the browse-side inline cooldown check imported
  resolveInlineProviderApiKeyUnusableUntil from the auth-profiles usage module,
  which several command/directive tests partially mock — the new export was
  absent from those mocks, so every mocking test threw at the browse path.
  Read the inline cooldown via usage-state primitives (resolveProfileUnusableUntil
  + isAuthCooldownBypassedForProvider) and build the usage id with the same
  normalizeProviderId the write side uses, so the hot browse path no longer
  depends on the mocked usage module while keeping key resolution identical.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(auth): make model-auth inline cooldown reads mock-independent

resolveApiKeyForProvider/assertInlineProviderApiKeyUsable read the inline
provider API-key cooldown, which this PR routes through the auth-profiles
usage module. Many existing suites partially mock that barrel (e.g.
run.overflow-compaction.test.ts) without the new export, so those tests
threw once merged code reached the inline path.

Read the cooldown via usage-state primitives (resolveProfileUnusableUntil +
isAuthCooldownBypassedForProvider) and build the `inline-api-key:<provider>`
usage id with the same normalizeProviderId the write side uses, so model-auth
no longer depends on the mocked usage module while key resolution stays
identical. Verified across all 30 suites that mock the auth-profiles module.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(test): use main's terminal failure shape for the inline-key billing case

Main reworked EmbeddedRunAttemptResult so attempt failures are reported via
`terminal: { kind: "failed", source, error }` instead of the old
`promptError`/`promptErrorSource` fields. The inline-key billing regression
test still used the removed fields, which merged cleanly but broke
check-test-types.

Switch it to the same shape the sibling helper in this file already uses.
Verified with `pnpm check:test-types` (exit 0).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Mert Basar <MertBasar0@users.noreply.github.com>
Co-authored-by: Ceviz Agent <ceviz@openclaw.local>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-08-03 18:19:06 -07:00

336 lines
11 KiB
TypeScript

/**
* Snapshot-aware and synthetic provider-auth availability.
*/
import { normalizeProviderId } from "@openclaw/model-catalog-core/provider-id";
import { normalizeUniqueStringEntries } from "@openclaw/normalization-core/string-normalization";
import { getRuntimeConfigSnapshot } from "../config/config.js";
import type { OpenClawConfig } from "../config/types.openclaw.js";
import { resolveProviderSyntheticAuthWithPlugin } from "../plugins/provider-runtime.js";
import { resolveRuntimeSyntheticAuthProviderRefState } from "../plugins/synthetic-auth.runtime.js";
import { mintSecretSentinel } from "../secrets/sentinel.js";
import type { AuthProfileStore } from "./auth-profiles.js";
import { resolveProviderEnvAuthLookupMaps } from "./model-auth-env-vars.js";
import { resolveEnvApiKey, type EnvApiKeyLookupOptions } from "./model-auth-env.js";
import { CUSTOM_LOCAL_AUTH_MARKER, isNonSecretApiKeyMarker } from "./model-auth-markers.js";
import { isAuthModeAllowedForModel } from "./model-auth-openai.js";
import * as authConfig from "./model-auth-provider-config.js";
import { resolveManagedSecretRefRuntimeProviderAuth } from "./model-auth-runtime-config.js";
import type { ResolvedProviderAuth } from "./model-auth-runtime-shared.js";
/** Precomputed provider-auth lookup tables reused during one runtime turn. */
export type RuntimeProviderAuthLookup = {
envApiKey: Pick<
EnvApiKeyLookupOptions,
"aliasMap" | "candidateMap" | "authEvidenceMap" | "skipSetupProviderFallback"
>;
setupProviderFallbackRefs?: readonly string[];
syntheticAuthProviderRefs?: readonly string[];
syntheticAuthProviderRefsComplete?: boolean;
};
/** Builds stable env/synthetic auth lookup data for repeated provider checks. */
export function createRuntimeProviderAuthLookup(params: {
cfg?: OpenClawConfig;
workspaceDir?: string;
env?: NodeJS.ProcessEnv;
includePluginSyntheticAuth?: boolean;
}): RuntimeProviderAuthLookup {
const env = params.env ?? process.env;
const lookupParams = {
config: params.cfg,
workspaceDir: params.workspaceDir,
env,
};
const syntheticAuthProviderRefs =
params.includePluginSyntheticAuth === false
? undefined
: resolveRuntimeSyntheticAuthProviderRefState(lookupParams);
const authLookupMaps = resolveProviderEnvAuthLookupMaps(lookupParams);
return {
envApiKey: {
aliasMap: authLookupMaps.aliasMap,
candidateMap: authLookupMaps.envCandidateMap,
authEvidenceMap: authLookupMaps.authEvidenceMap,
skipSetupProviderFallback: true,
},
setupProviderFallbackRefs: authLookupMaps.setupProviderFallbackRefs,
syntheticAuthProviderRefs: syntheticAuthProviderRefs?.complete
? syntheticAuthProviderRefs.refs
: undefined,
syntheticAuthProviderRefsComplete: syntheticAuthProviderRefs?.complete,
};
}
function runtimeLookupAllowsSetupProviderFallback(params: {
provider: string;
runtimeLookup?: RuntimeProviderAuthLookup;
}): boolean {
const refs = params.runtimeLookup?.setupProviderFallbackRefs;
if (!refs?.length) {
return false;
}
const normalizedProvider = normalizeProviderId(params.provider);
const aliasTarget = params.runtimeLookup?.envApiKey.aliasMap?.[normalizedProvider];
return refs.includes(normalizedProvider) || (aliasTarget ? refs.includes(aliasTarget) : false);
}
function resolveRuntimeEnvApiKeyLookupOptions(params: {
provider: string;
runtimeLookup?: RuntimeProviderAuthLookup;
}):
| Pick<
EnvApiKeyLookupOptions,
"aliasMap" | "candidateMap" | "authEvidenceMap" | "skipSetupProviderFallback"
>
| undefined {
const envApiKey = params.runtimeLookup?.envApiKey;
if (!envApiKey) {
return undefined;
}
const skipSetupProviderFallback =
envApiKey.skipSetupProviderFallback === true
? !runtimeLookupAllowsSetupProviderFallback(params)
: envApiKey.skipSetupProviderFallback;
return {
...envApiKey,
...(skipSetupProviderFallback !== undefined ? { skipSetupProviderFallback } : {}),
};
}
function listProviderSyntheticAuthRefs(params: {
cfg: OpenClawConfig | undefined;
provider: string;
modelApi?: string;
}): string[] {
const refs = [params.provider];
const providerConfig = authConfig.resolveProviderConfig(params.cfg, params.provider);
if (params.modelApi) {
refs.push(params.modelApi);
}
if (providerConfig?.api) {
refs.push(providerConfig.api);
}
return normalizeUniqueStringEntries(refs.map((ref) => normalizeProviderId(ref)));
}
function shouldResolvePluginSyntheticAuth(params: {
cfg: OpenClawConfig | undefined;
provider: string;
modelApi?: string;
runtimeLookup?: RuntimeProviderAuthLookup;
}): boolean {
const syntheticAuthProviderRefs = params.runtimeLookup?.syntheticAuthProviderRefs;
if (!syntheticAuthProviderRefs) {
return true;
}
const eligibleRefs = new Set(
normalizeUniqueStringEntries(syntheticAuthProviderRefs.map((ref) => normalizeProviderId(ref))),
);
if (eligibleRefs.size === 0) {
return false;
}
return listProviderSyntheticAuthRefs(params).some((ref) => eligibleRefs.has(ref));
}
/** Fast auth-availability check for runtime provider/model selection. */
export function hasRuntimeAvailableProviderAuth(params: {
provider: string;
cfg?: OpenClawConfig;
workspaceDir?: string;
env?: NodeJS.ProcessEnv;
allowPluginSyntheticAuth?: boolean;
runtimeLookup?: RuntimeProviderAuthLookup;
modelApi?: string;
store?: AuthProfileStore;
}): boolean {
const provider = normalizeProviderId(params.provider);
const authOverride = authConfig.resolveProviderAuthOverride(params.cfg, provider);
if (authOverride === "aws-sdk") {
return true;
}
// Callers that supply the auth store get inline provider keys hidden while
// their billing/auth cooldown is active, so browse and tool selection stop
// advertising a credential the resolver would refuse to hand back.
const inlineProviderApiKeyUsable = params.store
? (() => {
const unusableUntil = authConfig.resolveInlineProviderApiKeyCooldownUntil(
params.store,
provider,
);
return unusableUntil === null || unusableUntil <= Date.now();
})()
: true;
const envAuth = resolveEnvApiKey(provider, params.env, {
config: params.cfg,
workspaceDir: params.workspaceDir,
...resolveRuntimeEnvApiKeyLookupOptions({
provider,
runtimeLookup: params.runtimeLookup,
}),
});
if (
envAuth &&
isAuthModeAllowedForModel({
provider,
modelApi: params.modelApi,
mode: envAuth.source.includes("OAUTH_TOKEN") ? "oauth" : "api-key",
}) &&
(!authConfig.isConfigBackedInlineProviderApiKey({
cfg: params.cfg,
provider,
source: envAuth.source,
store: params.store,
}) ||
inlineProviderApiKeyUsable)
) {
return true;
}
if (
authConfig.resolveUsableCustomProviderApiKey({
cfg: params.cfg,
provider,
env: params.env,
}) &&
inlineProviderApiKeyUsable
) {
return true;
}
const managedRuntimeAuth = resolveManagedSecretRefRuntimeProviderAuth({
cfg: params.cfg,
provider,
});
if (
managedRuntimeAuth &&
(!authConfig.isConfigBackedInlineProviderApiKey({
cfg: params.cfg,
provider,
source: managedRuntimeAuth.source,
store: params.store,
}) ||
inlineProviderApiKeyUsable)
) {
return true;
}
if (authConfig.hasSyntheticLocalProviderAuthConfig({ cfg: params.cfg, provider })) {
return true;
}
if (
params.allowPluginSyntheticAuth !== false &&
shouldResolvePluginSyntheticAuth({
cfg: params.cfg,
provider,
runtimeLookup: params.runtimeLookup,
}) &&
resolveSyntheticLocalProviderAuth({ cfg: params.cfg, provider })
) {
return true;
}
return false;
}
type SyntheticProviderAuthResolution = {
auth?: ResolvedProviderAuth;
blockedOnManagedSecretRef?: boolean;
};
function resolveProviderSyntheticRuntimeAuth(params: {
cfg: OpenClawConfig | undefined;
provider: string;
modelApi?: string;
secretSentinels?: boolean;
}): SyntheticProviderAuthResolution {
const runtimeAuth = resolveManagedSecretRefRuntimeProviderAuth(params);
if (runtimeAuth) {
return { auth: runtimeAuth };
}
if (authConfig.hasSecretRefProviderApiKey(params.cfg, params.provider)) {
return { blockedOnManagedSecretRef: true };
}
const resolveFromConfig = (
config: OpenClawConfig | undefined,
): ResolvedProviderAuth | undefined => {
const providerConfig = authConfig.resolveProviderConfig(config, params.provider);
return (
resolveProviderSyntheticAuthWithPlugin({
provider: params.provider,
config,
context: {
config,
provider: params.provider,
providerConfig,
},
modelApi: params.modelApi,
}) ?? undefined
);
};
const directAuth = resolveFromConfig(params.cfg);
if (!directAuth) {
return {};
}
if (!authConfig.isManagedSecretRefApiKeyMarker(directAuth.apiKey)) {
return { auth: directAuth };
}
const runtimeConfig = getRuntimeConfigSnapshot();
if (!runtimeConfig || runtimeConfig === params.cfg) {
return { blockedOnManagedSecretRef: true };
}
const runtimePluginAuth = resolveFromConfig(runtimeConfig);
const runtimeApiKey = runtimePluginAuth?.apiKey;
if (!runtimePluginAuth || !runtimeApiKey || isNonSecretApiKeyMarker(runtimeApiKey)) {
return { blockedOnManagedSecretRef: true };
}
return {
auth: {
...runtimePluginAuth,
apiKey: params.secretSentinels
? mintSecretSentinel(runtimeApiKey, {
label: `model-auth:${params.provider}`,
})
: runtimeApiKey,
},
};
}
export function resolveSyntheticLocalProviderAuth(params: {
cfg: OpenClawConfig | undefined;
provider: string;
modelApi?: string;
secretSentinels?: boolean;
allowPluginSyntheticAuth?: boolean;
}): ResolvedProviderAuth | null {
// Prepared direct attempts may use local no-auth config, but must not widen
// back into an unprepared plugin-owned credential source.
const syntheticProviderAuth =
params.allowPluginSyntheticAuth === false ? {} : resolveProviderSyntheticRuntimeAuth(params);
if (syntheticProviderAuth.auth) {
return syntheticProviderAuth.auth;
}
if (syntheticProviderAuth.blockedOnManagedSecretRef) {
return null;
}
const providerConfig = authConfig.resolveProviderConfig(params.cfg, params.provider);
if (!providerConfig) {
return null;
}
// Custom providers pointing at a local server (e.g. llama.cpp, vLLM, LocalAI)
// typically don't require auth. Synthesize a local key so the auth resolver
// doesn't reject them when the user left the API key blank during setup.
if (authConfig.hasSyntheticLocalProviderAuthConfig(params)) {
return {
apiKey: CUSTOM_LOCAL_AUTH_MARKER,
source: `models.providers.${params.provider} (synthetic local key)`,
mode: "api-key",
};
}
return null;
}