diff --git a/docs/cli/doctor.md b/docs/cli/doctor.md index 1698d6f3164d..2215cfb67ad4 100644 --- a/docs/cli/doctor.md +++ b/docs/cli/doctor.md @@ -84,6 +84,7 @@ openclaw doctor --lint --json openclaw doctor --lint --all openclaw doctor --lint --allow-exec openclaw doctor --lint --only core/doctor/gateway-config --json +openclaw doctor --lint --only core/doctor/local-audio-acceleration --severity-min info ``` Human output is compact: @@ -125,6 +126,8 @@ Exit codes: `--all` controls which checks are selected before severity filtering. The default lint run excludes checks that are deep, historical, or more likely to surface repairable legacy residue; use `--all` for the complete inventory. `--only ` is the most precise selector and can run any registered check by id. +`core/doctor/local-audio-acceleration` reports the auto-selected local STT command, separate capable/requested/observed backend evidence, and fallback order without loading a speech model. It emits an informational finding, so include `--severity-min info` to display it. + ## Structured health checks Modern doctor checks use a small split contract: diff --git a/docs/nodes/audio.md b/docs/nodes/audio.md index e816a30c51d7..6d1c0ffcf9ab 100644 --- a/docs/nodes/audio.md +++ b/docs/nodes/audio.md @@ -23,11 +23,16 @@ If you have not configured models and `tools.media.audio.enabled` is not `false` 1. **Active reply model**, when its provider supports audio understanding. 2. **Configured provider auth** โ€” any `models.providers.*` entry with auth available for a provider that supports audio transcription. This is checked before local CLIs, so a configured API key always wins over a local binary on `PATH`. Provider priority when multiple are configured: Groq, OpenAI, xAI, Deepgram, Google, SenseAudio, ElevenLabs, Mistral. -3. **Local CLIs** (only if no provider auth resolved), checked in this order: - - `sherpa-onnx-offline` (requires `SHERPA_ONNX_MODEL_DIR` with `tokens.txt`, `encoder.onnx`, `decoder.onnx`, and `joiner.onnx`) - - `whisper-cli` (from `whisper-cpp`; uses `WHISPER_CPP_MODEL` or a bundled tiny model) +3. **Local CLIs** (only if no provider auth resolved). OpenClaw builds an ordered fallback list: + - `whisper-cli`, before CPU defaults only when an earlier model invocation in the current process observed Metal or CUDA + - `sherpa-onnx-offline` on its default CPU provider (requires `SHERPA_ONNX_MODEL_DIR` with `tokens.txt`, `encoder.onnx`, `decoder.onnx`, and `joiner.onnx`) + - `whisper-cli` when Metal/CUDA is only build-capable or the selected backend is otherwise unobserved + - `parakeet-mlx` on Apple Silicon (MLX-capable; device use remains unobserved) - `whisper` (Python CLI; downloads models automatically) +Install/link provenance is capability evidence, not execution evidence. It never moves a candidate ahead of CPU sherpa by itself. OpenClaw does not load a model during setup or status checks just to probe a backend. +Auto-detected whisper.cpp keeps its normal model-run logs enabled so OpenClaw can record the upstream `using โ€ฆ backend` line. Explicit CLI entries keep their configured output flags. + Gemini CLI auto-detect for media understanding was replaced by a sandboxed Antigravity CLI (`agy`) fallback for image/video; audio does not use a CLI fallback beyond the local binaries above. To disable auto-detection, set `tools.media.audio.enabled: false`. To customize, set `tools.media.audio.models`. @@ -36,6 +41,15 @@ To disable auto-detection, set `tools.media.audio.enabled: false`. To customize, Binary detection is best-effort across macOS/Linux/Windows. Make sure the CLI is on `PATH` (`~` is expanded), or set an explicit CLI model with a full command path. +Inspect the local selection without transcribing audio: + +```bash +openclaw capability audio providers +openclaw doctor --lint --only core/doctor/local-audio-acceleration --severity-min info +``` + +The provider inventory reports the local fallback winner separately from global provider selection, plus capable, requested, and observed backend fields. After transcription runs, `/status` reports the requested or observed backend in the media line. Explicit `tools.media.audio.models` CLI entries still bypass auto-selection; use their backend-specific flags such as sherpa `--provider=cuda` or whisper.cpp `--no-gpu`/`--device`. + ## Config examples ### Provider + CLI fallback (OpenAI + Whisper CLI) @@ -161,6 +175,11 @@ Binary detection is best-effort across macOS/Linux/Windows. Make sure the CLI is - `tools.media.audio.echoFormat` customizes the echo text (placeholder: `{transcript}`; default `๐Ÿ“ "{transcript}"`). - CLI stdout is capped at 5MB; keep CLI output concise. - CLI `args` should use `{{MediaPath}}` for the local audio file path. Run `openclaw doctor --fix` to migrate deprecated `{input}` placeholders from older `audio.transcription.command` configs (retired key: `audio.transcription`, replaced by `tools.media.audio.models`). +- `tools.media.concurrency` bounds media tasks; it is not a GPU scheduler. + +### Resident local STT + +Auto-detected local STT remains process-per-request. OpenClaw does not currently manage a resident whisper.cpp server because the standard Homebrew `whisper-cpp` package disables that server, while the upstream example has no configured bounded admission queue. A plugin-owned resident lifecycle needs a maintained packaged worker with health/startup, model residency, bounded queueing, cancellation/timeout, loopback-only no-auth operation, and no cloud fallback before it can be enabled safely. ### Proxy environment support diff --git a/docs/nodes/media-understanding.md b/docs/nodes/media-understanding.md index 4dfcde4c4954..0e46360c5648 100644 --- a/docs/nodes/media-understanding.md +++ b/docs/nodes/media-understanding.md @@ -164,11 +164,15 @@ When `tools.media..enabled` is not `false` and no models are configu Configured `models.providers.*` entries that support audio are tried before local CLIs. Bundled provider priority order (ties break alphabetically by provider id): Groq/OpenAI → xAI → Deepgram → OpenRouter → Google/SenseAudio → Deepinfra/ElevenLabs → Mistral. - First installed local binary, in this order: - - `sherpa-onnx-offline` (requires `SHERPA_ONNX_MODEL_DIR` with `tokens.txt`/`encoder.onnx`/`decoder.onnx`/`joiner.onnx`) - - `whisper-cli` (`whisper-cpp`; uses `WHISPER_CPP_MODEL` or a bundled tiny model) + Ready local binaries become an ordered fallback list: + - `whisper-cli` first only after an earlier model invocation in the current process observed Metal or CUDA + - CPU-default `sherpa-onnx-offline` (requires `SHERPA_ONNX_MODEL_DIR` with `tokens.txt`/`encoder.onnx`/`decoder.onnx`/`joiner.onnx`) + - `whisper-cli` when acceleration is merely build-capable or unobserved + - `parakeet-mlx` on Apple Silicon (MLX-capable, device use unobserved) - `whisper` (Python CLI; defaults to the `turbo` model, downloads automatically) + Backend capability inspection is cached and does not load a model. Build capability, requested backend flags, and backend observed from a real invocation remain separate. Auto-detected whisper.cpp leaves model-run logs enabled so the upstream selected-backend line can be recorded. Explicit CLI entries keep their configured order, backend flags, and output flags. + Configured `models.providers.*` entries that support the capability are tried before the bundled fallback order. Image-only config providers with an image-capable model auto-register for media understanding even when they are not a bundled vendor plugin. @@ -421,7 +425,13 @@ When `mode: "all"`, outputs are labeled `[Image 1/2]`, `[Audio 2/2]`, etc. When media understanding runs, `/status` includes a per-capability summary line: ``` -๐Ÿ“Ž Media: image ok (openai/gpt-5.5) ยท audio skipped (maxBytes) +๐Ÿ“Ž Media: image ok (openai/gpt-5.5) ยท audio ok (whisper-cli observed=metal) +``` + +For preflight inventory, run `openclaw capability audio providers`. Local rows show the local fallback winner separately from global provider selection, readiness, and separate capable/requested/observed backend fields. The same local selection is available as an informational doctor finding: + +```bash +openclaw doctor --lint --only core/doctor/local-audio-acceleration --severity-min info ``` ## Notes diff --git a/src/auto-reply/status.test.ts b/src/auto-reply/status.test.ts index a03bfb40c205..83f9cd9ac82b 100644 --- a/src/auto-reply/status.test.ts +++ b/src/auto-reply/status.test.ts @@ -1529,6 +1529,37 @@ describe("buildStatusMessage", () => { expect(normalized).toContain("Media: image ok (openai/gpt-5.4) ยท audio skipped (maxBytes)"); }); + it("distinguishes observed local STT backends from requested backends", () => { + const text = buildStatusMessage({ + agent: { model: "anthropic/claude-opus-4-6" }, + sessionEntry: { sessionId: "media-local-stt", updatedAt: 0 }, + sessionKey: "agent:main:main", + queue: { mode: "none" }, + mediaDecisions: [ + { + capability: "audio", + outcome: "success", + attachments: [ + { + attachmentIndex: 0, + attempts: [], + chosen: { + type: "cli", + provider: "whisper-cli", + model: "whisper-cli", + requestedBackend: "device:0", + observedBackend: "metal", + outcome: "success", + }, + }, + ], + }, + ], + }); + + expect(normalizeTestText(text)).toContain("Media: audio ok (whisper-cli observed=metal)"); + }); + it("includes failed media understanding decisions with the surfaced reason", () => { const text = buildStatusMessage({ agent: { model: "anthropic/claude-opus-4-6" }, diff --git a/src/cli/capability-cli.test.ts b/src/cli/capability-cli.test.ts index de929e0e0fdf..3256b15fa3fe 100644 --- a/src/cli/capability-cli.test.ts +++ b/src/cli/capability-cli.test.ts @@ -4,6 +4,7 @@ import os from "node:os"; import path from "node:path"; import { Command } from "commander"; import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; +import type { LocalAudioSelection } from "../media-understanding/local-audio.js"; import { runRegisteredCli } from "../test-utils/command-runner.js"; import { CAPABILITY_METADATA, registerCapabilityCli } from "./capability-cli.js"; @@ -147,6 +148,10 @@ const mocks = vi.hoisted(() => ({ ]), listEmbeddingProviders: vi.fn(() => []), buildMediaUnderstandingRegistry: vi.fn(() => new Map()), + inspectLocalAudioSelection: vi.fn<() => Promise>(async () => ({ + candidates: [], + entries: [], + })), convertHeicToJpeg: vi.fn(async () => Buffer.from("jpeg-normalized")), isWebSearchProviderConfigured: vi.fn(() => false), isWebFetchProviderConfigured: vi.fn(() => false), @@ -314,6 +319,10 @@ vi.mock("../media-understanding/provider-registry.js", () => ({ mocks.buildMediaUnderstandingRegistry as typeof import("../media-understanding/provider-registry.js").buildMediaUnderstandingRegistry, })); +vi.mock("../media-understanding/local-audio.js", () => ({ + inspectLocalAudioSelection: mocks.inspectLocalAudioSelection, +})); + vi.mock("../media/media-services.js", async (importOriginal) => { const actual = await importOriginal(); return { @@ -522,6 +531,7 @@ describe("capability cli", () => { mocks.resolveExplicitTtsOverrides.mockClear(); mocks.getProviderEnvVars.mockClear(); mocks.buildMediaUnderstandingRegistry.mockReset().mockReturnValue(new Map()); + mocks.inspectLocalAudioSelection.mockReset().mockResolvedValue({ candidates: [], entries: [] }); mocks.convertHeicToJpeg.mockClear(); mocks.createEmbeddingProvider.mockClear(); mocks.listMemoryEmbeddingProviders @@ -3351,6 +3361,70 @@ describe("capability cli", () => { ]); }); + it("distinguishes the local STT fallback winner from global provider selection", async () => { + vi.stubEnv("DEEPGRAM_API_KEY", "deepgram-test-key"); + mocks.buildMediaUnderstandingRegistry.mockReturnValueOnce( + new Map([ + [ + "deepgram", + { + id: "deepgram", + capabilities: ["audio"], + defaultModels: { audio: "nova-3" }, + }, + ], + ]), + ); + const candidate = { + id: "whisper-cli" as const, + command: "whisper-cli", + resolvedCommand: "/opt/homebrew/bin/whisper-cli", + available: true, + ready: true, + capableBackend: "metal" as const, + evidence: "Apple Silicon Homebrew whisper-cpp runtime with Metal support", + selected: true, + entry: { + type: "cli" as const, + command: "whisper-cli", + args: ["{{MediaPath}}"], + }, + }; + mocks.inspectLocalAudioSelection.mockResolvedValueOnce({ + candidates: [candidate], + entries: [candidate.entry], + selected: candidate, + }); + + await runRegisteredCli({ + register: registerCapabilityCli as (program: Command) => void, + argv: ["capability", "audio", "providers", "--json"], + }); + + expect(firstJsonOutput()).toEqual([ + { + available: true, + configured: true, + selected: false, + id: "deepgram", + capabilities: ["audio"], + defaultModels: { audio: "nova-3" }, + }, + { + available: true, + configured: true, + selected: false, + localFallbackSelected: true, + id: "local/whisper-cli", + transport: "local-cli", + command: "whisper-cli", + capableBackend: "metal", + observedBackend: "unknown", + evidence: "Apple Silicon Homebrew whisper-cpp runtime with Metal support", + }, + ]); + }); + it("resolves plugin web search SecretRefs before running infer web search", async () => { const unresolvedConfig = { tools: { web: { search: { provider: "tavily", enabled: true } } }, diff --git a/src/cli/capability-cli.ts b/src/cli/capability-cli.ts index 5f129cfaa28b..cd561d06efad 100644 --- a/src/cli/capability-cli.ts +++ b/src/cli/capability-cli.ts @@ -60,6 +60,7 @@ import { parseStrictFiniteNumber, parseStrictPositiveInteger, } from "../infra/parse-finite-number.js"; +import { inspectLocalAudioSelection } from "../media-understanding/local-audio.js"; import { buildMediaUnderstandingRegistry } from "../media-understanding/provider-registry.js"; import type { RunMediaUnderstandingFileResult } from "../media-understanding/runtime-types.js"; import { @@ -2529,7 +2530,7 @@ export function registerCapabilityCli(program: Command) { .action(async (opts) => { await runCommandWithRuntime(defaultRuntime, async () => { const cfg = getRuntimeConfig(); - const providers = [...buildMediaUnderstandingRegistry(undefined, cfg).values()] + const remoteProviders = [...buildMediaUnderstandingRegistry(undefined, cfg).values()] .filter((provider) => provider.capabilities?.includes("audio")) .map((provider) => ({ available: true, @@ -2546,6 +2547,28 @@ export function registerCapabilityCli(program: Command) { capabilities: provider.capabilities, defaultModels: provider.defaultModels, })); + const localSelection = await inspectLocalAudioSelection(); + const localProviders = localSelection.candidates + .filter((candidate) => candidate.available) + .map((candidate) => + Object.assign( + { + available: candidate.available, + configured: candidate.ready, + selected: false, + localFallbackSelected: candidate.selected, + id: `local/${candidate.id}`, + transport: "local-cli", + command: candidate.command, + observedBackend: candidate.observedBackend ?? "unknown", + evidence: candidate.evidence, + }, + candidate.capableBackend ? { capableBackend: candidate.capableBackend } : {}, + candidate.requestedBackend ? { requestedBackend: candidate.requestedBackend } : {}, + candidate.reason ? { reason: candidate.reason } : {}, + ), + ); + const providers = [...remoteProviders, ...localProviders]; emitJsonOrText(defaultRuntime, Boolean(opts.json), providers, providerSummaryText); }); }); diff --git a/src/flows/doctor-core-checks.runtime.ts b/src/flows/doctor-core-checks.runtime.ts index 8511dc564c94..5e8d47d959c3 100644 --- a/src/flows/doctor-core-checks.runtime.ts +++ b/src/flows/doctor-core-checks.runtime.ts @@ -43,6 +43,10 @@ import { import { resolveGatewayService, readGatewayServiceState } from "../daemon/service.js"; import { buildGatewayProbeConnectionDetails } from "../gateway/call.js"; import { formatErrorMessage } from "../infra/errors.js"; +import { + formatLocalAudioSelection, + inspectLocalAudioSelection, +} from "../media-understanding/local-audio.js"; import type { ProviderRuntimeModel } from "../plugins/provider-runtime-model.types.js"; import { getPluginToolMeta, setPluginToolMeta } from "../plugins/tools.js"; import type { ProviderCatalogOrder, ProviderPlugin } from "../plugins/types.js"; @@ -68,6 +72,38 @@ export function detectUnavailableSkills(cfg: OpenClawConfig): SkillStatusEntry[] return collectUnavailableAgentSkills(report); } +export async function collectLocalAudioAccelerationFindings(): Promise { + const selection = await inspectLocalAudioSelection(); + const available = selection.candidates.filter((candidate) => candidate.available); + if (available.length === 0) { + return []; + } + const summary = formatLocalAudioSelection(selection); + if (summary) { + return [ + { + checkId: "core/doctor/local-audio-acceleration", + severity: "info", + message: `Local STT auto-selection: ${summary}.`, + path: "tools.media.audio.models", + }, + ]; + } + const blockers = available + .map((candidate) => `${candidate.command}: ${candidate.reason}`) + .join("; "); + return [ + { + checkId: "core/doctor/local-audio-acceleration", + severity: "info", + message: `Local STT commands were found but none are ready for auto-selection: ${blockers}.`, + path: "tools.media.audio.models", + fixHint: + "Install the matching local model/runtime, or configure an explicit tools.media.audio.models CLI entry.", + }, + ]; +} + export async function collectGatewayHealthFindings( ctx: Pick, ): Promise { diff --git a/src/flows/doctor-core-checks.test.ts b/src/flows/doctor-core-checks.test.ts index 82ae68be51f6..2a2e940c619c 100644 --- a/src/flows/doctor-core-checks.test.ts +++ b/src/flows/doctor-core-checks.test.ts @@ -95,6 +95,9 @@ function createDeps(overrides: Partial = {}): CoreHealthChe async collectProviderCatalogProjectionFindings() { return []; }, + async collectLocalAudioAccelerationFindings() { + return []; + }, async collectGatewayHealthFindings() { return []; }, @@ -170,6 +173,27 @@ describe("CORE_HEALTH_CHECKS", () => { ).toBe(false); }); + it("reports local STT auto-selection diagnostics", async () => { + const finding: HealthFinding = { + checkId: "core/doctor/local-audio-acceleration", + severity: "info", + message: + "Local STT auto-selection: whisper-cli (capable=metal, observed=unknown); build capability is not runtime observation.", + }; + const check = getCheck( + createCoreHealthChecks( + createDeps({ + async collectLocalAudioAccelerationFindings() { + return [finding]; + }, + }), + ), + "core/doctor/local-audio-acceleration", + ); + + await expect(check.detect({ mode: "lint", runtime, cfg: {} })).resolves.toEqual([finding]); + }); + it("warns when autonomous Skill Workshop capture is enabled but policy hides its tool", async () => { const check = getCheck( createCoreHealthChecks(createDeps()), diff --git a/src/flows/doctor-core-checks.ts b/src/flows/doctor-core-checks.ts index 4dda5aa9eab9..b2256b54a7a7 100644 --- a/src/flows/doctor-core-checks.ts +++ b/src/flows/doctor-core-checks.ts @@ -77,6 +77,7 @@ export type CoreHealthCheckDeps = { readonly collectProviderCatalogProjectionFindings: ( ctx: HealthCheckContext, ) => Promise; + readonly collectLocalAudioAccelerationFindings: () => Promise; readonly collectGatewayHealthFindings: ( ctx: HealthCheckContext, ) => Promise; @@ -127,6 +128,13 @@ async function collectProviderCatalogProjectionFindingsWithRuntime( return runtime.collectProviderCatalogProjectionFindings(ctx.cfg); } +async function collectLocalAudioAccelerationFindingsWithRuntime(): Promise< + readonly HealthFinding[] +> { + const runtime = await loadDoctorCoreChecksRuntimeModule(); + return runtime.collectLocalAudioAccelerationFindings(); +} + async function collectGatewayHealthFindingsWithRuntime( ctx: HealthCheckContext, ): Promise { @@ -147,6 +155,7 @@ const defaultCoreHealthCheckDeps: CoreHealthCheckDeps = { collectWorkspaceSuggestionNotes: collectWorkspaceSuggestionNotesWithRuntime, collectRuntimeToolSchemaFindings: collectRuntimeToolSchemaFindingsWithRuntime, collectProviderCatalogProjectionFindings: collectProviderCatalogProjectionFindingsWithRuntime, + collectLocalAudioAccelerationFindings: collectLocalAudioAccelerationFindingsWithRuntime, collectGatewayHealthFindings: collectGatewayHealthFindingsWithRuntime, collectGatewayDaemonFindings: collectGatewayDaemonFindingsWithRuntime, }; @@ -1137,6 +1146,15 @@ function createConvertedWorkflowChecks( hooksModelCheck, bootstrapSizeCheck, createProviderCatalogProjectionCheck(deps), + { + id: "core/doctor/local-audio-acceleration", + kind: "core", + description: "Local STT auto-selection and acceleration evidence are visible.", + source: "doctor", + async detect() { + return await deps.collectLocalAudioAccelerationFindings(); + }, + }, createRuntimeToolSchemaCheck(deps), createWorkspaceSuggestionsCheck(deps), skillWorkshopToolPolicyCheck, diff --git a/src/media-understanding/apply.test.ts b/src/media-understanding/apply.test.ts index e61b5984a4ac..3ed8f852a48d 100644 --- a/src/media-understanding/apply.test.ts +++ b/src/media-understanding/apply.test.ts @@ -120,6 +120,14 @@ function getRunExecCall(index = 0) { return call; } +function getRunExecCallForCommand(command: string) { + const call = mockedRunExec.mock.calls.find(([calledCommand]) => calledCommand === command); + if (!call) { + throw new Error(`expected runExec call for ${command}`); + } + return call; +} + function getRunFfmpegArgs(index = 0) { const [args] = mockedRunFfmpeg.mock.calls[index] ?? []; if (!Array.isArray(args)) { @@ -245,7 +253,10 @@ async function setupAudioAutoDetectCase(stdout?: string): Promise<{ } function mockWhisperCliTranscript(transcript: string) { - mockedRunExec.mockImplementationOnce(async (_command, args) => { + mockedRunExec.mockImplementation(async (command, args) => { + if (command === "readelf" || command === "otool") { + return { stdout: "", stderr: "" }; + } const outputBaseIndex = args.indexOf("-of"); const outputBase = outputBaseIndex >= 0 ? args[outputBaseIndex + 1] : undefined; if (typeof outputBase !== "string") { @@ -869,7 +880,7 @@ describe("applyMediaUnderstanding", () => { ); expect(ctx.Transcript).toBe("whisper cpp ok"); - const [command, args, options] = getRunExecCall(); + const [command, args, options] = getRunExecCallForCommand("whisper-cli"); expect(command).toBe("whisper-cli"); if (!Array.isArray(args)) { throw new Error("expected whisper-cli args"); @@ -877,7 +888,17 @@ describe("applyMediaUnderstanding", () => { expect(args.slice(0, 4)).toEqual(["-m", modelPath, "-otxt", "-of"]); expect(typeof args[4]).toBe("string"); expect(String(args[4]).endsWith("sample")).toBe(true); - expect(args.slice(5)).toEqual(["-np", "-nt", await fs.realpath(ctx.MediaPath ?? "")]); + expect(args.slice(5)).toEqual(["-nt", await fs.realpath(ctx.MediaPath ?? "")]); + if (process.platform === "linux") { + expect(mockedRunExec.mock.calls).toContainEqual([ + "readelf", + ["-d", expect.stringContaining("whisper-cli")], + expect.objectContaining({ timeoutMs: 1500 }), + ]); + expect(mockedRunExec.mock.calls.some(([calledCommand]) => calledCommand === "ldd")).toBe( + false, + ); + } expectCliRunOptions(options); }); @@ -935,14 +956,14 @@ describe("applyMediaUnderstanding", () => { expect(String(ffmpegArgs[11])).toContain("telegram-voice.wav"); expect(String(ffmpegArgs[11]).endsWith(".part")).toBe(true); - const [command, args, options] = getRunExecCall(); + const [command, args, options] = getRunExecCallForCommand("whisper-cli"); expect(command).toBe("whisper-cli"); if (!Array.isArray(args)) { throw new Error("expected whisper-cli transcode args"); } expect(args.slice(0, 4)).toEqual(["-m", modelPath, "-otxt", "-of"]); - expect(args.slice(5, 7)).toEqual(["-np", "-nt"]); - expect(String(args[7]).endsWith("telegram-voice.wav")).toBe(true); + expect(args[5]).toBe("-nt"); + expect(String(args[6]).endsWith("telegram-voice.wav")).toBe(true); expectCliRunOptions(options); }); diff --git a/src/media-understanding/local-audio.test.ts b/src/media-understanding/local-audio.test.ts new file mode 100644 index 000000000000..7b1407e06141 --- /dev/null +++ b/src/media-understanding/local-audio.test.ts @@ -0,0 +1,264 @@ +import fs from "node:fs/promises"; +import os from "node:os"; +import path from "node:path"; +import { afterEach, describe, expect, it } from "vitest"; +import { + clearLocalAudioInspectionCacheForTests, + inspectLocalAudioSelection, + recordLocalAudioBackendObservation, +} from "./local-audio.js"; + +let tempDirs: string[] = []; + +async function createTempDir(): Promise { + const tempDir = await fs.mkdtemp(path.join(os.tmpdir(), "openclaw-local-audio-")); + tempDirs.push(tempDir); + return tempDir; +} + +afterEach(async () => { + clearLocalAudioInspectionCacheForTests(); + await Promise.all(tempDirs.map(async (tempDir) => await fs.rm(tempDir, { recursive: true }))); + tempDirs = []; +}); + +describe("local audio selection", () => { + it("expands home-directory shorthand in PATH entries", async () => { + const tempDir = await createTempDir(); + const binDir = path.join(tempDir, "bin"); + const modelPath = path.join(tempDir, "whisper.bin"); + const commandPath = path.join(binDir, "whisper-cli"); + await fs.mkdir(binDir); + await fs.writeFile(modelPath, "model"); + await fs.writeFile(commandPath, "#!/bin/sh\n"); + await fs.chmod(commandPath, 0o755); + + const selection = await inspectLocalAudioSelection({ + env: { + HOME: tempDir, + PATH: "~/bin", + WHISPER_CPP_MODEL: modelPath, + }, + platform: process.platform, + arch: process.arch, + inspectLinkedLibraries: async () => null, + }); + + expect(selection.selected).toMatchObject({ + id: "whisper-cli", + resolvedCommand: commandPath, + }); + }); + + it("does not resolve auto-detected commands from empty PATH entries", async () => { + const tempDir = await createTempDir(); + const modelPath = path.join(tempDir, "whisper.bin"); + await fs.writeFile(modelPath, "model"); + const checkedPaths: string[] = []; + + const selection = await inspectLocalAudioSelection({ + env: { + PATH: path.delimiter, + WHISPER_CPP_MODEL: modelPath, + }, + platform: process.platform, + arch: process.arch, + checkExecutable: async (filePath) => { + checkedPaths.push(filePath); + return true; + }, + inspectLinkedLibraries: async () => null, + }); + + expect(checkedPaths).toEqual([]); + expect(selection.candidates.find((candidate) => candidate.id === "whisper-cli")).toMatchObject({ + available: false, + ready: false, + }); + }); + + it("does not rank Metal-capable whisper ahead of sherpa until a run observes Metal", async () => { + const tempDir = await createTempDir(); + const modelPath = path.join(tempDir, "whisper.bin"); + const sherpaDir = path.join(tempDir, "sherpa"); + await fs.writeFile(modelPath, "model"); + await fs.mkdir(sherpaDir); + await Promise.all( + ["tokens.txt", "encoder.onnx", "decoder.onnx", "joiner.onnx"].map(async (fileName) => { + await fs.writeFile(path.join(sherpaDir, fileName), "model"); + }), + ); + + const selection = await inspectLocalAudioSelection({ + env: { + WHISPER_CPP_MODEL: modelPath, + SHERPA_ONNX_MODEL_DIR: sherpaDir, + }, + platform: "darwin", + arch: "arm64", + resolveBinary: async (name) => + name === "whisper-cli" + ? "/opt/homebrew/bin/whisper-cli" + : name === "sherpa-onnx-offline" + ? "/usr/local/bin/sherpa-onnx-offline" + : null, + resolveRealpath: async () => "/opt/homebrew/Cellar/whisper-cpp/1.9.1/bin/whisper-cli", + inspectLinkedLibraries: async () => null, + }); + + expect(selection.selected).toMatchObject({ + id: "sherpa-onnx-offline", + requestedBackend: "cpu", + }); + const capableWhisper = selection.candidates.find((candidate) => candidate.id === "whisper-cli"); + expect(capableWhisper).toMatchObject({ capableBackend: "metal" }); + expect(capableWhisper).toHaveProperty("observedBackend", undefined); + expect(selection.entries.map((entry) => entry.command)).toEqual([ + "sherpa-onnx-offline", + "whisper-cli", + ]); + + recordLocalAudioBackendObservation({ + command: "/custom/bin/whisper-cli", + args: ["-m", modelPath, "-otxt", "-of", "{{OutputBase}}", "-nt", "{{MediaPath}}"], + output: "whisper_backend_init_gpu: using MTL0 backend", + }); + const mismatchedCommandSelection = await inspectLocalAudioSelection({ + env: { + WHISPER_CPP_MODEL: modelPath, + SHERPA_ONNX_MODEL_DIR: sherpaDir, + }, + platform: "darwin", + arch: "arm64", + resolveBinary: async (name) => + name === "whisper-cli" + ? "/opt/homebrew/bin/whisper-cli" + : name === "sherpa-onnx-offline" + ? "/usr/local/bin/sherpa-onnx-offline" + : null, + resolveRealpath: async () => "/opt/homebrew/Cellar/whisper-cpp/1.9.1/bin/whisper-cli", + inspectLinkedLibraries: async () => null, + }); + expect(mismatchedCommandSelection.selected).toMatchObject({ id: "sherpa-onnx-offline" }); + + recordLocalAudioBackendObservation({ + command: "whisper-cli", + args: ["-m", modelPath, "-otxt", "-of", "{{OutputBase}}", "-nt", "{{MediaPath}}"], + output: "whisper_backend_init_gpu: using MTL0 backend", + }); + const observedSelection = await inspectLocalAudioSelection({ + env: { + WHISPER_CPP_MODEL: modelPath, + SHERPA_ONNX_MODEL_DIR: sherpaDir, + }, + platform: "darwin", + arch: "arm64", + resolveBinary: async (name) => + name === "whisper-cli" + ? "/opt/homebrew/bin/whisper-cli" + : name === "sherpa-onnx-offline" + ? "/usr/local/bin/sherpa-onnx-offline" + : null, + resolveRealpath: async () => "/opt/homebrew/Cellar/whisper-cpp/1.9.1/bin/whisper-cli", + inspectLinkedLibraries: async () => null, + }); + expect(observedSelection.selected).toMatchObject({ + id: "whisper-cli", + capableBackend: "metal", + observedBackend: "metal", + }); + }); + + it("reports Parakeet as MLX-capable without treating capability as observation", async () => { + const tempDir = await createTempDir(); + const whisperModel = path.join(tempDir, "whisper.bin"); + const sherpaDir = path.join(tempDir, "sherpa"); + await fs.writeFile(whisperModel, "model"); + await fs.mkdir(sherpaDir); + await Promise.all( + ["tokens.txt", "encoder.onnx", "decoder.onnx", "joiner.onnx"].map(async (fileName) => { + await fs.writeFile(path.join(sherpaDir, fileName), "model"); + }), + ); + + const selection = await inspectLocalAudioSelection({ + env: { + WHISPER_CPP_MODEL: whisperModel, + SHERPA_ONNX_MODEL_DIR: sherpaDir, + }, + platform: "darwin", + arch: "arm64", + resolveBinary: async (name) => `/usr/local/bin/${name}`, + resolveRealpath: async (filePath) => filePath, + inspectLinkedLibraries: async () => null, + }); + + expect(selection.selected).toMatchObject({ + id: "sherpa-onnx-offline", + requestedBackend: "cpu", + }); + const parakeet = selection.candidates.find((candidate) => candidate.id === "parakeet-mlx"); + expect(parakeet).toMatchObject({ capableBackend: "mlx" }); + expect(parakeet).not.toHaveProperty("observedBackend"); + expect(selection.entries.map((entry) => entry.command)).toEqual([ + "sherpa-onnx-offline", + "whisper-cli", + "parakeet-mlx", + "whisper", + ]); + }); + + it("keeps an unproven whisper runtime behind CPU sherpa", async () => { + const tempDir = await createTempDir(); + const whisperModel = path.join(tempDir, "whisper.bin"); + const sherpaDir = path.join(tempDir, "sherpa"); + await fs.writeFile(whisperModel, "model"); + await fs.mkdir(sherpaDir); + await Promise.all( + ["tokens.txt", "encoder.onnx", "decoder.onnx", "joiner.onnx"].map(async (fileName) => { + await fs.writeFile(path.join(sherpaDir, fileName), "model"); + }), + ); + + const selection = await inspectLocalAudioSelection({ + env: { + WHISPER_CPP_MODEL: whisperModel, + SHERPA_ONNX_MODEL_DIR: sherpaDir, + }, + platform: "linux", + arch: "x64", + resolveBinary: async (name) => + name === "whisper-cli" || name === "sherpa-onnx-offline" ? `/usr/local/bin/${name}` : null, + inspectLinkedLibraries: async () => "libggml-cpu.so", + }); + + expect(selection.selected).toMatchObject({ + id: "sherpa-onnx-offline", + requestedBackend: "cpu", + }); + expect(selection.entries.map((entry) => entry.command)).toEqual([ + "sherpa-onnx-offline", + "whisper-cli", + ]); + }); + + it("reports a dynamically linked CUDA runtime as capable but unobserved", async () => { + const tempDir = await createTempDir(); + const whisperModel = path.join(tempDir, "whisper.bin"); + await fs.writeFile(whisperModel, "model"); + + const selection = await inspectLocalAudioSelection({ + env: { WHISPER_CPP_MODEL: whisperModel }, + platform: "linux", + arch: "x64", + resolveBinary: async (name) => (name === "whisper-cli" ? "/usr/local/bin/whisper-cli" : null), + inspectLinkedLibraries: async () => "libggml-cuda.so => /usr/local/lib/libggml-cuda.so", + }); + + expect(selection.selected).toMatchObject({ + id: "whisper-cli", + capableBackend: "cuda", + }); + expect(selection.selected).toHaveProperty("observedBackend", undefined); + }); +}); diff --git a/src/media-understanding/local-audio.ts b/src/media-understanding/local-audio.ts new file mode 100644 index 000000000000..54b3888bc2bc --- /dev/null +++ b/src/media-understanding/local-audio.ts @@ -0,0 +1,469 @@ +import { constants as fsConstants } from "node:fs"; +import fs from "node:fs/promises"; +import path from "node:path"; +import type { MediaUnderstandingModelConfig } from "../config/types.tools.js"; +import { runExec } from "../process/exec.js"; +import { fileExists } from "./fs.js"; + +export type LocalAudioCandidate = { + id: "parakeet-mlx" | "whisper-cli" | "sherpa-onnx-offline" | "whisper"; + command: string; + resolvedCommand?: string; + available: boolean; + ready: boolean; + capableBackend?: "cuda" | "metal" | "mlx"; + requestedBackend?: string; + observedBackend?: "cpu" | "cuda" | "metal"; + evidence: string; + selected: boolean; + reason?: string; + entry?: MediaUnderstandingModelConfig; +}; + +export type LocalAudioSelection = { + candidates: LocalAudioCandidate[]; + entries: MediaUnderstandingModelConfig[]; + selected?: LocalAudioCandidate; +}; + +type InspectionOptions = { + env?: NodeJS.ProcessEnv; + platform?: NodeJS.Platform; + arch?: string; + resolveBinary?: (name: string, env: NodeJS.ProcessEnv) => Promise; + checkExecutable?: (filePath: string, platform: NodeJS.Platform) => Promise; + resolveRealpath?: (filePath: string) => Promise; + inspectLinkedLibraries?: (filePath: string, platform: NodeJS.Platform) => Promise; +}; + +const binaryCache = new Map>(); +const libraryCache = new Map>(); +const observedBackendCache = new Map(); + +export function clearLocalAudioInspectionCacheForTests(): void { + binaryCache.clear(); + libraryCache.clear(); + observedBackendCache.clear(); +} + +function commandId(command: string): string { + return path.basename(command.trim()).toLowerCase(); +} + +export function resolveRequestedLocalAudioBackend(params: { + command: string; + args: readonly string[]; +}): string | undefined { + const command = commandId(params.command); + if (command === "sherpa-onnx-offline") { + const providerIndex = params.args.findIndex((arg) => arg === "--provider"); + return ( + (providerIndex >= 0 ? params.args[providerIndex + 1] : undefined) ?? + params.args.find((arg) => arg.startsWith("--provider="))?.slice("--provider=".length) ?? + "cpu" + ) + .trim() + .toLowerCase(); + } + if (command === "whisper-cli") { + if (params.args.includes("-ng") || params.args.includes("--no-gpu")) { + return "cpu"; + } + const deviceIndex = params.args.findIndex((arg) => arg === "-dev" || arg === "--device"); + const device = + (deviceIndex >= 0 ? params.args[deviceIndex + 1] : undefined) ?? + params.args.find((arg) => arg.startsWith("--device="))?.slice("--device=".length); + return device?.trim() ? `device:${device.trim()}` : undefined; + } + return undefined; +} + +function observationKey(params: { command: string; args: readonly string[] }): string { + return `${params.command.trim()}\0${resolveRequestedLocalAudioBackend(params) ?? "default"}`; +} + +export function recordLocalAudioBackendObservation(params: { + command: string; + args: readonly string[]; + output: string; +}): "cpu" | "cuda" | "metal" | undefined { + if (commandId(params.command) !== "whisper-cli") { + return undefined; + } + const backend = /using\s+(?:MTL\d+|Metal)\s+backend/i.test(params.output) + ? "metal" + : /using\s+CUDA\d*\s+backend/i.test(params.output) + ? "cuda" + : /using\s+CPU\s+backend|no GPU found/i.test(params.output) + ? "cpu" + : undefined; + if (backend) { + observedBackendCache.set(observationKey(params), backend); + } + return backend; +} + +function getObservedBackend(params: { + command: string; + args: readonly string[]; +}): "cpu" | "cuda" | "metal" | undefined { + return observedBackendCache.get(observationKey(params)); +} + +async function isExecutable(filePath: string, platform: NodeJS.Platform): Promise { + try { + const stat = await fs.stat(filePath); + if (!stat.isFile()) { + return false; + } + if (platform !== "win32") { + await fs.access(filePath, fsConstants.X_OK); + } + return true; + } catch { + return false; + } +} + +function binaryNames(name: string, platform: NodeJS.Platform, env: NodeJS.ProcessEnv): string[] { + if (platform !== "win32" || path.extname(name)) { + return [name]; + } + const extensions = (env.PATHEXT ?? ".EXE;.CMD;.BAT;.COM") + .split(";") + .map((extension) => extension.trim()) + .filter(Boolean); + return [name, ...extensions.map((extension) => `${name}${extension}`)]; +} + +function expandHomeDir(value: string, env: NodeJS.ProcessEnv): string { + const trimmed = value.trim().replace(/^"(.*)"$/, "$1"); + if (trimmed === "~") { + return env.HOME ?? trimmed; + } + if (trimmed.startsWith("~/") || trimmed.startsWith("~\\")) { + return env.HOME ? path.join(env.HOME, trimmed.slice(2)) : trimmed; + } + return trimmed; +} + +async function findBinary( + name: string, + env: NodeJS.ProcessEnv, + platform: NodeJS.Platform, + checkExecutable: (filePath: string, platform: NodeJS.Platform) => Promise = isExecutable, +): Promise { + const key = `${platform}\0${env.PATH ?? ""}\0${env.PATHEXT ?? ""}\0${name}`; + const cached = binaryCache.get(key); + if (cached) { + return await cached; + } + const lookup = (async () => { + const direct = name.trim(); + const candidates = binaryNames(direct, platform, env); + if (direct.includes("/") || direct.includes("\\")) { + for (const candidate of candidates) { + const expanded = + candidate === "~" || candidate.startsWith("~/") || candidate.startsWith("~\\") + ? path.join(env.HOME ?? "~", candidate.slice(candidate === "~" ? 1 : 2)) + : candidate; + if (await checkExecutable(expanded, platform)) { + return expanded; + } + } + return null; + } + for (const directory of (env.PATH ?? "").split(path.delimiter)) { + const expandedDirectory = expandHomeDir(directory, env); + if (!expandedDirectory) { + continue; + } + for (const candidate of candidates) { + const fullPath = path.join(expandedDirectory, candidate); + if (await checkExecutable(fullPath, platform)) { + return fullPath; + } + } + } + return null; + })(); + binaryCache.set(key, lookup); + return await lookup; +} + +async function inspectLinkedLibraries( + filePath: string, + platform: NodeJS.Platform, +): Promise { + const key = `${platform}\0${filePath}`; + const cached = libraryCache.get(key); + if (cached) { + return await cached; + } + const inspection = (async () => { + const command = platform === "darwin" ? "otool" : platform === "linux" ? "readelf" : null; + if (!command) { + return null; + } + try { + const args = platform === "darwin" ? ["-L", filePath] : ["-d", filePath]; + const result = await runExec(command, args, { timeoutMs: 1500 }); + return `${result.stdout}\n${result.stderr ?? ""}`; + } catch { + return null; + } + })(); + libraryCache.set(key, inspection); + return await inspection; +} + +async function inspectWhisperBackend(params: { + command: string; + platform: NodeJS.Platform; + arch: string; + realpath: (filePath: string) => Promise; + libraries: (filePath: string, platform: NodeJS.Platform) => Promise; +}): Promise> { + const libraries = await params.libraries(params.command, params.platform); + if (/(?:ggml[-_]?cuda|libcuda|libcudart)/i.test(libraries ?? "")) { + return { + capableBackend: "cuda", + evidence: "whisper-cli links a CUDA ggml runtime", + }; + } + if (params.platform === "darwin" && params.arch === "arm64") { + const realCommand = await params.realpath(params.command).catch(() => params.command); + if ( + /(?:ggml[-_]?metal|Metal\.framework)/i.test(libraries ?? "") || + /\/Cellar\/whisper-cpp\/[^/]+\/bin\/whisper-cli$/.test(realCommand) + ) { + return { + capableBackend: "metal", + evidence: "Apple Silicon Homebrew whisper-cpp runtime with Metal support", + }; + } + } + return { + evidence: "whisper-cli backend cannot be proven without loading a model", + }; +} + +function rank(candidate: LocalAudioCandidate): number { + if ( + candidate.id === "whisper-cli" && + (candidate.observedBackend === "metal" || candidate.observedBackend === "cuda") + ) { + return 0; + } + if (candidate.id === "sherpa-onnx-offline") { + return 1; + } + if (candidate.id === "whisper-cli") { + return 2; + } + return candidate.id === "parakeet-mlx" ? 3 : 4; +} + +export async function inspectLocalAudioSelection( + options: InspectionOptions = {}, +): Promise { + const env = options.env ?? process.env; + const platform = options.platform ?? process.platform; + const arch = options.arch ?? process.arch; + const resolveBinary = async (name: string) => + options.resolveBinary + ? await options.resolveBinary(name, env) + : await findBinary(name, env, platform, options.checkExecutable); + const [parakeetCommand, whisperCommand, sherpaCommand, pythonCommand] = await Promise.all( + ["parakeet-mlx", "whisper-cli", "sherpa-onnx-offline", "whisper"].map(resolveBinary), + ); + + const envModel = env.WHISPER_CPP_MODEL?.trim(); + const defaultWhisperModel = "/opt/homebrew/share/whisper-cpp/for-tests-ggml-tiny.bin"; + const whisperModel = envModel && (await fileExists(envModel)) ? envModel : defaultWhisperModel; + const whisperReady = Boolean(whisperCommand) && (await fileExists(whisperModel)); + const whisperBackend = whisperCommand + ? await inspectWhisperBackend({ + command: whisperCommand, + platform, + arch, + realpath: options.resolveRealpath ?? fs.realpath, + libraries: options.inspectLinkedLibraries ?? inspectLinkedLibraries, + }) + : { + evidence: "whisper-cli command not found", + }; + + const sherpaDir = env.SHERPA_ONNX_MODEL_DIR?.trim(); + const sherpaFiles = sherpaDir + ? ["tokens.txt", "encoder.onnx", "decoder.onnx", "joiner.onnx"].map((file) => + path.join(sherpaDir, file), + ) + : []; + const sherpaReady = + Boolean(sherpaCommand) && + sherpaFiles.length === 4 && + (await Promise.all(sherpaFiles.map(fileExists))).every(Boolean); + const parakeetReady = Boolean(parakeetCommand) && platform === "darwin" && arch === "arm64"; + const parakeetArgs = [ + "{{MediaPath}}", + "--output-format", + "txt", + "--output-dir", + "{{OutputDir}}", + "--output-template", + "{filename}", + ]; + const whisperArgs = [ + "-m", + whisperModel, + "-otxt", + "-of", + "{{OutputBase}}", + "-nt", + "{{MediaPath}}", + ]; + const sherpaArgs = [ + `--tokens=${sherpaFiles[0]}`, + `--encoder=${sherpaFiles[1]}`, + `--decoder=${sherpaFiles[2]}`, + `--joiner=${sherpaFiles[3]}`, + "{{MediaPath}}", + ]; + const pythonArgs = [ + "--model", + "turbo", + "--output_format", + "txt", + "--output_dir", + "{{OutputDir}}", + "--verbose", + "False", + "{{MediaPath}}", + ]; + + const candidates: LocalAudioCandidate[] = [ + { + id: "parakeet-mlx", + command: "parakeet-mlx", + resolvedCommand: parakeetCommand ?? undefined, + available: Boolean(parakeetCommand), + ready: parakeetReady, + capableBackend: parakeetReady ? "mlx" : undefined, + evidence: parakeetReady + ? "parakeet-mlx is an MLX runtime on Apple Silicon; device use is unobserved" + : "parakeet-mlx acceleration is only supported on Apple Silicon", + selected: false, + reason: parakeetCommand + ? parakeetReady + ? undefined + : "unsupported platform for MLX acceleration" + : "command not found", + entry: parakeetReady + ? { + type: "cli", + command: "parakeet-mlx", + args: parakeetArgs, + } + : undefined, + }, + { + id: "whisper-cli", + command: "whisper-cli", + resolvedCommand: whisperCommand ?? undefined, + available: Boolean(whisperCommand), + ready: whisperReady, + ...whisperBackend, + requestedBackend: resolveRequestedLocalAudioBackend({ + command: "whisper-cli", + args: whisperArgs, + }), + observedBackend: getObservedBackend({ command: "whisper-cli", args: whisperArgs }), + selected: false, + reason: whisperCommand + ? whisperReady + ? undefined + : "model file not found" + : "command not found", + entry: whisperReady + ? { + type: "cli", + command: "whisper-cli", + args: whisperArgs, + } + : undefined, + }, + { + id: "sherpa-onnx-offline", + command: "sherpa-onnx-offline", + resolvedCommand: sherpaCommand ?? undefined, + available: Boolean(sherpaCommand), + ready: sherpaReady, + requestedBackend: "cpu", + evidence: "OpenClaw auto args omit --provider, so sherpa-onnx uses its CPU default", + selected: false, + reason: sherpaCommand + ? sherpaReady + ? undefined + : "SHERPA_ONNX_MODEL_DIR is missing required model files" + : "command not found", + entry: sherpaReady + ? { + type: "cli", + command: "sherpa-onnx-offline", + args: sherpaArgs, + } + : undefined, + }, + { + id: "whisper", + command: "whisper", + resolvedCommand: pythonCommand ?? undefined, + available: Boolean(pythonCommand), + ready: Boolean(pythonCommand), + evidence: "Python Whisper chooses its runtime device when the model loads", + selected: false, + reason: pythonCommand ? undefined : "command not found", + entry: pythonCommand + ? { + type: "cli", + command: "whisper", + args: pythonArgs, + } + : undefined, + }, + ]; + candidates.sort((left, right) => rank(left) - rank(right)); + const selected = candidates.find((candidate) => candidate.ready && candidate.entry); + if (selected) { + selected.selected = true; + } + return { + candidates, + entries: candidates.flatMap((candidate) => + candidate.ready && candidate.entry ? [candidate.entry] : [], + ), + selected, + }; +} + +export function formatLocalAudioSelection(selection: LocalAudioSelection): string | null { + const selected = selection.selected; + if (!selected) { + return null; + } + const describeBackend = (candidate: LocalAudioCandidate) => + [ + candidate.capableBackend ? `capable=${candidate.capableBackend}` : null, + candidate.requestedBackend ? `requested=${candidate.requestedBackend}` : null, + `observed=${candidate.observedBackend ?? "unknown"}`, + ] + .filter(Boolean) + .join(", "); + const fallbacks = selection.candidates + .filter((candidate) => candidate.ready && candidate !== selected) + .map((candidate) => `${candidate.command} (${describeBackend(candidate)})`); + return `${selected.command} (${describeBackend(selected)}); ${selected.evidence}${ + fallbacks.length > 0 ? `; fallbacks: ${fallbacks.join(", ")}` : "" + }`; +} diff --git a/src/media-understanding/runner.cli-audio.test.ts b/src/media-understanding/runner.cli-audio.test.ts index 2a1c9548efdf..a33b6c69e3e2 100644 --- a/src/media-understanding/runner.cli-audio.test.ts +++ b/src/media-understanding/runner.cli-audio.test.ts @@ -175,6 +175,28 @@ describe("media-understanding CLI audio entry", () => { expect(result?.text).toBe("file transcript"); }); + it("records the backend observed during a whisper.cpp model run", async () => { + runExecMock.mockImplementationOnce(async (_command, args: string[]) => { + await fs.writeFile(`${args[2]}.txt`, "observed transcript\n"); + return { + stdout: "", + stderr: "whisper_backend_init_gpu: using MTL0 backend", + }; + }); + + const result = await runAudioEntry({ + command: "whisper-cli", + args: ["-otxt", "-of", "{{OutputBase}}", "{{MediaPath}}"], + }); + + expect(result).toMatchObject({ + provider: "whisper-cli", + model: "whisper-cli", + observedBackend: "metal", + text: "observed transcript", + }); + }); + it("reads parakeet txt output selected through its upstream environment", async () => { const testCase: TranscriptFileCase = { name: "parakeet environment output", diff --git a/src/media-understanding/runner.entries.guards.test.ts b/src/media-understanding/runner.entries.guards.test.ts index ca8104f7ed0a..85daa725ca96 100644 --- a/src/media-understanding/runner.entries.guards.test.ts +++ b/src/media-understanding/runner.entries.guards.test.ts @@ -1,7 +1,11 @@ // Runner entry guard tests cover malformed decision data formatting without // depending on provider execution. import { describe, expect, it } from "vitest"; -import { formatDecisionSummary, formatMissingProviderHint } from "./runner.entries.js"; +import { + buildModelDecision, + formatDecisionSummary, + formatMissingProviderHint, +} from "./runner.entries.js"; import type { MediaUnderstandingDecision } from "./types.js"; describe("media-understanding formatDecisionSummary guards", () => { @@ -46,6 +50,42 @@ describe("media-understanding formatDecisionSummary guards", () => { }); }); +describe("media-understanding CLI backend decisions", () => { + it.each([ + { + command: "sherpa-onnx-offline", + args: ["--provider=cuda", "{{MediaPath}}"], + requestedBackend: "cuda", + }, + { + command: "sherpa-onnx-offline", + args: ["{{MediaPath}}"], + requestedBackend: "cpu", + }, + { + command: "whisper-cli", + args: ["--no-gpu", "{{MediaPath}}"], + requestedBackend: "cpu", + }, + { + command: "whisper-cli", + args: ["--device", "GPU0", "{{MediaPath}}"], + requestedBackend: "device:GPU0", + }, + ])( + "reports $command backend request as $requestedBackend", + ({ command, args, requestedBackend }) => { + expect( + buildModelDecision({ + entry: { type: "cli", command, args }, + entryType: "cli", + outcome: "success", + }), + ).toMatchObject({ provider: command, model: command, requestedBackend }); + }, + ); +}); + describe("media-understanding formatMissingProviderHint", () => { it("returns the catalog hint for a provider with mediaUnderstandingProviders contract (groq)", () => { const hint = formatMissingProviderHint("groq"); diff --git a/src/media-understanding/runner.entries.ts b/src/media-understanding/runner.entries.ts index 1e92a95d2f8f..c941897ac9e2 100644 --- a/src/media-understanding/runner.entries.ts +++ b/src/media-understanding/runner.entries.ts @@ -54,6 +54,10 @@ import { } from "./defaults.constants.js"; import { normalizeImageDescriptionInput } from "./image-input-normalize.js"; import { describeImageWithModel } from "./image-runtime.js"; +import { + recordLocalAudioBackendObservation, + resolveRequestedLocalAudioBackend, +} from "./local-audio.js"; import { resolveOpenAiAudioAuthModelApi } from "./openai-audio-api.js"; import { normalizeMediaExecutionProviderId } from "./provider-id.js"; import { getMediaUnderstandingProvider, normalizeMediaProviderId } from "./provider-registry.js"; @@ -393,10 +397,17 @@ export function buildModelDecision(params: { }): MediaUnderstandingModelDecision { if (params.entryType === "cli") { const command = params.entry.command?.trim(); + const requestedBackend = command + ? resolveRequestedLocalAudioBackend({ + command, + args: params.entry.args ?? [], + }) + : undefined; return { type: "cli", provider: command ?? "cli", model: params.entry.model ?? command, + ...(requestedBackend ? { requestedBackend } : {}), outcome: params.outcome, reason: params.reason, }; @@ -653,11 +664,20 @@ export function formatDecisionSummary(decision: MediaUnderstandingDecision): str const chosen = attachments.find((entry) => entry?.chosen)?.chosen; const provider = typeof chosen?.provider === "string" ? chosen.provider.trim() : undefined; const model = typeof chosen?.model === "string" ? chosen.model.trim() : undefined; - const modelLabel = provider ? (model ? `${provider}/${model}` : provider) : undefined; + const modelLabel = provider + ? model && model !== provider + ? `${provider}/${model}` + : provider + : undefined; + const backendLabel = chosen?.observedBackend + ? ` observed=${chosen.observedBackend}` + : chosen?.requestedBackend + ? ` requested=${chosen.requestedBackend}` + : ""; const reason = findDecisionReason(decision, decision.outcome === "failed" ? "failed" : undefined); const shortReason = summarizeDecisionReason(reason); const countLabel = total > 0 ? ` (${success}/${total})` : ""; - const viaLabel = modelLabel ? ` via ${modelLabel}` : ""; + const viaLabel = modelLabel ? ` via ${modelLabel}${backendLabel}` : ""; const reasonLabel = shortReason ? ` reason=${shortReason}` : ""; return `${decision.capability}: ${decision.outcome}${countLabel}${viaLabel}${reasonLabel}`; } @@ -1052,11 +1072,26 @@ export async function runCliEntry(params: { if (shouldLogVerbose()) { logVerbose(`Media understanding via CLI: ${argv.join(" ")}`); } - const { stdout } = await runExec(argv[0], argv.slice(1), { + const { stdout, stderr } = await runExec(argv[0], argv.slice(1), { timeoutMs, maxBuffer: CLI_OUTPUT_MAX_BUFFER, cwd: isAntigravityCliCommand(command) ? path.dirname(mediaPath) : undefined, }); + const requestedBackend = + capability === "audio" + ? resolveRequestedLocalAudioBackend({ + command, + args: argv.slice(1), + }) + : undefined; + const observedBackend = + capability === "audio" + ? recordLocalAudioBackendObservation({ + command, + args: argv.slice(1), + output: `${stderr ?? ""}\n${stdout}`, + }) + : undefined; const resolved = await resolveCliOutput({ command, args: argv.slice(1), @@ -1071,8 +1106,10 @@ export async function runCliEntry(params: { kind: capability === "audio" ? "audio.transcription" : `${capability}.description`, attachmentIndex: params.attachmentIndex, text, - provider: "cli", + provider: capability === "audio" ? commandBase(command) : "cli", model: command, + ...(requestedBackend ? { requestedBackend } : {}), + ...(observedBackend ? { observedBackend } : {}), }; } finally { await fs.rm(outputDir, { recursive: true, force: true }).catch(() => {}); diff --git a/src/media-understanding/runner.ts b/src/media-understanding/runner.ts index ba885315d089..8fe7581838fa 100644 --- a/src/media-understanding/runner.ts +++ b/src/media-understanding/runner.ts @@ -43,7 +43,10 @@ import { getDefaultMediaLocalRoots } from "../media/local-roots.js"; import { runExec } from "../process/exec.js"; import { createLazyRuntimeModule, createLazyRuntimeNamedExport } from "../shared/lazy-runtime.js"; import { MediaAttachmentCache, selectAttachments } from "./attachments.js"; -import { fileExists } from "./fs.js"; +import { + clearLocalAudioInspectionCacheForTests, + inspectLocalAudioSelection, +} from "./local-audio.js"; import { resolveOpenAiAudioAuthModelApi } from "./openai-audio-api.js"; import { normalizeMediaExecutionProviderId, normalizeMediaProviderId } from "./provider-id.js"; import { @@ -335,6 +338,7 @@ const antigravityCliCache = new Map>(); export function clearMediaUnderstandingBinaryCacheForTests(): void { binaryCache.clear(); antigravityCliCache.clear(); + clearLocalAudioInspectionCacheForTests(); } function expandHomeDir(value: string): string { @@ -425,10 +429,6 @@ async function findBinary(name: string): Promise { return resolved; } -async function hasBinary(name: string): Promise { - return Boolean(await findBinary(name)); -} - async function probeAntigravityCliCandidate(command: string): Promise { const resolved = await findBinary(command); if (!resolved) { @@ -476,93 +476,6 @@ async function resolveAntigravityCliBinary(): Promise { return resolved; } -async function resolveLocalWhisperCppEntry(): Promise { - if (!(await hasBinary("whisper-cli"))) { - return null; - } - const envModel = process.env.WHISPER_CPP_MODEL?.trim(); - const defaultModel = "/opt/homebrew/share/whisper-cpp/for-tests-ggml-tiny.bin"; - const modelPath = envModel && (await fileExists(envModel)) ? envModel : defaultModel; - if (!(await fileExists(modelPath))) { - return null; - } - return { - type: "cli", - command: "whisper-cli", - args: ["-m", modelPath, "-otxt", "-of", "{{OutputBase}}", "-np", "-nt", "{{MediaPath}}"], - }; -} - -async function resolveLocalWhisperEntry(): Promise { - if (!(await hasBinary("whisper"))) { - return null; - } - return { - type: "cli", - command: "whisper", - args: [ - "--model", - "turbo", - "--output_format", - "txt", - "--output_dir", - "{{OutputDir}}", - "--verbose", - "False", - "{{MediaPath}}", - ], - }; -} - -async function resolveSherpaOnnxEntry(): Promise { - if (!(await hasBinary("sherpa-onnx-offline"))) { - return null; - } - const modelDir = process.env.SHERPA_ONNX_MODEL_DIR?.trim(); - if (!modelDir) { - return null; - } - const tokens = path.join(modelDir, "tokens.txt"); - const encoder = path.join(modelDir, "encoder.onnx"); - const decoder = path.join(modelDir, "decoder.onnx"); - const joiner = path.join(modelDir, "joiner.onnx"); - if (!(await fileExists(tokens))) { - return null; - } - if (!(await fileExists(encoder))) { - return null; - } - if (!(await fileExists(decoder))) { - return null; - } - if (!(await fileExists(joiner))) { - return null; - } - return { - type: "cli", - command: "sherpa-onnx-offline", - args: [ - `--tokens=${tokens}`, - `--encoder=${encoder}`, - `--decoder=${decoder}`, - `--joiner=${joiner}`, - "{{MediaPath}}", - ], - }; -} - -async function resolveLocalAudioEntry(): Promise { - const sherpa = await resolveSherpaOnnxEntry(); - if (sherpa) { - return sherpa; - } - const whisperCpp = await resolveLocalWhisperCppEntry(); - if (whisperCpp) { - return whisperCpp; - } - return await resolveLocalWhisperEntry(); -} - async function resolveAntigravityCliEntry( capability: MediaUnderstandingCapability, ): Promise { @@ -798,9 +711,9 @@ async function resolveAutoEntries(params: { if (keyEntry) { return [keyEntry]; } - const localAudio = await resolveLocalAudioEntry(); - if (localAudio) { - return [localAudio]; + const localAudio = await inspectLocalAudioSelection(); + if (localAudio.entries.length > 0) { + return localAudio.entries; } } const keys = await resolveKeyEntry(params); @@ -989,6 +902,12 @@ async function runAttachmentEntries(params: { if (result.model) { decision.model = result.model; } + if (result.requestedBackend) { + decision.requestedBackend = result.requestedBackend; + } + if (result.observedBackend) { + decision.observedBackend = result.observedBackend; + } attempts.push(decision); return { output: result, attempts }; } diff --git a/src/media-understanding/types.ts b/src/media-understanding/types.ts index 9e9f9e6c88d1..5d90a349ef53 100644 --- a/src/media-understanding/types.ts +++ b/src/media-understanding/types.ts @@ -29,6 +29,8 @@ export type MediaUnderstandingOutput = { text: string; provider: string; model?: string; + requestedBackend?: string; + observedBackend?: string; }; type MediaUnderstandingDecisionOutcome = @@ -42,6 +44,8 @@ type MediaUnderstandingDecisionOutcome = export type MediaUnderstandingModelDecision = { provider?: string; model?: string; + requestedBackend?: string; + observedBackend?: string; type: "provider" | "cli"; outcome: "success" | "skipped" | "failed"; reason?: string; diff --git a/src/status/status-message.ts b/src/status/status-message.ts index 6b0ef541a26a..3c6ebb997fb7 100644 --- a/src/status/status-message.ts +++ b/src/status/status-message.ts @@ -422,8 +422,19 @@ const formatMediaUnderstandingLine = (decisions?: ReadonlyArray entry.chosen)?.chosen; const provider = chosen?.provider?.trim(); const model = chosen?.model?.trim(); - const modelLabel = provider ? (model ? `${provider}/${model}` : provider) : null; - return `${decision.capability}${countLabel} ok${modelLabel ? ` (${modelLabel})` : ""}`; + const modelLabel = provider + ? model && model !== provider + ? `${provider}/${model}` + : provider + : null; + const backendLabel = chosen?.observedBackend + ? ` observed=${chosen.observedBackend}` + : chosen?.requestedBackend + ? ` requested=${chosen.requestedBackend}` + : ""; + return `${decision.capability}${countLabel} ok${ + modelLabel ? ` (${modelLabel}${backendLabel})` : "" + }`; } if (decision.outcome === "no-attachment") { return `${decision.capability} none`;