mirror of
https://github.com/openclaw/openclaw.git
synced 2026-08-12 21:53:00 -06:00
fix(media): verify local STT acceleration before prioritizing it (#104210)
* fix(media): observe local STT backend selection * docs(media): explain local STT acceleration evidence * fix(media): expand home paths in local STT discovery * fix(media): scope local STT backend observations * fix(security): avoid executing STT binaries during inspection * fix(media): widen local STT selection state * fix(security): ignore empty local STT PATH entries * test(media): type local STT capability mocks * test(media): isolate local STT PATH lookup * style(cli): simplify local STT provider rows * fix(cli): distinguish local STT fallback selection
This commit is contained in:
Vincent Koc
@@ -84,6 +84,7 @@ openclaw doctor --lint --json
|
||||
openclaw doctor --lint --all
|
||||
openclaw doctor --lint --allow-exec
|
||||
openclaw doctor --lint --only core/doctor/gateway-config --json
|
||||
openclaw doctor --lint --only core/doctor/local-audio-acceleration --severity-min info
|
||||
```
|
||||
|
||||
Human output is compact:
|
||||
@@ -125,6 +126,8 @@ Exit codes:
|
||||
|
||||
`--all` controls which checks are selected before severity filtering. The default lint run excludes checks that are deep, historical, or more likely to surface repairable legacy residue; use `--all` for the complete inventory. `--only <id>` is the most precise selector and can run any registered check by id.
|
||||
|
||||
`core/doctor/local-audio-acceleration` reports the auto-selected local STT command, separate capable/requested/observed backend evidence, and fallback order without loading a speech model. It emits an informational finding, so include `--severity-min info` to display it.
|
||||
|
||||
## Structured health checks
|
||||
|
||||
Modern doctor checks use a small split contract:
|
||||
|
||||
+22
-3
@@ -23,11 +23,16 @@ If you have not configured models and `tools.media.audio.enabled` is not `false`
|
||||
1. **Active reply model**, when its provider supports audio understanding.
|
||||
2. **Configured provider auth** — any `models.providers.*` entry with auth available for a provider that supports audio transcription. This is checked before local CLIs, so a configured API key always wins over a local binary on `PATH`.
|
||||
Provider priority when multiple are configured: Groq, OpenAI, xAI, Deepgram, Google, SenseAudio, ElevenLabs, Mistral.
|
||||
3. **Local CLIs** (only if no provider auth resolved), checked in this order:
|
||||
- `sherpa-onnx-offline` (requires `SHERPA_ONNX_MODEL_DIR` with `tokens.txt`, `encoder.onnx`, `decoder.onnx`, and `joiner.onnx`)
|
||||
- `whisper-cli` (from `whisper-cpp`; uses `WHISPER_CPP_MODEL` or a bundled tiny model)
|
||||
3. **Local CLIs** (only if no provider auth resolved). OpenClaw builds an ordered fallback list:
|
||||
- `whisper-cli`, before CPU defaults only when an earlier model invocation in the current process observed Metal or CUDA
|
||||
- `sherpa-onnx-offline` on its default CPU provider (requires `SHERPA_ONNX_MODEL_DIR` with `tokens.txt`, `encoder.onnx`, `decoder.onnx`, and `joiner.onnx`)
|
||||
- `whisper-cli` when Metal/CUDA is only build-capable or the selected backend is otherwise unobserved
|
||||
- `parakeet-mlx` on Apple Silicon (MLX-capable; device use remains unobserved)
|
||||
- `whisper` (Python CLI; downloads models automatically)
|
||||
|
||||
Install/link provenance is capability evidence, not execution evidence. It never moves a candidate ahead of CPU sherpa by itself. OpenClaw does not load a model during setup or status checks just to probe a backend.
|
||||
Auto-detected whisper.cpp keeps its normal model-run logs enabled so OpenClaw can record the upstream `using … backend` line. Explicit CLI entries keep their configured output flags.
|
||||
|
||||
Gemini CLI auto-detect for media understanding was replaced by a sandboxed Antigravity CLI (`agy`) fallback for image/video; audio does not use a CLI fallback beyond the local binaries above.
|
||||
|
||||
To disable auto-detection, set `tools.media.audio.enabled: false`. To customize, set `tools.media.audio.models`.
|
||||
@@ -36,6 +41,15 @@ To disable auto-detection, set `tools.media.audio.enabled: false`. To customize,
|
||||
Binary detection is best-effort across macOS/Linux/Windows. Make sure the CLI is on `PATH` (`~` is expanded), or set an explicit CLI model with a full command path.
|
||||
</Note>
|
||||
|
||||
Inspect the local selection without transcribing audio:
|
||||
|
||||
```bash
|
||||
openclaw capability audio providers
|
||||
openclaw doctor --lint --only core/doctor/local-audio-acceleration --severity-min info
|
||||
```
|
||||
|
||||
The provider inventory reports the local fallback winner separately from global provider selection, plus capable, requested, and observed backend fields. After transcription runs, `/status` reports the requested or observed backend in the media line. Explicit `tools.media.audio.models` CLI entries still bypass auto-selection; use their backend-specific flags such as sherpa `--provider=cuda` or whisper.cpp `--no-gpu`/`--device`.
|
||||
|
||||
## Config examples
|
||||
|
||||
### Provider + CLI fallback (OpenAI + Whisper CLI)
|
||||
@@ -161,6 +175,11 @@ Binary detection is best-effort across macOS/Linux/Windows. Make sure the CLI is
|
||||
- `tools.media.audio.echoFormat` customizes the echo text (placeholder: `{transcript}`; default `📝 "{transcript}"`).
|
||||
- CLI stdout is capped at 5MB; keep CLI output concise.
|
||||
- CLI `args` should use `{{MediaPath}}` for the local audio file path. Run `openclaw doctor --fix` to migrate deprecated `{input}` placeholders from older `audio.transcription.command` configs (retired key: `audio.transcription`, replaced by `tools.media.audio.models`).
|
||||
- `tools.media.concurrency` bounds media tasks; it is not a GPU scheduler.
|
||||
|
||||
### Resident local STT
|
||||
|
||||
Auto-detected local STT remains process-per-request. OpenClaw does not currently manage a resident whisper.cpp server because the standard Homebrew `whisper-cpp` package disables that server, while the upstream example has no configured bounded admission queue. A plugin-owned resident lifecycle needs a maintained packaged worker with health/startup, model residency, bounded queueing, cancellation/timeout, loopback-only no-auth operation, and no cloud fallback before it can be enabled safely.
|
||||
|
||||
### Proxy environment support
|
||||
|
||||
|
||||
@@ -164,11 +164,15 @@ When `tools.media.<capability>.enabled` is not `false` and no models are configu
|
||||
Configured `models.providers.*` entries that support audio are tried before local CLIs. Bundled provider priority order (ties break alphabetically by provider id): Groq/OpenAI → xAI → Deepgram → OpenRouter → Google/SenseAudio → Deepinfra/ElevenLabs → Mistral.
|
||||
</Step>
|
||||
<Step title="Local CLIs (audio only)">
|
||||
First installed local binary, in this order:
|
||||
- `sherpa-onnx-offline` (requires `SHERPA_ONNX_MODEL_DIR` with `tokens.txt`/`encoder.onnx`/`decoder.onnx`/`joiner.onnx`)
|
||||
- `whisper-cli` (`whisper-cpp`; uses `WHISPER_CPP_MODEL` or a bundled tiny model)
|
||||
Ready local binaries become an ordered fallback list:
|
||||
- `whisper-cli` first only after an earlier model invocation in the current process observed Metal or CUDA
|
||||
- CPU-default `sherpa-onnx-offline` (requires `SHERPA_ONNX_MODEL_DIR` with `tokens.txt`/`encoder.onnx`/`decoder.onnx`/`joiner.onnx`)
|
||||
- `whisper-cli` when acceleration is merely build-capable or unobserved
|
||||
- `parakeet-mlx` on Apple Silicon (MLX-capable, device use unobserved)
|
||||
- `whisper` (Python CLI; defaults to the `turbo` model, downloads automatically)
|
||||
|
||||
Backend capability inspection is cached and does not load a model. Build capability, requested backend flags, and backend observed from a real invocation remain separate. Auto-detected whisper.cpp leaves model-run logs enabled so the upstream selected-backend line can be recorded. Explicit CLI entries keep their configured order, backend flags, and output flags.
|
||||
|
||||
</Step>
|
||||
<Step title="Provider auth (image/video)">
|
||||
Configured `models.providers.*` entries that support the capability are tried before the bundled fallback order. Image-only config providers with an image-capable model auto-register for media understanding even when they are not a bundled vendor plugin.
|
||||
@@ -421,7 +425,13 @@ When `mode: "all"`, outputs are labeled `[Image 1/2]`, `[Audio 2/2]`, etc.
|
||||
When media understanding runs, `/status` includes a per-capability summary line:
|
||||
|
||||
```
|
||||
📎 Media: image ok (openai/gpt-5.5) · audio skipped (maxBytes)
|
||||
📎 Media: image ok (openai/gpt-5.5) · audio ok (whisper-cli observed=metal)
|
||||
```
|
||||
|
||||
For preflight inventory, run `openclaw capability audio providers`. Local rows show the local fallback winner separately from global provider selection, readiness, and separate capable/requested/observed backend fields. The same local selection is available as an informational doctor finding:
|
||||
|
||||
```bash
|
||||
openclaw doctor --lint --only core/doctor/local-audio-acceleration --severity-min info
|
||||
```
|
||||
|
||||
## Notes
|
||||
|
||||
@@ -1529,6 +1529,37 @@ describe("buildStatusMessage", () => {
|
||||
expect(normalized).toContain("Media: image ok (openai/gpt-5.4) · audio skipped (maxBytes)");
|
||||
});
|
||||
|
||||
it("distinguishes observed local STT backends from requested backends", () => {
|
||||
const text = buildStatusMessage({
|
||||
agent: { model: "anthropic/claude-opus-4-6" },
|
||||
sessionEntry: { sessionId: "media-local-stt", updatedAt: 0 },
|
||||
sessionKey: "agent:main:main",
|
||||
queue: { mode: "none" },
|
||||
mediaDecisions: [
|
||||
{
|
||||
capability: "audio",
|
||||
outcome: "success",
|
||||
attachments: [
|
||||
{
|
||||
attachmentIndex: 0,
|
||||
attempts: [],
|
||||
chosen: {
|
||||
type: "cli",
|
||||
provider: "whisper-cli",
|
||||
model: "whisper-cli",
|
||||
requestedBackend: "device:0",
|
||||
observedBackend: "metal",
|
||||
outcome: "success",
|
||||
},
|
||||
},
|
||||
],
|
||||
},
|
||||
],
|
||||
});
|
||||
|
||||
expect(normalizeTestText(text)).toContain("Media: audio ok (whisper-cli observed=metal)");
|
||||
});
|
||||
|
||||
it("includes failed media understanding decisions with the surfaced reason", () => {
|
||||
const text = buildStatusMessage({
|
||||
agent: { model: "anthropic/claude-opus-4-6" },
|
||||
|
||||
@@ -4,6 +4,7 @@ import os from "node:os";
|
||||
import path from "node:path";
|
||||
import { Command } from "commander";
|
||||
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import type { LocalAudioSelection } from "../media-understanding/local-audio.js";
|
||||
import { runRegisteredCli } from "../test-utils/command-runner.js";
|
||||
import { CAPABILITY_METADATA, registerCapabilityCli } from "./capability-cli.js";
|
||||
|
||||
@@ -147,6 +148,10 @@ const mocks = vi.hoisted(() => ({
|
||||
]),
|
||||
listEmbeddingProviders: vi.fn(() => []),
|
||||
buildMediaUnderstandingRegistry: vi.fn(() => new Map()),
|
||||
inspectLocalAudioSelection: vi.fn<() => Promise<LocalAudioSelection>>(async () => ({
|
||||
candidates: [],
|
||||
entries: [],
|
||||
})),
|
||||
convertHeicToJpeg: vi.fn(async () => Buffer.from("jpeg-normalized")),
|
||||
isWebSearchProviderConfigured: vi.fn(() => false),
|
||||
isWebFetchProviderConfigured: vi.fn(() => false),
|
||||
@@ -314,6 +319,10 @@ vi.mock("../media-understanding/provider-registry.js", () => ({
|
||||
mocks.buildMediaUnderstandingRegistry as typeof import("../media-understanding/provider-registry.js").buildMediaUnderstandingRegistry,
|
||||
}));
|
||||
|
||||
vi.mock("../media-understanding/local-audio.js", () => ({
|
||||
inspectLocalAudioSelection: mocks.inspectLocalAudioSelection,
|
||||
}));
|
||||
|
||||
vi.mock("../media/media-services.js", async (importOriginal) => {
|
||||
const actual = await importOriginal<typeof import("../media/media-services.js")>();
|
||||
return {
|
||||
@@ -522,6 +531,7 @@ describe("capability cli", () => {
|
||||
mocks.resolveExplicitTtsOverrides.mockClear();
|
||||
mocks.getProviderEnvVars.mockClear();
|
||||
mocks.buildMediaUnderstandingRegistry.mockReset().mockReturnValue(new Map());
|
||||
mocks.inspectLocalAudioSelection.mockReset().mockResolvedValue({ candidates: [], entries: [] });
|
||||
mocks.convertHeicToJpeg.mockClear();
|
||||
mocks.createEmbeddingProvider.mockClear();
|
||||
mocks.listMemoryEmbeddingProviders
|
||||
@@ -3351,6 +3361,70 @@ describe("capability cli", () => {
|
||||
]);
|
||||
});
|
||||
|
||||
it("distinguishes the local STT fallback winner from global provider selection", async () => {
|
||||
vi.stubEnv("DEEPGRAM_API_KEY", "deepgram-test-key");
|
||||
mocks.buildMediaUnderstandingRegistry.mockReturnValueOnce(
|
||||
new Map([
|
||||
[
|
||||
"deepgram",
|
||||
{
|
||||
id: "deepgram",
|
||||
capabilities: ["audio"],
|
||||
defaultModels: { audio: "nova-3" },
|
||||
},
|
||||
],
|
||||
]),
|
||||
);
|
||||
const candidate = {
|
||||
id: "whisper-cli" as const,
|
||||
command: "whisper-cli",
|
||||
resolvedCommand: "/opt/homebrew/bin/whisper-cli",
|
||||
available: true,
|
||||
ready: true,
|
||||
capableBackend: "metal" as const,
|
||||
evidence: "Apple Silicon Homebrew whisper-cpp runtime with Metal support",
|
||||
selected: true,
|
||||
entry: {
|
||||
type: "cli" as const,
|
||||
command: "whisper-cli",
|
||||
args: ["{{MediaPath}}"],
|
||||
},
|
||||
};
|
||||
mocks.inspectLocalAudioSelection.mockResolvedValueOnce({
|
||||
candidates: [candidate],
|
||||
entries: [candidate.entry],
|
||||
selected: candidate,
|
||||
});
|
||||
|
||||
await runRegisteredCli({
|
||||
register: registerCapabilityCli as (program: Command) => void,
|
||||
argv: ["capability", "audio", "providers", "--json"],
|
||||
});
|
||||
|
||||
expect(firstJsonOutput()).toEqual([
|
||||
{
|
||||
available: true,
|
||||
configured: true,
|
||||
selected: false,
|
||||
id: "deepgram",
|
||||
capabilities: ["audio"],
|
||||
defaultModels: { audio: "nova-3" },
|
||||
},
|
||||
{
|
||||
available: true,
|
||||
configured: true,
|
||||
selected: false,
|
||||
localFallbackSelected: true,
|
||||
id: "local/whisper-cli",
|
||||
transport: "local-cli",
|
||||
command: "whisper-cli",
|
||||
capableBackend: "metal",
|
||||
observedBackend: "unknown",
|
||||
evidence: "Apple Silicon Homebrew whisper-cpp runtime with Metal support",
|
||||
},
|
||||
]);
|
||||
});
|
||||
|
||||
it("resolves plugin web search SecretRefs before running infer web search", async () => {
|
||||
const unresolvedConfig = {
|
||||
tools: { web: { search: { provider: "tavily", enabled: true } } },
|
||||
|
||||
@@ -60,6 +60,7 @@ import {
|
||||
parseStrictFiniteNumber,
|
||||
parseStrictPositiveInteger,
|
||||
} from "../infra/parse-finite-number.js";
|
||||
import { inspectLocalAudioSelection } from "../media-understanding/local-audio.js";
|
||||
import { buildMediaUnderstandingRegistry } from "../media-understanding/provider-registry.js";
|
||||
import type { RunMediaUnderstandingFileResult } from "../media-understanding/runtime-types.js";
|
||||
import {
|
||||
@@ -2529,7 +2530,7 @@ export function registerCapabilityCli(program: Command) {
|
||||
.action(async (opts) => {
|
||||
await runCommandWithRuntime(defaultRuntime, async () => {
|
||||
const cfg = getRuntimeConfig();
|
||||
const providers = [...buildMediaUnderstandingRegistry(undefined, cfg).values()]
|
||||
const remoteProviders = [...buildMediaUnderstandingRegistry(undefined, cfg).values()]
|
||||
.filter((provider) => provider.capabilities?.includes("audio"))
|
||||
.map((provider) => ({
|
||||
available: true,
|
||||
@@ -2546,6 +2547,28 @@ export function registerCapabilityCli(program: Command) {
|
||||
capabilities: provider.capabilities,
|
||||
defaultModels: provider.defaultModels,
|
||||
}));
|
||||
const localSelection = await inspectLocalAudioSelection();
|
||||
const localProviders = localSelection.candidates
|
||||
.filter((candidate) => candidate.available)
|
||||
.map((candidate) =>
|
||||
Object.assign(
|
||||
{
|
||||
available: candidate.available,
|
||||
configured: candidate.ready,
|
||||
selected: false,
|
||||
localFallbackSelected: candidate.selected,
|
||||
id: `local/${candidate.id}`,
|
||||
transport: "local-cli",
|
||||
command: candidate.command,
|
||||
observedBackend: candidate.observedBackend ?? "unknown",
|
||||
evidence: candidate.evidence,
|
||||
},
|
||||
candidate.capableBackend ? { capableBackend: candidate.capableBackend } : {},
|
||||
candidate.requestedBackend ? { requestedBackend: candidate.requestedBackend } : {},
|
||||
candidate.reason ? { reason: candidate.reason } : {},
|
||||
),
|
||||
);
|
||||
const providers = [...remoteProviders, ...localProviders];
|
||||
emitJsonOrText(defaultRuntime, Boolean(opts.json), providers, providerSummaryText);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -43,6 +43,10 @@ import {
|
||||
import { resolveGatewayService, readGatewayServiceState } from "../daemon/service.js";
|
||||
import { buildGatewayProbeConnectionDetails } from "../gateway/call.js";
|
||||
import { formatErrorMessage } from "../infra/errors.js";
|
||||
import {
|
||||
formatLocalAudioSelection,
|
||||
inspectLocalAudioSelection,
|
||||
} from "../media-understanding/local-audio.js";
|
||||
import type { ProviderRuntimeModel } from "../plugins/provider-runtime-model.types.js";
|
||||
import { getPluginToolMeta, setPluginToolMeta } from "../plugins/tools.js";
|
||||
import type { ProviderCatalogOrder, ProviderPlugin } from "../plugins/types.js";
|
||||
@@ -68,6 +72,38 @@ export function detectUnavailableSkills(cfg: OpenClawConfig): SkillStatusEntry[]
|
||||
return collectUnavailableAgentSkills(report);
|
||||
}
|
||||
|
||||
export async function collectLocalAudioAccelerationFindings(): Promise<readonly HealthFinding[]> {
|
||||
const selection = await inspectLocalAudioSelection();
|
||||
const available = selection.candidates.filter((candidate) => candidate.available);
|
||||
if (available.length === 0) {
|
||||
return [];
|
||||
}
|
||||
const summary = formatLocalAudioSelection(selection);
|
||||
if (summary) {
|
||||
return [
|
||||
{
|
||||
checkId: "core/doctor/local-audio-acceleration",
|
||||
severity: "info",
|
||||
message: `Local STT auto-selection: ${summary}.`,
|
||||
path: "tools.media.audio.models",
|
||||
},
|
||||
];
|
||||
}
|
||||
const blockers = available
|
||||
.map((candidate) => `${candidate.command}: ${candidate.reason}`)
|
||||
.join("; ");
|
||||
return [
|
||||
{
|
||||
checkId: "core/doctor/local-audio-acceleration",
|
||||
severity: "info",
|
||||
message: `Local STT commands were found but none are ready for auto-selection: ${blockers}.`,
|
||||
path: "tools.media.audio.models",
|
||||
fixHint:
|
||||
"Install the matching local model/runtime, or configure an explicit tools.media.audio.models CLI entry.",
|
||||
},
|
||||
];
|
||||
}
|
||||
|
||||
export async function collectGatewayHealthFindings(
|
||||
ctx: Pick<HealthCheckContext, "cfg" | "configPath">,
|
||||
): Promise<readonly HealthFinding[]> {
|
||||
|
||||
@@ -95,6 +95,9 @@ function createDeps(overrides: Partial<CoreHealthCheckDeps> = {}): CoreHealthChe
|
||||
async collectProviderCatalogProjectionFindings() {
|
||||
return [];
|
||||
},
|
||||
async collectLocalAudioAccelerationFindings() {
|
||||
return [];
|
||||
},
|
||||
async collectGatewayHealthFindings() {
|
||||
return [];
|
||||
},
|
||||
@@ -170,6 +173,27 @@ describe("CORE_HEALTH_CHECKS", () => {
|
||||
).toBe(false);
|
||||
});
|
||||
|
||||
it("reports local STT auto-selection diagnostics", async () => {
|
||||
const finding: HealthFinding = {
|
||||
checkId: "core/doctor/local-audio-acceleration",
|
||||
severity: "info",
|
||||
message:
|
||||
"Local STT auto-selection: whisper-cli (capable=metal, observed=unknown); build capability is not runtime observation.",
|
||||
};
|
||||
const check = getCheck(
|
||||
createCoreHealthChecks(
|
||||
createDeps({
|
||||
async collectLocalAudioAccelerationFindings() {
|
||||
return [finding];
|
||||
},
|
||||
}),
|
||||
),
|
||||
"core/doctor/local-audio-acceleration",
|
||||
);
|
||||
|
||||
await expect(check.detect({ mode: "lint", runtime, cfg: {} })).resolves.toEqual([finding]);
|
||||
});
|
||||
|
||||
it("warns when autonomous Skill Workshop capture is enabled but policy hides its tool", async () => {
|
||||
const check = getCheck(
|
||||
createCoreHealthChecks(createDeps()),
|
||||
|
||||
@@ -77,6 +77,7 @@ export type CoreHealthCheckDeps = {
|
||||
readonly collectProviderCatalogProjectionFindings: (
|
||||
ctx: HealthCheckContext,
|
||||
) => Promise<readonly HealthFinding[]>;
|
||||
readonly collectLocalAudioAccelerationFindings: () => Promise<readonly HealthFinding[]>;
|
||||
readonly collectGatewayHealthFindings: (
|
||||
ctx: HealthCheckContext,
|
||||
) => Promise<readonly HealthFinding[]>;
|
||||
@@ -127,6 +128,13 @@ async function collectProviderCatalogProjectionFindingsWithRuntime(
|
||||
return runtime.collectProviderCatalogProjectionFindings(ctx.cfg);
|
||||
}
|
||||
|
||||
async function collectLocalAudioAccelerationFindingsWithRuntime(): Promise<
|
||||
readonly HealthFinding[]
|
||||
> {
|
||||
const runtime = await loadDoctorCoreChecksRuntimeModule();
|
||||
return runtime.collectLocalAudioAccelerationFindings();
|
||||
}
|
||||
|
||||
async function collectGatewayHealthFindingsWithRuntime(
|
||||
ctx: HealthCheckContext,
|
||||
): Promise<readonly HealthFinding[]> {
|
||||
@@ -147,6 +155,7 @@ const defaultCoreHealthCheckDeps: CoreHealthCheckDeps = {
|
||||
collectWorkspaceSuggestionNotes: collectWorkspaceSuggestionNotesWithRuntime,
|
||||
collectRuntimeToolSchemaFindings: collectRuntimeToolSchemaFindingsWithRuntime,
|
||||
collectProviderCatalogProjectionFindings: collectProviderCatalogProjectionFindingsWithRuntime,
|
||||
collectLocalAudioAccelerationFindings: collectLocalAudioAccelerationFindingsWithRuntime,
|
||||
collectGatewayHealthFindings: collectGatewayHealthFindingsWithRuntime,
|
||||
collectGatewayDaemonFindings: collectGatewayDaemonFindingsWithRuntime,
|
||||
};
|
||||
@@ -1137,6 +1146,15 @@ function createConvertedWorkflowChecks(
|
||||
hooksModelCheck,
|
||||
bootstrapSizeCheck,
|
||||
createProviderCatalogProjectionCheck(deps),
|
||||
{
|
||||
id: "core/doctor/local-audio-acceleration",
|
||||
kind: "core",
|
||||
description: "Local STT auto-selection and acceleration evidence are visible.",
|
||||
source: "doctor",
|
||||
async detect() {
|
||||
return await deps.collectLocalAudioAccelerationFindings();
|
||||
},
|
||||
},
|
||||
createRuntimeToolSchemaCheck(deps),
|
||||
createWorkspaceSuggestionsCheck(deps),
|
||||
skillWorkshopToolPolicyCheck,
|
||||
|
||||
@@ -120,6 +120,14 @@ function getRunExecCall(index = 0) {
|
||||
return call;
|
||||
}
|
||||
|
||||
function getRunExecCallForCommand(command: string) {
|
||||
const call = mockedRunExec.mock.calls.find(([calledCommand]) => calledCommand === command);
|
||||
if (!call) {
|
||||
throw new Error(`expected runExec call for ${command}`);
|
||||
}
|
||||
return call;
|
||||
}
|
||||
|
||||
function getRunFfmpegArgs(index = 0) {
|
||||
const [args] = mockedRunFfmpeg.mock.calls[index] ?? [];
|
||||
if (!Array.isArray(args)) {
|
||||
@@ -245,7 +253,10 @@ async function setupAudioAutoDetectCase(stdout?: string): Promise<{
|
||||
}
|
||||
|
||||
function mockWhisperCliTranscript(transcript: string) {
|
||||
mockedRunExec.mockImplementationOnce(async (_command, args) => {
|
||||
mockedRunExec.mockImplementation(async (command, args) => {
|
||||
if (command === "readelf" || command === "otool") {
|
||||
return { stdout: "", stderr: "" };
|
||||
}
|
||||
const outputBaseIndex = args.indexOf("-of");
|
||||
const outputBase = outputBaseIndex >= 0 ? args[outputBaseIndex + 1] : undefined;
|
||||
if (typeof outputBase !== "string") {
|
||||
@@ -869,7 +880,7 @@ describe("applyMediaUnderstanding", () => {
|
||||
);
|
||||
|
||||
expect(ctx.Transcript).toBe("whisper cpp ok");
|
||||
const [command, args, options] = getRunExecCall();
|
||||
const [command, args, options] = getRunExecCallForCommand("whisper-cli");
|
||||
expect(command).toBe("whisper-cli");
|
||||
if (!Array.isArray(args)) {
|
||||
throw new Error("expected whisper-cli args");
|
||||
@@ -877,7 +888,17 @@ describe("applyMediaUnderstanding", () => {
|
||||
expect(args.slice(0, 4)).toEqual(["-m", modelPath, "-otxt", "-of"]);
|
||||
expect(typeof args[4]).toBe("string");
|
||||
expect(String(args[4]).endsWith("sample")).toBe(true);
|
||||
expect(args.slice(5)).toEqual(["-np", "-nt", await fs.realpath(ctx.MediaPath ?? "")]);
|
||||
expect(args.slice(5)).toEqual(["-nt", await fs.realpath(ctx.MediaPath ?? "")]);
|
||||
if (process.platform === "linux") {
|
||||
expect(mockedRunExec.mock.calls).toContainEqual([
|
||||
"readelf",
|
||||
["-d", expect.stringContaining("whisper-cli")],
|
||||
expect.objectContaining({ timeoutMs: 1500 }),
|
||||
]);
|
||||
expect(mockedRunExec.mock.calls.some(([calledCommand]) => calledCommand === "ldd")).toBe(
|
||||
false,
|
||||
);
|
||||
}
|
||||
expectCliRunOptions(options);
|
||||
});
|
||||
|
||||
@@ -935,14 +956,14 @@ describe("applyMediaUnderstanding", () => {
|
||||
expect(String(ffmpegArgs[11])).toContain("telegram-voice.wav");
|
||||
expect(String(ffmpegArgs[11]).endsWith(".part")).toBe(true);
|
||||
|
||||
const [command, args, options] = getRunExecCall();
|
||||
const [command, args, options] = getRunExecCallForCommand("whisper-cli");
|
||||
expect(command).toBe("whisper-cli");
|
||||
if (!Array.isArray(args)) {
|
||||
throw new Error("expected whisper-cli transcode args");
|
||||
}
|
||||
expect(args.slice(0, 4)).toEqual(["-m", modelPath, "-otxt", "-of"]);
|
||||
expect(args.slice(5, 7)).toEqual(["-np", "-nt"]);
|
||||
expect(String(args[7]).endsWith("telegram-voice.wav")).toBe(true);
|
||||
expect(args[5]).toBe("-nt");
|
||||
expect(String(args[6]).endsWith("telegram-voice.wav")).toBe(true);
|
||||
expectCliRunOptions(options);
|
||||
});
|
||||
|
||||
|
||||
@@ -0,0 +1,264 @@
|
||||
import fs from "node:fs/promises";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
import { afterEach, describe, expect, it } from "vitest";
|
||||
import {
|
||||
clearLocalAudioInspectionCacheForTests,
|
||||
inspectLocalAudioSelection,
|
||||
recordLocalAudioBackendObservation,
|
||||
} from "./local-audio.js";
|
||||
|
||||
let tempDirs: string[] = [];
|
||||
|
||||
async function createTempDir(): Promise<string> {
|
||||
const tempDir = await fs.mkdtemp(path.join(os.tmpdir(), "openclaw-local-audio-"));
|
||||
tempDirs.push(tempDir);
|
||||
return tempDir;
|
||||
}
|
||||
|
||||
afterEach(async () => {
|
||||
clearLocalAudioInspectionCacheForTests();
|
||||
await Promise.all(tempDirs.map(async (tempDir) => await fs.rm(tempDir, { recursive: true })));
|
||||
tempDirs = [];
|
||||
});
|
||||
|
||||
describe("local audio selection", () => {
|
||||
it("expands home-directory shorthand in PATH entries", async () => {
|
||||
const tempDir = await createTempDir();
|
||||
const binDir = path.join(tempDir, "bin");
|
||||
const modelPath = path.join(tempDir, "whisper.bin");
|
||||
const commandPath = path.join(binDir, "whisper-cli");
|
||||
await fs.mkdir(binDir);
|
||||
await fs.writeFile(modelPath, "model");
|
||||
await fs.writeFile(commandPath, "#!/bin/sh\n");
|
||||
await fs.chmod(commandPath, 0o755);
|
||||
|
||||
const selection = await inspectLocalAudioSelection({
|
||||
env: {
|
||||
HOME: tempDir,
|
||||
PATH: "~/bin",
|
||||
WHISPER_CPP_MODEL: modelPath,
|
||||
},
|
||||
platform: process.platform,
|
||||
arch: process.arch,
|
||||
inspectLinkedLibraries: async () => null,
|
||||
});
|
||||
|
||||
expect(selection.selected).toMatchObject({
|
||||
id: "whisper-cli",
|
||||
resolvedCommand: commandPath,
|
||||
});
|
||||
});
|
||||
|
||||
it("does not resolve auto-detected commands from empty PATH entries", async () => {
|
||||
const tempDir = await createTempDir();
|
||||
const modelPath = path.join(tempDir, "whisper.bin");
|
||||
await fs.writeFile(modelPath, "model");
|
||||
const checkedPaths: string[] = [];
|
||||
|
||||
const selection = await inspectLocalAudioSelection({
|
||||
env: {
|
||||
PATH: path.delimiter,
|
||||
WHISPER_CPP_MODEL: modelPath,
|
||||
},
|
||||
platform: process.platform,
|
||||
arch: process.arch,
|
||||
checkExecutable: async (filePath) => {
|
||||
checkedPaths.push(filePath);
|
||||
return true;
|
||||
},
|
||||
inspectLinkedLibraries: async () => null,
|
||||
});
|
||||
|
||||
expect(checkedPaths).toEqual([]);
|
||||
expect(selection.candidates.find((candidate) => candidate.id === "whisper-cli")).toMatchObject({
|
||||
available: false,
|
||||
ready: false,
|
||||
});
|
||||
});
|
||||
|
||||
it("does not rank Metal-capable whisper ahead of sherpa until a run observes Metal", async () => {
|
||||
const tempDir = await createTempDir();
|
||||
const modelPath = path.join(tempDir, "whisper.bin");
|
||||
const sherpaDir = path.join(tempDir, "sherpa");
|
||||
await fs.writeFile(modelPath, "model");
|
||||
await fs.mkdir(sherpaDir);
|
||||
await Promise.all(
|
||||
["tokens.txt", "encoder.onnx", "decoder.onnx", "joiner.onnx"].map(async (fileName) => {
|
||||
await fs.writeFile(path.join(sherpaDir, fileName), "model");
|
||||
}),
|
||||
);
|
||||
|
||||
const selection = await inspectLocalAudioSelection({
|
||||
env: {
|
||||
WHISPER_CPP_MODEL: modelPath,
|
||||
SHERPA_ONNX_MODEL_DIR: sherpaDir,
|
||||
},
|
||||
platform: "darwin",
|
||||
arch: "arm64",
|
||||
resolveBinary: async (name) =>
|
||||
name === "whisper-cli"
|
||||
? "/opt/homebrew/bin/whisper-cli"
|
||||
: name === "sherpa-onnx-offline"
|
||||
? "/usr/local/bin/sherpa-onnx-offline"
|
||||
: null,
|
||||
resolveRealpath: async () => "/opt/homebrew/Cellar/whisper-cpp/1.9.1/bin/whisper-cli",
|
||||
inspectLinkedLibraries: async () => null,
|
||||
});
|
||||
|
||||
expect(selection.selected).toMatchObject({
|
||||
id: "sherpa-onnx-offline",
|
||||
requestedBackend: "cpu",
|
||||
});
|
||||
const capableWhisper = selection.candidates.find((candidate) => candidate.id === "whisper-cli");
|
||||
expect(capableWhisper).toMatchObject({ capableBackend: "metal" });
|
||||
expect(capableWhisper).toHaveProperty("observedBackend", undefined);
|
||||
expect(selection.entries.map((entry) => entry.command)).toEqual([
|
||||
"sherpa-onnx-offline",
|
||||
"whisper-cli",
|
||||
]);
|
||||
|
||||
recordLocalAudioBackendObservation({
|
||||
command: "/custom/bin/whisper-cli",
|
||||
args: ["-m", modelPath, "-otxt", "-of", "{{OutputBase}}", "-nt", "{{MediaPath}}"],
|
||||
output: "whisper_backend_init_gpu: using MTL0 backend",
|
||||
});
|
||||
const mismatchedCommandSelection = await inspectLocalAudioSelection({
|
||||
env: {
|
||||
WHISPER_CPP_MODEL: modelPath,
|
||||
SHERPA_ONNX_MODEL_DIR: sherpaDir,
|
||||
},
|
||||
platform: "darwin",
|
||||
arch: "arm64",
|
||||
resolveBinary: async (name) =>
|
||||
name === "whisper-cli"
|
||||
? "/opt/homebrew/bin/whisper-cli"
|
||||
: name === "sherpa-onnx-offline"
|
||||
? "/usr/local/bin/sherpa-onnx-offline"
|
||||
: null,
|
||||
resolveRealpath: async () => "/opt/homebrew/Cellar/whisper-cpp/1.9.1/bin/whisper-cli",
|
||||
inspectLinkedLibraries: async () => null,
|
||||
});
|
||||
expect(mismatchedCommandSelection.selected).toMatchObject({ id: "sherpa-onnx-offline" });
|
||||
|
||||
recordLocalAudioBackendObservation({
|
||||
command: "whisper-cli",
|
||||
args: ["-m", modelPath, "-otxt", "-of", "{{OutputBase}}", "-nt", "{{MediaPath}}"],
|
||||
output: "whisper_backend_init_gpu: using MTL0 backend",
|
||||
});
|
||||
const observedSelection = await inspectLocalAudioSelection({
|
||||
env: {
|
||||
WHISPER_CPP_MODEL: modelPath,
|
||||
SHERPA_ONNX_MODEL_DIR: sherpaDir,
|
||||
},
|
||||
platform: "darwin",
|
||||
arch: "arm64",
|
||||
resolveBinary: async (name) =>
|
||||
name === "whisper-cli"
|
||||
? "/opt/homebrew/bin/whisper-cli"
|
||||
: name === "sherpa-onnx-offline"
|
||||
? "/usr/local/bin/sherpa-onnx-offline"
|
||||
: null,
|
||||
resolveRealpath: async () => "/opt/homebrew/Cellar/whisper-cpp/1.9.1/bin/whisper-cli",
|
||||
inspectLinkedLibraries: async () => null,
|
||||
});
|
||||
expect(observedSelection.selected).toMatchObject({
|
||||
id: "whisper-cli",
|
||||
capableBackend: "metal",
|
||||
observedBackend: "metal",
|
||||
});
|
||||
});
|
||||
|
||||
it("reports Parakeet as MLX-capable without treating capability as observation", async () => {
|
||||
const tempDir = await createTempDir();
|
||||
const whisperModel = path.join(tempDir, "whisper.bin");
|
||||
const sherpaDir = path.join(tempDir, "sherpa");
|
||||
await fs.writeFile(whisperModel, "model");
|
||||
await fs.mkdir(sherpaDir);
|
||||
await Promise.all(
|
||||
["tokens.txt", "encoder.onnx", "decoder.onnx", "joiner.onnx"].map(async (fileName) => {
|
||||
await fs.writeFile(path.join(sherpaDir, fileName), "model");
|
||||
}),
|
||||
);
|
||||
|
||||
const selection = await inspectLocalAudioSelection({
|
||||
env: {
|
||||
WHISPER_CPP_MODEL: whisperModel,
|
||||
SHERPA_ONNX_MODEL_DIR: sherpaDir,
|
||||
},
|
||||
platform: "darwin",
|
||||
arch: "arm64",
|
||||
resolveBinary: async (name) => `/usr/local/bin/${name}`,
|
||||
resolveRealpath: async (filePath) => filePath,
|
||||
inspectLinkedLibraries: async () => null,
|
||||
});
|
||||
|
||||
expect(selection.selected).toMatchObject({
|
||||
id: "sherpa-onnx-offline",
|
||||
requestedBackend: "cpu",
|
||||
});
|
||||
const parakeet = selection.candidates.find((candidate) => candidate.id === "parakeet-mlx");
|
||||
expect(parakeet).toMatchObject({ capableBackend: "mlx" });
|
||||
expect(parakeet).not.toHaveProperty("observedBackend");
|
||||
expect(selection.entries.map((entry) => entry.command)).toEqual([
|
||||
"sherpa-onnx-offline",
|
||||
"whisper-cli",
|
||||
"parakeet-mlx",
|
||||
"whisper",
|
||||
]);
|
||||
});
|
||||
|
||||
it("keeps an unproven whisper runtime behind CPU sherpa", async () => {
|
||||
const tempDir = await createTempDir();
|
||||
const whisperModel = path.join(tempDir, "whisper.bin");
|
||||
const sherpaDir = path.join(tempDir, "sherpa");
|
||||
await fs.writeFile(whisperModel, "model");
|
||||
await fs.mkdir(sherpaDir);
|
||||
await Promise.all(
|
||||
["tokens.txt", "encoder.onnx", "decoder.onnx", "joiner.onnx"].map(async (fileName) => {
|
||||
await fs.writeFile(path.join(sherpaDir, fileName), "model");
|
||||
}),
|
||||
);
|
||||
|
||||
const selection = await inspectLocalAudioSelection({
|
||||
env: {
|
||||
WHISPER_CPP_MODEL: whisperModel,
|
||||
SHERPA_ONNX_MODEL_DIR: sherpaDir,
|
||||
},
|
||||
platform: "linux",
|
||||
arch: "x64",
|
||||
resolveBinary: async (name) =>
|
||||
name === "whisper-cli" || name === "sherpa-onnx-offline" ? `/usr/local/bin/${name}` : null,
|
||||
inspectLinkedLibraries: async () => "libggml-cpu.so",
|
||||
});
|
||||
|
||||
expect(selection.selected).toMatchObject({
|
||||
id: "sherpa-onnx-offline",
|
||||
requestedBackend: "cpu",
|
||||
});
|
||||
expect(selection.entries.map((entry) => entry.command)).toEqual([
|
||||
"sherpa-onnx-offline",
|
||||
"whisper-cli",
|
||||
]);
|
||||
});
|
||||
|
||||
it("reports a dynamically linked CUDA runtime as capable but unobserved", async () => {
|
||||
const tempDir = await createTempDir();
|
||||
const whisperModel = path.join(tempDir, "whisper.bin");
|
||||
await fs.writeFile(whisperModel, "model");
|
||||
|
||||
const selection = await inspectLocalAudioSelection({
|
||||
env: { WHISPER_CPP_MODEL: whisperModel },
|
||||
platform: "linux",
|
||||
arch: "x64",
|
||||
resolveBinary: async (name) => (name === "whisper-cli" ? "/usr/local/bin/whisper-cli" : null),
|
||||
inspectLinkedLibraries: async () => "libggml-cuda.so => /usr/local/lib/libggml-cuda.so",
|
||||
});
|
||||
|
||||
expect(selection.selected).toMatchObject({
|
||||
id: "whisper-cli",
|
||||
capableBackend: "cuda",
|
||||
});
|
||||
expect(selection.selected).toHaveProperty("observedBackend", undefined);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,469 @@
|
||||
import { constants as fsConstants } from "node:fs";
|
||||
import fs from "node:fs/promises";
|
||||
import path from "node:path";
|
||||
import type { MediaUnderstandingModelConfig } from "../config/types.tools.js";
|
||||
import { runExec } from "../process/exec.js";
|
||||
import { fileExists } from "./fs.js";
|
||||
|
||||
export type LocalAudioCandidate = {
|
||||
id: "parakeet-mlx" | "whisper-cli" | "sherpa-onnx-offline" | "whisper";
|
||||
command: string;
|
||||
resolvedCommand?: string;
|
||||
available: boolean;
|
||||
ready: boolean;
|
||||
capableBackend?: "cuda" | "metal" | "mlx";
|
||||
requestedBackend?: string;
|
||||
observedBackend?: "cpu" | "cuda" | "metal";
|
||||
evidence: string;
|
||||
selected: boolean;
|
||||
reason?: string;
|
||||
entry?: MediaUnderstandingModelConfig;
|
||||
};
|
||||
|
||||
export type LocalAudioSelection = {
|
||||
candidates: LocalAudioCandidate[];
|
||||
entries: MediaUnderstandingModelConfig[];
|
||||
selected?: LocalAudioCandidate;
|
||||
};
|
||||
|
||||
type InspectionOptions = {
|
||||
env?: NodeJS.ProcessEnv;
|
||||
platform?: NodeJS.Platform;
|
||||
arch?: string;
|
||||
resolveBinary?: (name: string, env: NodeJS.ProcessEnv) => Promise<string | null>;
|
||||
checkExecutable?: (filePath: string, platform: NodeJS.Platform) => Promise<boolean>;
|
||||
resolveRealpath?: (filePath: string) => Promise<string>;
|
||||
inspectLinkedLibraries?: (filePath: string, platform: NodeJS.Platform) => Promise<string | null>;
|
||||
};
|
||||
|
||||
const binaryCache = new Map<string, Promise<string | null>>();
|
||||
const libraryCache = new Map<string, Promise<string | null>>();
|
||||
const observedBackendCache = new Map<string, "cpu" | "cuda" | "metal">();
|
||||
|
||||
export function clearLocalAudioInspectionCacheForTests(): void {
|
||||
binaryCache.clear();
|
||||
libraryCache.clear();
|
||||
observedBackendCache.clear();
|
||||
}
|
||||
|
||||
function commandId(command: string): string {
|
||||
return path.basename(command.trim()).toLowerCase();
|
||||
}
|
||||
|
||||
export function resolveRequestedLocalAudioBackend(params: {
|
||||
command: string;
|
||||
args: readonly string[];
|
||||
}): string | undefined {
|
||||
const command = commandId(params.command);
|
||||
if (command === "sherpa-onnx-offline") {
|
||||
const providerIndex = params.args.findIndex((arg) => arg === "--provider");
|
||||
return (
|
||||
(providerIndex >= 0 ? params.args[providerIndex + 1] : undefined) ??
|
||||
params.args.find((arg) => arg.startsWith("--provider="))?.slice("--provider=".length) ??
|
||||
"cpu"
|
||||
)
|
||||
.trim()
|
||||
.toLowerCase();
|
||||
}
|
||||
if (command === "whisper-cli") {
|
||||
if (params.args.includes("-ng") || params.args.includes("--no-gpu")) {
|
||||
return "cpu";
|
||||
}
|
||||
const deviceIndex = params.args.findIndex((arg) => arg === "-dev" || arg === "--device");
|
||||
const device =
|
||||
(deviceIndex >= 0 ? params.args[deviceIndex + 1] : undefined) ??
|
||||
params.args.find((arg) => arg.startsWith("--device="))?.slice("--device=".length);
|
||||
return device?.trim() ? `device:${device.trim()}` : undefined;
|
||||
}
|
||||
return undefined;
|
||||
}
|
||||
|
||||
function observationKey(params: { command: string; args: readonly string[] }): string {
|
||||
return `${params.command.trim()}\0${resolveRequestedLocalAudioBackend(params) ?? "default"}`;
|
||||
}
|
||||
|
||||
export function recordLocalAudioBackendObservation(params: {
|
||||
command: string;
|
||||
args: readonly string[];
|
||||
output: string;
|
||||
}): "cpu" | "cuda" | "metal" | undefined {
|
||||
if (commandId(params.command) !== "whisper-cli") {
|
||||
return undefined;
|
||||
}
|
||||
const backend = /using\s+(?:MTL\d+|Metal)\s+backend/i.test(params.output)
|
||||
? "metal"
|
||||
: /using\s+CUDA\d*\s+backend/i.test(params.output)
|
||||
? "cuda"
|
||||
: /using\s+CPU\s+backend|no GPU found/i.test(params.output)
|
||||
? "cpu"
|
||||
: undefined;
|
||||
if (backend) {
|
||||
observedBackendCache.set(observationKey(params), backend);
|
||||
}
|
||||
return backend;
|
||||
}
|
||||
|
||||
function getObservedBackend(params: {
|
||||
command: string;
|
||||
args: readonly string[];
|
||||
}): "cpu" | "cuda" | "metal" | undefined {
|
||||
return observedBackendCache.get(observationKey(params));
|
||||
}
|
||||
|
||||
async function isExecutable(filePath: string, platform: NodeJS.Platform): Promise<boolean> {
|
||||
try {
|
||||
const stat = await fs.stat(filePath);
|
||||
if (!stat.isFile()) {
|
||||
return false;
|
||||
}
|
||||
if (platform !== "win32") {
|
||||
await fs.access(filePath, fsConstants.X_OK);
|
||||
}
|
||||
return true;
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
function binaryNames(name: string, platform: NodeJS.Platform, env: NodeJS.ProcessEnv): string[] {
|
||||
if (platform !== "win32" || path.extname(name)) {
|
||||
return [name];
|
||||
}
|
||||
const extensions = (env.PATHEXT ?? ".EXE;.CMD;.BAT;.COM")
|
||||
.split(";")
|
||||
.map((extension) => extension.trim())
|
||||
.filter(Boolean);
|
||||
return [name, ...extensions.map((extension) => `${name}${extension}`)];
|
||||
}
|
||||
|
||||
function expandHomeDir(value: string, env: NodeJS.ProcessEnv): string {
|
||||
const trimmed = value.trim().replace(/^"(.*)"$/, "$1");
|
||||
if (trimmed === "~") {
|
||||
return env.HOME ?? trimmed;
|
||||
}
|
||||
if (trimmed.startsWith("~/") || trimmed.startsWith("~\\")) {
|
||||
return env.HOME ? path.join(env.HOME, trimmed.slice(2)) : trimmed;
|
||||
}
|
||||
return trimmed;
|
||||
}
|
||||
|
||||
async function findBinary(
|
||||
name: string,
|
||||
env: NodeJS.ProcessEnv,
|
||||
platform: NodeJS.Platform,
|
||||
checkExecutable: (filePath: string, platform: NodeJS.Platform) => Promise<boolean> = isExecutable,
|
||||
): Promise<string | null> {
|
||||
const key = `${platform}\0${env.PATH ?? ""}\0${env.PATHEXT ?? ""}\0${name}`;
|
||||
const cached = binaryCache.get(key);
|
||||
if (cached) {
|
||||
return await cached;
|
||||
}
|
||||
const lookup = (async () => {
|
||||
const direct = name.trim();
|
||||
const candidates = binaryNames(direct, platform, env);
|
||||
if (direct.includes("/") || direct.includes("\\")) {
|
||||
for (const candidate of candidates) {
|
||||
const expanded =
|
||||
candidate === "~" || candidate.startsWith("~/") || candidate.startsWith("~\\")
|
||||
? path.join(env.HOME ?? "~", candidate.slice(candidate === "~" ? 1 : 2))
|
||||
: candidate;
|
||||
if (await checkExecutable(expanded, platform)) {
|
||||
return expanded;
|
||||
}
|
||||
}
|
||||
return null;
|
||||
}
|
||||
for (const directory of (env.PATH ?? "").split(path.delimiter)) {
|
||||
const expandedDirectory = expandHomeDir(directory, env);
|
||||
if (!expandedDirectory) {
|
||||
continue;
|
||||
}
|
||||
for (const candidate of candidates) {
|
||||
const fullPath = path.join(expandedDirectory, candidate);
|
||||
if (await checkExecutable(fullPath, platform)) {
|
||||
return fullPath;
|
||||
}
|
||||
}
|
||||
}
|
||||
return null;
|
||||
})();
|
||||
binaryCache.set(key, lookup);
|
||||
return await lookup;
|
||||
}
|
||||
|
||||
async function inspectLinkedLibraries(
|
||||
filePath: string,
|
||||
platform: NodeJS.Platform,
|
||||
): Promise<string | null> {
|
||||
const key = `${platform}\0${filePath}`;
|
||||
const cached = libraryCache.get(key);
|
||||
if (cached) {
|
||||
return await cached;
|
||||
}
|
||||
const inspection = (async () => {
|
||||
const command = platform === "darwin" ? "otool" : platform === "linux" ? "readelf" : null;
|
||||
if (!command) {
|
||||
return null;
|
||||
}
|
||||
try {
|
||||
const args = platform === "darwin" ? ["-L", filePath] : ["-d", filePath];
|
||||
const result = await runExec(command, args, { timeoutMs: 1500 });
|
||||
return `${result.stdout}\n${result.stderr ?? ""}`;
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
})();
|
||||
libraryCache.set(key, inspection);
|
||||
return await inspection;
|
||||
}
|
||||
|
||||
async function inspectWhisperBackend(params: {
|
||||
command: string;
|
||||
platform: NodeJS.Platform;
|
||||
arch: string;
|
||||
realpath: (filePath: string) => Promise<string>;
|
||||
libraries: (filePath: string, platform: NodeJS.Platform) => Promise<string | null>;
|
||||
}): Promise<Pick<LocalAudioCandidate, "capableBackend" | "evidence">> {
|
||||
const libraries = await params.libraries(params.command, params.platform);
|
||||
if (/(?:ggml[-_]?cuda|libcuda|libcudart)/i.test(libraries ?? "")) {
|
||||
return {
|
||||
capableBackend: "cuda",
|
||||
evidence: "whisper-cli links a CUDA ggml runtime",
|
||||
};
|
||||
}
|
||||
if (params.platform === "darwin" && params.arch === "arm64") {
|
||||
const realCommand = await params.realpath(params.command).catch(() => params.command);
|
||||
if (
|
||||
/(?:ggml[-_]?metal|Metal\.framework)/i.test(libraries ?? "") ||
|
||||
/\/Cellar\/whisper-cpp\/[^/]+\/bin\/whisper-cli$/.test(realCommand)
|
||||
) {
|
||||
return {
|
||||
capableBackend: "metal",
|
||||
evidence: "Apple Silicon Homebrew whisper-cpp runtime with Metal support",
|
||||
};
|
||||
}
|
||||
}
|
||||
return {
|
||||
evidence: "whisper-cli backend cannot be proven without loading a model",
|
||||
};
|
||||
}
|
||||
|
||||
function rank(candidate: LocalAudioCandidate): number {
|
||||
if (
|
||||
candidate.id === "whisper-cli" &&
|
||||
(candidate.observedBackend === "metal" || candidate.observedBackend === "cuda")
|
||||
) {
|
||||
return 0;
|
||||
}
|
||||
if (candidate.id === "sherpa-onnx-offline") {
|
||||
return 1;
|
||||
}
|
||||
if (candidate.id === "whisper-cli") {
|
||||
return 2;
|
||||
}
|
||||
return candidate.id === "parakeet-mlx" ? 3 : 4;
|
||||
}
|
||||
|
||||
export async function inspectLocalAudioSelection(
|
||||
options: InspectionOptions = {},
|
||||
): Promise<LocalAudioSelection> {
|
||||
const env = options.env ?? process.env;
|
||||
const platform = options.platform ?? process.platform;
|
||||
const arch = options.arch ?? process.arch;
|
||||
const resolveBinary = async (name: string) =>
|
||||
options.resolveBinary
|
||||
? await options.resolveBinary(name, env)
|
||||
: await findBinary(name, env, platform, options.checkExecutable);
|
||||
const [parakeetCommand, whisperCommand, sherpaCommand, pythonCommand] = await Promise.all(
|
||||
["parakeet-mlx", "whisper-cli", "sherpa-onnx-offline", "whisper"].map(resolveBinary),
|
||||
);
|
||||
|
||||
const envModel = env.WHISPER_CPP_MODEL?.trim();
|
||||
const defaultWhisperModel = "/opt/homebrew/share/whisper-cpp/for-tests-ggml-tiny.bin";
|
||||
const whisperModel = envModel && (await fileExists(envModel)) ? envModel : defaultWhisperModel;
|
||||
const whisperReady = Boolean(whisperCommand) && (await fileExists(whisperModel));
|
||||
const whisperBackend = whisperCommand
|
||||
? await inspectWhisperBackend({
|
||||
command: whisperCommand,
|
||||
platform,
|
||||
arch,
|
||||
realpath: options.resolveRealpath ?? fs.realpath,
|
||||
libraries: options.inspectLinkedLibraries ?? inspectLinkedLibraries,
|
||||
})
|
||||
: {
|
||||
evidence: "whisper-cli command not found",
|
||||
};
|
||||
|
||||
const sherpaDir = env.SHERPA_ONNX_MODEL_DIR?.trim();
|
||||
const sherpaFiles = sherpaDir
|
||||
? ["tokens.txt", "encoder.onnx", "decoder.onnx", "joiner.onnx"].map((file) =>
|
||||
path.join(sherpaDir, file),
|
||||
)
|
||||
: [];
|
||||
const sherpaReady =
|
||||
Boolean(sherpaCommand) &&
|
||||
sherpaFiles.length === 4 &&
|
||||
(await Promise.all(sherpaFiles.map(fileExists))).every(Boolean);
|
||||
const parakeetReady = Boolean(parakeetCommand) && platform === "darwin" && arch === "arm64";
|
||||
const parakeetArgs = [
|
||||
"{{MediaPath}}",
|
||||
"--output-format",
|
||||
"txt",
|
||||
"--output-dir",
|
||||
"{{OutputDir}}",
|
||||
"--output-template",
|
||||
"{filename}",
|
||||
];
|
||||
const whisperArgs = [
|
||||
"-m",
|
||||
whisperModel,
|
||||
"-otxt",
|
||||
"-of",
|
||||
"{{OutputBase}}",
|
||||
"-nt",
|
||||
"{{MediaPath}}",
|
||||
];
|
||||
const sherpaArgs = [
|
||||
`--tokens=${sherpaFiles[0]}`,
|
||||
`--encoder=${sherpaFiles[1]}`,
|
||||
`--decoder=${sherpaFiles[2]}`,
|
||||
`--joiner=${sherpaFiles[3]}`,
|
||||
"{{MediaPath}}",
|
||||
];
|
||||
const pythonArgs = [
|
||||
"--model",
|
||||
"turbo",
|
||||
"--output_format",
|
||||
"txt",
|
||||
"--output_dir",
|
||||
"{{OutputDir}}",
|
||||
"--verbose",
|
||||
"False",
|
||||
"{{MediaPath}}",
|
||||
];
|
||||
|
||||
const candidates: LocalAudioCandidate[] = [
|
||||
{
|
||||
id: "parakeet-mlx",
|
||||
command: "parakeet-mlx",
|
||||
resolvedCommand: parakeetCommand ?? undefined,
|
||||
available: Boolean(parakeetCommand),
|
||||
ready: parakeetReady,
|
||||
capableBackend: parakeetReady ? "mlx" : undefined,
|
||||
evidence: parakeetReady
|
||||
? "parakeet-mlx is an MLX runtime on Apple Silicon; device use is unobserved"
|
||||
: "parakeet-mlx acceleration is only supported on Apple Silicon",
|
||||
selected: false,
|
||||
reason: parakeetCommand
|
||||
? parakeetReady
|
||||
? undefined
|
||||
: "unsupported platform for MLX acceleration"
|
||||
: "command not found",
|
||||
entry: parakeetReady
|
||||
? {
|
||||
type: "cli",
|
||||
command: "parakeet-mlx",
|
||||
args: parakeetArgs,
|
||||
}
|
||||
: undefined,
|
||||
},
|
||||
{
|
||||
id: "whisper-cli",
|
||||
command: "whisper-cli",
|
||||
resolvedCommand: whisperCommand ?? undefined,
|
||||
available: Boolean(whisperCommand),
|
||||
ready: whisperReady,
|
||||
...whisperBackend,
|
||||
requestedBackend: resolveRequestedLocalAudioBackend({
|
||||
command: "whisper-cli",
|
||||
args: whisperArgs,
|
||||
}),
|
||||
observedBackend: getObservedBackend({ command: "whisper-cli", args: whisperArgs }),
|
||||
selected: false,
|
||||
reason: whisperCommand
|
||||
? whisperReady
|
||||
? undefined
|
||||
: "model file not found"
|
||||
: "command not found",
|
||||
entry: whisperReady
|
||||
? {
|
||||
type: "cli",
|
||||
command: "whisper-cli",
|
||||
args: whisperArgs,
|
||||
}
|
||||
: undefined,
|
||||
},
|
||||
{
|
||||
id: "sherpa-onnx-offline",
|
||||
command: "sherpa-onnx-offline",
|
||||
resolvedCommand: sherpaCommand ?? undefined,
|
||||
available: Boolean(sherpaCommand),
|
||||
ready: sherpaReady,
|
||||
requestedBackend: "cpu",
|
||||
evidence: "OpenClaw auto args omit --provider, so sherpa-onnx uses its CPU default",
|
||||
selected: false,
|
||||
reason: sherpaCommand
|
||||
? sherpaReady
|
||||
? undefined
|
||||
: "SHERPA_ONNX_MODEL_DIR is missing required model files"
|
||||
: "command not found",
|
||||
entry: sherpaReady
|
||||
? {
|
||||
type: "cli",
|
||||
command: "sherpa-onnx-offline",
|
||||
args: sherpaArgs,
|
||||
}
|
||||
: undefined,
|
||||
},
|
||||
{
|
||||
id: "whisper",
|
||||
command: "whisper",
|
||||
resolvedCommand: pythonCommand ?? undefined,
|
||||
available: Boolean(pythonCommand),
|
||||
ready: Boolean(pythonCommand),
|
||||
evidence: "Python Whisper chooses its runtime device when the model loads",
|
||||
selected: false,
|
||||
reason: pythonCommand ? undefined : "command not found",
|
||||
entry: pythonCommand
|
||||
? {
|
||||
type: "cli",
|
||||
command: "whisper",
|
||||
args: pythonArgs,
|
||||
}
|
||||
: undefined,
|
||||
},
|
||||
];
|
||||
candidates.sort((left, right) => rank(left) - rank(right));
|
||||
const selected = candidates.find((candidate) => candidate.ready && candidate.entry);
|
||||
if (selected) {
|
||||
selected.selected = true;
|
||||
}
|
||||
return {
|
||||
candidates,
|
||||
entries: candidates.flatMap((candidate) =>
|
||||
candidate.ready && candidate.entry ? [candidate.entry] : [],
|
||||
),
|
||||
selected,
|
||||
};
|
||||
}
|
||||
|
||||
export function formatLocalAudioSelection(selection: LocalAudioSelection): string | null {
|
||||
const selected = selection.selected;
|
||||
if (!selected) {
|
||||
return null;
|
||||
}
|
||||
const describeBackend = (candidate: LocalAudioCandidate) =>
|
||||
[
|
||||
candidate.capableBackend ? `capable=${candidate.capableBackend}` : null,
|
||||
candidate.requestedBackend ? `requested=${candidate.requestedBackend}` : null,
|
||||
`observed=${candidate.observedBackend ?? "unknown"}`,
|
||||
]
|
||||
.filter(Boolean)
|
||||
.join(", ");
|
||||
const fallbacks = selection.candidates
|
||||
.filter((candidate) => candidate.ready && candidate !== selected)
|
||||
.map((candidate) => `${candidate.command} (${describeBackend(candidate)})`);
|
||||
return `${selected.command} (${describeBackend(selected)}); ${selected.evidence}${
|
||||
fallbacks.length > 0 ? `; fallbacks: ${fallbacks.join(", ")}` : ""
|
||||
}`;
|
||||
}
|
||||
@@ -175,6 +175,28 @@ describe("media-understanding CLI audio entry", () => {
|
||||
expect(result?.text).toBe("file transcript");
|
||||
});
|
||||
|
||||
it("records the backend observed during a whisper.cpp model run", async () => {
|
||||
runExecMock.mockImplementationOnce(async (_command, args: string[]) => {
|
||||
await fs.writeFile(`${args[2]}.txt`, "observed transcript\n");
|
||||
return {
|
||||
stdout: "",
|
||||
stderr: "whisper_backend_init_gpu: using MTL0 backend",
|
||||
};
|
||||
});
|
||||
|
||||
const result = await runAudioEntry({
|
||||
command: "whisper-cli",
|
||||
args: ["-otxt", "-of", "{{OutputBase}}", "{{MediaPath}}"],
|
||||
});
|
||||
|
||||
expect(result).toMatchObject({
|
||||
provider: "whisper-cli",
|
||||
model: "whisper-cli",
|
||||
observedBackend: "metal",
|
||||
text: "observed transcript",
|
||||
});
|
||||
});
|
||||
|
||||
it("reads parakeet txt output selected through its upstream environment", async () => {
|
||||
const testCase: TranscriptFileCase = {
|
||||
name: "parakeet environment output",
|
||||
|
||||
@@ -1,7 +1,11 @@
|
||||
// Runner entry guard tests cover malformed decision data formatting without
|
||||
// depending on provider execution.
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { formatDecisionSummary, formatMissingProviderHint } from "./runner.entries.js";
|
||||
import {
|
||||
buildModelDecision,
|
||||
formatDecisionSummary,
|
||||
formatMissingProviderHint,
|
||||
} from "./runner.entries.js";
|
||||
import type { MediaUnderstandingDecision } from "./types.js";
|
||||
|
||||
describe("media-understanding formatDecisionSummary guards", () => {
|
||||
@@ -46,6 +50,42 @@ describe("media-understanding formatDecisionSummary guards", () => {
|
||||
});
|
||||
});
|
||||
|
||||
describe("media-understanding CLI backend decisions", () => {
|
||||
it.each([
|
||||
{
|
||||
command: "sherpa-onnx-offline",
|
||||
args: ["--provider=cuda", "{{MediaPath}}"],
|
||||
requestedBackend: "cuda",
|
||||
},
|
||||
{
|
||||
command: "sherpa-onnx-offline",
|
||||
args: ["{{MediaPath}}"],
|
||||
requestedBackend: "cpu",
|
||||
},
|
||||
{
|
||||
command: "whisper-cli",
|
||||
args: ["--no-gpu", "{{MediaPath}}"],
|
||||
requestedBackend: "cpu",
|
||||
},
|
||||
{
|
||||
command: "whisper-cli",
|
||||
args: ["--device", "GPU0", "{{MediaPath}}"],
|
||||
requestedBackend: "device:GPU0",
|
||||
},
|
||||
])(
|
||||
"reports $command backend request as $requestedBackend",
|
||||
({ command, args, requestedBackend }) => {
|
||||
expect(
|
||||
buildModelDecision({
|
||||
entry: { type: "cli", command, args },
|
||||
entryType: "cli",
|
||||
outcome: "success",
|
||||
}),
|
||||
).toMatchObject({ provider: command, model: command, requestedBackend });
|
||||
},
|
||||
);
|
||||
});
|
||||
|
||||
describe("media-understanding formatMissingProviderHint", () => {
|
||||
it("returns the catalog hint for a provider with mediaUnderstandingProviders contract (groq)", () => {
|
||||
const hint = formatMissingProviderHint("groq");
|
||||
|
||||
@@ -54,6 +54,10 @@ import {
|
||||
} from "./defaults.constants.js";
|
||||
import { normalizeImageDescriptionInput } from "./image-input-normalize.js";
|
||||
import { describeImageWithModel } from "./image-runtime.js";
|
||||
import {
|
||||
recordLocalAudioBackendObservation,
|
||||
resolveRequestedLocalAudioBackend,
|
||||
} from "./local-audio.js";
|
||||
import { resolveOpenAiAudioAuthModelApi } from "./openai-audio-api.js";
|
||||
import { normalizeMediaExecutionProviderId } from "./provider-id.js";
|
||||
import { getMediaUnderstandingProvider, normalizeMediaProviderId } from "./provider-registry.js";
|
||||
@@ -393,10 +397,17 @@ export function buildModelDecision(params: {
|
||||
}): MediaUnderstandingModelDecision {
|
||||
if (params.entryType === "cli") {
|
||||
const command = params.entry.command?.trim();
|
||||
const requestedBackend = command
|
||||
? resolveRequestedLocalAudioBackend({
|
||||
command,
|
||||
args: params.entry.args ?? [],
|
||||
})
|
||||
: undefined;
|
||||
return {
|
||||
type: "cli",
|
||||
provider: command ?? "cli",
|
||||
model: params.entry.model ?? command,
|
||||
...(requestedBackend ? { requestedBackend } : {}),
|
||||
outcome: params.outcome,
|
||||
reason: params.reason,
|
||||
};
|
||||
@@ -653,11 +664,20 @@ export function formatDecisionSummary(decision: MediaUnderstandingDecision): str
|
||||
const chosen = attachments.find((entry) => entry?.chosen)?.chosen;
|
||||
const provider = typeof chosen?.provider === "string" ? chosen.provider.trim() : undefined;
|
||||
const model = typeof chosen?.model === "string" ? chosen.model.trim() : undefined;
|
||||
const modelLabel = provider ? (model ? `${provider}/${model}` : provider) : undefined;
|
||||
const modelLabel = provider
|
||||
? model && model !== provider
|
||||
? `${provider}/${model}`
|
||||
: provider
|
||||
: undefined;
|
||||
const backendLabel = chosen?.observedBackend
|
||||
? ` observed=${chosen.observedBackend}`
|
||||
: chosen?.requestedBackend
|
||||
? ` requested=${chosen.requestedBackend}`
|
||||
: "";
|
||||
const reason = findDecisionReason(decision, decision.outcome === "failed" ? "failed" : undefined);
|
||||
const shortReason = summarizeDecisionReason(reason);
|
||||
const countLabel = total > 0 ? ` (${success}/${total})` : "";
|
||||
const viaLabel = modelLabel ? ` via ${modelLabel}` : "";
|
||||
const viaLabel = modelLabel ? ` via ${modelLabel}${backendLabel}` : "";
|
||||
const reasonLabel = shortReason ? ` reason=${shortReason}` : "";
|
||||
return `${decision.capability}: ${decision.outcome}${countLabel}${viaLabel}${reasonLabel}`;
|
||||
}
|
||||
@@ -1052,11 +1072,26 @@ export async function runCliEntry(params: {
|
||||
if (shouldLogVerbose()) {
|
||||
logVerbose(`Media understanding via CLI: ${argv.join(" ")}`);
|
||||
}
|
||||
const { stdout } = await runExec(argv[0], argv.slice(1), {
|
||||
const { stdout, stderr } = await runExec(argv[0], argv.slice(1), {
|
||||
timeoutMs,
|
||||
maxBuffer: CLI_OUTPUT_MAX_BUFFER,
|
||||
cwd: isAntigravityCliCommand(command) ? path.dirname(mediaPath) : undefined,
|
||||
});
|
||||
const requestedBackend =
|
||||
capability === "audio"
|
||||
? resolveRequestedLocalAudioBackend({
|
||||
command,
|
||||
args: argv.slice(1),
|
||||
})
|
||||
: undefined;
|
||||
const observedBackend =
|
||||
capability === "audio"
|
||||
? recordLocalAudioBackendObservation({
|
||||
command,
|
||||
args: argv.slice(1),
|
||||
output: `${stderr ?? ""}\n${stdout}`,
|
||||
})
|
||||
: undefined;
|
||||
const resolved = await resolveCliOutput({
|
||||
command,
|
||||
args: argv.slice(1),
|
||||
@@ -1071,8 +1106,10 @@ export async function runCliEntry(params: {
|
||||
kind: capability === "audio" ? "audio.transcription" : `${capability}.description`,
|
||||
attachmentIndex: params.attachmentIndex,
|
||||
text,
|
||||
provider: "cli",
|
||||
provider: capability === "audio" ? commandBase(command) : "cli",
|
||||
model: command,
|
||||
...(requestedBackend ? { requestedBackend } : {}),
|
||||
...(observedBackend ? { observedBackend } : {}),
|
||||
};
|
||||
} finally {
|
||||
await fs.rm(outputDir, { recursive: true, force: true }).catch(() => {});
|
||||
|
||||
@@ -43,7 +43,10 @@ import { getDefaultMediaLocalRoots } from "../media/local-roots.js";
|
||||
import { runExec } from "../process/exec.js";
|
||||
import { createLazyRuntimeModule, createLazyRuntimeNamedExport } from "../shared/lazy-runtime.js";
|
||||
import { MediaAttachmentCache, selectAttachments } from "./attachments.js";
|
||||
import { fileExists } from "./fs.js";
|
||||
import {
|
||||
clearLocalAudioInspectionCacheForTests,
|
||||
inspectLocalAudioSelection,
|
||||
} from "./local-audio.js";
|
||||
import { resolveOpenAiAudioAuthModelApi } from "./openai-audio-api.js";
|
||||
import { normalizeMediaExecutionProviderId, normalizeMediaProviderId } from "./provider-id.js";
|
||||
import {
|
||||
@@ -335,6 +338,7 @@ const antigravityCliCache = new Map<string, Promise<string | null>>();
|
||||
export function clearMediaUnderstandingBinaryCacheForTests(): void {
|
||||
binaryCache.clear();
|
||||
antigravityCliCache.clear();
|
||||
clearLocalAudioInspectionCacheForTests();
|
||||
}
|
||||
|
||||
function expandHomeDir(value: string): string {
|
||||
@@ -425,10 +429,6 @@ async function findBinary(name: string): Promise<string | null> {
|
||||
return resolved;
|
||||
}
|
||||
|
||||
async function hasBinary(name: string): Promise<boolean> {
|
||||
return Boolean(await findBinary(name));
|
||||
}
|
||||
|
||||
async function probeAntigravityCliCandidate(command: string): Promise<string | null> {
|
||||
const resolved = await findBinary(command);
|
||||
if (!resolved) {
|
||||
@@ -476,93 +476,6 @@ async function resolveAntigravityCliBinary(): Promise<string | null> {
|
||||
return resolved;
|
||||
}
|
||||
|
||||
async function resolveLocalWhisperCppEntry(): Promise<MediaUnderstandingModelConfig | null> {
|
||||
if (!(await hasBinary("whisper-cli"))) {
|
||||
return null;
|
||||
}
|
||||
const envModel = process.env.WHISPER_CPP_MODEL?.trim();
|
||||
const defaultModel = "/opt/homebrew/share/whisper-cpp/for-tests-ggml-tiny.bin";
|
||||
const modelPath = envModel && (await fileExists(envModel)) ? envModel : defaultModel;
|
||||
if (!(await fileExists(modelPath))) {
|
||||
return null;
|
||||
}
|
||||
return {
|
||||
type: "cli",
|
||||
command: "whisper-cli",
|
||||
args: ["-m", modelPath, "-otxt", "-of", "{{OutputBase}}", "-np", "-nt", "{{MediaPath}}"],
|
||||
};
|
||||
}
|
||||
|
||||
async function resolveLocalWhisperEntry(): Promise<MediaUnderstandingModelConfig | null> {
|
||||
if (!(await hasBinary("whisper"))) {
|
||||
return null;
|
||||
}
|
||||
return {
|
||||
type: "cli",
|
||||
command: "whisper",
|
||||
args: [
|
||||
"--model",
|
||||
"turbo",
|
||||
"--output_format",
|
||||
"txt",
|
||||
"--output_dir",
|
||||
"{{OutputDir}}",
|
||||
"--verbose",
|
||||
"False",
|
||||
"{{MediaPath}}",
|
||||
],
|
||||
};
|
||||
}
|
||||
|
||||
async function resolveSherpaOnnxEntry(): Promise<MediaUnderstandingModelConfig | null> {
|
||||
if (!(await hasBinary("sherpa-onnx-offline"))) {
|
||||
return null;
|
||||
}
|
||||
const modelDir = process.env.SHERPA_ONNX_MODEL_DIR?.trim();
|
||||
if (!modelDir) {
|
||||
return null;
|
||||
}
|
||||
const tokens = path.join(modelDir, "tokens.txt");
|
||||
const encoder = path.join(modelDir, "encoder.onnx");
|
||||
const decoder = path.join(modelDir, "decoder.onnx");
|
||||
const joiner = path.join(modelDir, "joiner.onnx");
|
||||
if (!(await fileExists(tokens))) {
|
||||
return null;
|
||||
}
|
||||
if (!(await fileExists(encoder))) {
|
||||
return null;
|
||||
}
|
||||
if (!(await fileExists(decoder))) {
|
||||
return null;
|
||||
}
|
||||
if (!(await fileExists(joiner))) {
|
||||
return null;
|
||||
}
|
||||
return {
|
||||
type: "cli",
|
||||
command: "sherpa-onnx-offline",
|
||||
args: [
|
||||
`--tokens=${tokens}`,
|
||||
`--encoder=${encoder}`,
|
||||
`--decoder=${decoder}`,
|
||||
`--joiner=${joiner}`,
|
||||
"{{MediaPath}}",
|
||||
],
|
||||
};
|
||||
}
|
||||
|
||||
async function resolveLocalAudioEntry(): Promise<MediaUnderstandingModelConfig | null> {
|
||||
const sherpa = await resolveSherpaOnnxEntry();
|
||||
if (sherpa) {
|
||||
return sherpa;
|
||||
}
|
||||
const whisperCpp = await resolveLocalWhisperCppEntry();
|
||||
if (whisperCpp) {
|
||||
return whisperCpp;
|
||||
}
|
||||
return await resolveLocalWhisperEntry();
|
||||
}
|
||||
|
||||
async function resolveAntigravityCliEntry(
|
||||
capability: MediaUnderstandingCapability,
|
||||
): Promise<MediaUnderstandingModelConfig | null> {
|
||||
@@ -798,9 +711,9 @@ async function resolveAutoEntries(params: {
|
||||
if (keyEntry) {
|
||||
return [keyEntry];
|
||||
}
|
||||
const localAudio = await resolveLocalAudioEntry();
|
||||
if (localAudio) {
|
||||
return [localAudio];
|
||||
const localAudio = await inspectLocalAudioSelection();
|
||||
if (localAudio.entries.length > 0) {
|
||||
return localAudio.entries;
|
||||
}
|
||||
}
|
||||
const keys = await resolveKeyEntry(params);
|
||||
@@ -989,6 +902,12 @@ async function runAttachmentEntries(params: {
|
||||
if (result.model) {
|
||||
decision.model = result.model;
|
||||
}
|
||||
if (result.requestedBackend) {
|
||||
decision.requestedBackend = result.requestedBackend;
|
||||
}
|
||||
if (result.observedBackend) {
|
||||
decision.observedBackend = result.observedBackend;
|
||||
}
|
||||
attempts.push(decision);
|
||||
return { output: result, attempts };
|
||||
}
|
||||
|
||||
@@ -29,6 +29,8 @@ export type MediaUnderstandingOutput = {
|
||||
text: string;
|
||||
provider: string;
|
||||
model?: string;
|
||||
requestedBackend?: string;
|
||||
observedBackend?: string;
|
||||
};
|
||||
|
||||
type MediaUnderstandingDecisionOutcome =
|
||||
@@ -42,6 +44,8 @@ type MediaUnderstandingDecisionOutcome =
|
||||
export type MediaUnderstandingModelDecision = {
|
||||
provider?: string;
|
||||
model?: string;
|
||||
requestedBackend?: string;
|
||||
observedBackend?: string;
|
||||
type: "provider" | "cli";
|
||||
outcome: "success" | "skipped" | "failed";
|
||||
reason?: string;
|
||||
|
||||
@@ -422,8 +422,19 @@ const formatMediaUnderstandingLine = (decisions?: ReadonlyArray<MediaUnderstandi
|
||||
const chosen = decision.attachments.find((entry) => entry.chosen)?.chosen;
|
||||
const provider = chosen?.provider?.trim();
|
||||
const model = chosen?.model?.trim();
|
||||
const modelLabel = provider ? (model ? `${provider}/${model}` : provider) : null;
|
||||
return `${decision.capability}${countLabel} ok${modelLabel ? ` (${modelLabel})` : ""}`;
|
||||
const modelLabel = provider
|
||||
? model && model !== provider
|
||||
? `${provider}/${model}`
|
||||
: provider
|
||||
: null;
|
||||
const backendLabel = chosen?.observedBackend
|
||||
? ` observed=${chosen.observedBackend}`
|
||||
: chosen?.requestedBackend
|
||||
? ` requested=${chosen.requestedBackend}`
|
||||
: "";
|
||||
return `${decision.capability}${countLabel} ok${
|
||||
modelLabel ? ` (${modelLabel}${backendLabel})` : ""
|
||||
}`;
|
||||
}
|
||||
if (decision.outcome === "no-attachment") {
|
||||
return `${decision.capability} none`;
|
||||
|
||||
Reference in New Issue
Block a user