ewilliams28/openclaw
1
0
Fork 0
mirror of https://github.com/openclaw/openclaw.git synced 2026-08-12 21:53:00 -06:00
Code Issues Packages Projects Releases Wiki Activity

feat(meetings): enable Linux Chrome talk-back (#118451)

Browse Source 
* feat(meetings): add Linux audio routing

* chore(plugin-sdk): refresh meeting audio API baseline
This commit is contained in:
Peter Steinberger
2026-08-09 02:13:17 -07:00
committed by GitHub
parent c708b41af4
commit 7d4066639e
73 changed files with 2699 additions and 661 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/.generated/plugin-sdk-api-baseline.sha256
+1 -1
View File
@@ -89,7 +89,7 @@ f74d7295fe716aa140aa0bc9300d6259d71dab826de0808fca6bb02592bf5d6e module/media-m
6a52f93107335f88751704352cc01e62add06f854a5b7d765e2a5ee87c0313b6 module/media-store
bb9f787cd2362268894393f5cf8d16ea582609cb1bb27fbdfc8ea97a76cea214 module/media-understanding
7fc60001aee2f0f95caa7eebe4e8dc74d3a4dcac71f16ce63447d73226f9c683 module/media-understanding-runtime
815f77ed6b11ba4d34e45e2f2c6c4aa388c40f64f69a050bcc74eff4523e7849 module/meeting-runtime
daa4f31dbdf3bcd7866e1f625928c66d898af4aca5287c05b53bf79387160405 module/meeting-runtime
327414d8448f9bbac5d6770f67be92bc7e343415779476611b61acf6e842b0ad module/memory-core-host-engine-foundation
c73e056b3c97d04cc2c975a054ed1f8ee17a52031325027422f0f09dbeb36a22 module/memory-host-core
1efa0aadc4261d1c6073058cbf3dcc9fa681424819bdd14333e19b249bbc4b18 module/messaging-targets
docs/plugins/google-meet.md
+35 -22
View File
@@ -18,7 +18,7 @@ The `google-meet` plugin joins explicit Meet URLs on behalf of an OpenClaw agent
## Quick start
Install the plugin and local audio dependencies, then set a realtime provider key. OpenAI is the default transcription provider for `agent` mode; Google Gemini Live is available as the `bidi`-mode voice provider:
Install the plugin and the native audio dependencies for the Chrome host, then set a realtime provider key. OpenAI is the default transcription provider for `agent` mode; Google Gemini Live is available as the `bidi`-mode voice provider. On macOS:
```bash
openclaw plugins install npm:@openclaw/google-meet
@@ -41,6 +41,17 @@ system_profiler SPAudioDataType | grep -i BlackHole
command -v sox
```
On a Linux desktop with PipeWire-Pulse:
```bash
sudo apt install pipewire-audio pulseaudio-utils # Debian/Ubuntu
systemctl --user --now enable pipewire pipewire-pulse wireplumber
pactl info
command -v pactl pacat parec
```
OpenClaw provisions an `OpenClaw Meeting Audio` null sink and matching source in that desktop user's audio session. Run the Gateway or paired node as the same user that runs Chrome.
The plugin is enabled by default after installation. Add an entry only to customize it:
```json5
@@ -64,7 +75,7 @@ openclaw googlemeet setup
openclaw googlemeet join https://meet.google.com/abc-defg-hij
```
`setup` output is agent-readable and mode/transport-aware: it reports Chrome profile, node pinning, and, for realtime Chrome joins, the BlackHole/SoX audio bridge and the delayed-intro check. Observe-only joins skip realtime prerequisites:
`setup` output is agent-readable and mode/transport-aware: it reports Chrome profile, node pinning, and, for realtime Chrome joins, the native virtual-audio backend and delayed-intro check. Observe-only joins skip realtime prerequisites:
```bash
openclaw googlemeet setup --transport chrome-node --mode transcribe
@@ -87,7 +98,7 @@ Or let an agent join through the `google_meet` tool:
}
```
On non-macOS Gateway hosts, `google_meet` stays visible for artifact, calendar, setup, transcribe, Twilio, and `chrome-node` actions, but local Chrome talk-back (`transport: "chrome"` with `mode: "agent"` or `"bidi"`) is blocked before it reaches the audio bridge, because that path currently depends on macOS `BlackHole 2ch`. Use `mode: "transcribe"`, Twilio dial-in, or a macOS `chrome-node` host instead.
Local Chrome talk-back supports macOS with `BlackHole 2ch` and SoX, or Linux with PipeWire-Pulse and `pactl`/`pacat`/`parec`. On other operating systems, use `mode: "transcribe"`, Twilio dial-in, or a supported macOS/Linux `chrome-node` host.
### Create a meeting
@@ -121,7 +132,7 @@ If the browser fallback hits a Google login or Meet permission blocker, the tool
### Observe-only join
Set `"mode": "transcribe"` to skip the duplex realtime bridge (no BlackHole/SoX requirement, no talk-back). Transcribe-mode Chrome joins also skip OpenClaw's microphone/camera permission grant and the Meet **Use microphone** path; if Meet shows the audio-choice interstitial, automation tries **Continue without microphone** first. Managed Chrome transports install a best-effort Meet caption observer in every mode so durable notes are available without changing the live agent-consult path. `googlemeet status --json` and `googlemeet doctor` report `captioning`, `captionsEnabledAttempted`, `transcriptLines`, `lastCaptionAt`, `lastCaptionSpeaker`, `lastCaptionText`, and a `recentTranscript` tail.
Set `"mode": "transcribe"` to skip the duplex realtime bridge (no virtual-audio requirement, no talk-back). Transcribe-mode Chrome joins also skip OpenClaw's microphone/camera permission grant and the Meet **Use microphone** path; if Meet shows the audio-choice interstitial, automation tries **Continue without microphone** first. Managed Chrome transports install a best-effort Meet caption observer in every mode so durable notes are available without changing the live agent-consult path. `googlemeet status --json` and `googlemeet doctor` report `captioning`, `captionsEnabledAttempted`, `transcriptLines`, `lastCaptionAt`, `lastCaptionSpeaker`, `lastCaptionText`, and a `recentTranscript` tail.
For the bounded session transcript, read the exact tracked Meet tab:
@@ -149,7 +160,7 @@ It joins in transcribe mode, waits for fresh caption/transcript movement, and re
During talk-back sessions, `google_meet` status reports Chrome/audio bridge health: `inCall`, `manualAction`, `providerConnected`, `realtimeReady`, `audioInputActive`, `audioOutputActive`, last input/output timestamps, byte counters, and bridge-closed state. Managed Chrome sessions only speak the intro/test phrase after health reports `inCall: true`; otherwise `speechReady: false` and the speech attempt is blocked rather than silently no-opping.
Local Chrome joins through the signed-in OpenClaw browser profile and needs `BlackHole 2ch` for the mic/speaker path. A single BlackHole device is enough for a first smoke test but can echo; use separate virtual devices or a Loopback-style graph for clean duplex audio.
Local Chrome joins through the signed-in OpenClaw browser profile and routes its microphone and speaker through the native backend selected by `chrome.audioBackend`. The default shared loopback device is enough for a first smoke test but can echo; use separate virtual devices or a Loopback-style graph for clean duplex audio.
## Local Gateway + Parallels Chrome
@@ -252,18 +263,19 @@ If `chromeNode.node` is omitted, OpenClaw auto-selects only when exactly one con
| -------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `Configured Google Meet node ... is not usable: offline` | The pinned node is known but unavailable. Report the setup blocker; do not silently fall back to another transport unless asked. |
| `No connected Google Meet-capable node` | Install `npm:@openclaw/google-meet` in the VM, run `openclaw plugins enable browser`, start `openclaw node run`, and approve pairing. If Google Meet was explicitly disabled, enable it too. Confirm `gateway.nodes.commands.allow` includes `googlemeet.chrome` and `browser.proxy`. |
| `BlackHole 2ch audio device not found` | Install `blackhole-2ch` on the host being checked and reboot. |
| `BlackHole 2ch audio device not found on the node` | Install `blackhole-2ch` in the VM and reboot the VM. |
| `BlackHole 2ch audio device not found` | On macOS, install `blackhole-2ch` on the host being checked and reboot. |
| `PipeWire-Pulse is unavailable` | On Linux, start the desktop user's `pipewire-pulse` service and install `pulseaudio-utils`; do not run the node as root or outside the Chrome user's audio session. |
| Chrome opens but cannot join | Sign in to the browser profile in the VM, or keep `chrome.guestName` set. Guest auto-join uses OpenClaw browser automation through the node browser proxy; point the node's `browser.defaultProfile` (or a named existing-session profile) at the profile you want. |
| Duplicate Meet tabs | Leave `chrome.reuseExistingTab: true`. OpenClaw activates an existing tab for the same URL, and creation reuses an in-progress `.../new` or Google account prompt tab, before opening another. |
| No audio | Route Meet mic/speaker through the virtual audio path used by OpenClaw; use separate virtual devices or Loopback-style routing for clean duplex audio. |
## Install notes
The Chrome talk-back default uses two external tools that OpenClaw does not bundle or redistribute; install them as host dependencies through Homebrew:
The Chrome talk-back default uses host audio tools that OpenClaw does not bundle or redistribute:
- `sox`: command-line audio utility. The plugin issues explicit CoreAudio device commands for the default 24 kHz PCM16 audio bridge.
- `blackhole-2ch`: macOS virtual audio driver providing the `BlackHole 2ch` device Chrome/Meet route through.
- `pactl`, `pacat`, and `parec`: Linux PulseAudio utilities used against PipeWire-Pulse to provision and stream through `OpenClaw Meeting Audio`.
SoX is licensed `LGPL-2.0-only AND GPL-2.0-only`; BlackHole is GPL-3.0. If you build an installer or appliance that bundles BlackHole with OpenClaw, review BlackHole's upstream licensing or get a separate license from Existential Audio.
@@ -277,14 +289,14 @@ SoX is licensed `LGPL-2.0-only AND GPL-2.0-only`; BlackHole is GPL-3.0. If you b
### Chrome
Opens the Meet URL through OpenClaw browser control and joins as the signed-in OpenClaw browser profile. On macOS, the plugin checks for `BlackHole 2ch` before launch and, if configured, runs an audio bridge health/startup command before opening Chrome. For local Chrome, pick the profile with `browser.defaultProfile`; `chrome.browserProfile` is passed to `chrome-node` hosts instead.
Opens the Meet URL through OpenClaw browser control and joins as the signed-in OpenClaw browser profile. Before launch, the plugin checks or provisions the host's native virtual-audio backend and then runs any configured audio bridge health/startup command. For local Chrome, pick the profile with `browser.defaultProfile`; `chrome.browserProfile` is passed to `chrome-node` hosts instead.
```bash
openclaw googlemeet join https://meet.google.com/abc-defg-hij --transport chrome
openclaw googlemeet join https://meet.google.com/abc-defg-hij --transport chrome-node
```
Chrome mic/speaker audio routes through the local OpenClaw audio bridge. If `BlackHole 2ch` is not installed, the join fails with a setup error instead of joining without an audio path.
Chrome mic/speaker audio routes through the local OpenClaw audio bridge. If the native backend is unavailable, the join fails with a setup error instead of joining without an audio path.
### Twilio
@@ -370,7 +382,7 @@ openclaw googlemeet join https://meet.google.com/abc-defg-hij \
## OAuth and preflight
OAuth is optional for creating a Meet link, because `googlemeet create` can fall back to browser automation. Configure OAuth for official API create, space resolution, or Meet Media API preflight. Chrome/Chrome-node joins never depend on OAuth; they use a signed-in Chrome profile, BlackHole/SoX, and (for `chrome-node`) a connected node either way.
OAuth is optional for creating a Meet link, because `googlemeet create` can fall back to browser automation. Configure OAuth for official API create, space resolution, or Meet Media API preflight. Chrome/Chrome-node joins never depend on OAuth; they use a signed-in Chrome profile, the host's native virtual-audio backend, and (for `chrome-node`) a connected node either way.
### Create Google credentials
@@ -684,15 +696,15 @@ The common Chrome agent path only needs the plugin enabled, BlackHole, SoX, a re
| `defaultMode` | `"agent"` | `"realtime"` is accepted as a legacy alias for `"agent"`; new callers should say `"agent"` |
| `chromeNode.node` | unset | Node id/name/IP for `chrome-node`; required when more than one capable node may be connected |
| `chrome.launch` | `true` | Launch Chrome for the join; set `false` only when reusing an already-open session |
| `chrome.audioBackend` | `"blackhole-2ch"` | |
| `chrome.audioBackend` | `"auto"` | Selects `blackhole-2ch` on macOS or `pipewire-pulse` on Linux; set an explicit backend when a paired Chrome node uses a different OS than the Gateway |
| `chrome.guestName` | `"OpenClaw Agent"` | Shown on the signed-out Meet guest screen |
| `chrome.autoJoin` | `true` | Best-effort guest-name fill and Join Now click on `chrome-node` |
| `chrome.reuseExistingTab` | `true` | Activates an existing Meet tab instead of opening duplicates |
| `chrome.waitForInCallMs` | `20000` | Wait for the Meet tab to report in-call before the talk-back intro fires |
| `chrome.audioFormat` | `"pcm16-24khz"` | Command-pair audio format; `"g711-ulaw-8khz"` is only for legacy/custom command pairs that emit telephony audio |
| `chrome.audioBufferBytes` | `4096` | SoX processing buffer for generated command-pair audio commands (half SoX's default 8192-byte buffer, lowering pipe latency); values are clamped to a minimum of 17 bytes |
| `chrome.audioInputCommand` | generated SoX command | Reads from CoreAudio `BlackHole 2ch`, writes audio in `chrome.audioFormat` |
| `chrome.audioOutputCommand` | generated SoX command | Reads audio in `chrome.audioFormat`, writes to CoreAudio `BlackHole 2ch` |
| `chrome.audioBufferBytes` | `4096` | Processing buffer used to derive generated command latency; values are clamped to a minimum of 17 bytes |
| `chrome.audioInputCommand` | generated native command | SoX/CoreAudio on macOS; `parec` from the OpenClaw PipeWire-Pulse source on Linux |
| `chrome.audioOutputCommand` | generated native command | SoX/CoreAudio on macOS; `pacat` into the OpenClaw PipeWire-Pulse sink on Linux |
| `chrome.bargeInInputCommand` | unset | Optional local microphone command writing signed 16-bit little-endian mono PCM for human barge-in detection during assistant playback; applies to the Gateway-hosted command-pair bridge |
| `chrome.bargeInRmsThreshold` | `650` | RMS level counted as human interruption |
| `chrome.bargeInPeakThreshold` | `2500` | Peak level counted as human interruption |
@@ -917,8 +929,9 @@ Speaking on demand:
| `speechReady` / `speechBlockedReason` / `speechBlockedMessage` | Whether managed Chrome speech is allowed now; `speechReady: false` means OpenClaw did not send the intro/test phrase |
| `providerConnected` / `realtimeReady` | Realtime voice bridge state |
| `lastInputAt` / `lastOutputAt` | Last audio seen from/sent to the bridge |
| `audioOutputRouted` / `audioOutputDeviceLabel` | Whether the Meet tab's media output was actively routed to the bridge's BlackHole device |
| `lastOutputLoopbackAt` / `outputLoopbackSignalBytes` | Fresh output whose waveform fingerprint was correlated on the BlackHole microphone capture path |
| `audioInputRouted` / `audioInputDeviceLabel` | Whether Meet's microphone is the verified native virtual-audio input |
| `audioOutputRouted` / `audioOutputDeviceLabel` | Whether the Meet tab's media output was actively routed to the native virtual-audio backend |
| `lastOutputLoopbackAt` / `outputLoopbackSignalBytes` | Fresh output whose waveform fingerprint was correlated on the virtual microphone capture path |
| `lastOutputLoopbackCorrelation` | Correlation score tying the captured signal to the current assistant-output generation |
| `outputGeneration` / `verifiedOutputGeneration` | Monotonic ids; equality means the current output, rather than an older utterance, passed loopback proof |
| `lastOutputLoopbackRms` / `lastOutputLoopbackPeak` | Audio-energy diagnostics for the latest verified loopback capture chunk |
@@ -1021,7 +1034,7 @@ openclaw plugins list | grep google-meet
openclaw googlemeet setup
```
On non-macOS Gateway hosts, `google_meet` stays visible, but local Chrome talk-back actions are blocked before they hit the audio bridge. Use `mode: "transcribe"`, Twilio dial-in, or a macOS `chrome-node` host instead of the default local Chrome agent path.
On Linux, local Chrome talk-back requires PipeWire-Pulse in the Chrome desktop user's session plus `pactl`, `pacat`, and `parec`. On unsupported operating systems, use `mode: "transcribe"`, Twilio dial-in, or a supported macOS/Linux `chrome-node` host.
### No connected Google Meet-capable node
@@ -1101,7 +1114,7 @@ Use `mode: "agent"` for the STT -> OpenClaw agent -> TTS path, `mode: "bidi"` fo
`googlemeet test-speech` always checks the realtime path and reports whether bridge output bytes were observed for that invocation. If `speechOutputVerified` is false and `speechOutputTimedOut` is true, the realtime provider may have accepted the utterance but OpenClaw did not see new output bytes reach the Chrome audio bridge.
Also verify: a realtime provider key (`OPENAI_API_KEY` or `GEMINI_API_KEY`) is available on the Gateway host; `BlackHole 2ch` is visible on the Chrome host; `sox` exists there; Meet mic/speaker are routed through the virtual audio path (`doctor` should show `meet output routed: yes` for local Chrome realtime joins).
Also verify: a realtime provider key (`OPENAI_API_KEY` or `GEMINI_API_KEY`) is available on the Gateway host; the native audio backend is ready on the Chrome host; and Meet mic/speaker are routed through the virtual audio path (`doctor` should show both input and output routed for local Chrome realtime joins).
`googlemeet doctor [session-id]` prints session, node, in-call state, manual action reason, realtime provider connection, `realtimeReady`, audio input/output activity, last audio timestamps, byte counters, and browser URL. Use `googlemeet status [session-id] --json` for raw JSON, and `googlemeet doctor --oauth` (add `--meeting` or `--create-space`) to verify OAuth refresh without exposing tokens.
@@ -1213,14 +1226,14 @@ If webhooks do not arrive, debug the Voice Call plugin first: the provider must
Google Meet's official media API is receive-oriented, so speaking into a call still needs a participant path. This plugin keeps that boundary visible: Chrome handles browser participation and local audio routing; Twilio handles phone dial-in participation.
Chrome talk-back modes need `BlackHole 2ch` plus either:
Chrome talk-back modes need a supported native virtual-audio backend plus either:
- `chrome.audioInputCommand` plus `chrome.audioOutputCommand`: OpenClaw owns the bridge and pipes audio in `chrome.audioFormat` between those commands and the selected provider. `agent` mode uses realtime transcription plus regular TTS; `bidi` mode uses the realtime voice provider. The default path is 24 kHz PCM16 with `chrome.audioBufferBytes: 4096`; 8 kHz G.711 mu-law remains available for legacy command pairs.
- `chrome.audioBridgeCommand`: an external bridge command owns the whole local audio path and must exit after starting or validating its daemon. Valid only for `bidi`, because `agent` mode needs direct command-pair access for TTS.
With the command-pair Chrome bridge, `chrome.bargeInInputCommand` can listen to a separate local microphone and clear assistant playback when a human starts talking, keeping human speech ahead of assistant output even while the shared BlackHole loopback input is temporarily suppressed during assistant playback. Like `chrome.audioInputCommand`/`chrome.audioOutputCommand`, it is an operator-configured local command: use an explicit trusted command path or argument list, never a script from an untrusted location.
With the command-pair Chrome bridge, `chrome.bargeInInputCommand` can listen to a separate local microphone and clear assistant playback when a human starts talking, keeping human speech ahead of assistant output even while the shared virtual loopback input is temporarily suppressed during assistant playback. Like `chrome.audioInputCommand`/`chrome.audioOutputCommand`, it is an operator-configured local command: use an explicit trusted command path or argument list, never a script from an untrusted location.
For clean duplex audio, route Meet output and Meet microphone through separate virtual devices or a Loopback-style virtual device graph; a single shared BlackHole device can echo other participants back into the call.
For clean duplex audio, route Meet output and Meet microphone through separate virtual devices or a Loopback-style virtual device graph; the default shared loopback device can echo other participants back into the call.
`googlemeet speak` triggers the active talk-back audio bridge for a Chrome session; `googlemeet leave` stops it (and, for Twilio sessions delegated through Voice Call, hangs up the underlying call). Use `googlemeet end-active-conference` to also close the active Google Meet conference for an API-managed space.
docs/plugins/meeting-plugins.md
+20 -8
View File
@@ -3,7 +3,7 @@ summary: "Choose and configure Google Meet, Microsoft Teams, or Zoom meeting par
read_when:
- You want an OpenClaw agent to join a video meeting
- You are choosing between the Google Meet, Microsoft Teams meetings, and Zoom meetings plugins
- You need the shared Chrome, BlackHole, SoX, or meeting-mode setup
- You need the shared Chrome, virtual-audio, or meeting-mode setup
title: "Meeting plugins"
---
@@ -25,11 +25,11 @@ Choose Google Meet when you need meeting creation, Google API artifacts, or a Tw
The three plugins share the same modes:
| Mode | Behavior | Audio requirements |
| ------------ | ----------------------------------------------------------------------------------------------------- | ------------------------------------------------------- |
| `agent` | Realtime transcription goes to the configured OpenClaw agent; regular OpenClaw TTS speaks the reply. | Chrome talk-back requires the BlackHole and SoX bridge. |
| `bidi` | A realtime voice model listens and replies directly. | Chrome talk-back requires the BlackHole and SoX bridge. |
| `transcribe` | Joins observe-only and exposes a bounded live-caption transcript when the platform provides captions. | No BlackHole or SoX talk-back bridge. |
| Mode | Behavior | Audio requirements |
| ------------ | ----------------------------------------------------------------------------------------------------- | ------------------------------------------------------------ |
| `agent` | Realtime transcription goes to the configured OpenClaw agent; regular OpenClaw TTS speaks the reply. | Chrome talk-back requires a supported virtual-audio backend. |
| `bidi` | A realtime voice model listens and replies directly. | Chrome talk-back requires a supported virtual-audio backend. |
| `transcribe` | Joins observe-only and exposes a bounded live-caption transcript when the platform provides captions. | No virtual-audio bridge. |
Use `transcribe` when the agent only needs meeting text. Use `agent` for normal OpenClaw reasoning and tools. Use `bidi` when low-latency direct voice is more important than routing each turn through the regular agent.
@@ -55,7 +55,7 @@ Chrome can run on the Gateway host or on a paired node. A remote Chrome node mus
| Microsoft Teams | `teamsmeetings.chrome` |
| Zoom | `zoommeetings.chrome` |
For `agent` or `bidi` mode through Chrome, run Chrome on macOS and install the shared audio dependencies on that same host:
For `agent` or `bidi` mode through Chrome, install the native audio dependencies on the host that runs Chrome. On macOS:
```bash
brew install blackhole-2ch sox
@@ -64,6 +64,18 @@ system_profiler SPAudioDataType | grep -i BlackHole
command -v sox
```
On a Linux desktop with PipeWire-Pulse, install the PulseAudio command-line tools. OpenClaw creates and reuses an `OpenClaw Meeting Audio` null sink and matching source in the desktop user's audio session:
```bash
# Debian/Ubuntu
sudo apt install pipewire-audio pulseaudio-utils
systemctl --user --now enable pipewire pipewire-pulse wireplumber
pactl info
command -v pactl pacat parec
```
Run the Gateway or paired node as the same desktop user that runs Chrome. A root service or headless service without that user's `XDG_RUNTIME_DIR` cannot access the PipeWire-Pulse socket and fails setup with an actionable error.
The Gateway host still owns the OpenClaw agent and model credentials when Chrome runs on a paired node. Configure a realtime transcription provider and OpenClaw TTS for `agent` mode, or a realtime voice provider for `bidi` mode. The platform guides contain the provider and audio-command options.
## Install or disable plugins
@@ -97,7 +109,7 @@ Restart the Gateway if your plugin-management path does not restart it automatic
Treat any failed setup check as a blocker for that transport and mode. For an observe-only smoke test, select `transcribe` mode and confirm that status reports an in-call session before expecting caption text.
For talk-back smoke tests, verified speech requires more than bytes accepted by the playback command. The shared command-pair bridge correlates a bounded waveform fingerprint from the current output generation with audio returning on the BlackHole microphone capture path; Google Meet, Teams, and Zoom do not report `speechOutputVerified: true` when only the output-byte counter advances or unrelated participant audio is present.
For talk-back smoke tests, verified speech requires more than bytes accepted by the playback command. The shared command-pair bridge correlates a bounded waveform fingerprint from the current output generation with audio returning on the selected virtual microphone capture path; Google Meet, Teams, and Zoom do not report `speechOutputVerified: true` when only the output-byte counter advances or unrelated participant audio is present.
## Handle platform policy prompts
docs/plugins/teams-meetings.md
+10 -3
View File
@@ -2,7 +2,7 @@
summary: "Microsoft Teams meetings plugin: join work or consumer meetings as a Chrome browser guest"
read_when:
- You want an OpenClaw agent to join a Microsoft Teams meeting
- You are configuring Chrome, BlackHole, or SoX for Teams meeting talk-back
- You are configuring Chrome or virtual audio for Teams meeting talk-back
title: "Microsoft Teams meetings plugin"
---
@@ -10,7 +10,7 @@ The `teams-meetings` plugin joins Microsoft Teams links as a guest in the OpenCl
## Setup
Talk-back uses the same local audio prerequisites as the [Google Meet plugin](/plugins/google-meet): macOS, the `BlackHole 2ch` virtual audio device, and SoX.
Talk-back uses the shared [meeting-plugin audio setup](/plugins/meeting-plugins#prepare-chrome-and-audio): `BlackHole 2ch` plus SoX on macOS, or PipeWire-Pulse plus `pactl`/`pacat`/`parec` on Linux.
```bash
openclaw plugins install @openclaw/teams-meetings
@@ -21,6 +21,13 @@ system_profiler SPAudioDataType | grep -i BlackHole
command -v sox
```
On Linux, verify the desktop user's PipeWire-Pulse session instead:
```bash
pactl info
command -v pactl pacat parec
```
The plugin is enabled by default after installation. Add an entry only to customize it, then check setup:
```json5
@@ -45,7 +52,7 @@ openclaw teamsmeetings setup
openclaw teamsmeetings join 'https://teams.microsoft.com/l/meetup-join/...'
```
Use `chromeNode.node` to run Chrome, BlackHole, and SoX on a paired macOS node. The node must allow `teamsmeetings.chrome` and `browser.proxy`.
Use `chromeNode.node` to run Chrome and its native virtual-audio backend on a paired macOS or Linux node. The node must allow `teamsmeetings.chrome` and `browser.proxy`; backend setup and generated commands resolve on that node, not on the Gateway host.
## Modes
docs/plugins/zoom-meetings.md
+10 -3
View File
@@ -2,7 +2,7 @@
summary: "Zoom meetings plugin: join meetings as a Chrome browser guest"
read_when:
- You want an OpenClaw agent to join a Zoom meeting
- You are configuring Chrome, BlackHole, or SoX for Zoom meeting talk-back
- You are configuring Chrome or virtual audio for Zoom meeting talk-back
title: "Zoom meetings plugin"
---
@@ -10,7 +10,7 @@ The `zoom-meetings` plugin joins Zoom meeting links as a guest through the Zoom
## Setup
Talk-back uses the same local audio prerequisites as the [Google Meet plugin](/plugins/google-meet): macOS, the `BlackHole 2ch` virtual audio device, and SoX.
Talk-back uses the shared [meeting-plugin audio setup](/plugins/meeting-plugins#prepare-chrome-and-audio): `BlackHole 2ch` plus SoX on macOS, or PipeWire-Pulse plus `pactl`/`pacat`/`parec` on Linux.
```bash
openclaw plugins install @openclaw/zoom-meetings
@@ -21,6 +21,13 @@ system_profiler SPAudioDataType | grep -i BlackHole
command -v sox
```
On Linux, verify the desktop user's PipeWire-Pulse session instead:
```bash
pactl info
command -v pactl pacat parec
```
The plugin is enabled by default after installation. Add an entry only to customize it, then check setup:
```json5
@@ -45,7 +52,7 @@ openclaw zoommeetings setup
openclaw zoommeetings join 'https://zoom.us/j/1234567890'
```
Use `chromeNode.node` to run Chrome, BlackHole, and SoX on a paired macOS node. The node must allow `zoommeetings.chrome` and `browser.proxy`.
Use `chromeNode.node` to run Chrome and its native virtual-audio backend on a paired macOS or Linux node. The node must allow `zoommeetings.chrome` and `browser.proxy`; backend setup and generated commands resolve on that node, not on the Gateway host.
## Modes
extensions/google-meet/index.test.ts
+48 -28
View File
@@ -1009,7 +1009,7 @@ describe("google-meet plugin", () => {
});
it("defaults to chrome agent mode with safe read-only tools", () => {
const config = resolveGoogleMeetConfig({});
const config = withPlatform("darwin", () => resolveGoogleMeetConfig({}));
expect(config.enabled).toBe(true);
expect(config.defaults).toEqual({});
@@ -1017,7 +1017,9 @@ describe("google-meet plugin", () => {
expect(config.defaultTransport).toBe("chrome");
expect(config.defaultMode).toBe("agent");
expect(config.chrome).toEqual({
audioBackend: "blackhole-2ch",
audioBackend: "auto",
audioBridgeCommand: undefined,
audioBridgeHealthCommand: undefined,
launch: true,
guestName: "OpenClaw Agent",
reuseExistingTab: true,
@@ -1047,6 +1049,7 @@ describe("google-meet plugin", () => {
"-L",
"-",
],
audioInputCommandOverride: undefined,
audioOutputCommand: [
"sox",
"-q",
@@ -1068,9 +1071,12 @@ describe("google-meet plugin", () => {
"coreaudio",
"BlackHole 2ch",
],
audioOutputCommandOverride: undefined,
bargeInRmsThreshold: 650,
bargeInPeakThreshold: 2500,
bargeInCooldownMs: 900,
bargeInInputCommand: undefined,
browserProfile: undefined,
});
expect(config.chromeNode).toEqual({});
expect(config.twilio).toEqual({});
@@ -1343,7 +1349,9 @@ describe("google-meet plugin", () => {
});
it("lets generated Chrome audio commands use a configured SoX buffer", () => {
const config = resolveGoogleMeetConfig({ chrome: { audioBufferBytes: 2048 } });
const config = resolveGoogleMeetConfig({
chrome: { audioBackend: "blackhole-2ch", audioBufferBytes: 2048 },
});
expect(config.chrome.audioBufferBytes).toBe(2048);
expect(config.chrome.audioInputCommand).toEqual([
@@ -1376,7 +1384,9 @@ describe("google-meet plugin", () => {
});
it("clamps configured Chrome audio buffers above SoX's minimum", () => {
const config = resolveGoogleMeetConfig({ chrome: { audioBufferBytes: 1 } });
const config = resolveGoogleMeetConfig({
chrome: { audioBackend: "blackhole-2ch", audioBufferBytes: 1 },
});
expect(config.chrome.audioBufferBytes).toBe(17);
expect(config.chrome.audioInputCommand?.slice(0, 4)).toEqual(["sox", "-q", "--buffer", "17"]);
@@ -1462,9 +1472,8 @@ describe("google-meet plugin", () => {
});
});
it("keeps the agent tool visible on non-macOS hosts but blocks local Chrome talk-back joins", async () => {
it("keeps local Chrome talk-back available on Linux and blocks unsupported hosts", () => {
const { cliRegistrations, methods, tools } = setup(undefined, { registerPlatform: "linux" });
const tool = getMeetTool({ tools });
expect(tools).toHaveLength(1);
expect(cliRegistrations).toHaveLength(1);
@@ -1475,10 +1484,7 @@ describe("google-meet plugin", () => {
raw: { action: "join" },
platform: "linux",
}),
).toBe(true);
const blocked = await tool.execute("id", { action: "join" });
expect(JSON.stringify(blocked)).toContain("local Chrome talk-back audio is macOS-only");
).toBe(false);
expect(
googleMeetPluginTesting.isGoogleMeetAgentToolActionUnsupportedOnHost({
@@ -1487,6 +1493,13 @@ describe("google-meet plugin", () => {
platform: "linux",
}),
).toBe(false);
expect(
googleMeetPluginTesting.isGoogleMeetAgentToolActionUnsupportedOnHost({
config: resolveGoogleMeetConfig({}),
raw: { action: "join" },
platform: "win32",
}),
).toBe(true);
expect(
googleMeetPluginTesting.isGoogleMeetAgentToolActionUnsupportedOnHost({
config: resolveGoogleMeetConfig({}),
@@ -2490,7 +2503,7 @@ describe("google-meet plugin", () => {
},
{
runCommandWithTimeoutHandler: async (argv) => {
if (argv[0] === "/usr/sbin/system_profiler") {
if (argv[0]?.endsWith("system_profiler")) {
return { code: 0, stdout: "BlackHole 2ch", stderr: "" };
}
return { code: 0, stdout: "", stderr: "" };
@@ -2716,7 +2729,7 @@ describe("google-meet plugin", () => {
it.each([
{
config: { defaultTransport: "chrome" },
failingCommand: "/usr/sbin/system_profiler",
failingCommand: "system_profiler",
expectedCheckId: "chrome-local-audio-device",
expectedMessage: "BlackHole 2ch audio device not found",
},
@@ -2746,11 +2759,9 @@ describe("google-meet plugin", () => {
await withPlatform("darwin", async () => {
const { tools } = setup(config, {
runCommandWithTimeoutHandler: async (argv) => {
if (argv[0] === "/usr/sbin/system_profiler") {
if (argv[0]?.endsWith("system_profiler")) {
const stdout =
failingCommand === "/usr/sbin/system_profiler"
? "Built-in Output"
: "BlackHole 2ch";
failingCommand === "system_profiler" ? "Built-in Output" : "BlackHole 2ch";
return { code: 0, stdout, stderr: "" };
}
if (argv[0] === "/bin/sh" && argv.at(-1) === failingCommand) {
@@ -3726,7 +3737,7 @@ describe("google-meet plugin", () => {
{
runCommandWithTimeoutHandler: async (argv) => {
events.push(`command:${argv.join(" ")}`);
return argv[0] === "/usr/sbin/system_profiler"
return argv[0]?.endsWith("system_profiler")
? { code: 0, stdout: "BlackHole 2ch", stderr: "" }
: { code: 0, stdout: "", stderr: "" };
},
@@ -3810,7 +3821,7 @@ describe("google-meet plugin", () => {
{
runCommandWithTimeoutHandler: async (argv) => {
events.push(`command:${argv.join(" ")}`);
return argv[0] === "/usr/sbin/system_profiler"
return argv[0]?.endsWith("system_profiler")
? { code: 0, stdout: "BlackHole 2ch", stderr: "" }
: { code: 0, stdout: "", stderr: "" };
},
@@ -4363,7 +4374,7 @@ describe("google-meet plugin", () => {
expect(joinElsewhere.click).not.toHaveBeenCalled();
});
it("uses the local Meet microphone control instead of remote participant mute buttons", async () => {
it("does not unmute local or remote microphones without a verified virtual input", async () => {
const makeButton = (label: string, disabled = false) => ({
disabled,
innerText: "",
@@ -4410,9 +4421,11 @@ describe("google-meet plugin", () => {
const result = JSON.parse(await inspect()) as { micMuted?: boolean; notes?: string[] };
expect(result.micMuted).toBe(true);
expect(localMic.click).toHaveBeenCalledTimes(1);
expect(localMic.click).not.toHaveBeenCalled();
expect(remoteMute.click).not.toHaveBeenCalled();
expect(result.notes).toContain("Attempted to turn on the Meet microphone for talk-back mode.");
expect(result.notes).not.toContain(
"Turned on the Meet microphone after verifying the virtual audio input.",
);
});
it.each(
@@ -4517,7 +4530,7 @@ describe("google-meet plugin", () => {
expect(health.speechReady).toBe(true);
});
it("preserves the paired-node bridge for an externally managed Meet session", async () => {
it("does not start an externally managed paired-node bridge without a recoverable tab", async () => {
const { chrome, nodesInvoke } = await runGoogleMeetJoinScenario(
{
defaultTransport: "chrome-node",
@@ -4533,14 +4546,14 @@ describe("google-meet plugin", () => {
);
expect(
matchingNodeInvocations(nodesInvoke, { command: "googlemeet.chrome", action: "start" }),
).toHaveLength(1);
).toHaveLength(0);
expect(
nodesInvoke.mock.calls.some(([rawCall]) => {
const call = requireRecord(rawCall, "node invoke");
return call.command === "browser.proxy";
}),
).toBe(false);
expect(requireRecord(chrome.audioBridge, "join audio bridge").type).toBe("external-command");
).toBe(true);
expect(chrome.audioBridge).toBeUndefined();
});
it("joins Chrome on a paired node without local Chrome or BlackHole", async () => {
@@ -6099,7 +6112,12 @@ describe("google-meet plugin", () => {
transport: "chrome",
found: true,
message: "Existing Meet tab is in-call.",
browser: { inCall: true, micMuted: false },
browser: {
inCall: true,
micMuted: false,
audioInputRouted: true,
audioOutputRouted: true,
},
});
try {
const runtime = createChromeLifecycleRuntime();
@@ -6210,6 +6228,8 @@ describe("google-meet plugin", () => {
? {
inCall: true,
micMuted: false,
audioInputRouted: true,
audioOutputRouted: true,
title: "Meet call",
url: MEET_URL,
}
@@ -6363,10 +6383,10 @@ describe("google-meet plugin", () => {
});
expectRespondedOk(respond);
expect(runCommandWithTimeout).toHaveBeenNthCalledWith(2, ["bridge", "status"], {
expect(runCommandWithTimeout).toHaveBeenCalledWith(["bridge", "status"], {
timeoutMs: 30000,
});
expect(runCommandWithTimeout).toHaveBeenNthCalledWith(3, ["bridge", "start"], {
expect(runCommandWithTimeout).toHaveBeenCalledWith(["bridge", "start"], {
timeoutMs: 30000,
});
const openRequests = callGatewayFromCli.mock.calls.filter((call) => {
extensions/google-meet/index.ts
+1 -1
View File
@@ -266,7 +266,7 @@ export default definePluginEntry({
name: "google_meet",
label: "Google Meet",
description:
"Join and track Google Meet sessions through Chrome or Twilio. Call setup_status before join/create/test_listen/test_speech; if it reports a Chrome node offline, local audio missing, or missing Twilio dial plan, surface that blocker instead of retrying or switching transports. Twilio cannot dial a Meet URL directly: provide dialInNumber plus optional pin/dtmfSequence, or configure twilio.defaultDialInNumber. Offline nodes are diagnostics only, not usable candidates. If local Chrome talk-back audio is unsupported on this OS, use mode=transcribe, transport=twilio, or a macOS chrome-node for agent/bidi Chrome. If a Meet tab is already open after a timeout, call recover_current_tab before retrying join to report login, permission, or admission blockers without opening another tab.",
"Join and track Google Meet sessions through Chrome or Twilio. Call setup_status before join/create/test_listen/test_speech; if it reports a Chrome node offline, local audio missing, or missing Twilio dial plan, surface that blocker instead of retrying or switching transports. Twilio cannot dial a Meet URL directly: provide dialInNumber plus optional pin/dtmfSequence, or configure twilio.defaultDialInNumber. Offline nodes are diagnostics only, not usable candidates. Local Chrome talk-back needs macOS with BlackHole 2ch or Linux with PipeWire-Pulse; otherwise use mode=transcribe, transport=twilio, or a supported chrome-node. If a Meet tab is already open after a timeout, call recover_current_tab before retrying join to report login, permission, or admission blockers without opening another tab.",
parameters: GoogleMeetToolSchema,
async execute(_toolCallId, params) {
const raw = asParamRecord(params);
extensions/google-meet/node-host.test.ts
+2
View File
@@ -349,7 +349,9 @@ describe("google-meet node host bridge sessions", () => {
expect(typeof start.bridgeId).toBe("string");
expect(start.bridgeId.length).toBeGreaterThan(0);
expect(start).toEqual({
audioBackend: "blackhole-2ch",
audioBridge: { type: "node-command-pair", outputGeneration: true },
audioDeviceLabel: "BlackHole 2ch",
bridgeId: start.bridgeId,
launched: false,
});
extensions/google-meet/openclaw.plugin.json
+5 -47
View File
@@ -37,7 +37,7 @@
},
"chrome.audioBackend": {
"label": "Chrome Audio Backend",
"help": "BlackHole 2ch is required for local duplex audio routing."
"help": "Auto selects BlackHole 2ch on macOS or PipeWire-Pulse on Linux."
},
"chrome.launch": {
"label": "Launch Chrome"
@@ -100,7 +100,7 @@
},
"chrome.audioBufferBytes": {
"label": "Audio Buffer Bytes",
"help": "SoX processing buffer for generated Chrome command-pair audio commands. Lower values reduce latency but may underrun on busy hosts.",
"help": "Processing buffer for generated Chrome command-pair audio. Lower values reduce latency but may underrun on busy hosts.",
"advanced": true
},
"chrome.audioBridgeCommand": {
@@ -113,7 +113,7 @@
},
"chromeNode.node": {
"label": "Chrome Node",
"help": "Node id/name/IP that owns Chrome, BlackHole, and SoX for chrome-node transport.",
"help": "Node id/name/IP that owns Chrome and the native virtual-audio backend for chrome-node transport.",
"advanced": true
},
"twilio.defaultDialInNumber": {
@@ -260,8 +260,8 @@
"properties": {
"audioBackend": {
"type": "string",
"enum": ["blackhole-2ch"],
"default": "blackhole-2ch"
"enum": ["auto", "blackhole-2ch", "pipewire-pulse"],
"default": "auto"
},
"launch": {
"type": "boolean",
@@ -301,54 +301,12 @@
},
"audioInputCommand": {
"type": "array",
"default": [
"sox",
"-q",
"--buffer",
"4096",
"-t",
"coreaudio",
"BlackHole 2ch",
"-t",
"raw",
"-r",
"24000",
"-c",
"1",
"-e",
"signed-integer",
"-b",
"16",
"-L",
"-"
],
"items": {
"type": "string"
}
},
"audioOutputCommand": {
"type": "array",
"default": [
"sox",
"-q",
"--buffer",
"4096",
"-t",
"raw",
"-r",
"24000",
"-c",
"1",
"-e",
"signed-integer",
"-b",
"16",
"-L",
"-",
"-t",
"coreaudio",
"BlackHole 2ch"
],
"items": {
"type": "string"
}
extensions/google-meet/src/cli-runtime-commands.ts
+22 -7
View File
@@ -2,6 +2,9 @@ import { parseStrictNonNegativeInteger } from "openclaw/plugin-sdk/number-runtim
import type { GoogleMeetCliCommandContext } from "./cli-command-context.js";
import {
callGoogleMeetGateway,
parseGoogleMeetBrowserTransport,
parseGoogleMeetMode,
parseGoogleMeetTransport,
parsePositiveNumber,
type JoinOptions,
type RecoverTabOptions,
@@ -30,8 +33,8 @@ export function registerGoogleMeetProbeCommands(context: GoogleMeetCliCommandCon
.action(async (url: string | undefined, options: JoinOptions) => {
const payload = {
url: resolveMeetingInput(params.config, url),
transport: options.transport,
mode: options.mode,
transport: parseGoogleMeetTransport(options.transport),
mode: parseGoogleMeetMode(options.mode),
message: options.message,
dialInNumber: options.dialInNumber,
pin: options.pin,
@@ -66,8 +69,8 @@ export function registerGoogleMeetProbeCommands(context: GoogleMeetCliCommandCon
.action(async (url: string | undefined, options: JoinOptions) => {
const payload = {
url: resolveMeetingInput(params.config, url),
transport: options.transport,
mode: options.mode,
transport: parseGoogleMeetTransport(options.transport),
mode: parseGoogleMeetMode(options.mode),
message: options.message,
};
const delegated = await callGoogleMeetGateway({
@@ -92,7 +95,7 @@ export function registerGoogleMeetProbeCommands(context: GoogleMeetCliCommandCon
.action(async (url: string | undefined, options: JoinOptions) => {
const payload = {
url: resolveMeetingInput(params.config, url),
transport: options.transport,
transport: parseGoogleMeetBrowserTransport(options.transport),
timeoutMs: parsePositiveNumber(options.timeoutMs, "timeout-ms"),
};
const delegated = await callGoogleMeetGateway({
@@ -189,7 +192,10 @@ export function registerGoogleMeetLifecycleCommands(context: GoogleMeetCliComman
.option("--json", "Print JSON output", false)
.action(async (url: string | undefined, options: RecoverTabOptions) => {
const rt = await params.ensureRuntime();
const result = await rt.recoverCurrentTab({ url, transport: options.transport });
const result = await rt.recoverCurrentTab({
url,
transport: parseGoogleMeetBrowserTransport(options.transport),
});
if (options.json) {
writeStdoutJson(result);
return;
@@ -205,12 +211,21 @@ export function registerGoogleMeetLifecycleCommands(context: GoogleMeetCliComman
.option("--json", "Print JSON output", false)
.action(async (options: SetupOptions) => {
const rt = await params.ensureRuntime();
const status = await rt.setupStatus({ transport: options.transport, mode: options.mode });
const status = await rt.setupStatus({
transport: parseGoogleMeetTransport(options.transport),
mode: parseGoogleMeetMode(options.mode),
});
if (options.json) {
writeStdoutJson(status);
if (!status.ok) {
process.exitCode = 1;
}
return;
}
writeSetupStatus(status);
if (!status.ok) {
process.exitCode = 1;
}
});
root
extensions/google-meet/src/cli-runtime.test.ts
+19
View File
@@ -13,6 +13,7 @@ import {
describe("google-meet CLI", () => {
afterEach(() => {
vi.unstubAllGlobals();
process.exitCode = undefined;
});
afterAll(() => {
@@ -65,12 +66,30 @@ describe("google-meet CLI", () => {
id: "twilio-voice-call-plugin",
ok: false,
});
expect(process.exitCode).toBe(1);
} finally {
stdout.restore();
}
}
});
it("rejects unknown setup mode and transport values", async () => {
const setupStatus = vi.fn<NonNullable<GoogleMeetRuntime["setupStatus"]>>();
const cli = setupCli({ runtime: { setupStatus } });
await expect(
cli.parseAsync(["googlemeet", "setup", "--mode", "agnt"], { from: "user" }),
).rejects.toThrow("mode must be agent, bidi, transcribe, or realtime; received agnt");
await expect(
cli.parseAsync(["googlemeet", "setup", "--transport", "definitely-not-a-transport"], {
from: "user",
}),
).rejects.toThrow(
"transport must be chrome, chrome-node, or twilio; received definitely-not-a-transport",
);
expect(setupStatus).not.toHaveBeenCalled();
});
it("accepts --json on session status", async () => {
const stdout = captureStdout();
try {
extensions/google-meet/src/cli-shared.ts
+39 -8
View File
@@ -17,8 +17,8 @@ import {
import type { GoogleMeetRuntime } from "./runtime.js";
export type JoinOptions = {
transport?: GoogleMeetTransport;
mode?: GoogleMeetModeInput;
transport?: string;
mode?: string;
message?: string;
timeoutMs?: string;
dialInNumber?: string;
@@ -122,8 +122,8 @@ export type GoogleMeetExportManifest = {
export type SetupOptions = {
json?: boolean;
mode?: GoogleMeetModeInput;
transport?: GoogleMeetTransport;
mode?: string;
transport?: string;
};
type GoogleMeetGatewayMethod =
@@ -158,7 +158,7 @@ export type JsonOptions = {
};
export type RecoverTabOptions = JsonOptions & {
transport?: GoogleMeetTransport;
transport?: string;
};
export type CreateOptions = {
@@ -170,8 +170,8 @@ export type CreateOptions = {
accessType?: string;
entryPointAccess?: string;
join?: boolean;
transport?: GoogleMeetTransport;
mode?: GoogleMeetModeInput;
transport?: string;
mode?: string;
message?: string;
dialInNumber?: string;
pin?: string;
@@ -179,6 +179,38 @@ export type CreateOptions = {
json?: boolean;
};
export function parseGoogleMeetMode(value: string | undefined): GoogleMeetModeInput | undefined {
if (
value === undefined ||
value === "agent" ||
value === "bidi" ||
value === "transcribe" ||
value === "realtime"
) {
return value;
}
throw new Error(`mode must be agent, bidi, transcribe, or realtime; received ${value}`);
}
export function parseGoogleMeetTransport(
value: string | undefined,
): GoogleMeetTransport | undefined {
if (value === undefined || value === "chrome" || value === "chrome-node" || value === "twilio") {
return value;
}
throw new Error(`transport must be chrome, chrome-node, or twilio; received ${value}`);
}
export function parseGoogleMeetBrowserTransport(
value: string | undefined,
): "chrome" | "chrome-node" | undefined {
const transport = parseGoogleMeetTransport(value);
if (transport === "twilio") {
throw new Error(`transport must be chrome or chrome-node; received ${value}`);
}
return transport;
}
export function writeStdoutJson(value: unknown): void {
process.stdout.write(`${JSON.stringify(value, null, 2)}\n`);
}
@@ -451,7 +483,6 @@ export function writeRecoverCurrentTabResult(
: "signed-in Google Chrome profile",
realtime: { enabled: false, toolPolicy: "safe-read-only" },
chrome: {
audioBackend: "blackhole-2ch",
launched: true,
nodeId: result.nodeId,
health: result.browser,
extensions/google-meet/src/cli-space-commands.ts
+6 -4
View File
@@ -8,6 +8,8 @@ import {
import { writeCalendarEventsSummary, writeLatestConferenceRecordSummary } from "./cli-export.js";
import {
callGoogleMeetGateway,
parseGoogleMeetMode,
parseGoogleMeetTransport,
type CreateOptions,
type JsonOptions,
type ResolveSpaceOptions,
@@ -137,8 +139,8 @@ export function registerGoogleMeetCreateCommands(context: GoogleMeetCliCommandCo
options.join !== false
? await rt.join({
url: result.meetingUri,
transport: options.transport,
mode: options.mode,
transport: parseGoogleMeetTransport(options.transport),
mode: parseGoogleMeetMode(options.mode),
message: options.message,
dialInNumber: options.dialInNumber,
pin: options.pin,
@@ -176,8 +178,8 @@ export function registerGoogleMeetCreateCommands(context: GoogleMeetCliCommandCo
await params.ensureRuntime()
).join({
url: result.meetingUri,
transport: options.transport,
mode: options.mode,
transport: parseGoogleMeetTransport(options.transport),
mode: parseGoogleMeetMode(options.mode),
message: options.message,
dialInNumber: options.dialInNumber,
pin: options.pin,
extensions/google-meet/src/config.test.ts
+32
View File
@@ -4,6 +4,38 @@ import { describe, expect, it } from "vitest";
import { resolveGoogleMeetConfig, resolveGoogleMeetGatewayOperationTimeoutMs } from "./config.js";
describe("google meet gateway operation timeout", () => {
it("keeps sparse and legacy audio config compatible", () => {
const sparse = resolveGoogleMeetConfig({});
expect(sparse.chrome.audioBackend).toBe("auto");
expect(sparse.chrome.audioInputCommandOverride).toBeUndefined();
expect(sparse.chrome.audioOutputCommandOverride).toBeUndefined();
const legacy = resolveGoogleMeetConfig({
chrome: { audioBackend: "blackhole-2ch" },
});
expect(legacy.chrome.audioBackend).toBe("blackhole-2ch");
expect(legacy.chrome.audioInputCommand).toContain("BlackHole 2ch");
expect(legacy.chrome.audioOutputCommand).toContain("BlackHole 2ch");
});
it("builds PipeWire-Pulse commands and retains explicit overrides", () => {
const linux = resolveGoogleMeetConfig({
chrome: { audioBackend: "pipewire-pulse" },
});
expect(linux.chrome.audioInputCommand).toContain("parec");
expect(linux.chrome.audioOutputCommand).toContain("pacat");
const custom = resolveGoogleMeetConfig({
chrome: { audioInputCommand: ["capture"], audioOutputCommand: ["play"] },
});
expect(custom.chrome).toMatchObject({
audioInputCommand: ["capture"],
audioOutputCommand: ["play"],
audioInputCommandOverride: ["capture"],
audioOutputCommandOverride: ["play"],
});
});
it("caps timer config fields before runtime polling uses them", () => {
const config = resolveGoogleMeetConfig({
chrome: {
extensions/google-meet/src/config.ts
+53 -17
View File
@@ -21,6 +21,7 @@ export type GoogleMeetModeInput = GoogleMeetMode | "realtime";
type GoogleMeetRealtimeStrategy = "agent" | "bidi";
type GoogleMeetChromeAudioFormat = "pcm16-24khz" | "g711-ulaw-8khz";
type GoogleMeetToolPolicy = RealtimeVoiceAgentConsultToolPolicy;
type MeetingAudioBackendSelection = "auto" | "blackhole-2ch" | "pipewire-pulse";
export type GoogleMeetConfig = {
enabled: boolean;
@@ -33,7 +34,7 @@ export type GoogleMeetConfig = {
defaultTransport: GoogleMeetTransport;
defaultMode: GoogleMeetMode;
chrome: {
audioBackend: "blackhole-2ch";
audioBackend: MeetingAudioBackendSelection;
audioFormat: GoogleMeetChromeAudioFormat;
audioBufferBytes: number;
launch: boolean;
@@ -45,6 +46,8 @@ export type GoogleMeetConfig = {
waitForInCallMs: number;
audioInputCommand?: string[];
audioOutputCommand?: string[];
audioInputCommandOverride?: string[];
audioOutputCommandOverride?: string[];
bargeInInputCommand?: string[];
bargeInRmsThreshold: number;
bargeInPeakThreshold: number;
@@ -109,9 +112,35 @@ const SOX_MIN_BUFFER_BYTES = 17;
const DEFAULT_GOOGLE_MEET_AUDIO_BUFFER_BYTES = SOX_DEFAULT_BUFFER_BYTES / 2;
const PLAIN_DECIMAL_NUMBER_RE = /^\d+(?:\.\d+)?$/;
function buildGoogleMeetSoxAudioCommands(format: GoogleMeetChromeAudioFormat, bufferBytes: number) {
function buildGoogleMeetAudioCommands(
backend: MeetingAudioBackendSelection,
format: GoogleMeetChromeAudioFormat,
bufferBytes: number,
) {
// Config parsing runs during registration; command construction must not load
// the full meeting runtime before a Google Meet action needs it.
const pipeWire =
backend === "pipewire-pulse" || (backend === "auto" && process.platform === "linux");
const sampleRate = format === "g711-ulaw-8khz" ? 8_000 : 24_000;
const bits = format === "g711-ulaw-8khz" ? 8 : 16;
if (pipeWire) {
const pulseFormat = format === "g711-ulaw-8khz" ? "ulaw" : "s16le";
const latencyMs = Math.max(
1,
Math.ceil((bufferBytes / (sampleRate * Math.ceil(bits / 8))) * 1_000),
);
const common = [
"--device=openclaw_meeting_audio",
`--format=${pulseFormat}`,
`--rate=${sampleRate}`,
"--channels=1",
`--latency-msec=${latencyMs}`,
];
return {
inputCommand: ["parec", "--raw", ...common],
outputCommand: ["pacat", "--raw", "--playback", ...common],
};
}
const wire =
format === "g711-ulaw-8khz"
? ["-t", "raw", "-r", "8000", "-c", "1", "-e", "mu-law", "-b", "8", "-"]
@@ -123,27 +152,22 @@ function buildGoogleMeetSoxAudioCommands(format: GoogleMeetChromeAudioFormat, bu
String(bufferBytes),
...args,
];
if (format === "g711-ulaw-8khz") {
return {
inputCommand: withBuffer("rec", wire),
outputCommand: withBuffer("play", wire),
};
}
return {
inputCommand: withBuffer("sox", ["-t", "coreaudio", "BlackHole 2ch", ...wire]),
outputCommand: withBuffer("sox", [...wire, "-t", "coreaudio", "BlackHole 2ch"]),
};
}
const DEFAULT_GOOGLE_MEET_SOX_COMMANDS = buildGoogleMeetSoxAudioCommands(
const DEFAULT_GOOGLE_MEET_AUDIO_COMMANDS = buildGoogleMeetAudioCommands(
"blackhole-2ch",
"pcm16-24khz",
DEFAULT_GOOGLE_MEET_AUDIO_BUFFER_BYTES,
);
export const DEFAULT_GOOGLE_MEET_AUDIO_INPUT_COMMAND =
DEFAULT_GOOGLE_MEET_SOX_COMMANDS.inputCommand;
DEFAULT_GOOGLE_MEET_AUDIO_COMMANDS.inputCommand;
export const DEFAULT_GOOGLE_MEET_AUDIO_OUTPUT_COMMAND =
DEFAULT_GOOGLE_MEET_SOX_COMMANDS.outputCommand;
DEFAULT_GOOGLE_MEET_AUDIO_COMMANDS.outputCommand;
const DEFAULT_GOOGLE_MEET_CHROME_AUDIO_FORMAT: GoogleMeetChromeAudioFormat = "pcm16-24khz";
const DEFAULT_GOOGLE_MEET_BARGE_IN_RMS_THRESHOLD = 650;
@@ -162,7 +186,7 @@ const DEFAULT_GOOGLE_MEET_CONFIG: GoogleMeetConfig = {
defaultTransport: "chrome",
defaultMode: "agent",
chrome: {
audioBackend: "blackhole-2ch",
audioBackend: "auto",
audioFormat: DEFAULT_GOOGLE_MEET_CHROME_AUDIO_FORMAT,
audioBufferBytes: DEFAULT_GOOGLE_MEET_AUDIO_BUFFER_BYTES,
launch: true,
@@ -352,17 +376,24 @@ function resolveAudioBufferBytes(value: unknown, fallback: number): number {
}
function defaultAudioInputCommand(
backend: MeetingAudioBackendSelection,
format: GoogleMeetChromeAudioFormat,
bufferBytes: number,
): string[] {
return buildGoogleMeetSoxAudioCommands(format, bufferBytes).inputCommand;
return buildGoogleMeetAudioCommands(backend, format, bufferBytes).inputCommand;
}
function defaultAudioOutputCommand(
backend: MeetingAudioBackendSelection,
format: GoogleMeetChromeAudioFormat,
bufferBytes: number,
): string[] {
return buildGoogleMeetSoxAudioCommands(format, bufferBytes).outputCommand;
return buildGoogleMeetAudioCommands(backend, format, bufferBytes).outputCommand;
}
function resolveAudioBackend(value: unknown): MeetingAudioBackendSelection {
const normalized = normalizeOptionalLowercaseString(value)?.replaceAll("_", "-");
return normalized === "blackhole-2ch" || normalized === "pipewire-pulse" ? normalized : "auto";
}
export function resolveGoogleMeetConfig(input: unknown): GoogleMeetConfig {
@@ -388,6 +419,7 @@ function resolveGoogleMeetConfigWithEnv(
chrome.audioBufferBytes,
DEFAULT_GOOGLE_MEET_CONFIG.chrome.audioBufferBytes,
);
const audioBackend = resolveAudioBackend(chrome.audioBackend);
const chromeNode = asRecord(raw.chromeNode);
const twilio = asRecord(raw.twilio);
const voiceCall = asRecord(raw.voiceCall);
@@ -417,7 +449,7 @@ function resolveGoogleMeetConfigWithEnv(
),
defaultMode: resolveMode(raw.defaultMode, DEFAULT_GOOGLE_MEET_CONFIG.defaultMode),
chrome: {
audioBackend: "blackhole-2ch",
audioBackend,
audioFormat,
audioBufferBytes,
launch: resolveBoolean(chrome.launch, DEFAULT_GOOGLE_MEET_CONFIG.chrome.launch),
@@ -438,9 +470,13 @@ function resolveGoogleMeetConfigWithEnv(
DEFAULT_GOOGLE_MEET_CONFIG.chrome.waitForInCallMs,
),
audioInputCommand:
configuredAudioInputCommand ?? defaultAudioInputCommand(audioFormat, audioBufferBytes),
configuredAudioInputCommand ??
defaultAudioInputCommand(audioBackend, audioFormat, audioBufferBytes),
audioOutputCommand:
configuredAudioOutputCommand ?? defaultAudioOutputCommand(audioFormat, audioBufferBytes),
configuredAudioOutputCommand ??
defaultAudioOutputCommand(audioBackend, audioFormat, audioBufferBytes),
audioInputCommandOverride: configuredAudioInputCommand,
audioOutputCommandOverride: configuredAudioOutputCommand,
bargeInInputCommand: resolveStringArray(chrome.bargeInInputCommand),
bargeInRmsThreshold: resolveNumber(
chrome.bargeInRmsThreshold,
extensions/google-meet/src/node-host.ts
+10 -33
View File
@@ -1,41 +1,12 @@
// Google Meet injects device/URL labels into the shared node-side browser audio host.
import { spawnSync } from "node:child_process";
import { createMeetingNodeHost } from "openclaw/plugin-sdk/meeting-runtime";
import { MeetingPlatformAdapter } from "openclaw/plugin-sdk/meeting-runtime";
import {
DEFAULT_GOOGLE_MEET_AUDIO_INPUT_COMMAND,
DEFAULT_GOOGLE_MEET_AUDIO_OUTPUT_COMMAND,
} from "./config.js";
import {
GOOGLE_MEET_SYSTEM_PROFILER_COMMAND,
outputMentionsBlackHole2ch,
} from "./transports/chrome-audio-device.js";
import { GOOGLE_MEET_PLATFORM_ADAPTER } from "./transports/google-meet-platform-adapter.js";
import { GOOGLE_MEET_NODE_COMMAND } from "./transports/google-meet-platform-constants.js";
function assertBlackHoleAvailable(timeoutMs: number) {
if (process.platform !== "darwin") {
throw new Error("Chrome Meet transport with blackhole-2ch audio is currently macOS-only");
}
const result = spawnSync(GOOGLE_MEET_SYSTEM_PROFILER_COMMAND, ["SPAudioDataType"], {
encoding: "utf8",
timeout: timeoutMs,
});
const stderr =
result.stderr ??
(result.error
? result.error instanceof Error
? result.error.message
: String(result.error)
: "");
const output = `${result.stdout ?? ""}\n${stderr}`;
if (
(typeof result.status === "number" ? result.status : result.error ? 1 : 0) !== 0 ||
!outputMentionsBlackHole2ch(output)
) {
throw new Error("BlackHole 2ch audio device not found on the node.");
}
}
function normalizeMeetKey(value?: string): string | undefined {
if (!value) {
return undefined;
@@ -52,18 +23,24 @@ function normalizeMeetKey(value?: string): string | undefined {
}
}
const googleMeetNodeHost = createMeetingNodeHost({
const googleMeetNodeHost = MeetingPlatformAdapter.createNodeHostHandler({
commandName: GOOGLE_MEET_NODE_COMMAND,
displayName: "Google Meet",
browserLabel: "Meet",
bridgeIdPrefix: "meet_node_",
defaultAudioInputCommand: DEFAULT_GOOGLE_MEET_AUDIO_INPUT_COMMAND,
defaultAudioOutputCommand: DEFAULT_GOOGLE_MEET_AUDIO_OUTPUT_COMMAND,
defaultAudio: {
backend: "auto",
bufferBytes: 4_096,
format: "pcm16-24khz",
},
meetingLabel: "Google Meet",
sharePrerequisiteDeadline: true,
talkBackModes: new Set(["agent", "bidi", "realtime"]),
agentMode: "agent",
normalizeUrl: (url) => GOOGLE_MEET_PLATFORM_ADAPTER.urls.validateAndNormalize(url),
normalizeMeetingKey: normalizeMeetKey,
assertAudioAvailable: assertBlackHoleAvailable,
browser: {
application: "Google Chrome",
buildProfileArgs: (profile) => ["--args", `--profile-directory=${profile}`],
@@ -75,5 +52,5 @@ const googleMeetNodeHost = createMeetingNodeHost({
});
export async function handleGoogleMeetNodeHostCommand(paramsJSON?: string | null): Promise<string> {
return await googleMeetNodeHost.handleCommand(paramsJSON);
return await googleMeetNodeHost(paramsJSON);
}
extensions/google-meet/src/node-invoke-policy.test.ts
+11 -1
View File
@@ -60,6 +60,9 @@ describe("Google Meet node invoke policy", () => {
launch: false,
browserProfile: "Trusted Profile",
joinTimeoutMs: 45_000,
audioBackend: "auto",
audioBufferBytes: 4_096,
audioFormat: "g711-ulaw-8khz",
audioInputCommand: ["trusted-capture", "--raw"],
audioOutputCommand: ["trusted-play", "--raw"],
},
@@ -118,7 +121,14 @@ describe("Google Meet node invoke policy", () => {
await policy.handle(ctx);
expect(invokeNode).toHaveBeenCalledWith({ params: { action: "setup" } });
expect(invokeNode).toHaveBeenCalledWith({
params: {
action: "setup",
audioBackend: "auto",
audioBufferBytes: 4_096,
audioFormat: "pcm16-24khz",
},
});
});
it("rejects malformed bridge control before node dispatch", async () => {
extensions/google-meet/src/node-invoke-policy.ts
+1
View File
@@ -17,6 +17,7 @@ export function createGoogleMeetChromeNodeInvokePolicy(
deniedCode: "GOOGLE_MEET_NODE_POLICY_DENIED",
supportedModes: START_MODES,
normalizeUrl: (url) => GOOGLE_MEET_PLATFORM_ADAPTER.urls.validateAndNormalize(url),
useConfiguredSetupCommands: true,
start: config.chrome,
});
}
extensions/google-meet/src/plugin-registration.ts
+2 -2
View File
@@ -114,7 +114,7 @@ function isGoogleMeetAgentToolActionUnsupportedOnHost(params: {
platform?: NodeJS.Platform;
}): boolean {
const platform = params.platform ?? googleMeetToolDeps.platform();
if (platform === "darwin") {
if (platform === "darwin" || platform === "linux") {
return false;
}
const action = params.raw.action;
@@ -141,7 +141,7 @@ export function assertGoogleMeetAgentToolActionSupported(params: {
return;
}
throw new Error(
"Google Meet local Chrome talk-back audio is macOS-only. On this host, use mode: transcribe, transport: twilio, or transport: chrome-node backed by a macOS node.",
"Google Meet local Chrome talk-back audio requires macOS with BlackHole 2ch or Linux with PipeWire-Pulse. On this host, use mode: transcribe, transport: twilio, or a supported chrome-node.",
);
}
extensions/google-meet/src/plugin-schema.ts
+3 -3
View File
@@ -26,7 +26,7 @@ export const googleMeetConfigSchema = {
},
"chrome.audioBackend": {
label: "Chrome Audio Backend",
help: "BlackHole 2ch is required for local duplex audio routing.",
help: "Auto selects BlackHole 2ch on macOS or PipeWire-Pulse on Linux.",
},
"chrome.launch": { label: "Launch Chrome" },
"chrome.browserProfile": { label: "Chrome Profile", advanced: true },
@@ -54,7 +54,7 @@ export const googleMeetConfigSchema = {
},
"chrome.audioBufferBytes": {
label: "Audio Buffer Bytes",
help: "SoX processing buffer for generated Chrome command-pair audio commands. Lower values reduce latency but may underrun on busy hosts.",
help: "Processing buffer for generated Chrome command-pair audio. Lower values reduce latency but may underrun on busy hosts.",
advanced: true,
},
"chrome.audioInputCommand": {
@@ -94,7 +94,7 @@ export const googleMeetConfigSchema = {
},
"chromeNode.node": {
label: "Chrome Node",
help: "Node id/name/IP that owns Chrome, BlackHole, and SoX for chrome-node transport.",
help: "Node id/name/IP that owns Chrome and the native virtual-audio backend for chrome-node transport.",
advanced: true,
},
"twilio.defaultDialInNumber": {
extensions/google-meet/src/runtime-setup.test.ts
+46
View File
@@ -0,0 +1,46 @@
import type { PluginRuntime } from "openclaw/plugin-sdk/plugin-runtime";
import { describe, expect, it, vi } from "vitest";
import { resolveGoogleMeetConfig } from "./config.js";
import { getGoogleMeetRuntimeSetupStatus } from "./runtime-setup.js";
describe("Google Meet runtime setup", () => {
it("lets the Chrome node generate commands for its own platform", async () => {
const invoke = vi.fn(async () => ({ ok: true }));
const runtime = {
nodes: {
invoke,
list: vi.fn(async () => ({
nodes: [
{
caps: ["browser"],
commands: ["browser.proxy", "googlemeet.chrome"],
connected: true,
displayName: "meet-node",
nodeId: "node-1",
},
],
})),
},
} as unknown as PluginRuntime;
const status = await getGoogleMeetRuntimeSetupStatus({
config: resolveGoogleMeetConfig({ chromeNode: { node: "meet-node" } }),
fullConfig: {},
runtime,
options: { mode: "agent", transport: "chrome-node" },
});
expect(invoke).toHaveBeenCalledWith({
command: "googlemeet.chrome",
nodeId: "node-1",
params: {
action: "setup",
audioBackend: "auto",
audioBufferBytes: 4_096,
audioFormat: "pcm16-24khz",
},
timeoutMs: 12_000,
});
expect(status.ok).toBe(true);
});
});
extensions/google-meet/src/runtime-setup.ts
+31 -3
View File
@@ -5,7 +5,8 @@ import { uniqueStrings } from "openclaw/plugin-sdk/string-coerce-runtime";
import type { GoogleMeetConfig, GoogleMeetModeInput, GoogleMeetTransport } from "./config.js";
import { addGoogleMeetSetupCheck, getGoogleMeetSetupStatus } from "./setup.js";
import { resolveChromeNodeInfo } from "./transports/chrome-browser-proxy.js";
import { assertBlackHole2chAvailable } from "./transports/chrome.js";
import { assertGoogleMeetAudioAvailable } from "./transports/chrome.js";
import { GOOGLE_MEET_NODE_COMMAND } from "./transports/google-meet-platform-constants.js";
import { normalizeDialInNumber } from "./transports/twilio.js";
function collectChromeAudioCommands(config: GoogleMeetConfig): string[] {
@@ -62,6 +63,32 @@ export async function getGoogleMeetRuntimeSetupStatus(params: {
ok: true,
message: `Connected Google Meet node ready: ${label}`,
});
if ((mode === "agent" || mode === "bidi") && node.nodeId) {
const setup = await params.runtime.nodes.invoke({
nodeId: node.nodeId,
command: GOOGLE_MEET_NODE_COMMAND,
params: {
action: "setup",
audioBackend: params.config.chrome.audioBackend,
audioFormat: params.config.chrome.audioFormat,
audioBufferBytes: params.config.chrome.audioBufferBytes,
...(params.config.chrome.audioInputCommandOverride
? { audioInputCommand: params.config.chrome.audioInputCommandOverride }
: {}),
...(params.config.chrome.audioOutputCommandOverride
? { audioOutputCommand: params.config.chrome.audioOutputCommandOverride }
: {}),
},
timeoutMs: 12_000,
});
status = addGoogleMeetSetupCheck(status, {
id: "chrome-node-audio-prerequisites",
ok: true,
message: setup
? "Remote virtual audio backend and command-pair prerequisites are ready"
: "Remote audio setup completed",
});
}
} catch (error) {
status = addGoogleMeetSetupCheck(status, {
id: "chrome-node-connected",
@@ -74,14 +101,15 @@ export async function getGoogleMeetRuntimeSetupStatus(params: {
return status;
}
try {
await assertBlackHole2chAvailable({
await assertGoogleMeetAudioAvailable({
runtime: params.runtime,
config: params.config,
timeoutMs: Math.min(params.config.chrome.joinTimeoutMs, 10_000),
});
status = addGoogleMeetSetupCheck(status, {
id: "chrome-local-audio-device",
ok: true,
message: "BlackHole 2ch audio device found",
message: "Virtual meeting audio backend is ready",
});
} catch (error) {
status = addGoogleMeetSetupCheck(status, {
extensions/google-meet/src/runtime.ts
+3 -5
View File
@@ -353,7 +353,7 @@ export class GoogleMeetRuntime {
tab,
});
session.chrome = {
audioBackend: this.params.config.chrome.audioBackend,
audioBackend: result.audioBackend,
launched: result.launched,
nodeId,
browserProfile: this.params.config.chrome.browserProfile,
@@ -370,7 +370,7 @@ export class GoogleMeetRuntime {
? "Chrome node transport joins as the signed-in Google profile on the selected node and routes realtime audio through the node bridge."
: "Chrome transport joins as the signed-in Google profile and routes realtime audio through the configured bridge."
: MeetingPlatformAdapter.isTalkBackMode(session.mode)
? "Chrome transport joins as the signed-in Google profile and expects BlackHole 2ch audio routing."
? "Chrome transport is waiting for verified virtual input/output audio routing."
: "Chrome transport joins as the signed-in Google profile without starting the realtime audio bridge.",
);
this.#sessions.refreshSpeechReadiness(session);
@@ -475,9 +475,7 @@ export class GoogleMeetRuntime {
session.state !== "active" ||
!session.chrome ||
session.chrome.audioBridge ||
session.chrome.health?.inCall !== true ||
session.chrome.health.micMuted !== false ||
session.chrome.health.manualAction
!MeetingPlatformAdapter.isRealtimeRouteReady(session.mode, session.chrome.health)
) {
return undefined;
}
extensions/google-meet/src/test-support/fixtures.test-helpers.ts
+2
View File
@@ -120,6 +120,8 @@ export function meetBrowserState(overrides: Record<string, unknown> = {}) {
return {
inCall: true,
micMuted: false,
audioInputRouted: true,
audioOutputRouted: true,
title: "Meet call",
url: MEET_URL,
...overrides,
extensions/google-meet/src/test-support/plugin-harness.ts
+7 -5
View File
@@ -127,14 +127,16 @@ export function setupGoogleMeetPlugin(
result: {
ok: true,
targetId: proxy.body?.targetId ?? "tab-1",
result: JSON.stringify(
options.browserActResult ?? {
result: JSON.stringify({
audioInputRouted: true,
audioOutputRouted: true,
...(options.browserActResult ?? {
inCall: true,
micMuted: false,
title: "Meet call",
url: MEET_URL,
},
),
}),
}),
},
},
};
@@ -148,7 +150,7 @@ export function setupGoogleMeetPlugin(
if (options.runCommandWithTimeoutHandler) {
return options.runCommandWithTimeoutHandler(argv, runOptions);
}
if (argv[0] === "/usr/sbin/system_profiler") {
if (argv[0]?.endsWith("system_profiler")) {
return { code: 0, stdout: "BlackHole 2ch", stderr: "" };
}
return { code: 0, stdout: "", stderr: "" };
extensions/google-meet/src/transports/chrome-audio-device.ts
-6
View File
@@ -1,6 +0,0 @@
// Google Meet plugin module implements chrome audio device behavior.
export const GOOGLE_MEET_SYSTEM_PROFILER_COMMAND = "/usr/sbin/system_profiler";
export function outputMentionsBlackHole2ch(output: string): boolean {
return /\bBlackHole\s+2ch\b/i.test(output);
}
extensions/google-meet/src/transports/chrome.ts
+143 -59
View File
@@ -12,6 +12,7 @@ import {
MeetingPlatformAdapter,
startMeetingAgentRealtimeEngine,
startMeetingRealtimeEngine,
type MeetingBrowserRequestCaller,
type MeetingRealtimeAudioEngineHandle,
} from "openclaw/plugin-sdk/meeting-runtime";
import { addTimerTimeoutGraceMs } from "openclaw/plugin-sdk/number-runtime";
@@ -19,10 +20,6 @@ import type { PluginRuntime } from "openclaw/plugin-sdk/plugin-runtime";
import type { RuntimeLogger } from "openclaw/plugin-sdk/plugin-runtime";
import { resolveTranscriptsConfig } from "openclaw/plugin-sdk/transcripts";
import type { GoogleMeetConfig, GoogleMeetMode } from "../config.js";
import {
GOOGLE_MEET_SYSTEM_PROFILER_COMMAND,
outputMentionsBlackHole2ch,
} from "./chrome-audio-device.js";
import {
callBrowserProxyOnNode,
resolveChromeNode,
@@ -40,6 +37,13 @@ type ChromeRealtimeAudioBridgeHandle = MeetingRealtimeAudioEngineHandle & {
inputCommand: string[];
outputCommand: string[];
};
type MeetingAudioBackend = "blackhole-2ch" | "pipewire-pulse";
type MeetingAudioRuntime = {
backend: MeetingAudioBackend;
deviceLabel: string;
inputCommand: string[];
outputCommand: string[];
};
type ChromeNodeRealtimeAudioBridgeHandle = MeetingRealtimeAudioEngineHandle & {
type: "node-command-pair";
@@ -53,35 +57,35 @@ function shouldCaptureCaptions(mode: GoogleMeetMode, fullConfig?: OpenClawConfig
);
}
export async function assertBlackHole2chAvailable(params: {
async function prepareGoogleMeetAudioRuntime(params: {
runtime: PluginRuntime;
config: GoogleMeetConfig;
timeoutMs: number;
}): Promise<MeetingAudioRuntime> {
const audio = MeetingPlatformAdapter.resolveAudioRuntimeForFormat({
backend: params.config.chrome.audioBackend,
bufferBytes: params.config.chrome.audioBufferBytes,
format: params.config.chrome.audioFormat,
inputCommand: params.config.chrome.audioInputCommandOverride,
outputCommand: params.config.chrome.audioOutputCommandOverride,
});
await MeetingPlatformAdapter.ensureAudioBackend({
backend: audio.backend,
timeoutMs: params.timeoutMs,
run: async (argv, timeoutMs) => {
const result = await params.runtime.system.runCommandWithTimeout(argv, { timeoutMs });
return { ...result, code: result.code ?? 1 };
},
});
return audio;
}
export async function assertGoogleMeetAudioAvailable(params: {
runtime: PluginRuntime;
config: GoogleMeetConfig;
timeoutMs: number;
}): Promise<void> {
if (process.platform !== "darwin") {
throw new Error("Chrome Meet transport with blackhole-2ch audio is currently macOS-only");
}
const result = await params.runtime.system.runCommandWithTimeout(
[GOOGLE_MEET_SYSTEM_PROFILER_COMMAND, "SPAudioDataType"],
{ timeoutMs: params.timeoutMs },
);
const output = `${result.stdout ?? ""}\n${result.stderr ?? ""}`;
if (result.code !== 0 || !outputMentionsBlackHole2ch(output)) {
const hint =
params.runtime.system.formatNativeDependencyHint?.({
packageName: "BlackHole 2ch",
downloadCommand: "brew install blackhole-2ch",
}) ?? "";
throw new Error(
[
"BlackHole 2ch audio device not found.",
"Install BlackHole 2ch and route Chrome input/output through the OpenClaw audio bridge.",
hint,
]
.filter(Boolean)
.join(" "),
);
}
await prepareGoogleMeetAudioRuntime(params);
}
export async function launchChromeMeet(params: {
@@ -95,18 +99,21 @@ export async function launchChromeMeet(params: {
logger: RuntimeLogger;
}): Promise<{
launched: boolean;
audioBackend?: MeetingAudioBackend;
audioBridge?:
| { type: "external-command" }
| ({ type: "command-pair" } & ChromeRealtimeAudioBridgeHandle);
browser?: GoogleMeetChromeHealth;
tab?: GoogleMeetBrowserTab;
}> {
let audio: MeetingAudioRuntime | undefined;
const checkRealtimeAudioPrerequisites = async () => {
if (!MeetingPlatformAdapter.isTalkBackMode(params.mode)) {
return;
}
await assertBlackHole2chAvailable({
audio = await prepareGoogleMeetAudioRuntime({
runtime: params.runtime,
config: params.config,
timeoutMs: Math.min(params.config.chrome.joinTimeoutMs, 10_000),
});
@@ -153,9 +160,12 @@ export async function launchChromeMeet(params: {
"Chrome talk-back mode requires chrome.audioInputCommand and chrome.audioOutputCommand, or chrome.audioBridgeCommand for an external bridge.",
);
}
if (!audio) {
throw new Error("Google Meet audio backend was not prepared.");
}
const transport = createLocalMeetingRealtimeAudioTransport({
inputCommand: params.config.chrome.audioInputCommand,
outputCommand: params.config.chrome.audioOutputCommand,
inputCommand: audio.inputCommand,
outputCommand: audio.outputCommand,
audioFormat: params.config.chrome.audioFormat,
bargeInInputCommand: params.config.chrome.bargeInInputCommand,
bargeInRmsThreshold: params.config.chrome.bargeInRmsThreshold,
@@ -196,8 +206,8 @@ export async function launchChromeMeet(params: {
});
return {
type: "command-pair",
inputCommand: params.config.chrome.audioInputCommand,
outputCommand: params.config.chrome.audioOutputCommand,
inputCommand: audio.inputCommand,
outputCommand: audio.outputCommand,
...engine,
};
};
@@ -205,9 +215,30 @@ export async function launchChromeMeet(params: {
await checkRealtimeAudioPrerequisites();
if (!params.config.chrome.launch) {
// An external owner supplies the already-open call, so no browser health exists to gate on.
// Configuring this mode explicitly authorizes its realtime bridge to start immediately.
return { launched: false, audioBridge: await startRealtimeAudioBridge() };
const recovered = await recoverMeetingBrowserTab({
adapter: GOOGLE_MEET_PLATFORM_ADAPTER,
allowSessionAdoption: true,
autoJoin: params.config.chrome.autoJoin,
callBrowser: await resolveLocalMeetingBrowserRequest(params.runtime),
captureCaptions: shouldCaptureCaptions(params.mode, params.fullConfig),
config: params.config.chrome,
locationLabel: "in local Chrome",
meetingSessionId: params.meetingSessionId,
mode: params.mode,
requestedMeetingUrl: params.url,
trackedMeetingUrl: params.url,
trackedTargetId: undefined,
});
const audioBridge = MeetingPlatformAdapter.isRealtimeRouteReady(params.mode, recovered.browser)
? await startRealtimeAudioBridge()
: undefined;
return {
launched: false,
audioBackend: audio?.backend,
audioBridge,
browser: recovered.browser,
tab: recovered.targetId ? { targetId: recovered.targetId, openedByPlugin: false } : undefined,
};
}
const result = await openMeetingWithBrowser({
@@ -221,18 +252,18 @@ export async function launchChromeMeet(params: {
url: params.url,
},
});
const shouldStartRealtimeBridge =
MeetingPlatformAdapter.isTalkBackMode(params.mode) &&
result.browser?.inCall === true &&
result.browser.micMuted === false &&
result.browser.manualAction === undefined;
const shouldStartRealtimeBridge = MeetingPlatformAdapter.isRealtimeRouteReady(
params.mode,
result.browser,
);
const audioBridge = shouldStartRealtimeBridge ? await startRealtimeAudioBridge() : undefined;
return { ...result, audioBridge };
return { ...result, audioBackend: audio?.backend, audioBridge };
}
function parseNodeStartResult(raw: unknown): {
launched?: boolean;
bridgeId?: string;
audioBackend?: MeetingAudioBackend;
audioBridge?: { type?: string; outputGeneration?: boolean };
browser?: GoogleMeetChromeHealth;
} {
@@ -246,6 +277,7 @@ function parseNodeStartResult(raw: unknown): {
return value as {
launched?: boolean;
bridgeId?: string;
audioBackend?: MeetingAudioBackend;
audioBridge?: { type?: string; outputGeneration?: boolean };
browser?: GoogleMeetChromeHealth;
};
@@ -365,17 +397,39 @@ async function openMeetWithBrowserProxy(params: {
meetingSessionId: string;
url: string;
}): Promise<{ launched: boolean; browser?: GoogleMeetChromeHealth; tab?: GoogleMeetBrowserTab }> {
const callBrowser: MeetingBrowserRequestCaller = async (request) =>
await callBrowserProxyOnNode({
runtime: params.runtime,
nodeId: params.nodeId,
method: request.method,
path: request.path,
body: request.body,
timeoutMs: request.timeoutMs,
});
if (!params.config.chrome.launch) {
const recovered = await recoverMeetingBrowserTab({
adapter: GOOGLE_MEET_PLATFORM_ADAPTER,
allowSessionAdoption: true,
autoJoin: params.config.chrome.autoJoin,
callBrowser,
captureCaptions: params.captureCaptions,
config: params.config.chrome,
locationLabel: "on the selected Chrome node",
meetingSessionId: params.meetingSessionId,
mode: params.mode,
requestedMeetingUrl: params.url,
trackedMeetingUrl: params.url,
trackedTargetId: undefined,
});
return {
launched: false,
browser: recovered.browser,
tab: recovered.targetId ? { targetId: recovered.targetId, openedByPlugin: false } : undefined,
};
}
return await openMeetingWithBrowser({
adapter: GOOGLE_MEET_PLATFORM_ADAPTER,
callBrowser: async (request) =>
await callBrowserProxyOnNode({
runtime: params.runtime,
nodeId: params.nodeId,
method: request.method,
path: request.path,
body: request.body,
timeoutMs: request.timeoutMs,
}),
callBrowser,
config: params.config.chrome,
session: {
captureCaptions: params.captureCaptions,
@@ -481,6 +535,7 @@ export async function launchChromeMeetOnNode(params: {
}): Promise<{
nodeId: string;
launched: boolean;
audioBackend?: MeetingAudioBackend;
audioBridge?:
| { type: "external-command" }
| ({ type: "node-command-pair" } & ChromeNodeRealtimeAudioBridgeHandle);
@@ -509,6 +564,27 @@ export async function launchChromeMeetOnNode(params: {
}`,
);
}
const setup = MeetingPlatformAdapter.isTalkBackMode(params.mode)
? parseNodeStartResult(
await params.runtime.nodes.invoke({
nodeId,
command: GOOGLE_MEET_NODE_COMMAND,
params: {
action: "setup",
audioBackend: params.config.chrome.audioBackend,
audioFormat: params.config.chrome.audioFormat,
audioBufferBytes: params.config.chrome.audioBufferBytes,
...(params.config.chrome.audioInputCommandOverride
? { audioInputCommand: params.config.chrome.audioInputCommandOverride }
: {}),
...(params.config.chrome.audioOutputCommandOverride
? { audioOutputCommand: params.config.chrome.audioOutputCommandOverride }
: {}),
},
timeoutMs: 12_000,
}),
)
: undefined;
const browserControl = await openMeetWithBrowserProxy({
runtime: params.runtime,
nodeId,
@@ -521,15 +597,13 @@ export async function launchChromeMeetOnNode(params: {
// launch:false explicitly delegates call state to an already-open session.
// Browser-managed joins require explicit unmuted health before node audio starts.
if (
params.config.chrome.launch &&
MeetingPlatformAdapter.isTalkBackMode(params.mode) &&
(browserControl.browser?.inCall !== true ||
browserControl.browser.micMuted !== false ||
browserControl.browser.manualAction)
!MeetingPlatformAdapter.isRealtimeRouteReady(params.mode, browserControl.browser)
) {
return {
nodeId,
launched: browserControl.launched,
audioBackend: setup?.audioBackend,
browser: browserControl.browser,
tab: browserControl.tab,
};
@@ -544,8 +618,15 @@ export async function launchChromeMeetOnNode(params: {
launch: false,
browserProfile: params.config.chrome.browserProfile,
joinTimeoutMs: params.config.chrome.joinTimeoutMs,
audioInputCommand: params.config.chrome.audioInputCommand,
audioOutputCommand: params.config.chrome.audioOutputCommand,
audioBackend: params.config.chrome.audioBackend,
audioFormat: params.config.chrome.audioFormat,
audioBufferBytes: params.config.chrome.audioBufferBytes,
...(params.config.chrome.audioInputCommandOverride
? { audioInputCommand: params.config.chrome.audioInputCommandOverride }
: {}),
...(params.config.chrome.audioOutputCommandOverride
? { audioOutputCommand: params.config.chrome.audioOutputCommandOverride }
: {}),
audioBridgeCommand: params.config.chrome.audioBridgeCommand,
audioBridgeHealthCommand: params.config.chrome.audioBridgeHealthCommand,
},
@@ -614,6 +695,7 @@ export async function launchChromeMeetOnNode(params: {
return {
nodeId,
launched: browserControl.launched || result.launched === true,
audioBackend: result.audioBackend ?? setup?.audioBackend,
audioBridge: bridge,
browser: browserControl.browser ?? result.browser,
tab: browserControl.tab,
@@ -623,6 +705,7 @@ export async function launchChromeMeetOnNode(params: {
return {
nodeId,
launched: browserControl.launched || result.launched === true,
audioBackend: result.audioBackend ?? setup?.audioBackend,
audioBridge: { type: "external-command" },
browser: browserControl.browser ?? result.browser,
tab: browserControl.tab,
@@ -631,6 +714,7 @@ export async function launchChromeMeetOnNode(params: {
return {
nodeId,
launched: browserControl.launched || result.launched === true,
audioBackend: result.audioBackend ?? setup?.audioBackend,
browser: browserControl.browser ?? result.browser,
tab: browserControl.tab,
};
extensions/google-meet/src/transports/google-meet-page-scripts.ts
+206 -22
View File
@@ -19,7 +19,7 @@ export function meetStatusScript(params: {
const captionSessionId = ${JSON.stringify(params.captionSessionId)};
const captureCaptions = ${JSON.stringify(params.captureCaptions)};
const readOnly = ${JSON.stringify(Boolean(params.readOnly))};
const buttons = [...document.querySelectorAll('button')];
const buttons = () => [...document.querySelectorAll('button')];
const buttonLabel = (button) =>
[
button.getAttribute("aria-label"),
@@ -28,21 +28,40 @@ export function meetStatusScript(params: {
]
.filter(Boolean)
.join(" ");
const buttonLabels = buttons.map(buttonLabel).filter(Boolean);
const buttonLabels = buttons().map(buttonLabel).filter(Boolean);
const notes = [];
let audioInputRouted;
let audioInputDeviceLabel;
let audioInputRouteError;
let audioOutputRouted;
let audioOutputDeviceLabel;
let audioOutputRouteError;
const findButton = (pattern) =>
buttons.find((button) => {
buttons().find((button) => {
const label = buttonLabel(button);
return pattern.test(label) && !button.disabled;
});
const findCallControlButton = (pattern) =>
buttons.find((button) => {
buttons().find((button) => {
const label = buttonLabel(button);
return pattern.test(label) && !/remotely mute|someone else/i.test(label) && !button.disabled;
});
const audioDeviceFamily = (value) => {
const label = String(value || '');
if (/\\bOpenClaw Meeting Audio\\b/i.test(label)) return 'openclaw-meeting-audio';
if (/\\bBlackHole\\s+2ch\\b/i.test(label)) return 'blackhole-2ch';
return undefined;
};
const isMeetingAudioDevice = (value) => Boolean(audioDeviceFamily(value));
const deviceNodeLabel = (node) => [
node?.getAttribute?.('aria-label'),
node?.getAttribute?.('data-tooltip'),
node?.getAttribute?.('title'),
node?.label,
node?.textContent,
node?.innerText,
].filter(Boolean).join(' ').trim();
const waitForUi = () => new Promise((resolve) => setTimeout(resolve, 100));
const input = [...document.querySelectorAll('input')].find((el) =>
/your name/i.test(el.getAttribute('aria-label') || el.placeholder || '')
);
@@ -64,29 +83,181 @@ export function meetStatusScript(params: {
/^\\s*turn (?:off|on) microphone\\b/i.test(buttonLabel(button))
);
}
if (!readOnly && allowMicrophone && mic && /turn on microphone/i.test(buttonLabel(mic))) {
mic.click();
notes.push("Attempted to turn on the Meet microphone for talk-back mode.");
}
if (!readOnly && !allowMicrophone && mic && /turn off microphone/i.test(mic.getAttribute('aria-label') || text(mic))) {
mic.click();
notes.push("Muted Meet microphone for observe-only mode.");
}
const joinElsewhere = findButton(/join here too/i);
const join = !readOnly && ${JSON.stringify(params.autoJoin)}
? findButton(/join now|ask to join/i)
: null;
if (join) join.click();
const microphoneChoice = findButton(/\\buse microphone\\b/i);
const noMicrophoneChoice = findButton(/\\b(continue|join|use) without (microphone|mic)\\b|\\bnot now\\b/i);
if (!readOnly && allowMicrophone && microphoneChoice) {
microphoneChoice.click();
notes.push("Accepted Meet microphone prompt with browser automation.");
await waitForUi();
} else if (!readOnly && !allowMicrophone && noMicrophoneChoice) {
noMicrophoneChoice.click();
notes.push("Skipped Meet microphone prompt for observe-only mode.");
await waitForUi();
}
const inCall = buttons.some((button) => /leave call/i.test(button.getAttribute('aria-label') || text(button)));
const findMicrophoneDeviceControl = () => {
const selectors = [
'select[aria-label*="microphone" i]',
'select[name*="microphone" i]',
'[role="combobox"][aria-label*="microphone" i]',
'[role="listbox"][aria-label*="microphone" i]',
'[aria-haspopup="listbox"][aria-label*="microphone" i]',
];
for (const selector of selectors) {
const control = document.querySelector(selector);
if (control) return control;
}
return undefined;
};
const selectedMicrophoneLabel = () => {
const control = findMicrophoneDeviceControl();
const nativeSelected = control?.selectedOptions?.[0];
if (nativeSelected && isMeetingAudioDevice(deviceNodeLabel(nativeSelected))) {
return deviceNodeLabel(nativeSelected);
}
const controlledId = control?.getAttribute?.('aria-controls');
const optionsRoot = controlledId ? document.getElementById?.(controlledId) : control;
const selected = [...(optionsRoot?.querySelectorAll?.(
'[role="option"][aria-selected="true"], [role="menuitemradio"][aria-checked="true"], [role="radio"][aria-checked="true"]'
) || [])].find((node) => isMeetingAudioDevice(deviceNodeLabel(node)));
if (selected) return deviceNodeLabel(selected);
const role = control?.getAttribute?.('role');
if (
!control?.options &&
role !== 'listbox' &&
control &&
isMeetingAudioDevice(deviceNodeLabel(control))
) {
return deviceNodeLabel(control);
}
return undefined;
};
const openMicrophoneDeviceSettings = async () => {
if (findMicrophoneDeviceControl()) return;
const candidates = () => [
...document.querySelectorAll('button, [role="menuitem"], [role="tab"]'),
];
const direct = candidates().find((node) =>
/\\b(?:audio|microphone|device) settings\\b/i.test(deviceNodeLabel(node)) && !node.disabled
);
let settings = direct || candidates().find((node) =>
/^\\s*settings\\s*$/i.test(deviceNodeLabel(node)) && !node.disabled
);
if (!settings && !readOnly) {
const moreOptions = candidates().find((node) =>
/^\\s*more options\\s*$/i.test(deviceNodeLabel(node)) && !node.disabled
);
if (moreOptions) {
moreOptions.click();
await waitForUi();
settings = candidates().find((node) =>
/^\\s*settings\\s*$/i.test(deviceNodeLabel(node)) && !node.disabled
);
}
}
if (!settings || readOnly) return;
settings.click();
await waitForUi();
const audioTab = [...document.querySelectorAll('button, [role="tab"]')].find((node) =>
/^\\s*audio\\s*$/i.test(deviceNodeLabel(node)) && !node.disabled
);
if (audioTab) {
audioTab.click();
await waitForUi();
}
};
const routeMeetAudioInput = async () => {
if (
!allowMicrophone ||
typeof navigator === 'undefined' ||
!navigator.mediaDevices?.enumerateDevices
) return;
try {
const devices = await navigator.mediaDevices.enumerateDevices();
const input = devices.find((device) =>
device.kind === 'audioinput' && isMeetingAudioDevice(device.label)
);
if (!input?.deviceId) {
audioInputRouted = false;
audioInputRouteError = 'A supported virtual microphone was not visible to Meet.';
return;
}
audioInputDeviceLabel = input.label || 'OpenClaw meeting audio';
const inputFamily = audioDeviceFamily(audioInputDeviceLabel);
if (audioDeviceFamily(selectedMicrophoneLabel()) === inputFamily) {
audioInputRouted = true;
return;
}
if (readOnly) {
audioInputRouted = false;
return;
}
await openMicrophoneDeviceSettings();
const control = findMicrophoneDeviceControl();
if (!control) {
audioInputRouted = false;
audioInputRouteError = 'Meet microphone device selector was not available.';
return;
}
const nativeOptions = [...(control.options || [])];
const nativeOption = nativeOptions.find(
(option) => audioDeviceFamily(deviceNodeLabel(option)) === inputFamily
);
if (nativeOption) {
control.value = nativeOption.value;
nativeOption.selected = true;
control.dispatchEvent(new Event('input', { bubbles: true }));
control.dispatchEvent(new Event('change', { bubbles: true }));
} else {
control.click?.();
await waitForUi();
const choices = [...document.querySelectorAll(
'[role="option"], [role="menuitemradio"], [role="radio"]'
)];
const choice = choices.find(
(node) => audioDeviceFamily(deviceNodeLabel(node)) === inputFamily && !node.disabled
);
choice?.click?.();
}
await waitForUi();
audioInputRouted = audioDeviceFamily(selectedMicrophoneLabel()) === inputFamily;
if (audioInputRouted) {
notes.push(\`Selected \${audioInputDeviceLabel} as the Meet microphone.\`);
} else {
audioInputRouteError = \`Meet did not confirm \${audioInputDeviceLabel} as its microphone.\`;
}
} catch (error) {
audioInputRouted = false;
audioInputRouteError = error?.message || String(error);
notes.push(\`Could not select the Meet virtual microphone: \${audioInputRouteError}\`);
}
};
await routeMeetAudioInput();
mic = findCallControlButton(/^\\s*turn (?:off|on) microphone\\b/i) || mic;
if (
!readOnly &&
allowMicrophone &&
audioInputRouted !== true &&
mic &&
/turn off microphone/i.test(buttonLabel(mic))
) {
mic.click();
notes.push("Muted the Meet microphone because the virtual audio input was not verified.");
mic = findCallControlButton(/^\\s*turn (?:off|on) microphone\\b/i) || mic;
}
if (!readOnly && allowMicrophone && audioInputRouted === true && mic && /turn on microphone/i.test(buttonLabel(mic))) {
mic.click();
notes.push("Turned on the Meet microphone after verifying the virtual audio input.");
}
if (!readOnly && !allowMicrophone && mic && /turn off microphone/i.test(mic.getAttribute('aria-label') || text(mic))) {
mic.click();
notes.push("Muted Meet microphone for observe-only mode.");
}
const join = !readOnly && ${JSON.stringify(params.autoJoin)}
? findButton(/join now|ask to join/i)
: null;
if (join) join.click();
const inCall = buttons().some((button) => /leave call/i.test(button.getAttribute('aria-label') || text(button)));
const routeMeetAudioOutput = async () => {
if (
!allowMicrophone ||
@@ -98,14 +269,18 @@ export function meetStatusScript(params: {
if (mediaElements.length === 0) return;
try {
const devices = await navigator.mediaDevices.enumerateDevices();
const inputFamily = audioDeviceFamily(audioInputDeviceLabel);
const output = devices.find((device) =>
device.kind === 'audiooutput' && /\\bBlackHole\\s+2ch\\b/i.test(device.label || '')
device.kind === 'audiooutput' &&
inputFamily &&
audioDeviceFamily(device.label) === inputFamily
) || devices.find((device) =>
device.kind === 'audiooutput' && /\\bBlackHole\\b/i.test(device.label || '')
device.kind === 'audiooutput' && isMeetingAudioDevice(device.label)
);
if (!output?.deviceId) {
audioOutputRouted = false;
if (devices.some((device) => device.kind === 'audiooutput')) {
notes.push("BlackHole 2ch speaker output was not visible to Meet.");
notes.push("A supported virtual speaker output was not visible to Meet.");
}
return;
}
@@ -120,7 +295,7 @@ export function meetStatusScript(params: {
}
}
audioOutputRouted = mediaElements.some((element) => element.sinkId === output.deviceId);
audioOutputDeviceLabel = output.label || "BlackHole 2ch";
audioOutputDeviceLabel = output.label || "OpenClaw meeting audio";
if (!readOnly && audioOutputRouted) {
notes.push(
routed > 0
@@ -129,8 +304,9 @@ export function meetStatusScript(params: {
);
}
} catch (error) {
audioOutputRouted = false;
audioOutputRouteError = error?.message || String(error);
notes.push(\`Could not route Meet speaker output to BlackHole 2ch: \${audioOutputRouteError}\`);
notes.push(\`Could not route Meet speaker output to the virtual audio device: \${audioOutputRouteError}\`);
}
};
if (inCall) {
@@ -322,6 +498,11 @@ export function meetStatusScript(params: {
manualAction = manualActionFor("meet-admission-required", "Admit the OpenClaw browser participant in Google Meet, then retry speech.");
} else if (permissionNeeded) {
manualAction = manualActionFor("meet-permission-required", allowMicrophone ? "Allow microphone/camera/speaker permissions for Meet in the OpenClaw browser profile, then retry." : "Join without microphone/camera permissions in the OpenClaw browser profile, then retry.");
} else if (inCall && allowMicrophone && (audioInputRouted !== true || audioOutputRouted !== true)) {
manualAction = manualActionFor(
"meet-audio-choice-required",
"Select BlackHole 2ch or OpenClaw Meeting Audio as both the Meet microphone and speaker, then retry."
);
} else if (!inCall && (allowMicrophone ? !microphoneChoice : !noMicrophoneChoice) && /do you want people to hear you in the meeting/i.test(pageText)) {
manualAction = manualActionFor("meet-audio-choice-required", allowMicrophone ? "Meet is showing the microphone choice. Click Use microphone in the OpenClaw browser profile, then retry." : "Meet is showing the microphone choice. Choose the no-microphone option in the OpenClaw browser profile, then retry.");
}
@@ -339,6 +520,9 @@ export function meetStatusScript(params: {
lastCaptionSpeaker,
lastCaptionText,
recentTranscript,
audioInputRouted,
audioInputDeviceLabel,
audioInputRouteError,
audioOutputRouted,
audioOutputDeviceLabel,
audioOutputRouteError,
extensions/google-meet/src/transports/google-meet-platform-adapter.test.ts
+203 -1
View File
@@ -1,6 +1,135 @@
import { describe, expect, it } from "vitest";
import { runInNewContext } from "node:vm";
import { MeetingPlatformAdapter } from "openclaw/plugin-sdk/meeting-runtime";
import { describe, expect, it, vi } from "vitest";
import { meetStatusScript } from "./google-meet-page-scripts.js";
import { GOOGLE_MEET_PLATFORM_ADAPTER } from "./google-meet-platform-adapter.js";
const MEETING_URL = "https://meet.google.com/abc-defg-hij";
function pageNode(label: string) {
return {
disabled: false,
textContent: label,
click: vi.fn(),
getAttribute: (name: string) => (name === "aria-label" ? label : null),
};
}
function microphoneSelect(labels: string[]) {
const options = labels.map((label, index) => ({
label,
selected: index === 0,
textContent: label,
value: `device-${index}`,
}));
let value = options[0]?.value;
return {
dispatchEvent: vi.fn(),
get options() {
return options;
},
get selectedOptions() {
return options.filter((option) => option.selected);
},
getAttribute: (name: string) => (name === "aria-label" ? "Microphone" : null),
textContent: "",
get value() {
return value;
},
set value(next: string | undefined) {
value = next;
for (const option of options) {
option.selected = option.value === next;
}
},
};
}
async function runAudioStatus(
label: string,
selectLabels = ["MacBook Microphone", label],
initialMicrophoneLabel = "Turn on microphone",
) {
let microphoneLabel = initialMicrophoneLabel;
const microphone = pageNode("");
microphone.getAttribute = (name: string) => (name === "aria-label" ? microphoneLabel : null);
microphone.click.mockImplementation(() => {
microphoneLabel = /turn on microphone/i.test(microphoneLabel)
? "Turn off microphone"
: "Turn on microphone";
});
const leave = pageNode("Leave call");
const select = microphoneSelect(selectLabels);
const media = {
sinkId: "",
setSinkId: vi.fn(async (value: string) => {
media.sinkId = value;
}),
};
const buttons = [microphone, leave];
const document = {
body: { textContent: "" },
title: "Meet",
querySelector(selector: string) {
if (selector.includes("select") && selector.includes("microphone")) {
return select;
}
return null;
},
querySelectorAll(selector: string) {
if (selector === "button") {
return buttons;
}
if (selector === "input") {
return [];
}
if (selector === "audio, video") {
return [media];
}
if (selector.includes("button") || selector.includes('[role="')) {
return buttons;
}
return [];
},
};
const outputLabel = label.includes("OpenClaw") ? "OpenClaw Meeting Audio" : label;
const result = await runInNewContext(
`(${meetStatusScript({
allowMicrophone: true,
autoJoin: false,
captureCaptions: false,
guestName: "OpenClaw Agent",
})})()`,
{
Event: globalThis.Event,
JSON,
String,
document,
location: { href: MEETING_URL, hostname: "meet.google.com" },
navigator: {
mediaDevices: {
enumerateDevices: async () => [
{ deviceId: "input-1", kind: "audioinput", label },
{ deviceId: "output-1", kind: "audiooutput", label: outputLabel },
],
},
},
setTimeout: (callback: () => void) => {
callback();
return 1;
},
clearTimeout,
window: {},
},
);
return {
health: JSON.parse(result) as Record<string, unknown>,
media,
microphone,
select,
};
}
describe("GOOGLE_MEET_PLATFORM_ADAPTER captions", () => {
it("enables caption capture for durable notes in every browser mode", () => {
expect(GOOGLE_MEET_PLATFORM_ADAPTER.browser.captions.enabled("agent")).toBe(true);
@@ -8,3 +137,76 @@ describe("GOOGLE_MEET_PLATFORM_ADAPTER captions", () => {
expect(GOOGLE_MEET_PLATFORM_ADAPTER.browser.captions.enabled("transcribe")).toBe(true);
});
});
describe("GOOGLE_MEET_PLATFORM_ADAPTER audio routing", () => {
it.each(["BlackHole 2ch", "Monitor of OpenClaw Meeting Audio"])(
"selects and verifies %s for bidirectional Meet audio",
async (label) => {
const { health, media, microphone, select } = await runAudioStatus(label);
expect(health).toMatchObject({
audioInputRouted: true,
audioInputDeviceLabel: label,
audioOutputRouted: true,
audioOutputDeviceLabel: label.includes("OpenClaw") ? "OpenClaw Meeting Audio" : label,
micMuted: false,
});
expect(health.manualAction).toBeUndefined();
expect(select.dispatchEvent).toHaveBeenCalledTimes(2);
expect(microphone.click).toHaveBeenCalledOnce();
expect(media.setSinkId).toHaveBeenCalledWith("output-1");
},
);
it("parses input-route health and retries until both routes are ready", () => {
const pending = GOOGLE_MEET_PLATFORM_ADAPTER.browser.parseStatus({
result: JSON.stringify({
inCall: true,
micMuted: false,
audioInputRouted: true,
audioInputDeviceLabel: "OpenClaw Meeting Audio",
audioOutputRouted: false,
manualAction: {
reason: "meet-audio-choice-required",
message: "Select the virtual speaker",
},
}),
});
expect(pending).toMatchObject({
audioInputRouted: true,
audioInputDeviceLabel: "OpenClaw Meeting Audio",
audioOutputRouted: false,
});
if (!pending) {
throw new Error("expected parsed Google Meet browser health");
}
expect(GOOGLE_MEET_PLATFORM_ADAPTER.browser.shouldRetryJoinStatus?.(pending)).toBe(true);
expect(
MeetingPlatformAdapter.isRealtimeRouteReady("agent", {
...pending,
audioOutputRouted: true,
manualAction: undefined,
}),
).toBe(true);
expect(MeetingPlatformAdapter.isRealtimeRouteReady("agent", pending)).toBe(false);
});
it("keeps Meet muted and reports manual action when input selection cannot be verified", async () => {
const { health, microphone } = await runAudioStatus(
"OpenClaw Meeting Audio",
["MacBook Microphone"],
"Turn off microphone",
);
expect(health).toMatchObject({
audioInputRouted: false,
audioInputRouteError: "Meet did not confirm OpenClaw Meeting Audio as its microphone.",
audioOutputRouted: true,
micMuted: true,
manualAction: { reason: "meet-audio-choice-required" },
});
expect(microphone.click).toHaveBeenCalledOnce();
expect(MeetingPlatformAdapter.isRealtimeRouteReady("agent", health)).toBe(false);
});
});
extensions/google-meet/src/transports/google-meet-platform-adapter.ts
+4
View File
@@ -139,6 +139,10 @@ export const GOOGLE_MEET_PLATFORM_ADAPTER = MeetingPlatformAdapter.create<
guestName: params.guestName,
readOnly: params.readOnly,
}),
shouldRetryJoinStatus: (health) =>
health.inCall === true &&
health.manualAction?.reason === "meet-audio-choice-required" &&
(health.audioInputRouted !== true || health.audioOutputRouted !== true),
browserControlUnavailable: () => ({
category: "browser-control-unavailable",
reason: "browser-control-unavailable",
extensions/google-meet/src/transports/types.ts
+4 -1
View File
@@ -93,6 +93,9 @@ export type GoogleMeetChromeHealth = MeetingBrowserHealth<
providerConnected?: boolean;
realtimeReady?: boolean;
audioInputActive?: boolean;
audioInputRouted?: boolean;
audioInputDeviceLabel?: string;
audioInputRouteError?: string;
audioOutputActive?: boolean;
audioOutputRouted?: boolean;
audioOutputDeviceLabel?: string;
@@ -131,7 +134,7 @@ export type GoogleMeetSession = MeetingSessionRecord<
> & {
/** Canonical agent owner and shared fields retain their byte-compatible wire names. */
chrome?: {
audioBackend: "blackhole-2ch";
audioBackend?: "blackhole-2ch" | "pipewire-pulse";
launched: boolean;
nodeId?: string;
browserProfile?: string;
extensions/teams-meetings/openclaw.plugin.json
+7 -3
View File
@@ -20,13 +20,17 @@
"help": "Agent consults OpenClaw, bidi uses direct realtime voice, and transcribe observes only."
},
"chrome.browserProfile": { "label": "Chrome Profile", "advanced": true },
"chrome.audioBackend": {
"label": "Chrome Audio Backend",
"help": "Auto selects BlackHole 2ch on macOS or PipeWire-Pulse on Linux."
},
"chrome.guestName": { "label": "Guest Name" },
"chrome.waitForInCallMs": { "label": "Wait For In-Call (ms)", "advanced": true },
"chrome.audioInputCommand": { "label": "Audio Input Command", "advanced": true },
"chrome.audioOutputCommand": { "label": "Audio Output Command", "advanced": true },
"chromeNode.node": {
"label": "Chrome Node",
"help": "Node id/name/IP that owns Chrome, BlackHole, and SoX.",
"help": "Node id/name/IP that owns Chrome and the native virtual-audio backend.",
"advanced": true
},
"realtime.transcriptionProvider": { "label": "Realtime Transcription Provider" },
@@ -53,8 +57,8 @@
"properties": {
"audioBackend": {
"type": "string",
"enum": ["blackhole-2ch"],
"default": "blackhole-2ch"
"enum": ["auto", "blackhole-2ch", "pipewire-pulse"],
"default": "auto"
},
"audioFormat": {
"type": "string",
extensions/teams-meetings/src/config.test.ts
+31 -3
View File
@@ -7,15 +7,38 @@ const resolveTeamsMeetingsGatewayOperationTimeoutMs =
teamsMeetingsConfig.resolveGatewayOperationTimeoutMs;
describe("Microsoft Teams meetings config", () => {
it("keeps sparse and legacy audio config compatible", () => {
const sparse = resolveTeamsMeetingsConfig({});
expect(sparse.chrome.audioBackend).toBe("auto");
expect(sparse.chrome.audioInputCommandOverride).toBeUndefined();
expect(sparse.chrome.audioOutputCommandOverride).toBeUndefined();
const legacy = resolveTeamsMeetingsConfig({
chrome: { audioBackend: "blackhole-2ch" },
});
expect(legacy.chrome.audioBackend).toBe("blackhole-2ch");
expect(legacy.chrome.audioInputCommand).toContain("BlackHole 2ch");
expect(legacy.chrome.audioOutputCommand).toContain("BlackHole 2ch");
});
it("allows the live Teams web client enough time to reach prejoin and in-call UI", () => {
expect(resolveTeamsMeetingsConfig({}).chrome.waitForInCallMs).toBe(60_000);
});
it("builds matching SoX commands from the selected audio format", () => {
const config = resolveTeamsMeetingsConfig({ chrome: { audioBufferBytes: 2048 } });
it("builds native command pairs for the selected audio backend", () => {
const config = resolveTeamsMeetingsConfig({
chrome: { audioBackend: "blackhole-2ch", audioBufferBytes: 2048 },
});
expect(config.chrome.audioInputCommand).toContain("sox");
expect(config.chrome.audioInputCommand).toContain("2048");
expect(config.chrome.audioOutputCommand).toContain("BlackHole 2ch");
const linux = resolveTeamsMeetingsConfig({
chrome: { audioBackend: "pipewire-pulse", audioBufferBytes: 2_048 },
});
expect(linux.chrome.audioInputCommand).toContain("parec");
expect(linux.chrome.audioOutputCommand).toContain("pacat");
expect(linux.chrome.audioInputCommand).toContain("--latency-msec=43");
});
it("preserves explicit command overrides and realtime passthrough", () => {
@@ -31,7 +54,12 @@ describe("Microsoft Teams meetings config", () => {
});
expect(config).toMatchObject({
defaultMode: "bidi",
chrome: { audioInputCommand: ["capture"], audioOutputCommand: ["play"] },
chrome: {
audioInputCommand: ["capture"],
audioOutputCommand: ["play"],
audioInputCommandOverride: ["capture"],
audioOutputCommandOverride: ["play"],
},
chromeNode: { node: "mac-node" },
realtime: {
voiceProvider: "google",
extensions/teams-meetings/src/config.ts
-2
View File
@@ -6,8 +6,6 @@ export const teamsMeetingsConfig = MeetingPlatformAdapter.createPluginConfigSche
defaultRealtimeInstructions: `You are joining a private Microsoft Teams meeting as an OpenClaw voice transport. Keep spoken replies brief and natural. In agent mode, wait for OpenClaw consult results and speak them exactly. In bidi mode, answer directly and call ${REALTIME_VOICE_AGENT_CONSULT_TOOL_NAME} for deeper reasoning, current information, or tools.`,
resolveGatewayOperationTimeoutMs: (config) =>
Math.max(60_000, addTimerTimeoutGraceMs(config.chrome.joinTimeoutMs, 30_000) ?? 1),
resolveSoxAudioDevice: ({ format }) =>
format === "g711-ulaw-8khz" ? undefined : { device: "BlackHole 2ch", deviceType: "coreaudio" },
});
export type TeamsMeetingsConfig = ReturnType<typeof teamsMeetingsConfig.resolveConfig>;
extensions/teams-meetings/src/node-host.test.ts
+13 -3
View File
@@ -41,12 +41,16 @@ describe("Teams meeting node-host prerequisite deadline", () => {
it("shares one timeout budget across every prerequisite probe", async () => {
const now = vi.spyOn(Date, "now");
for (const value of [1_000, 1_000, 4_000, 4_000, 8_000, 8_000]) {
for (const value of [1_000, 1_000, 4_000, 8_000]) {
now.mockReturnValueOnce(value);
}
await expect(handleTeamsMeetingsNodeHostCommand(setupParams())).resolves.toBe(
JSON.stringify({ ok: true }),
JSON.stringify({
ok: true,
audioBackend: "blackhole-2ch",
audioDeviceLabel: "BlackHole 2ch",
}),
);
expect(
@@ -63,7 +67,13 @@ describe("Teams meeting node-host prerequisite deadline", () => {
audioOutputCommand: teamsMeetingsConfig.defaultAudioOutputCommand,
}),
),
).resolves.toBe(JSON.stringify({ ok: true }));
).resolves.toBe(
JSON.stringify({
ok: true,
audioBackend: "blackhole-2ch",
audioDeviceLabel: "BlackHole 2ch",
}),
);
expect(spawnSyncMock).toHaveBeenCalledTimes(2);
expect(spawnSyncMock.mock.calls[1]?.[1]).toEqual([
extensions/teams-meetings/src/runtime-setup.test.ts
+28 -1
View File
@@ -47,6 +47,9 @@ describe("Microsoft Teams meetings runtime setup", () => {
nodeId: "node-1",
params: {
action: "setup",
audioBackend: "auto",
audioBufferBytes: 4_096,
audioFormat: "pcm16-24khz",
audioInputCommand: ["custom-input", "--read"],
audioOutputCommand: ["custom-output", "--write"],
bargeInInputCommand: ["custom-barge-in"],
@@ -55,7 +58,7 @@ describe("Microsoft Teams meetings runtime setup", () => {
});
expect(status.checks).toContainEqual({
id: "chrome-node-audio-prerequisites",
message: "Remote macOS, BlackHole 2ch, and SoX prerequisites are ready",
message: "Remote virtual audio backend and command-pair prerequisites are ready",
ok: true,
});
expect(status.ok).toBe(true);
@@ -79,4 +82,28 @@ describe("Microsoft Teams meetings runtime setup", () => {
ok: false,
});
});
it("returns structured diagnostics when local talk-back is unsupported", async () => {
const platform = vi.spyOn(process, "platform", "get").mockReturnValue("win32");
const runCommandWithTimeout = vi.fn();
try {
const status = await getTeamsMeetingsSetupStatus({
config: resolveTeamsMeetingsConfig({}),
fullConfig: {},
runtime: { system: { runCommandWithTimeout } } as unknown as PluginRuntime,
options: { mode: "agent", transport: "chrome" },
});
expect(status.ok).toBe(false);
expect(status.checks).toContainEqual({
id: "chrome-local-audio-device",
message: expect.stringContaining("unsupported on win32"),
ok: false,
});
expect(status.checks.some((check) => check.id === "chrome-local-audio-commands")).toBe(false);
expect(runCommandWithTimeout).not.toHaveBeenCalled();
} finally {
platform.mockRestore();
}
});
});
extensions/teams-meetings/src/runtime-setup.ts
+2 -2
View File
@@ -1,13 +1,13 @@
import { MeetingPlatformAdapter } from "openclaw/plugin-sdk/meeting-runtime";
import type { TeamsMeetingsConfig, TeamsMeetingsMode } from "./config.js";
import { assertBlackHole2chAvailable } from "./transports/chrome.js";
import { assertTeamsMeetingsAudioAvailable } from "./transports/chrome.js";
import { TEAMS_MEETINGS_BROWSER_NODE_ADAPTER } from "./transports/teams-meetings-platform-constants.js";
export const getTeamsMeetingsSetupStatus = MeetingPlatformAdapter.createRuntimeSetup<
TeamsMeetingsConfig,
TeamsMeetingsMode
>({
assertAudioDeviceAvailable: assertBlackHole2chAvailable,
assertAudioDeviceAvailable: assertTeamsMeetingsAudioAvailable,
captionsMessage: (mode) =>
mode === "transcribe"
? "Teams caption scraping is disabled pending live selector validation; transcript snapshots are empty"
extensions/teams-meetings/src/runtime.ts
+1 -1
View File
@@ -40,7 +40,7 @@ export const TeamsMeetingsRuntime = MeetingPlatformAdapter.createRuntimeFacade<
durableTranscripts: { providerId: "teams", providerName: "Microsoft Teams" },
joined: {
local:
"Teams guest joined in local Chrome with realtime audio through BlackHole 2ch and SoX.",
"Teams guest joined in local Chrome with realtime audio through the native virtual-audio backend.",
node: "Teams guest joined in Chrome on the selected node with realtime audio through the node bridge.",
transcribe: "Teams guest joined observe-only with live-caption transcript capture.",
waiting:
extensions/teams-meetings/src/transports/chrome.ts
+1 -1
View File
@@ -8,7 +8,7 @@ const chromeTransport = MeetingPlatformAdapter.createPluginChromeTransport({
runtime: MeetingPlatformAdapter.createChromeRuntimeBindings(),
});
export const assertBlackHole2chAvailable = chromeTransport.assertAudioDeviceAvailable;
export const assertTeamsMeetingsAudioAvailable = chromeTransport.assertAudioDeviceAvailable;
export const launchTeamsMeetingInChrome = chromeTransport.launchInChrome;
export const launchTeamsMeetingOnNode = chromeTransport.launchOnNode;
export const leaveTeamsMeetingInBrowser = chromeTransport.leaveInBrowser;
extensions/teams-meetings/src/transports/teams-meetings-caption-capture.test.ts
+2 -1
View File
@@ -37,7 +37,8 @@ describe("Microsoft Teams meeting captions and permissions", () => {
expect(script).toContain("continue on this browser");
expect(script).toContain("someone will let you in shortly");
expect(script).toContain("setSinkId");
expect(script).toContain("BlackHole");
expect(script).toContain("blackhole 2ch");
expect(script).toContain("openclaw meeting audio");
});
it("enables live captions and captures the validated Teams caption row DOM", async () => {
extensions/teams-meetings/src/transports/teams-meetings-platform-adapter.test.ts
+65 -34
View File
@@ -756,41 +756,72 @@ describe("Microsoft Teams meeting platform adapter", () => {
).toBe(false);
});
it("reports verified routes only after the exact input marker and output sink agree", async () => {
const leave = control({ label: "Leave" });
const microphone = control({ label: "Turn microphone off", pressed: true });
const media = {
sinkId: "",
srcObject: liveMediaStream(),
async setSinkId(value: string) {
media.sinkId = value;
},
};
const { result } = await runStatusScript({
allowMicrophone: true,
devices: [
{ deviceId: "blackhole-input", kind: "audioinput", label: "BlackHole 2ch" },
{ deviceId: "blackhole-output", kind: "audiooutput", label: "BlackHole 2ch" },
],
leave,
media: [media],
microphone,
microphoneDevice: control({ label: "BlackHole 2ch" }),
priorMeeting: {
audioInputDeviceId: "blackhole-input",
identity: "teams-work:19:meeting_test@thread.v2",
},
});
it.each([
["BlackHole 2ch", "BlackHole 2ch (Virtual)"],
["OpenClaw Meeting Audio", "OpenClaw Meeting Audio"],
])(
"reports verified %s routes only after the exact input marker and output sink agree",
async (deviceLabel, selectedInputLabel) => {
const leave = control({ label: "Leave" });
const microphone = control({ label: "Turn microphone off", pressed: true });
const media = {
sinkId: "",
srcObject: liveMediaStream(),
async setSinkId(value: string) {
media.sinkId = value;
},
};
const { result } = await runStatusScript({
allowMicrophone: true,
devices: [
{ deviceId: "virtual-input", kind: "audioinput", label: deviceLabel },
{ deviceId: "virtual-output", kind: "audiooutput", label: deviceLabel },
],
leave,
media: [media],
microphone,
microphoneDevice: control({ label: selectedInputLabel }),
priorMeeting: {
audioInputDeviceId: "virtual-input",
identity: "teams-work:19:meeting_test@thread.v2",
},
});
expect(result).toMatchObject({
audioInputRouted: true,
audioOutputRouted: true,
inCall: true,
micMuted: false,
});
expect(result.manualAction).toBeUndefined();
expect(media.sinkId).toBe("blackhole-output");
});
expect(result).toMatchObject({
audioInputRouted: true,
audioInputDeviceLabel: deviceLabel,
audioOutputRouted: true,
audioOutputDeviceLabel: deviceLabel,
inCall: true,
micMuted: false,
});
expect(result.manualAction).toBeUndefined();
expect(media.sinkId).toBe("virtual-output");
},
);
it.each(["OpenClaw Meeting Audio (Virtual)", "Monitor of OpenClaw Meeting Audio"])(
"rejects the non-contract virtual audio label %s",
async (deviceLabel) => {
const { result } = await runStatusScript({
allowMicrophone: true,
camera: control({ label: "Turn camera on", pressed: false }),
devices: [{ deviceId: "virtual-input", kind: "audioinput", label: deviceLabel }],
join: control({ label: "Join now" }),
microphone: control({ label: "Turn microphone on", pressed: false }),
microphoneDevice: control({ label: deviceLabel }),
});
expect(result).toMatchObject({
audioInputRouted: false,
manualAction: {
message:
"Select the OpenClaw virtual audio device as the Teams microphone and verify it is selected before enabling talk-back.",
reason: "teams-audio-choice-required",
},
});
},
);
it("reports the prepared session input during read-only status inspection", async () => {
const media: PageMedia = { sinkId: "blackhole-output", async setSinkId() {} };
extensions/teams-meetings/src/transports/teams-meetings-status-prejoin-source.ts
+20 -16
View File
@@ -133,15 +133,17 @@ export function teamsMeetingStatusPreludeSource(params: MeetingStatusPreludePara
if (identityVerified && !inCall && join && cameraState !== "off") {
controlManualAction = manualActionFor("teams-camera-required", "Turn the Teams camera off and verify the camera control shows it is off, then retry joining.");
}
const isBlackHole = (value) =>
/^blackhole 2ch(?: \\(virtual\\))?$/i.test(String(value || "").replace(/\\s+/g, " ").trim());
const isBlackHoleNode = (node) => [
const isVirtualAudioDevice = (value) =>
/^(?:blackhole 2ch(?: \\(virtual\\))?|openclaw meeting audio)$/i.test(
String(value || "").replace(/\\s+/g, " ").trim()
);
const isVirtualAudioDeviceNode = (node) => [
node?.getAttribute?.("aria-label"),
node?.getAttribute?.("title"),
node?.label,
node?.value,
text(node),
].some(isBlackHole);
].some(isVirtualAudioDevice);
const microphoneDeviceRoots = () => {
// Consumer in-call controls expose the listbox itself, without the prejoin
// selected-device button/combobox wrapper.
@@ -160,13 +162,13 @@ export function teamsMeetingStatusPreludeSource(params: MeetingStatusPreludePara
const selectedMicrophoneLabel = () => {
const { control, roots } = microphoneDeviceRoots();
const selectedOption = control?.selectedOptions?.[0];
if (selectedOption && isBlackHoleNode(selectedOption)) {
if (selectedOption && isVirtualAudioDeviceNode(selectedOption)) {
return label(selectedOption) || selectedOption.value;
}
if (control && isBlackHoleNode(control)) return label(control) || control.value;
if (control && isVirtualAudioDeviceNode(control)) return label(control) || control.value;
for (const root of roots) {
const selected = firstWithin(root, selectors.selectedMicrophoneDevice);
if (selected && isBlackHoleNode(selected)) {
if (selected && isVirtualAudioDeviceNode(selected)) {
return label(selected) || selected.value;
}
}
@@ -179,9 +181,11 @@ export function teamsMeetingStatusPreludeSource(params: MeetingStatusPreludePara
if (!navigator.mediaDevices?.enumerateDevices) return false;
try {
const devices = await navigator.mediaDevices.enumerateDevices();
const input = devices.find((device) => device.kind === "audioinput" && isBlackHole(device.label));
const input = devices.find(
(device) => device.kind === "audioinput" && isVirtualAudioDevice(device.label)
);
if (!input?.deviceId) return false;
audioInputDeviceLabel = input.label || "BlackHole 2ch";
audioInputDeviceLabel = input.label || "Virtual audio device";
// Teams hides the selected-device control after admission. Reopen the in-call audio
// options and verify the current selection before unmuting; installed devices alone
// do not prove which microphone Teams is using.
@@ -202,7 +206,7 @@ export function teamsMeetingStatusPreludeSource(params: MeetingStatusPreludePara
const { control } = microphoneDeviceRoots();
if (control?.tagName?.toLowerCase() === "select") {
const options = [...control.options];
const option = options.find(isBlackHoleNode);
const option = options.find(isVirtualAudioDeviceNode);
if (option) {
control.value = option.value;
control.dispatchEvent(new Event("change", { bubbles: true }));
@@ -217,7 +221,7 @@ export function teamsMeetingStatusPreludeSource(params: MeetingStatusPreludePara
...(root.querySelectorAll?.(selector) || []),
])
);
const choice = choices.find(isBlackHoleNode);
const choice = choices.find(isVirtualAudioDeviceNode);
if (choice && choice.getAttribute?.("aria-selected") !== "true") {
clickable(choice)?.click?.();
await waitForUi();
@@ -242,14 +246,14 @@ export function teamsMeetingStatusPreludeSource(params: MeetingStatusPreludePara
const currentMicrophone = first(selectors.microphone) || findTextButton(/mute|unmute|microphone/i);
microphoneState = toggleState(currentMicrophone, "microphone");
}
controlManualAction = manualActionFor("teams-audio-choice-required", "Select BlackHole 2ch as the Teams microphone and verify it is selected before enabling talk-back.");
controlManualAction = manualActionFor("teams-audio-choice-required", "Select the OpenClaw virtual audio device as the Teams microphone and verify it is selected before enabling talk-back.");
} else if (canMutateSession && microphoneState === "off") {
microphone.click();
await waitForUi();
const currentMicrophone = first(selectors.microphone) || findTextButton(/mute|unmute|microphone/i);
microphoneState = toggleState(currentMicrophone, "microphone");
if (microphoneState === "on") {
notes.push("Unmuted the Teams microphone after verifying BlackHole 2ch input.");
notes.push("Unmuted the Teams microphone after verifying the virtual audio input.");
}
}
if (audioInputRouted && microphoneState !== "on") {
@@ -283,7 +287,7 @@ export function teamsMeetingStatusPreludeSource(params: MeetingStatusPreludePara
const currentMicrophone = first(selectors.microphone) || findTextButton(/mute|unmute|microphone/i);
microphoneState = toggleState(currentMicrophone, "microphone");
if (microphoneState === "off") {
notes.push("Muted the Teams microphone because BlackHole 2ch input could not be reverified.");
notes.push("Muted the Teams microphone because the virtual audio input could not be reverified.");
}
}
}
@@ -294,7 +298,7 @@ export function teamsMeetingStatusPreludeSource(params: MeetingStatusPreludePara
if (!microphone) {
controlManualAction = manualActionFor("teams-microphone-required", "Open Teams device settings and verify the microphone control before enabling talk-back.");
} else if (audioInputRouted !== true) {
controlManualAction = manualActionFor("teams-audio-choice-required", "Select BlackHole 2ch as the Teams microphone and verify it is selected before enabling talk-back.");
controlManualAction = manualActionFor("teams-audio-choice-required", "Select the OpenClaw virtual audio device as the Teams microphone and verify it is selected before enabling talk-back.");
} else if (microphoneState !== "on") {
controlManualAction = manualActionFor("teams-microphone-required", "Unmute the Teams microphone and verify the microphone control shows it is on, then retry joining.");
}
@@ -321,7 +325,7 @@ export function teamsMeetingStatusPreludeSource(params: MeetingStatusPreludePara
first(selectors.permissionPrompt) || continueWithoutDevices
);
// Teams shows the same no-audio/video warning when only camera access is denied.
// A granted microphone plus the verified BlackHole input is sufficient for talk-back.
// A granted microphone plus the verified virtual audio input is sufficient for talk-back.
const permissionRequired = devicePermissionPrompt &&
(!allowMicrophone || microphonePermissionState !== "granted");
let manualAction;
extensions/zoom-meetings/openclaw.plugin.json
+7 -3
View File
@@ -20,13 +20,17 @@
"help": "Agent consults OpenClaw, bidi uses direct realtime voice, and transcribe observes only."
},
"chrome.browserProfile": { "label": "Chrome Profile", "advanced": true },
"chrome.audioBackend": {
"label": "Chrome Audio Backend",
"help": "Auto selects BlackHole 2ch on macOS or PipeWire-Pulse on Linux."
},
"chrome.guestName": { "label": "Guest Name" },
"chrome.waitForInCallMs": { "label": "Wait For In-Call (ms)", "advanced": true },
"chrome.audioInputCommand": { "label": "Audio Input Command", "advanced": true },
"chrome.audioOutputCommand": { "label": "Audio Output Command", "advanced": true },
"chromeNode.node": {
"label": "Chrome Node",
"help": "Node id/name/IP that owns Chrome, BlackHole, and SoX.",
"help": "Node id/name/IP that owns Chrome and the native virtual-audio backend.",
"advanced": true
},
"realtime.transcriptionProvider": { "label": "Realtime Transcription Provider" },
@@ -53,8 +57,8 @@
"properties": {
"audioBackend": {
"type": "string",
"enum": ["blackhole-2ch"],
"default": "blackhole-2ch"
"enum": ["auto", "blackhole-2ch", "pipewire-pulse"],
"default": "auto"
},
"audioFormat": {
"type": "string",
extensions/zoom-meetings/src/config.test.ts
+20 -4
View File
@@ -23,22 +23,38 @@ describe("Zoom meetings config", () => {
).toBe(310_000);
});
it("builds matching SoX commands from the selected audio format", () => {
const config = resolveZoomMeetingsConfig({ chrome: { audioBufferBytes: 2048 } });
it("builds native command pairs for the selected audio backend and format", () => {
const config = resolveZoomMeetingsConfig({
chrome: { audioBackend: "blackhole-2ch", audioBufferBytes: 2048 },
});
expect(config.chrome.audioInputCommand).toContain("sox");
expect(config.chrome.audioInputCommand).toContain("2048");
expect(config.chrome.audioOutputCommand).toContain("BlackHole 2ch");
const g711 = resolveZoomMeetingsConfig({ chrome: { audioFormat: "g711-ulaw-8khz" } });
const g711 = resolveZoomMeetingsConfig({
chrome: { audioBackend: "blackhole-2ch", audioFormat: "g711-ulaw-8khz" },
});
expect(g711.chrome.audioInputCommand).toContain("BlackHole 2ch");
expect(g711.chrome.audioOutputCommand).toContain("BlackHole 2ch");
expect(g711.chrome.audioInputCommand).toContain("mu-law");
const linux = resolveZoomMeetingsConfig({
chrome: { audioBackend: "pipewire-pulse", audioFormat: "g711-ulaw-8khz" },
});
expect(linux.chrome.audioInputCommand).toContain("parec");
expect(linux.chrome.audioInputCommand).toContain("--format=ulaw");
expect(linux.chrome.audioOutputCommand).toContain("pacat");
});
it("preserves explicit command overrides and realtime passthrough", () => {
const config = resolveZoomMeetingsConfig({
defaultMode: "bidi",
chrome: { audioInputCommand: ["capture"], audioOutputCommand: ["play"] },
chrome: {
audioInputCommand: ["capture"],
audioOutputCommand: ["play"],
audioInputCommandOverride: ["capture"],
audioOutputCommandOverride: ["play"],
},
chromeNode: { node: "mac-node" },
realtime: {
voiceProvider: "google",
extensions/zoom-meetings/src/config.ts
-1
View File
@@ -12,7 +12,6 @@ export const zoomMeetingsConfig = MeetingPlatformAdapter.createPluginConfigSchem
config.chrome.waitForInCallMs + config.chrome.joinTimeoutMs + 30_000,
) ?? 1,
),
resolveSoxAudioDevice: () => ({ device: "BlackHole 2ch", deviceType: "coreaudio" }),
});
export type ZoomMeetingsConfig = ReturnType<typeof zoomMeetingsConfig.resolveConfig>;
extensions/zoom-meetings/src/runtime-setup.test.ts
+28 -9
View File
@@ -47,11 +47,6 @@ describe("Zoom meetings runtime setup", () => {
it("probes remote talk-back prerequisites through the selected Chrome node", async () => {
const runtime = runtimeWithNode(async () => ({ ok: true }));
const config = resolveZoomMeetingsConfig({
chrome: {
audioInputCommand: ["custom-input", "--read"],
audioOutputCommand: ["custom-output", "--write"],
bargeInInputCommand: ["custom-barge-in"],
},
chromeNode: { node: "zoom-node" },
});
const status = await getZoomMeetingsSetupStatus({
@@ -66,15 +61,15 @@ describe("Zoom meetings runtime setup", () => {
nodeId: "node-1",
params: {
action: "setup",
audioInputCommand: ["custom-input", "--read"],
audioOutputCommand: ["custom-output", "--write"],
bargeInInputCommand: ["custom-barge-in"],
audioBackend: "auto",
audioBufferBytes: 4_096,
audioFormat: "pcm16-24khz",
},
timeoutMs: 12_000,
});
expect(status.checks).toContainEqual({
id: "chrome-node-audio-prerequisites",
message: "Remote macOS, BlackHole 2ch, and SoX prerequisites are ready",
message: "Remote virtual audio backend and command-pair prerequisites are ready",
ok: true,
});
expect(status.ok).toBe(true);
@@ -98,4 +93,28 @@ describe("Zoom meetings runtime setup", () => {
ok: false,
});
});
it("returns structured diagnostics when local talk-back is unsupported", async () => {
const platform = vi.spyOn(process, "platform", "get").mockReturnValue("win32");
const runCommandWithTimeout = vi.fn();
try {
const status = await getZoomMeetingsSetupStatus({
config: resolveZoomMeetingsConfig({}),
fullConfig: {},
runtime: { system: { runCommandWithTimeout } } as unknown as PluginRuntime,
options: { mode: "agent", transport: "chrome" },
});
expect(status.ok).toBe(false);
expect(status.checks).toContainEqual({
id: "chrome-local-audio-device",
message: expect.stringContaining("unsupported on win32"),
ok: false,
});
expect(status.checks.some((check) => check.id === "chrome-local-audio-commands")).toBe(false);
expect(runCommandWithTimeout).not.toHaveBeenCalled();
} finally {
platform.mockRestore();
}
});
});
extensions/zoom-meetings/src/runtime-setup.ts
+2 -2
View File
@@ -1,13 +1,13 @@
import { MeetingPlatformAdapter } from "openclaw/plugin-sdk/meeting-runtime";
import type { ZoomMeetingsConfig, ZoomMeetingsMode } from "./config.js";
import { assertBlackHole2chAvailable } from "./transports/chrome.js";
import { assertZoomMeetingsAudioAvailable } from "./transports/chrome.js";
import { ZOOM_MEETINGS_BROWSER_NODE_ADAPTER } from "./transports/zoom-meetings-platform-constants.js";
export const getZoomMeetingsSetupStatus = MeetingPlatformAdapter.createRuntimeSetup<
ZoomMeetingsConfig,
ZoomMeetingsMode
>({
assertAudioDeviceAvailable: assertBlackHole2chAvailable,
assertAudioDeviceAvailable: assertZoomMeetingsAudioAvailable,
captionsMessage: (mode) =>
mode === "transcribe"
? "Zoom live-caption capture is enabled and ready"
extensions/zoom-meetings/src/runtime.ts
+2 -1
View File
@@ -133,7 +133,8 @@ export const ZoomMeetingsRuntime = MeetingPlatformAdapter.createRuntimeFacade<
browserReadinessFailed: (error) => `Zoom browser readiness refresh failed: ${error}`,
durableTranscripts: { providerId: "zoom", providerName: "Zoom" },
joined: {
local: "Zoom guest joined in local Chrome with realtime audio through BlackHole 2ch and SoX.",
local:
"Zoom guest joined in local Chrome with realtime audio through the native virtual-audio backend.",
node: "Zoom guest joined in Chrome on the selected node with realtime audio through the node bridge.",
transcribe: "Zoom guest joined observe-only with live-caption transcript capture.",
waiting:
extensions/zoom-meetings/src/transports/chrome.ts
+1 -1
View File
@@ -8,7 +8,7 @@ const chromeTransport = MeetingPlatformAdapter.createPluginChromeTransport({
runtime: MeetingPlatformAdapter.createChromeRuntimeBindings(),
});
export const assertBlackHole2chAvailable = chromeTransport.assertAudioDeviceAvailable;
export const assertZoomMeetingsAudioAvailable = chromeTransport.assertAudioDeviceAvailable;
export const launchZoomMeetingInChrome = chromeTransport.launchInChrome;
export const launchZoomMeetingOnNode = chromeTransport.launchOnNode;
export const leaveZoomMeetingInBrowser = chromeTransport.leaveInBrowser;
extensions/zoom-meetings/src/transports/zoom-meetings-platform-adapter.test.ts
+64 -1
View File
@@ -224,7 +224,8 @@ describe("Zoom meeting platform adapter", () => {
expect(script).toContain("join from browser");
expect(script).toContain("host will let you in soon");
expect(script).toContain("setSinkId");
expect(script).toContain("BlackHole");
expect(script).toContain("blackhole 2ch");
expect(script).toContain("openclaw meeting audio");
});
it("enables caption snapshots for durable notes in every mode", () => {
@@ -584,6 +585,68 @@ describe("Zoom meeting platform adapter", () => {
expect(meetingState).not.toHaveProperty("audioInputDeviceId");
});
it.each(["BlackHole 2ch", "BlackHole 2ch (Virtual)", "OpenClaw Meeting Audio"])(
"recognizes the exact virtual audio input label %s",
async (deviceLabel) => {
const result = await runStatusFixture({
allowMicrophone: true,
document: statusDocument({
bodyText: "",
camera: pageControl("Start Video"),
leave: pageControl("Leave"),
microphone: pageControl("Mute my microphone"),
}),
navigator: {
mediaDevices: {
enumerateDevices: vi.fn(async () => [
{ deviceId: "virtual-input", kind: "audioinput", label: deviceLabel },
]),
},
},
readOnly: true,
});
expect(result).toMatchObject({
audioInputDeviceLabel: deviceLabel,
audioInputRouted: false,
manualAction: {
message:
"Verify the OpenClaw virtual audio device is selected as both the Zoom microphone and speaker before starting talk-back.",
reason: "zoom-audio-choice-required",
},
});
},
);
it.each(["OpenClaw Meeting Audio (Virtual)", "Monitor of OpenClaw Meeting Audio"])(
"rejects the non-contract virtual audio input label %s",
async (deviceLabel) => {
const result = await runStatusFixture({
allowMicrophone: true,
document: statusDocument({
bodyText: "",
camera: pageControl("Start Video"),
leave: pageControl("Leave"),
microphone: pageControl("Mute my microphone"),
}),
navigator: {
mediaDevices: {
enumerateDevices: vi.fn(async () => [
{ deviceId: "virtual-input", kind: "audioinput", label: deviceLabel },
]),
},
},
readOnly: true,
});
expect(result).not.toHaveProperty("audioInputDeviceLabel");
expect(result).toMatchObject({
audioInputRouted: false,
manualAction: { reason: "zoom-audio-choice-required" },
});
},
);
it("retains meeting ownership through an unbounded lobby wait", async () => {
const window: Record<string, unknown> = {};
const waiting = await runStatusFixture({
extensions/zoom-meetings/src/transports/zoom-meetings-status-prejoin-source.ts
+19 -15
View File
@@ -189,15 +189,17 @@ export function zoomMeetingStatusPreludeSource(params: MeetingStatusPreludeParam
) {
controlManualAction = manualActionFor("zoom-camera-required", inCall ? "Turn the Zoom camera off and verify the in-call camera control shows it is off." : "Turn the Zoom camera off and verify the camera control shows it is off, then retry joining.");
}
const isBlackHole = (value) =>
/^blackhole 2ch(?: \\(virtual\\))?$/i.test(String(value || "").replace(/\\s+/g, " ").trim());
const isBlackHoleNode = (node) => [
const isVirtualAudioDevice = (value) =>
/^(?:blackhole 2ch(?: \\(virtual\\))?|openclaw meeting audio)$/i.test(
String(value || "").replace(/\\s+/g, " ").trim()
);
const isVirtualAudioDeviceNode = (node) => [
node?.getAttribute?.("aria-label"),
node?.getAttribute?.("title"),
node?.label,
node?.value,
text(node),
].some(isBlackHole);
].some(isVirtualAudioDevice);
const microphoneDeviceRoots = () => {
// Consumer in-call controls expose the listbox itself, without the prejoin
// selected-device button/combobox wrapper.
@@ -216,13 +218,13 @@ export function zoomMeetingStatusPreludeSource(params: MeetingStatusPreludeParam
const selectedMicrophoneLabel = () => {
const { control, roots } = microphoneDeviceRoots();
const selectedOption = control?.selectedOptions?.[0];
if (selectedOption && isBlackHoleNode(selectedOption)) {
if (selectedOption && isVirtualAudioDeviceNode(selectedOption)) {
return label(selectedOption) || selectedOption.value;
}
if (control && isBlackHoleNode(control)) return label(control) || control.value;
if (control && isVirtualAudioDeviceNode(control)) return label(control) || control.value;
for (const root of roots) {
const selected = firstWithin(root, selectors.selectedMicrophoneDevice);
if (selected && isBlackHoleNode(selected)) {
if (selected && isVirtualAudioDeviceNode(selected)) {
return label(selected) || selected.value;
}
}
@@ -239,9 +241,11 @@ export function zoomMeetingStatusPreludeSource(params: MeetingStatusPreludeParam
if (!navigator.mediaDevices?.enumerateDevices) return false;
try {
const devices = await navigator.mediaDevices.enumerateDevices();
const input = devices.find((device) => device.kind === "audioinput" && isBlackHole(device.label));
const input = devices.find(
(device) => device.kind === "audioinput" && isVirtualAudioDevice(device.label)
);
if (!input?.deviceId) return false;
audioInputDeviceLabel = input.label || "BlackHole 2ch";
audioInputDeviceLabel = input.label || "Virtual audio device";
// Zoom hides the selected-device control after admission. Reopen the in-call audio
// options and verify the current selection before unmuting; installed devices alone
// do not prove which microphone Zoom is using.
@@ -255,7 +259,7 @@ export function zoomMeetingStatusPreludeSource(params: MeetingStatusPreludeParam
const { control } = microphoneDeviceRoots();
if (control?.tagName?.toLowerCase() === "select") {
const options = [...control.options];
const option = options.find(isBlackHoleNode);
const option = options.find(isVirtualAudioDeviceNode);
if (option) {
control.value = option.value;
control.dispatchEvent(new Event("change", { bubbles: true }));
@@ -270,7 +274,7 @@ export function zoomMeetingStatusPreludeSource(params: MeetingStatusPreludeParam
...(root.querySelectorAll?.(selector) || []),
])
);
const choice = choices.find(isBlackHoleNode);
const choice = choices.find(isVirtualAudioDeviceNode);
if (choice && choice.getAttribute?.("aria-selected") !== "true") {
clickable(choice)?.click?.();
await waitForUi();
@@ -292,14 +296,14 @@ export function zoomMeetingStatusPreludeSource(params: MeetingStatusPreludeParam
const currentMicrophone = first(selectors.microphone) || findTextButton(/mute|unmute|microphone/i);
microphoneState = toggleState(currentMicrophone, "microphone");
}
notes.push("BlackHole input will be selected from Zoom's in-call audio controls.");
notes.push("The virtual audio input will be selected from Zoom's in-call audio controls.");
} else if (canMutateSession && microphoneState === "off") {
microphone.click();
await waitForUi();
const currentMicrophone = first(selectors.microphone) || findTextButton(/mute|unmute|microphone/i);
microphoneState = toggleState(currentMicrophone, "microphone");
if (microphoneState === "on") {
notes.push("Unmuted the Zoom microphone after verifying BlackHole 2ch input.");
notes.push("Unmuted the Zoom microphone after verifying the virtual audio input.");
}
}
} else if (canMutateSession && identityVerified && !allowMicrophone && microphoneState === "on") {
@@ -330,7 +334,7 @@ export function zoomMeetingStatusPreludeSource(params: MeetingStatusPreludeParam
const currentMicrophone = first(selectors.microphone) || findTextButton(/mute|unmute|microphone/i);
microphoneState = toggleState(currentMicrophone, "microphone");
if (microphoneState === "off") {
notes.push("Muted the Zoom microphone because BlackHole 2ch input could not be reverified.");
notes.push("Muted the Zoom microphone because the virtual audio input could not be reverified.");
}
}
}
@@ -358,7 +362,7 @@ export function zoomMeetingStatusPreludeSource(params: MeetingStatusPreludeParam
first(selectors.permissionPrompt) || continueWithoutDevices
);
// Zoom shows the same no-audio/video warning when only camera access is denied.
// A granted microphone plus the verified BlackHole input is sufficient for talk-back.
// A granted microphone plus the verified virtual audio input is sufficient for talk-back.
const permissionRequired = devicePermissionPrompt &&
(!allowMicrophone || microphonePermissionState !== "granted");
let manualAction;
src/meeting-bot/audio-backend.test.ts
+162
View File
@@ -0,0 +1,162 @@
import { describe, expect, it, vi } from "vitest";
import { ensureMeetingAudioBackend, resolveMeetingAudioRuntimeForFormat } from "./audio-backend.js";
const PIPEWIRE_SINK_NAME = "openclaw_meeting_audio";
const PIPEWIRE_MONITOR_NAME = `${PIPEWIRE_SINK_NAME}.monitor`;
const PIPEWIRE_SOURCE_NAME = PIPEWIRE_SINK_NAME;
describe("meeting audio backend", () => {
it("selects the native backend for macOS and Linux", () => {
expect(
resolveMeetingAudioRuntimeForFormat({
backend: "auto",
bufferBytes: 4_096,
format: "pcm16-24khz",
platform: "darwin",
}).backend,
).toBe("blackhole-2ch");
expect(
resolveMeetingAudioRuntimeForFormat({
backend: "auto",
bufferBytes: 4_096,
format: "pcm16-24khz",
platform: "linux",
}).backend,
).toBe("pipewire-pulse");
expect(() =>
resolveMeetingAudioRuntimeForFormat({
backend: "auto",
bufferBytes: 4_096,
format: "pcm16-24khz",
platform: "win32",
}),
).toThrow("unsupported on win32");
});
it("builds PipeWire-Pulse raw command pairs", () => {
const runtime = resolveMeetingAudioRuntimeForFormat({
backend: "auto",
bufferBytes: 4_096,
format: "pcm16-24khz",
platform: "linux",
});
expect(runtime).toMatchObject({
backend: "pipewire-pulse",
deviceLabel: "OpenClaw Meeting Audio",
});
expect(runtime.inputCommand).toEqual([
"parec",
"--raw",
`--device=${PIPEWIRE_SOURCE_NAME}`,
"--format=s16le",
"--rate=24000",
"--channels=1",
"--latency-msec=86",
]);
expect(runtime.outputCommand).toEqual([
"pacat",
"--raw",
"--playback",
`--device=${PIPEWIRE_SINK_NAME}`,
"--format=s16le",
"--rate=24000",
"--channels=1",
"--latency-msec=86",
]);
});
it("preserves explicit command overrides while resolving the host backend", () => {
expect(
resolveMeetingAudioRuntimeForFormat({
backend: "auto",
bufferBytes: 4_096,
format: "pcm16-24khz",
inputCommand: ["capture"],
outputCommand: ["playback"],
platform: "linux",
}),
).toMatchObject({ inputCommand: ["capture"], outputCommand: ["playback"] });
});
it("idempotently provisions a PipeWire-Pulse null sink", async () => {
const calls: string[][] = [];
let sinkLoaded = false;
let sourceLoaded = false;
const run = vi.fn(async (argv: string[]) => {
calls.push(argv);
if (argv[1] === "info") {
return { code: 0, stdout: "Server Name: PulseAudio (on PipeWire)" };
}
if (argv[1] === "load-module") {
if (argv[2] === "module-null-sink") {
sinkLoaded = true;
} else if (argv[2] === "module-remap-source") {
sourceLoaded = true;
}
return { code: 0, stdout: "17" };
}
if (argv[1] === "list") {
const kind = argv[3];
return {
code: 0,
stdout:
kind === "sinks"
? sinkLoaded
? `42\t${PIPEWIRE_SINK_NAME}\tPipeWire\ts16le\tSUSPENDED`
: ""
: sourceLoaded
? `43\t${PIPEWIRE_SOURCE_NAME}\tPipeWire\ts16le\tSUSPENDED`
: "",
};
}
return { code: 0, stdout: "" };
});
await ensureMeetingAudioBackend({ backend: "pipewire-pulse", run, timeoutMs: 5_000 });
expect(calls).toContainEqual([
"pactl",
"load-module",
"module-null-sink",
`sink_name=${PIPEWIRE_SINK_NAME}`,
"rate=48000",
"channels=2",
"channel_map=front-left,front-right",
`sink_properties='device.description="OpenClaw Meeting Audio"'`,
]);
expect(calls).toContainEqual([
"pactl",
"load-module",
"module-remap-source",
`source_name=${PIPEWIRE_SOURCE_NAME}`,
`master=${PIPEWIRE_MONITOR_NAME}`,
"channels=2",
"master_channel_map=front-left,front-right",
"channel_map=front-left,front-right",
"remix=no",
`source_properties='device.description="OpenClaw Meeting Audio"'`,
]);
});
it("does not load a second module when the sink already exists", async () => {
const run = vi.fn(async (argv: string[]) => {
if (argv[1] === "info") {
return { code: 0, stdout: "Server Name: PulseAudio (on PipeWire)" };
}
if (argv[1] === "list") {
return {
code: 0,
stdout:
argv[3] === "sinks"
? `42\t${PIPEWIRE_SINK_NAME}\tPipeWire`
: `43\t${PIPEWIRE_SOURCE_NAME}\tPipeWire`,
};
}
return { code: 0, stdout: "" };
});
await ensureMeetingAudioBackend({ backend: "pipewire-pulse", run, timeoutMs: 5_000 });
expect(run.mock.calls.some(([argv]) => argv[1] === "load-module")).toBe(false);
});
});
src/meeting-bot/audio-backend.ts
+246
View File
@@ -0,0 +1,246 @@
import type { MeetingRealtimeAudioFormat } from "./realtime-audio-format.js";
import { buildMeetingSoxAudioCommands } from "./sox-audio-command.js";
export type MeetingAudioBackend = "blackhole-2ch" | "pipewire-pulse";
export type MeetingAudioBackendSelection = "auto" | MeetingAudioBackend;
type MeetingAudioCommandConfig = {
backend?: MeetingAudioBackendSelection;
bufferBytes: number;
format: {
sampleRate: number;
channels: number;
encoding: string;
bits: number;
endian?: "little" | "big";
};
inputCommand?: readonly string[];
outputCommand?: readonly string[];
};
export type MeetingAudioRuntime = {
backend: MeetingAudioBackend;
deviceLabel: string;
inputCommand: string[];
outputCommand: string[];
};
export type MeetingAudioCommandResult = {
code: number;
stdout?: string;
stderr?: string;
};
const PIPEWIRE_SINK_NAME = "openclaw_meeting_audio";
const PIPEWIRE_MONITOR_NAME = `${PIPEWIRE_SINK_NAME}.monitor`;
const PIPEWIRE_SOURCE_NAME = PIPEWIRE_SINK_NAME;
const PIPEWIRE_MEETING_AUDIO_DEVICE_LABEL = "OpenClaw Meeting Audio";
const BLACKHOLE_MEETING_AUDIO_DEVICE_LABEL = "BlackHole 2ch";
function resolvePulseFormat(config: MeetingAudioCommandConfig["format"]): string {
if (config.encoding === "mu-law" && config.bits === 8) {
return "ulaw";
}
if (config.encoding === "signed-integer" && config.bits === 16) {
return config.endian === "big" ? "s16be" : "s16le";
}
throw new Error(
`PipeWire-Pulse does not support meeting audio format ${config.encoding}/${config.bits}.`,
);
}
function resolveMeetingAudioBackend(
selection: MeetingAudioBackendSelection | undefined,
platform: NodeJS.Platform = process.platform,
): MeetingAudioBackend {
if (selection && selection !== "auto") {
if (selection === "blackhole-2ch" && platform !== "darwin") {
throw new Error("BlackHole 2ch meeting audio requires macOS.");
}
if (selection === "pipewire-pulse" && platform !== "linux") {
throw new Error("PipeWire-Pulse meeting audio requires Linux.");
}
return selection;
}
if (platform === "darwin") {
return "blackhole-2ch";
}
if (platform === "linux") {
return "pipewire-pulse";
}
throw new Error(
`Local Chrome meeting talk-back is unsupported on ${platform}; use transcribe mode or a macOS/Linux Chrome node.`,
);
}
function resolveMeetingAudioRuntime(
config: MeetingAudioCommandConfig,
platform: NodeJS.Platform = process.platform,
): MeetingAudioRuntime {
const backend = resolveMeetingAudioBackend(config.backend, platform);
const bytesPerSecond =
config.format.sampleRate * config.format.channels * Math.ceil(config.format.bits / 8);
const pulseLatencyMs = Math.max(1, Math.ceil((config.bufferBytes / bytesPerSecond) * 1_000));
const defaults =
backend === "blackhole-2ch"
? buildMeetingSoxAudioCommands({
bufferBytes: config.bufferBytes,
device: BLACKHOLE_MEETING_AUDIO_DEVICE_LABEL,
deviceType: "coreaudio",
format: config.format,
})
: {
inputCommand: [
"parec",
"--raw",
`--device=${PIPEWIRE_SOURCE_NAME}`,
`--format=${resolvePulseFormat(config.format)}`,
`--rate=${config.format.sampleRate}`,
`--channels=${config.format.channels}`,
`--latency-msec=${pulseLatencyMs}`,
],
outputCommand: [
"pacat",
"--raw",
"--playback",
`--device=${PIPEWIRE_SINK_NAME}`,
`--format=${resolvePulseFormat(config.format)}`,
`--rate=${config.format.sampleRate}`,
`--channels=${config.format.channels}`,
`--latency-msec=${pulseLatencyMs}`,
],
};
return {
backend,
deviceLabel:
backend === "blackhole-2ch"
? BLACKHOLE_MEETING_AUDIO_DEVICE_LABEL
: PIPEWIRE_MEETING_AUDIO_DEVICE_LABEL,
inputCommand: config.inputCommand ? [...config.inputCommand] : defaults.inputCommand,
outputCommand: config.outputCommand ? [...config.outputCommand] : defaults.outputCommand,
};
}
export function resolveMeetingAudioRuntimeForFormat(params: {
backend?: MeetingAudioBackendSelection;
bufferBytes: number;
format: MeetingRealtimeAudioFormat;
inputCommand?: readonly string[];
outputCommand?: readonly string[];
platform?: NodeJS.Platform;
}): MeetingAudioRuntime {
return resolveMeetingAudioRuntime(
{
backend: params.backend,
bufferBytes: params.bufferBytes,
format:
params.format === "g711-ulaw-8khz"
? { sampleRate: 8_000, channels: 1, encoding: "mu-law", bits: 8 }
: {
sampleRate: 24_000,
channels: 1,
encoding: "signed-integer",
bits: 16,
endian: "little",
},
inputCommand: params.inputCommand,
outputCommand: params.outputCommand,
},
params.platform,
);
}
function commandOutput(result: MeetingAudioCommandResult): string {
return `${result.stdout ?? ""}\n${result.stderr ?? ""}`;
}
function pulseListContains(output: string, name: string): boolean {
return output.split(/\r?\n/u).some((line) => line.split(/\s+/u)[1] === name);
}
function assertCommandSucceeded(result: MeetingAudioCommandResult, message: string): void {
if (result.code === 0) {
return;
}
const detail = commandOutput(result).trim();
throw new Error(detail ? `${message}: ${detail}` : message);
}
export async function ensureMeetingAudioBackend(params: {
backend: MeetingAudioBackend;
run(argv: string[], timeoutMs: number): Promise<MeetingAudioCommandResult>;
timeoutMs: number;
}): Promise<void> {
if (params.backend === "blackhole-2ch") {
const result = await params.run(
["/usr/sbin/system_profiler", "SPAudioDataType"],
params.timeoutMs,
);
if (result.code !== 0 || !commandOutput(result).toLowerCase().includes("blackhole 2ch")) {
throw new Error(
"BlackHole 2ch audio device not found. Install BlackHole 2ch and SoX, then reboot.",
);
}
return;
}
const info = await params.run(["pactl", "info"], params.timeoutMs);
assertCommandSucceeded(
info,
"PipeWire-Pulse is unavailable. Start pipewire-pulse and install pulseaudio-utils",
);
let sinks = await params.run(["pactl", "list", "short", "sinks"], params.timeoutMs);
if (!pulseListContains(sinks.stdout ?? "", PIPEWIRE_SINK_NAME)) {
const loaded = await params.run(
[
"pactl",
"load-module",
"module-null-sink",
`sink_name=${PIPEWIRE_SINK_NAME}`,
"rate=48000",
"channels=2",
"channel_map=front-left,front-right",
`sink_properties='device.description="${PIPEWIRE_MEETING_AUDIO_DEVICE_LABEL}"'`,
],
params.timeoutMs,
);
if (loaded.code !== 0) {
sinks = await params.run(["pactl", "list", "short", "sinks"], params.timeoutMs);
if (!pulseListContains(sinks.stdout ?? "", PIPEWIRE_SINK_NAME)) {
assertCommandSucceeded(loaded, "Could not create the OpenClaw PipeWire-Pulse sink");
}
}
}
let sources = await params.run(["pactl", "list", "short", "sources"], params.timeoutMs);
if (!pulseListContains(sources.stdout ?? "", PIPEWIRE_SOURCE_NAME)) {
const loaded = await params.run(
[
"pactl",
"load-module",
"module-remap-source",
`source_name=${PIPEWIRE_SOURCE_NAME}`,
`master=${PIPEWIRE_MONITOR_NAME}`,
"channels=2",
"master_channel_map=front-left,front-right",
"channel_map=front-left,front-right",
"remix=no",
`source_properties='device.description="${PIPEWIRE_MEETING_AUDIO_DEVICE_LABEL}"'`,
],
params.timeoutMs,
);
if (loaded.code !== 0) {
sources = await params.run(["pactl", "list", "short", "sources"], params.timeoutMs);
if (!pulseListContains(sources.stdout ?? "", PIPEWIRE_SOURCE_NAME)) {
assertCommandSucceeded(loaded, "Could not create the OpenClaw PipeWire-Pulse source");
}
}
}
sinks = await params.run(["pactl", "list", "short", "sinks"], params.timeoutMs);
sources = await params.run(["pactl", "list", "short", "sources"], params.timeoutMs);
if (
!pulseListContains(sinks.stdout ?? "", PIPEWIRE_SINK_NAME) ||
!pulseListContains(sources.stdout ?? "", PIPEWIRE_SOURCE_NAME)
) {
throw new Error("OpenClaw PipeWire-Pulse sink or monitor source was not created.");
}
}
src/meeting-bot/chrome-node-result.ts
+24
View File
@@ -0,0 +1,24 @@
import type { MeetingAudioBackend } from "./audio-backend.js";
type MeetingChromeNodeStartResult<Health> = {
launched?: boolean;
bridgeId?: string;
audioBackend?: MeetingAudioBackend;
audioBridge?: { type?: string; outputGeneration?: boolean };
browser?: Health;
};
/** Unwraps the node.invoke envelope while keeping result validation transport-generic. */
export function parseMeetingChromeNodeResult<Health>(
raw: unknown,
invalidMessage: string,
): MeetingChromeNodeStartResult<Health> {
const value =
raw && typeof raw === "object" && "payload" in raw
? (raw as { payload?: unknown }).payload
: raw;
if (!value || typeof value !== "object") {
throw new Error(invalidMessage);
}
return value as MeetingChromeNodeStartResult<Health>;
}
src/meeting-bot/chrome-transport.test.ts
+19 -4
View File
@@ -41,6 +41,8 @@ import type { MeetingBrowserHealth, MeetingTranscriptSnapshot } from "./session-
type TestMode = "agent" | "transcribe";
type TestConfig = MeetingRealtimeEngineConfig & {
chrome: MeetingRealtimeEngineConfig["chrome"] & {
audioBackend: "auto";
audioBufferBytes: number;
audioInputCommand: string[];
audioOutputCommand: string[];
autoJoin: boolean;
@@ -61,6 +63,8 @@ type TestConfig = MeetingRealtimeEngineConfig & {
const config = {
chrome: {
audioBackend: "auto",
audioBufferBytes: 4_096,
audioFormat: "pcm16-24khz",
audioInputCommand: [],
audioOutputCommand: [],
@@ -152,7 +156,6 @@ describe.each(cases)("$name Chrome transport parity", (testCase) => {
isTalkBackMode: () => true,
meetingLabel: `${testCase.name} meeting`,
nodeCommandName: platform.nodeCommandName,
outputMentionsAudioDevice: () => true,
platform,
preserveTrackedBrowserOnEngineFailure: testCase.preserveTrackedBrowserOnEngineFailure,
runtime: {
@@ -177,7 +180,6 @@ describe.each(cases)("$name Chrome transport parity", (testCase) => {
}) as unknown as typeof startMeetingAgentRealtimeEngine,
startRealtimeEngine: vi.fn() as unknown as typeof startMeetingRealtimeEngine,
},
systemProfilerCommand: "/usr/sbin/system_profiler",
});
const runtime = {
system: {
@@ -232,7 +234,6 @@ describe.each(cases)("$name Chrome transport parity", (testCase) => {
isTalkBackMode: () => true,
meetingLabel: `${testCase.name} meeting`,
nodeCommandName: platform.nodeCommandName,
outputMentionsAudioDevice: () => true,
platform,
preserveTrackedBrowserOnEngineFailure: testCase.preserveTrackedBrowserOnEngineFailure,
runtime: {
@@ -254,7 +255,6 @@ describe.each(cases)("$name Chrome transport parity", (testCase) => {
})) as unknown as typeof startMeetingAgentRealtimeEngine,
startRealtimeEngine: vi.fn() as unknown as typeof startMeetingRealtimeEngine,
},
systemProfilerCommand: "/usr/sbin/system_profiler",
});
const runtime = {
nodes: {
@@ -283,6 +283,21 @@ describe.each(cases)("$name Chrome transport parity", (testCase) => {
});
expect(result.audioBridge?.type).toBe("node-command-pair");
expect(runtime.nodes.invoke).toHaveBeenCalledWith(
expect.objectContaining({
params: expect.objectContaining({
action: "setup",
audioBackend: "auto",
audioBufferBytes: 4_096,
audioFormat: "pcm16-24khz",
}),
}),
);
const startCall = vi
.mocked(runtime.nodes.invoke)
.mock.calls.find(([call]) => (call.params as { action?: string }).action === "start")?.[0];
expect(startCall?.params).not.toHaveProperty("audioInputCommand");
expect(startCall?.params).not.toHaveProperty("audioOutputCommand");
expect(
Reflect.get(
nodeAudioTransport,
src/meeting-bot/chrome-transport.ts
+94 -52
View File
@@ -3,6 +3,13 @@ import type { OpenClawConfig } from "../config/types.openclaw.js";
import type { PluginRuntime, RuntimeLogger } from "../plugins/runtime/types.js";
import { resolveTranscriptsConfig } from "../transcripts/config.js";
import type { createMeetingRealtimeEngineBindings } from "./agent-consult.js";
import {
ensureMeetingAudioBackend,
resolveMeetingAudioRuntimeForFormat,
type MeetingAudioBackend,
type MeetingAudioBackendSelection,
type MeetingAudioRuntime,
} from "./audio-backend.js";
import { openMeetingWithBrowser, recoverMeetingBrowserTab } from "./browser-controller.js";
import { callMeetingBrowserProxyOnNode, resolveMeetingBrowserNode } from "./browser-node.js";
import { resolveLocalMeetingBrowserRequest } from "./browser-request.js";
@@ -10,6 +17,7 @@ import {
leaveMeetingWithBrowser,
readMeetingTranscriptWithBrowser,
} from "./browser-session-control.js";
import { parseMeetingChromeNodeResult } from "./chrome-node-result.js";
import type {
MeetingBrowserJoinSession,
MeetingBrowserRequestCaller,
@@ -32,8 +40,12 @@ import type {
export type MeetingChromeTransportConfig = MeetingRealtimeEngineConfig & {
chrome: MeetingRealtimeEngineConfig["chrome"] & {
audioBackend: MeetingAudioBackendSelection;
audioBufferBytes: number;
audioInputCommand: string[];
audioInputCommandOverride?: string[];
audioOutputCommand: string[];
audioOutputCommandOverride?: string[];
autoJoin: boolean;
bargeInCooldownMs: number;
bargeInInputCommand?: string[];
@@ -70,7 +82,6 @@ export type MeetingChromeTransportOptions<
isTalkBackMode(mode: Mode): boolean;
meetingLabel: string;
nodeCommandName: string;
outputMentionsAudioDevice(output: string): boolean;
platform: MeetingPlatformAdapter<MeetingBrowserJoinSession<Mode>, Mode, Health, Transcript> &
MeetingPlatformRuntimeMetadata;
preserveTrackedBrowserOnEngineFailure: boolean;
@@ -81,7 +92,6 @@ export type MeetingChromeTransportOptions<
startAgentRealtimeEngine: typeof startMeetingAgentRealtimeEngine;
startRealtimeEngine: typeof startMeetingRealtimeEngine;
};
systemProfilerCommand: string;
};
export function createMeetingChromeTransport<
@@ -177,30 +187,35 @@ export function createMeetingChromeTransport<
}
}
async function prepareAudioRuntime(params: {
runtime: PluginRuntime;
config: Config;
timeoutMs: number;
}): Promise<MeetingAudioRuntime> {
const audio = resolveMeetingAudioRuntimeForFormat({
backend: params.config.chrome.audioBackend,
bufferBytes: params.config.chrome.audioBufferBytes,
format: params.config.chrome.audioFormat,
inputCommand: params.config.chrome.audioInputCommandOverride,
outputCommand: params.config.chrome.audioOutputCommandOverride,
});
await ensureMeetingAudioBackend({
backend: audio.backend,
timeoutMs: params.timeoutMs,
run: async (argv, timeoutMs) => {
const result = await params.runtime.system.runCommandWithTimeout(argv, { timeoutMs });
return { ...result, code: result.code ?? 1 };
},
});
return audio;
}
async function assertAudioDeviceAvailable(params: {
runtime: PluginRuntime;
config: Config;
timeoutMs: number;
}): Promise<void> {
if (process.platform !== "darwin") {
throw new Error(`${options.meetingLabel} talk-back with BlackHole 2ch is macOS-only`);
}
const result = await params.runtime.system.runCommandWithTimeout(
[options.systemProfilerCommand, "SPAudioDataType"],
{ timeoutMs: params.timeoutMs },
);
const output = `${result.stdout ?? ""}\n${result.stderr ?? ""}`;
if (result.code !== 0 || !options.outputMentionsAudioDevice(output)) {
const hint =
params.runtime.system.formatNativeDependencyHint?.({
packageName: "BlackHole 2ch",
downloadCommand: "brew install blackhole-2ch",
}) ?? "";
throw new Error(
["BlackHole 2ch audio device not found.", "Install BlackHole 2ch and SoX.", hint]
.filter(Boolean)
.join(" "),
);
}
await prepareAudioRuntime(params);
}
async function startLocalAudioBridge(params: {
@@ -211,13 +226,14 @@ export function createMeetingChromeTransport<
requesterSessionKey?: string;
mode: Mode;
logger: RuntimeLogger;
audio: MeetingAudioRuntime;
}): Promise<LocalAudioBridge | undefined> {
if (!options.isTalkBackMode(params.mode)) {
return undefined;
}
const transport = options.runtime.createLocalAudioTransport({
inputCommand: params.config.chrome.audioInputCommand,
outputCommand: params.config.chrome.audioOutputCommand,
inputCommand: params.audio.inputCommand,
outputCommand: params.audio.outputCommand,
audioFormat: params.config.chrome.audioFormat,
bargeInInputCommand: params.config.chrome.bargeInInputCommand,
bargeInRmsThreshold: params.config.chrome.bargeInRmsThreshold,
@@ -276,16 +292,18 @@ export function createMeetingChromeTransport<
logger: RuntimeLogger;
}): Promise<{
launched: boolean;
audioBackend?: MeetingAudioBackend;
audioBridge?: LocalAudioBridge;
browser?: Health;
tab?: MeetingBrowserTab;
}> {
if (options.isTalkBackMode(params.mode)) {
await assertAudioDeviceAvailable({
runtime: params.runtime,
timeoutMs: Math.min(params.config.chrome.joinTimeoutMs, 10_000),
});
}
const audio = options.isTalkBackMode(params.mode)
? await prepareAudioRuntime({
runtime: params.runtime,
config: params.config,
timeoutMs: Math.min(params.config.chrome.joinTimeoutMs, 10_000),
})
: undefined;
const callBrowser = await resolveLocalMeetingBrowserRequest(params.runtime);
const result = await openOrRecoverMeeting({
callBrowser,
@@ -298,10 +316,14 @@ export function createMeetingChromeTransport<
url: params.url,
});
if (!options.isRealtimeRouteReady(params.mode, result.browser)) {
return result;
return { ...result, audioBackend: audio?.backend };
}
try {
return { ...result, audioBridge: await startLocalAudioBridge(params) };
return {
...result,
audioBackend: audio?.backend,
audioBridge: audio ? await startLocalAudioBridge({ ...params, audio }) : undefined,
};
} catch (error) {
if (!options.preserveTrackedBrowserOnEngineFailure || !params.trackedTargetId) {
await rollbackBrowserJoin({
@@ -341,23 +363,11 @@ export function createMeetingChromeTransport<
});
}
type MeetingNodeStartResult = {
launched?: boolean;
bridgeId?: string;
audioBridge?: { type?: string; outputGeneration?: boolean };
browser?: Health;
};
function parseNodeStartResult(raw: unknown): MeetingNodeStartResult {
const value =
raw && typeof raw === "object" && "payload" in raw
? (raw as { payload?: unknown }).payload
: raw;
if (!value || typeof value !== "object") {
throw new Error(`${options.meetingLabel} node returned an invalid start result.`);
}
return value as MeetingNodeStartResult;
}
const parseNodeResult = (raw: unknown) =>
parseMeetingChromeNodeResult<Health>(
raw,
`${options.meetingLabel} node returned an invalid start result.`,
);
async function launchOnNode(params: {
runtime: PluginRuntime;
@@ -372,6 +382,7 @@ export function createMeetingChromeTransport<
}): Promise<{
nodeId: string;
launched: boolean;
audioBackend?: MeetingAudioBackend;
audioBridge?: NodeAudioBridge;
browser?: Health;
tab?: MeetingBrowserTab;
@@ -394,6 +405,27 @@ export function createMeetingChromeTransport<
}`,
);
}
const audioSetup = options.isTalkBackMode(params.mode)
? parseNodeResult(
await params.runtime.nodes.invoke({
nodeId,
command: options.nodeCommandName,
params: {
action: "setup",
audioBackend: params.config.chrome.audioBackend,
audioFormat: params.config.chrome.audioFormat,
audioBufferBytes: params.config.chrome.audioBufferBytes,
...(params.config.chrome.audioInputCommandOverride
? { audioInputCommand: params.config.chrome.audioInputCommandOverride }
: {}),
...(params.config.chrome.audioOutputCommandOverride
? { audioOutputCommand: params.config.chrome.audioOutputCommandOverride }
: {}),
},
timeoutMs: 12_000,
}),
)
: undefined;
const callBrowser: MeetingBrowserRequestCaller = async (request) =>
await callNodeBrowser({
runtime: params.runtime,
@@ -417,6 +449,7 @@ export function createMeetingChromeTransport<
return {
nodeId,
launched: browser.launched,
audioBackend: audioSetup?.audioBackend,
browser: browser.browser,
tab: browser.tab,
};
@@ -432,16 +465,24 @@ export function createMeetingChromeTransport<
launch: false,
browserProfile: params.config.chrome.browserProfile,
joinTimeoutMs: params.config.chrome.joinTimeoutMs,
audioInputCommand: params.config.chrome.audioInputCommand,
audioOutputCommand: params.config.chrome.audioOutputCommand,
...(params.config.chrome.audioInputCommandOverride
? { audioInputCommand: params.config.chrome.audioInputCommandOverride }
: {}),
...(params.config.chrome.audioOutputCommandOverride
? { audioOutputCommand: params.config.chrome.audioOutputCommandOverride }
: {}),
audioBackend: params.config.chrome.audioBackend,
audioFormat: params.config.chrome.audioFormat,
audioBufferBytes: params.config.chrome.audioBufferBytes,
},
timeoutMs: addTimerTimeoutGraceMs(params.config.chrome.joinTimeoutMs) ?? 1,
});
const result = parseNodeStartResult(raw);
const result = parseNodeResult(raw);
if (result.audioBridge?.type !== "node-command-pair") {
return {
nodeId,
launched: browser.launched || result.launched === true,
audioBackend: result.audioBackend ?? audioSetup?.audioBackend,
browser: browser.browser ?? result.browser,
tab: browser.tab,
};
@@ -507,6 +548,7 @@ export function createMeetingChromeTransport<
return {
nodeId,
launched: browser.launched || result.launched === true,
audioBackend: result.audioBackend ?? audioSetup?.audioBackend,
audioBridge: {
type: "node-command-pair",
nodeId,
src/meeting-bot/configured-node-host.test.ts
+150
View File
@@ -0,0 +1,150 @@
import { afterEach, describe, expect, it, vi } from "vitest";
const childProcessMocks = vi.hoisted(() => ({
spawnSync: vi.fn(),
}));
vi.mock("node:child_process", async (importOriginal) => ({
...(await importOriginal<typeof import("node:child_process")>()),
spawnSync: childProcessMocks.spawnSync,
}));
import { createMeetingConfiguredNodeHost } from "./configured-node-host.js";
function createHost() {
return createMeetingConfiguredNodeHost({
agentMode: "agent",
bridgeIdPrefix: "test-node-",
browser: {
application: "Google Chrome",
buildProfileArgs: () => [],
openedNotes: [],
openedStatus: "opened",
},
browserLabel: "Test meeting",
commandName: "testmeeting.chrome",
defaultAudio: {
backend: "auto",
bufferBytes: 4_096,
format: "pcm16-24khz",
},
defaultAudioInputCommand: ["legacy-capture"],
defaultAudioOutputCommand: ["legacy-playback"],
displayName: "Test meeting",
meetingLabel: "Test meeting",
normalizeMeetingKey: (url) => url,
normalizeUrl: (value) => String(value),
sharePrerequisiteDeadline: true,
talkBackModes: new Set(["agent", "bidi"]),
});
}
describe("configured meeting node host", () => {
afterEach(() => {
vi.restoreAllMocks();
vi.resetAllMocks();
});
it("provisions PipeWire-Pulse and probes node-local default commands", async () => {
vi.spyOn(process, "platform", "get").mockReturnValue("linux");
let sinkLoaded = false;
let sourceLoaded = false;
childProcessMocks.spawnSync.mockImplementation((command: string, args: string[]) => {
if (command === "/bin/sh") {
return { status: 0, stdout: "", stderr: "" };
}
if (command !== "pactl") {
return { status: 1, stdout: "", stderr: `unexpected command: ${command}` };
}
if (args[0] === "info") {
return { status: 0, stdout: "Server Name: PulseAudio (on PipeWire)", stderr: "" };
}
if (args[0] === "load-module") {
if (args[1] === "module-null-sink") {
sinkLoaded = true;
}
if (args[1] === "module-remap-source") {
sourceLoaded = true;
}
return { status: 0, stdout: "42", stderr: "" };
}
if (args[0] === "list" && args[2] === "sinks") {
return {
status: 0,
stdout: sinkLoaded ? "42\topenclaw_meeting_audio\tPipeWire" : "",
stderr: "",
};
}
if (args[0] === "list" && args[2] === "sources") {
return {
status: 0,
stdout: sourceLoaded ? "43\topenclaw_meeting_audio\tPipeWire" : "",
stderr: "",
};
}
return { status: 0, stdout: "", stderr: "" };
});
const result = JSON.parse(await createHost()(JSON.stringify({ action: "setup" }))) as Record<
string,
unknown
>;
expect(result).toEqual({
ok: true,
audioBackend: "pipewire-pulse",
audioDeviceLabel: "OpenClaw Meeting Audio",
});
expect(childProcessMocks.spawnSync).toHaveBeenCalledWith(
"/bin/sh",
["-lc", 'command -v "$1" >/dev/null 2>&1', "sh", "parec"],
expect.objectContaining({ encoding: "utf8" }),
);
expect(childProcessMocks.spawnSync).toHaveBeenCalledWith(
"/bin/sh",
["-lc", 'command -v "$1" >/dev/null 2>&1', "sh", "pacat"],
expect.objectContaining({ encoding: "utf8" }),
);
expect(childProcessMocks.spawnSync).not.toHaveBeenCalledWith(
"/bin/sh",
expect.arrayContaining(["legacy-capture"]),
expect.anything(),
);
});
it("probes explicit command overrides after provisioning the backend", async () => {
vi.spyOn(process, "platform", "get").mockReturnValue("linux");
childProcessMocks.spawnSync.mockImplementation((command: string, args: string[]) => {
if (command === "/bin/sh") {
return { status: 0, stdout: "", stderr: "" };
}
if (args[0] === "info") {
return { status: 0, stdout: "Server Name: PulseAudio (on PipeWire)", stderr: "" };
}
if (args[0] === "list" && args[2] === "sinks") {
return { status: 0, stdout: "42\topenclaw_meeting_audio\tPipeWire", stderr: "" };
}
if (args[0] === "list" && args[2] === "sources") {
return { status: 0, stdout: "43\topenclaw_meeting_audio\tPipeWire", stderr: "" };
}
return { status: 0, stdout: "", stderr: "" };
});
await createHost()(
JSON.stringify({
action: "setup",
audioInputCommand: ["trusted-capture", "--raw"],
audioOutputCommand: ["trusted-playback", "--raw"],
bargeInInputCommand: ["trusted-barge-in"],
}),
);
for (const command of ["trusted-capture", "trusted-playback", "trusted-barge-in"]) {
expect(childProcessMocks.spawnSync).toHaveBeenCalledWith(
"/bin/sh",
["-lc", 'command -v "$1" >/dev/null 2>&1', "sh", command],
expect.objectContaining({ encoding: "utf8" }),
);
}
});
});
src/meeting-bot/configured-node-host.ts
+85 -83
View File
@@ -1,31 +1,52 @@
import { spawnSync } from "node:child_process";
import { asRecord } from "@openclaw/normalization-core/record-coerce";
import { createMeetingNodeHost, type MeetingNodeHostOptions } from "./node-host.js";
import {
ensureMeetingAudioBackend,
resolveMeetingAudioRuntimeForFormat,
type MeetingAudioCommandResult,
} from "./audio-backend.js";
import {
createMeetingNodeHost,
type MeetingNodeAudioConfig,
type MeetingNodeHostOptions,
} from "./node-host.js";
type MeetingConfiguredNodeHostOptions = Omit<MeetingNodeHostOptions, "assertAudioAvailable"> & {
type MeetingConfiguredNodeHostOptions = Omit<
MeetingNodeHostOptions,
"assertAudioAvailable" | "prepareAudio"
> & {
meetingLabel: string;
outputMentionsAudioDevice(output: string): boolean;
sharePrerequisiteDeadline: boolean;
systemProfilerCommand: string;
};
function readSetupCommand(params: Record<string, unknown>, name: string): string[] {
const value = params[name];
if (
!Array.isArray(value) ||
value.length === 0 ||
value.some((entry) => typeof entry !== "string")
) {
throw new Error(`${name} must be a non-empty string array.`);
}
return value as string[];
}
function isSpawnSyncTimeout(error: unknown): boolean {
return error instanceof Error && "code" in error && error.code === "ETIMEDOUT";
}
export function createMeetingConfiguredNodeHost(options: MeetingConfiguredNodeHostOptions) {
const runCommand = (argv: string[], timeoutMs: number): MeetingAudioCommandResult => {
const [command, ...args] = argv;
if (!command) {
return { code: 1, stderr: "command must not be empty" };
}
const result = spawnSync(command, args, { encoding: "utf8", timeout: timeoutMs });
if (isSpawnSyncTimeout(result.error)) {
throw new Error(`${options.meetingLabel} audio prerequisite check timed out on the node.`);
}
const error = result.error
? result.error instanceof Error
? result.error.message
: String(result.error)
: "";
const stderr = [error, result.stderr, result.signal ? `terminated by ${result.signal}` : ""]
.filter(Boolean)
.join(": ");
return {
code: typeof result.status === "number" ? result.status : 1,
stdout: result.stdout ?? "",
stderr,
};
};
const probeCommand = (command: string, timeoutMs: number): "found" | "missing" | "timed-out" => {
const result = spawnSync("/bin/sh", ["-lc", 'command -v "$1" >/dev/null 2>&1', "sh", command], {
encoding: "utf8",
@@ -36,52 +57,39 @@ export function createMeetingConfiguredNodeHost(options: MeetingConfiguredNodeHo
}
return result.status === 0 ? "found" : "missing";
};
const assertAudioAvailable = (
timeoutMs: number,
commands: readonly (readonly string[])[] = [
options.defaultAudioInputCommand,
options.defaultAudioOutputCommand,
],
) => {
if (process.platform !== "darwin") {
throw new Error(`${options.meetingLabel} talk-back with BlackHole 2ch is macOS-only`);
}
const prepareAudio = async (config: MeetingNodeAudioConfig, timeoutMs: number) => {
const deadline = Date.now() + timeoutMs;
const commandTimeout = () =>
options.sharePrerequisiteDeadline ? Math.max(1, deadline - Date.now()) : timeoutMs;
const result = spawnSync(options.systemProfilerCommand, ["SPAudioDataType"], {
encoding: "utf8",
timeout: commandTimeout(),
});
if (isSpawnSyncTimeout(result.error)) {
throw new Error(`${options.meetingLabel} audio prerequisite check timed out on the node.`);
}
const stderr =
result.stderr ??
(result.error
? result.error instanceof Error
? result.error.message
: String(result.error)
: "");
const output = `${result.stdout ?? ""}\n${stderr}`;
if (
(typeof result.status === "number" ? result.status : result.error ? 1 : 0) !== 0 ||
!options.outputMentionsAudioDevice(output)
) {
throw new Error("BlackHole 2ch audio device not found on the node.");
}
const commandNames = new Set<string>();
for (const argv of commands) {
const command = argv[0];
if (!command) {
throw new Error(`Configured audio command not found on the node: ${command || "<empty>"}`);
const commandTimeout = () => {
if (!options.sharePrerequisiteDeadline) {
return timeoutMs;
}
commandNames.add(command);
}
for (const command of commandNames) {
if (options.sharePrerequisiteDeadline && Date.now() >= deadline) {
const remainingMs = deadline - Date.now();
if (remainingMs <= 0) {
throw new Error(`${options.meetingLabel} audio prerequisite check timed out on the node.`);
}
return remainingMs;
};
const runtime = resolveMeetingAudioRuntimeForFormat({
backend: config.backend,
bufferBytes: config.bufferBytes,
format: config.format,
inputCommand: config.inputCommand,
outputCommand: config.outputCommand,
});
await ensureMeetingAudioBackend({
backend: runtime.backend,
run: async (argv, requestedTimeoutMs) => {
return runCommand(argv, Math.min(requestedTimeoutMs, commandTimeout()));
},
timeoutMs,
});
const commandNames = new Set(
[runtime.inputCommand, runtime.outputCommand, config.bargeInInputCommand].flatMap(
(command) => (command?.[0] ? [command[0]] : []),
),
);
for (const command of commandNames) {
const probeResult = probeCommand(command, commandTimeout());
if (probeResult === "timed-out") {
throw new Error(`${options.meetingLabel} audio prerequisite check timed out on the node.`);
@@ -90,30 +98,24 @@ export function createMeetingConfiguredNodeHost(options: MeetingConfiguredNodeHo
throw new Error(`Configured audio command not found on the node: ${command}`);
}
}
return runtime;
};
const host = createMeetingNodeHost({ ...options, assertAudioAvailable });
return async (paramsJSON?: string | null): Promise<string> => {
if (paramsJSON) {
let raw: unknown;
try {
raw = JSON.parse(paramsJSON) as unknown;
} catch {
throw new Error(`${options.displayName} node host received malformed params JSON.`);
}
const params = asRecord(raw);
if (params.action === "setup") {
const commands = [
readSetupCommand(params, "audioInputCommand"),
readSetupCommand(params, "audioOutputCommand"),
];
if (params.bargeInInputCommand !== undefined) {
commands.push(readSetupCommand(params, "bargeInInputCommand"));
}
assertAudioAvailable(10_000, commands);
return JSON.stringify({ ok: true });
}
}
return await host.handleCommand(paramsJSON);
};
const host = createMeetingNodeHost({
...options,
assertAudioAvailable: async (timeoutMs) => {
await prepareAudio(
{
backend: options.defaultAudio?.backend ?? "auto",
bufferBytes: options.defaultAudio?.bufferBytes ?? 4_096,
format: options.defaultAudio?.format ?? "pcm16-24khz",
},
timeoutMs,
);
},
prepareAudio,
});
return async (paramsJSON?: string | null): Promise<string> =>
await host.handleCommand(paramsJSON);
}
src/meeting-bot/node-audio-config.ts
+96
View File
@@ -0,0 +1,96 @@
import type { MeetingAudioBackendSelection, MeetingAudioRuntime } from "./audio-backend.js";
import type { MeetingRealtimeAudioFormat } from "./realtime-audio-format.js";
export type MeetingNodeAudioConfig = {
backend: MeetingAudioBackendSelection;
bufferBytes: number;
format: MeetingRealtimeAudioFormat;
inputCommand?: string[];
outputCommand?: string[];
bargeInInputCommand?: string[];
};
type MeetingNodeAudioPreparation = {
defaultAudioInputCommand: readonly string[];
defaultAudioOutputCommand: readonly string[];
defaultAudio?: {
backend?: MeetingAudioBackendSelection;
bufferBytes: number;
format: MeetingRealtimeAudioFormat;
};
assertAudioAvailable(timeoutMs: number): void | Promise<void>;
prepareAudio?(config: MeetingNodeAudioConfig, timeoutMs: number): Promise<MeetingAudioRuntime>;
};
const DEFAULT_NODE_AUDIO_BUFFER_BYTES = 4_096;
const DEFAULT_NODE_AUDIO_FORMAT: MeetingRealtimeAudioFormat = "pcm16-24khz";
export function readMeetingNodeCommand(value: unknown): string[] | undefined {
if (!Array.isArray(value)) {
return undefined;
}
const result = value.filter(
(entry): entry is string => typeof entry === "string" && entry.length > 0,
);
return result.length > 0 ? result : undefined;
}
function readAudioBackend(value: unknown): MeetingAudioBackendSelection | undefined {
if (value === undefined) {
return undefined;
}
if (value === "auto" || value === "blackhole-2ch" || value === "pipewire-pulse") {
return value;
}
throw new Error("audioBackend must be auto, blackhole-2ch, or pipewire-pulse");
}
function readAudioFormat(value: unknown): MeetingRealtimeAudioFormat | undefined {
if (value === undefined) {
return undefined;
}
if (value === "pcm16-24khz" || value === "g711-ulaw-8khz") {
return value;
}
throw new Error("audioFormat must be pcm16-24khz or g711-ulaw-8khz");
}
function readAudioBufferBytes(value: unknown): number | undefined {
if (value === undefined) {
return undefined;
}
if (typeof value === "number" && Number.isSafeInteger(value) && value > 0) {
return value;
}
throw new Error("audioBufferBytes must be a positive safe integer");
}
/** Resolves untrusted node params before any backend or command process starts. */
export async function prepareMeetingNodeAudio(
params: Record<string, unknown>,
timeoutMs: number,
options: MeetingNodeAudioPreparation,
): Promise<MeetingAudioRuntime> {
const defaults = options.defaultAudio;
const config: MeetingNodeAudioConfig = {
backend: readAudioBackend(params.audioBackend) ?? defaults?.backend ?? "auto",
bufferBytes:
readAudioBufferBytes(params.audioBufferBytes) ??
defaults?.bufferBytes ??
DEFAULT_NODE_AUDIO_BUFFER_BYTES,
format: readAudioFormat(params.audioFormat) ?? defaults?.format ?? DEFAULT_NODE_AUDIO_FORMAT,
inputCommand: readMeetingNodeCommand(params.audioInputCommand),
outputCommand: readMeetingNodeCommand(params.audioOutputCommand),
bargeInInputCommand: readMeetingNodeCommand(params.bargeInInputCommand),
};
if (options.prepareAudio) {
return await options.prepareAudio(config, timeoutMs);
}
await options.assertAudioAvailable(timeoutMs);
return {
backend: "blackhole-2ch",
deviceLabel: "BlackHole 2ch",
inputCommand: config.inputCommand ?? [...options.defaultAudioInputCommand],
outputCommand: config.outputCommand ?? [...options.defaultAudioOutputCommand],
};
}
src/meeting-bot/node-host.audio.test.ts
+119 -2
View File
@@ -19,7 +19,7 @@ vi.mock("node:crypto", async (importOriginal) => ({
randomUUID: cryptoMocks.randomUUID,
}));
import { createMeetingNodeHost } from "./node-host.js";
import { createMeetingNodeHost, type MeetingNodeHostOptions } from "./node-host.js";
const TEST_UUID = "00000000-0000-4000-8000-000000000001";
// Mirrors the node-host's private retention limits.
@@ -90,7 +90,7 @@ function createProcess(params: {
return proc;
}
function createHost() {
function createHost(overrides: Partial<MeetingNodeHostOptions> = {}) {
return createMeetingNodeHost({
agentMode: "agent",
assertAudioAvailable: vi.fn(),
@@ -109,6 +109,7 @@ function createHost() {
normalizeMeetingKey: (url) => url,
normalizeUrl: (input) => (typeof input === "string" ? input : "https://meeting.test"),
talkBackModes: new Set(["bidi"]),
...overrides,
});
}
@@ -147,6 +148,122 @@ function invokeBridge(
return invokeHost(bridge.host, { action, bridgeId: bridge.bridgeId, ...params });
}
describe("meeting node host audio backend", () => {
beforeEach(() => {
cryptoMocks.randomUUID.mockReturnValue(TEST_UUID);
});
afterEach(() => {
vi.resetAllMocks();
});
it("prepares the node-local backend and uses its generated command pair", async () => {
const inputProcess = createProcess({ stdout: new EventEmitter() });
const outputProcess = createProcess({ stdin: createStdin(true) });
childProcessMocks.spawn.mockReturnValueOnce(outputProcess).mockReturnValueOnce(inputProcess);
const prepareAudio = vi.fn(async () => ({
backend: "pipewire-pulse" as const,
deviceLabel: "OpenClaw Meeting Audio",
inputCommand: ["parec", "--node-default"],
outputCommand: ["pacat", "--node-default"],
}));
const host = createHost({ prepareAudio });
const started = await invokeHost(host, {
action: "start",
url: "https://meeting.test/linux",
mode: "bidi",
launch: false,
audioBackend: "auto",
audioBufferBytes: 2_048,
audioFormat: "g711-ulaw-8khz",
audioInputCommand: ["trusted-capture"],
audioOutputCommand: ["trusted-playback"],
});
expect(prepareAudio).toHaveBeenCalledWith(
{
backend: "auto",
bufferBytes: 2_048,
format: "g711-ulaw-8khz",
inputCommand: ["trusted-capture"],
outputCommand: ["trusted-playback"],
bargeInInputCommand: undefined,
},
10_000,
);
expect(childProcessMocks.spawn).toHaveBeenNthCalledWith(1, "pacat", ["--node-default"], {
stdio: ["pipe", "ignore", "pipe"],
});
expect(childProcessMocks.spawn).toHaveBeenNthCalledWith(2, "parec", ["--node-default"], {
stdio: ["ignore", "pipe", "pipe"],
});
expect(started).toMatchObject({
audioBackend: "pipewire-pulse",
audioDeviceLabel: "OpenClaw Meeting Audio",
audioBridge: { type: "node-command-pair" },
});
await invokeHost(host, { action: "stop", bridgeId: started.bridgeId });
});
it("provisions audio before preserving the configured external bridge path", async () => {
const events: string[] = [];
const prepareAudio = vi.fn(async () => {
events.push("prepare");
return {
backend: "pipewire-pulse" as const,
deviceLabel: "OpenClaw Meeting Audio",
inputCommand: ["parec"],
outputCommand: ["pacat"],
};
});
childProcessMocks.spawnSync.mockImplementation(() => {
events.push("bridge");
return { error: undefined, signal: null, status: 0, stderr: "", stdout: "" };
});
const host = createHost({ prepareAudio });
const started = await invokeHost(host, {
action: "start",
url: "https://meeting.test/external",
mode: "bidi",
launch: false,
audioBridgeCommand: ["trusted-bridge", "start"],
});
expect(events).toEqual(["prepare", "bridge"]);
expect(childProcessMocks.spawn).not.toHaveBeenCalled();
expect(started).toMatchObject({
audioBackend: "pipewire-pulse",
audioBridge: { type: "external-command" },
});
});
it("returns the concrete backend from setup without starting bridge processes", async () => {
const prepareAudio = vi.fn(async () => ({
backend: "pipewire-pulse" as const,
deviceLabel: "OpenClaw Meeting Audio",
inputCommand: ["parec"],
outputCommand: ["pacat"],
}));
const host = createHost({ prepareAudio });
await expect(
invokeHost(host, {
action: "setup",
audioBackend: "auto",
audioBufferBytes: 4_096,
audioFormat: "pcm16-24khz",
}),
).resolves.toEqual({
ok: true,
audioBackend: "pipewire-pulse",
audioDeviceLabel: "OpenClaw Meeting Audio",
});
expect(childProcessMocks.spawn).not.toHaveBeenCalled();
});
});
describe("meeting node host audio output", () => {
beforeEach(() => {
cryptoMocks.randomUUID.mockReturnValue(TEST_UUID);
src/meeting-bot/node-host.ts
+34 -24
View File
@@ -2,9 +2,18 @@ import { spawn, spawnSync, type ChildProcess } from "node:child_process";
import { randomUUID } from "node:crypto";
import { asOptionalRecord } from "@openclaw/normalization-core/record-coerce";
import { formatErrorMessage } from "../infra/errors.js";
import type { MeetingAudioBackendSelection, MeetingAudioRuntime } from "./audio-backend.js";
import { decodeMeetingAudioBase64 } from "./audio-base64.js";
import { terminateMeetingBridgeProcess } from "./bridge-process.js";
import {
prepareMeetingNodeAudio,
readMeetingNodeCommand,
type MeetingNodeAudioConfig,
} from "./node-audio-config.js";
import { MeetingNodeAudioPullWaiters } from "./node-audio-pull-waiters.js";
import type { MeetingRealtimeAudioFormat } from "./realtime-audio-format.js";
export type { MeetingNodeAudioConfig } from "./node-audio-config.js";
const NODE_BRIDGE_TERMINATION_GRACE_MS = 2_000;
const NODE_BRIDGE_INPUT_DRAIN_MS = NODE_BRIDGE_TERMINATION_GRACE_MS + 1_000;
@@ -52,11 +61,17 @@ export type MeetingNodeHostOptions = {
bridgeIdPrefix: string;
defaultAudioInputCommand: readonly string[];
defaultAudioOutputCommand: readonly string[];
defaultAudio?: {
backend?: MeetingAudioBackendSelection;
bufferBytes: number;
format: MeetingRealtimeAudioFormat;
};
talkBackModes: ReadonlySet<string>;
agentMode: string;
normalizeUrl(input: unknown): string;
normalizeMeetingKey(url?: string): string | undefined;
assertAudioAvailable(timeoutMs: number): void;
assertAudioAvailable(timeoutMs: number): void | Promise<void>;
prepareAudio?(config: MeetingNodeAudioConfig, timeoutMs: number): Promise<MeetingAudioRuntime>;
browser: {
application: string;
buildProfileArgs(profile: string): string[];
@@ -65,16 +80,6 @@ export type MeetingNodeHostOptions = {
};
};
function readStringArray(value: unknown): string[] | undefined {
if (!Array.isArray(value)) {
return undefined;
}
const result = value.filter(
(entry): entry is string => typeof entry === "string" && entry.length > 0,
);
return result.length > 0 ? result : undefined;
}
function readString(value: unknown): string | undefined {
return typeof value === "string" && value.length > 0 ? value : undefined;
}
@@ -496,18 +501,19 @@ export function createMeetingNodeHost(options: MeetingNodeHostOptions): {
return { bridgeId, ok: true, clearCount: session.clearCount };
};
const startBrowser = (params: Record<string, unknown>) => {
const startBrowser = async (params: Record<string, unknown>) => {
const url = options.normalizeUrl(params.url);
const timeoutMs = readNumber(params.joinTimeoutMs, 30_000);
const mode = readString(params.mode);
let audioRuntime: MeetingAudioRuntime | undefined;
let bridgeId: string | undefined;
let audioBridge:
| { type: "external-command" }
| { type: "node-command-pair"; outputGeneration: true }
| undefined;
if (mode && options.talkBackModes.has(mode)) {
options.assertAudioAvailable(Math.min(timeoutMs, 10_000));
const healthCommand = readStringArray(params.audioBridgeHealthCommand);
audioRuntime = await prepareMeetingNodeAudio(params, Math.min(timeoutMs, 10_000), options);
const healthCommand = readMeetingNodeCommand(params.audioBridgeHealthCommand);
if (healthCommand) {
const health = runCommandWithTimeout(healthCommand, timeoutMs);
if (health.code !== 0) {
@@ -516,7 +522,7 @@ export function createMeetingNodeHost(options: MeetingNodeHostOptions): {
);
}
}
const bridgeCommand = readStringArray(params.audioBridgeCommand);
const bridgeCommand = readMeetingNodeCommand(params.audioBridgeCommand);
if (bridgeCommand) {
if (mode === options.agentMode) {
throw new Error(
@@ -532,12 +538,8 @@ export function createMeetingNodeHost(options: MeetingNodeHostOptions): {
audioBridge = { type: "external-command" };
} else {
const session = startCommandPair({
inputCommand: readStringArray(params.audioInputCommand) ?? [
...options.defaultAudioInputCommand,
],
outputCommand: readStringArray(params.audioOutputCommand) ?? [
...options.defaultAudioOutputCommand,
],
inputCommand: audioRuntime.inputCommand,
outputCommand: audioRuntime.outputCommand,
url,
mode,
});
@@ -571,6 +573,8 @@ export function createMeetingNodeHost(options: MeetingNodeHostOptions): {
}
return {
launched: params.launch !== false,
audioBackend: audioRuntime?.backend,
audioDeviceLabel: audioRuntime?.deviceLabel,
bridgeId,
audioBridge,
browser:
@@ -689,11 +693,17 @@ export function createMeetingNodeHost(options: MeetingNodeHostOptions): {
let result: unknown;
switch (action) {
case "setup":
options.assertAudioAvailable(10_000);
result = { ok: true };
{
const audioRuntime = await prepareMeetingNodeAudio(params, 10_000, options);
result = {
ok: true,
audioBackend: audioRuntime.backend,
audioDeviceLabel: audioRuntime.deviceLabel,
};
}
break;
case "start":
result = startBrowser(params);
result = await startBrowser(params);
break;
case "status":
result = bridgeStatus(params);
src/meeting-bot/node-invoke-policy.test.ts
+105
View File
@@ -0,0 +1,105 @@
import { describe, expect, it, vi } from "vitest";
import type { OpenClawPluginNodeInvokePolicyContext } from "../plugins/plugin-registration.types.js";
import { createMeetingBrowserNodeInvokePolicy } from "./node-invoke-policy.js";
function createPolicy(
audio: {
audioInputCommandOverride?: string[];
audioOutputCommandOverride?: string[];
} = {},
) {
return createMeetingBrowserNodeInvokePolicy({
commandName: "testmeeting.chrome",
deniedCode: "TEST_MEETING_DENIED",
displayName: "Test meeting",
normalizeUrl: (value) => String(value),
start: {
launch: true,
joinTimeoutMs: 30_000,
audioBackend: "auto",
audioBufferBytes: 4_096,
audioFormat: "pcm16-24khz",
// Resolved local commands remain available to local Chrome but are not node overrides.
audioInputCommand: ["gateway-generated-capture"],
audioInputCommandOverride: audio.audioInputCommandOverride,
audioOutputCommand: ["gateway-generated-playback"],
audioOutputCommandOverride: audio.audioOutputCommandOverride,
bargeInInputCommand: ["trusted-barge-in"],
},
supportedModes: new Set(["agent", "bidi", "transcribe"]),
useConfiguredSetupCommands: true,
});
}
function createContext(params: Record<string, unknown>) {
const invokeNode = vi.fn(async () => ({ ok: true as const }));
return {
invokeNode,
context: {
command: "testmeeting.chrome",
config: {},
invokeNode,
nodeId: "node-1",
params,
} as OpenClawPluginNodeInvokePolicyContext,
};
}
describe("meeting browser node invoke policy audio", () => {
it("forwards trusted declarations without gateway-generated command defaults", async () => {
const policy = createPolicy();
const { context, invokeNode } = createContext({
action: "start",
url: "https://meeting.test/call",
mode: "bidi",
audioBackend: "blackhole-2ch",
audioBufferBytes: 1,
audioFormat: "g711-ulaw-8khz",
audioInputCommand: ["caller-capture"],
audioOutputCommand: ["caller-playback"],
});
await policy.handle(context);
expect(invokeNode).toHaveBeenCalledWith({
params: {
action: "start",
url: "https://meeting.test/call",
mode: "bidi",
launch: true,
browserProfile: undefined,
joinTimeoutMs: 30_000,
audioBackend: "auto",
audioBufferBytes: 4_096,
audioFormat: "pcm16-24khz",
bargeInInputCommand: ["trusted-barge-in"],
},
});
});
it("forwards only trusted explicit command overrides for setup", async () => {
const policy = createPolicy({
audioInputCommandOverride: ["trusted-capture", "--raw"],
audioOutputCommandOverride: ["trusted-playback", "--raw"],
});
const { context, invokeNode } = createContext({
action: "setup",
audioInputCommand: ["caller-capture"],
audioOutputCommand: ["caller-playback"],
});
await policy.handle(context);
expect(invokeNode).toHaveBeenCalledWith({
params: {
action: "setup",
audioBackend: "auto",
audioBufferBytes: 4_096,
audioFormat: "pcm16-24khz",
audioInputCommand: ["trusted-capture", "--raw"],
audioOutputCommand: ["trusted-playback", "--raw"],
bargeInInputCommand: ["trusted-barge-in"],
},
});
});
});
src/meeting-bot/node-invoke-policy.ts
+54 -21
View File
@@ -4,14 +4,21 @@ import type {
OpenClawPluginNodeInvokePolicy,
OpenClawPluginNodeInvokePolicyResult,
} from "../plugins/plugin-registration.types.js";
import type { MeetingAudioBackendSelection } from "./audio-backend.js";
import { isMeetingAudioBase64 } from "./audio-base64.js";
import type { MeetingRealtimeAudioFormat } from "./realtime-audio-format.js";
export type MeetingBrowserNodeStartConfig = {
launch: boolean;
browserProfile?: string;
joinTimeoutMs: number;
audioBackend?: MeetingAudioBackendSelection;
audioBufferBytes?: number;
audioFormat?: MeetingRealtimeAudioFormat;
audioInputCommand?: string[];
audioInputCommandOverride?: string[];
audioOutputCommand?: string[];
audioOutputCommandOverride?: string[];
bargeInInputCommand?: string[];
audioBridgeCommand?: string[];
audioBridgeHealthCommand?: string[];
@@ -47,6 +54,51 @@ function copyCommand(command: string[] | undefined): string[] | undefined {
return command && command.length > 0 ? [...command] : undefined;
}
function copyConfiguredAudio(
target: Record<string, unknown>,
start: MeetingBrowserNodeStartConfig,
): void {
if (
start.audioBackend === "auto" ||
start.audioBackend === "blackhole-2ch" ||
start.audioBackend === "pipewire-pulse"
) {
target.audioBackend = start.audioBackend;
}
if (typeof start.audioBufferBytes === "number" && start.audioBufferBytes > 0) {
target.audioBufferBytes = start.audioBufferBytes;
}
if (start.audioFormat === "pcm16-24khz" || start.audioFormat === "g711-ulaw-8khz") {
target.audioFormat = start.audioFormat;
}
const hasCommandOverrideFields =
"audioInputCommandOverride" in start || "audioOutputCommandOverride" in start;
const audioInputCommand = copyCommand(
start.audioInputCommandOverride ??
(hasCommandOverrideFields ? undefined : start.audioInputCommand),
);
const audioOutputCommand = copyCommand(
start.audioOutputCommandOverride ??
(hasCommandOverrideFields ? undefined : start.audioOutputCommand),
);
if (audioInputCommand) {
target.audioInputCommand = audioInputCommand;
}
if (audioOutputCommand) {
target.audioOutputCommand = audioOutputCommand;
}
for (const key of [
"bargeInInputCommand",
"audioBridgeCommand",
"audioBridgeHealthCommand",
] as const) {
const command = copyCommand(start[key]);
if (command) {
target[key] = command;
}
}
}
function denied(options: MeetingBrowserNodePolicyOptions, message: string) {
return { ok: false as const, code: options.deniedCode, message };
}
@@ -88,17 +140,7 @@ function buildStartParams(
if (mode) {
startParams.mode = mode;
}
for (const key of [
"audioInputCommand",
"audioOutputCommand",
"audioBridgeCommand",
"audioBridgeHealthCommand",
] as const) {
const command = copyCommand(options.start[key]);
if (command) {
startParams[key] = command;
}
}
copyConfiguredAudio(startParams, options.start);
return approved(startParams);
}
@@ -258,16 +300,7 @@ export function createMeetingBrowserNodeInvokePolicy(
const action = readString(params.action);
if (action === "setup" && options.useConfiguredSetupCommands) {
const setupParams: Record<string, unknown> = { action };
for (const key of [
"audioInputCommand",
"audioOutputCommand",
"bargeInInputCommand",
] as const) {
const command = copyCommand(options.start[key]);
if (command) {
setupParams[key] = command;
}
}
copyConfiguredAudio(setupParams, options.start);
return await ctx.invokeNode({ params: setupParams });
}
const decision =
src/meeting-bot/platform-adapter.ts
+3
View File
@@ -1,4 +1,5 @@
import { formatErrorMessage } from "../infra/errors.js";
import { ensureMeetingAudioBackend, resolveMeetingAudioRuntimeForFormat } from "./audio-backend.js";
import { createMeetingChromeTransport } from "./chrome-transport.js";
import { createMeetingConfiguredNodeHost } from "./configured-node-host.js";
import { isMeetingRealtimeRouteReady, isMeetingTalkBackMode } from "./meeting-modes.js";
@@ -399,4 +400,6 @@ export const MeetingPlatformAdapter = {
createStatusPreludeSource: createMeetingStatusPreludeSource,
isRealtimeRouteReady: isMeetingRealtimeRouteReady,
isTalkBackMode: isMeetingTalkBackMode,
ensureAudioBackend: ensureMeetingAudioBackend,
resolveAudioRuntimeForFormat: resolveMeetingAudioRuntimeForFormat,
};
src/meeting-bot/plugin-cli.test.ts
+64
View File
@@ -0,0 +1,64 @@
import { Command } from "commander";
import { afterEach, describe, expect, it, vi } from "vitest";
import type { callGatewayFromCli } from "../cli/gateway-rpc.js";
import { registerMeetingPluginCli } from "./plugin-cli.js";
import type { MeetingPluginConfig } from "./plugin-config.js";
function createCli(callGateway: typeof callGatewayFromCli) {
const program = new Command();
registerMeetingPluginCli({
callGateway,
commandName: "testmeetings",
config: {} as MeetingPluginConfig,
descriptions: {
root: "test meetings",
join: "join",
leave: "leave",
status: "status",
setup: "setup",
testSpeech: "test speech",
testListen: "test listen",
},
methodPrefix: "testmeetings",
program,
resolveGatewayTimeoutMs: () => 1_000,
});
return program;
}
describe("meeting plugin CLI options", () => {
afterEach(() => {
process.exitCode = undefined;
vi.restoreAllMocks();
});
it("rejects misspelled setup modes and transports before gateway dispatch", async () => {
const callGateway = vi.fn();
const cli = createCli(callGateway as unknown as typeof callGatewayFromCli);
await expect(
cli.parseAsync(["testmeetings", "setup", "--mode", "agnt"], { from: "user" }),
).rejects.toThrow("mode must be agent, bidi, or transcribe; received agnt");
await expect(
cli.parseAsync(["testmeetings", "setup", "--transport", "definitely-not-a-transport"], {
from: "user",
}),
).rejects.toThrow(
"transport must be chrome or chrome-node; received definitely-not-a-transport",
);
expect(callGateway).not.toHaveBeenCalled();
});
it("sets a failing process exit code for unhealthy setup status", async () => {
vi.spyOn(process.stdout, "write").mockImplementation(() => true);
const callGateway = vi.fn(async () => ({ ok: false, checks: [] }));
await createCli(callGateway as unknown as typeof callGatewayFromCli).parseAsync(
["testmeetings", "setup"],
{ from: "user" },
);
expect(callGateway).toHaveBeenCalledOnce();
expect(process.exitCode).toBe(1);
});
});
src/meeting-bot/plugin-cli.ts
+31 -6
View File
@@ -31,8 +31,8 @@ type MeetingCliOptions = {
};
type JoinOptions = {
transport?: MeetingCliTransport;
mode?: MeetingCliMode;
transport?: string;
mode?: string;
message?: string;
timeoutMs?: string;
};
@@ -48,11 +48,31 @@ function parseTimeout(value: string | undefined): number | undefined {
return parsed;
}
function parseMeetingCliMode(value: string | undefined): MeetingCliMode | undefined {
if (value === undefined || value === "agent" || value === "bidi" || value === "transcribe") {
return value;
}
throw new Error(`mode must be agent, bidi, or transcribe; received ${value}`);
}
function parseMeetingCliTransport(value: string | undefined): MeetingCliTransport | undefined {
if (value === undefined || value === "chrome" || value === "chrome-node") {
return value;
}
throw new Error(`transport must be chrome or chrome-node; received ${value}`);
}
function markMeetingSetupFailure(result: unknown): void {
if (result && typeof result === "object" && "ok" in result && result.ok === false) {
process.exitCode = 1;
}
}
function joinPayload(url: string, options: JoinOptions): Record<string, unknown> {
return {
url,
...(options.transport ? { transport: options.transport } : {}),
...(options.mode ? { mode: options.mode } : {}),
...(options.transport ? { transport: parseMeetingCliTransport(options.transport) } : {}),
...(options.mode ? { mode: parseMeetingCliMode(options.mode) } : {}),
...(options.message ? { message: options.message } : {}),
...(options.timeoutMs ? { timeoutMs: parseTimeout(options.timeoutMs) } : {}),
};
@@ -81,6 +101,7 @@ export function registerMeetingPluginCli(options: MeetingCliOptions): void {
{ progress: false, scopes: ["operator.admin"] },
);
process.stdout.write(`${JSON.stringify(result, null, 2)}\n`);
return result;
};
const method = (action: string) => `${options.methodPrefix}.${action}`;
const root = options.program.command(options.commandName).description(options.descriptions.root);
@@ -121,8 +142,12 @@ export function registerMeetingPluginCli(options: MeetingCliOptions): void {
command("setup", options.descriptions.setup)
.option("--transport <transport>", "chrome or chrome-node")
.option("--mode <mode>", "agent, bidi, or transcribe")
.action(async (setupOptions: { transport?: MeetingCliTransport; mode?: MeetingCliMode }) => {
await call(method("setup"), setupOptions);
.action(async (setupOptions: { transport?: string; mode?: string }) => {
const result = await call(method("setup"), {
transport: parseMeetingCliTransport(setupOptions.transport),
mode: parseMeetingCliMode(setupOptions.mode),
});
markMeetingSetupFailure(result);
});
for (const [name, action, description] of [
src/meeting-bot/plugin-config.ts
+60 -40
View File
@@ -10,12 +10,12 @@ import {
resolveRealtimeVoiceAgentConsultToolPolicy,
type RealtimeVoiceAgentConsultToolPolicy,
} from "../talk/agent-consult-tool.js";
import {
resolveMeetingAudioRuntimeForFormat,
type MeetingAudioBackendSelection,
} from "./audio-backend.js";
import type { MeetingRealtimeAudioFormat } from "./realtime-audio-format.js";
import type { MeetingRealtimeEngineConfig } from "./realtime-engine.js";
import {
buildMeetingSoxAudioCommands,
type MeetingSoxAudioCommandParams,
} from "./sox-audio-command.js";
type MeetingPluginMode = "agent" | "bidi" | "transcribe";
@@ -23,7 +23,7 @@ export type MeetingPluginConfig = MeetingRealtimeEngineConfig & {
enabled: boolean;
defaultMode: MeetingPluginMode;
chrome: MeetingRealtimeEngineConfig["chrome"] & {
audioBackend: "blackhole-2ch";
audioBackend: MeetingAudioBackendSelection;
audioBufferBytes: number;
launch: boolean;
browserProfile?: string;
@@ -34,6 +34,8 @@ export type MeetingPluginConfig = MeetingRealtimeEngineConfig & {
waitForInCallMs: number;
audioInputCommand: string[];
audioOutputCommand: string[];
audioInputCommandOverride?: string[];
audioOutputCommandOverride?: string[];
bargeInInputCommand?: string[];
bargeInRmsThreshold: number;
bargeInPeakThreshold: number;
@@ -47,21 +49,16 @@ export type MeetingPluginConfig = MeetingRealtimeEngineConfig & {
};
};
type MeetingSoxAudioDevice = Pick<MeetingSoxAudioCommandParams, "device" | "deviceType">;
type MeetingPluginConfigOptions = {
defaultRealtimeInstructions: string;
resolveGatewayOperationTimeoutMs(config: MeetingPluginConfig): number;
resolveSoxAudioDevice(params: {
format: MeetingRealtimeAudioFormat;
}): MeetingSoxAudioDevice | undefined;
};
const DEFAULT_AUDIO_BUFFER_BYTES = 4_096;
const DEFAULT_AUDIO_FORMAT: MeetingRealtimeAudioFormat = "pcm16-24khz";
const DEFAULT_MODE_HELP =
"Agent consults OpenClaw, bidi uses direct realtime voice, and transcribe observes only.";
const CHROME_NODE_HELP = "Node id/name/IP that owns Chrome, BlackHole, and SoX.";
const CHROME_NODE_HELP = "Node id/name/IP that owns Chrome and the native virtual-audio backend.";
function resolveBoolean(value: unknown, fallback: boolean): boolean {
return typeof value === "boolean" ? value : fallback;
@@ -87,6 +84,11 @@ function resolveAudioFormat(value: unknown): MeetingRealtimeAudioFormat {
return normalized === "g711-ulaw-8khz" ? normalized : DEFAULT_AUDIO_FORMAT;
}
function resolveAudioBackend(value: unknown): MeetingAudioBackendSelection {
const normalized = normalizeOptionalLowercaseString(value)?.replaceAll("_", "-");
return normalized === "blackhole-2ch" || normalized === "pipewire-pulse" ? normalized : "auto";
}
function resolveProviders(value: unknown): Record<string, Record<string, unknown>> {
const providers: Record<string, Record<string, unknown>> = {};
for (const [key, entry] of Object.entries(asRecord(value))) {
@@ -99,27 +101,38 @@ function resolveProviders(value: unknown): Record<string, Record<string, unknown
}
export function createMeetingPluginConfigSchema(options: MeetingPluginConfigOptions) {
const buildSoxCommands = (format: MeetingRealtimeAudioFormat, bufferBytes: number) =>
buildMeetingSoxAudioCommands({
bufferBytes,
...options.resolveSoxAudioDevice({ format }),
format:
format === "g711-ulaw-8khz"
? { sampleRate: 8_000, channels: 1, encoding: "mu-law", bits: 8 }
: {
sampleRate: 24_000,
channels: 1,
encoding: "signed-integer",
bits: 16,
endian: "little",
},
});
const defaultSoxCommands = buildSoxCommands(DEFAULT_AUDIO_FORMAT, DEFAULT_AUDIO_BUFFER_BYTES);
const buildAudioCommands = (
backend: MeetingAudioBackendSelection,
format: MeetingRealtimeAudioFormat,
bufferBytes: number,
) => {
const platform =
backend === "blackhole-2ch"
? "darwin"
: backend === "pipewire-pulse"
? "linux"
: process.platform;
try {
return resolveMeetingAudioRuntimeForFormat({ backend, bufferBytes, format, platform });
} catch {
return resolveMeetingAudioRuntimeForFormat({
backend: "blackhole-2ch",
bufferBytes,
format,
platform: "darwin",
});
}
};
const defaultAudioRuntime = buildAudioCommands(
"auto",
DEFAULT_AUDIO_FORMAT,
DEFAULT_AUDIO_BUFFER_BYTES,
);
const defaults: MeetingPluginConfig = {
enabled: true,
defaultMode: "agent",
chrome: {
audioBackend: "blackhole-2ch",
audioBackend: "auto",
audioFormat: DEFAULT_AUDIO_FORMAT,
audioBufferBytes: DEFAULT_AUDIO_BUFFER_BYTES,
launch: true,
@@ -128,8 +141,8 @@ export function createMeetingPluginConfigSchema(options: MeetingPluginConfigOpti
autoJoin: true,
joinTimeoutMs: 30_000,
waitForInCallMs: 60_000,
audioInputCommand: defaultSoxCommands.inputCommand,
audioOutputCommand: defaultSoxCommands.outputCommand,
audioInputCommand: defaultAudioRuntime.inputCommand,
audioOutputCommand: defaultAudioRuntime.outputCommand,
bargeInRmsThreshold: 650,
bargeInPeakThreshold: 2_500,
bargeInCooldownMs: 900,
@@ -155,13 +168,18 @@ export function createMeetingPluginConfigSchema(options: MeetingPluginConfigOpti
17,
Math.trunc(resolvePositiveNumber(chrome.audioBufferBytes, DEFAULT_AUDIO_BUFFER_BYTES)),
);
const generatedCommands = buildSoxCommands(audioFormat, audioBufferBytes);
const audioBackend = resolveAudioBackend(chrome.audioBackend);
const generatedCommands = buildAudioCommands(audioBackend, audioFormat, audioBufferBytes);
const audioInputCommandOverride = normalizeOptionalTrimmedStringList(chrome.audioInputCommand);
const audioOutputCommandOverride = normalizeOptionalTrimmedStringList(
chrome.audioOutputCommand,
);
const provider = normalizeOptionalString(realtime.provider) ?? defaults.realtime.provider;
return {
enabled: resolveBoolean(raw.enabled, defaults.enabled),
defaultMode: resolveMode(raw.defaultMode),
chrome: {
audioBackend: "blackhole-2ch",
audioBackend,
audioFormat,
audioBufferBytes,
launch: resolveBoolean(chrome.launch, defaults.chrome.launch),
@@ -171,12 +189,10 @@ export function createMeetingPluginConfigSchema(options: MeetingPluginConfigOpti
autoJoin: resolveBoolean(chrome.autoJoin, defaults.chrome.autoJoin),
joinTimeoutMs: resolveTimer(chrome.joinTimeoutMs, defaults.chrome.joinTimeoutMs),
waitForInCallMs: resolveTimer(chrome.waitForInCallMs, defaults.chrome.waitForInCallMs),
audioInputCommand:
normalizeOptionalTrimmedStringList(chrome.audioInputCommand) ??
generatedCommands.inputCommand,
audioOutputCommand:
normalizeOptionalTrimmedStringList(chrome.audioOutputCommand) ??
generatedCommands.outputCommand,
audioInputCommand: audioInputCommandOverride ?? generatedCommands.inputCommand,
audioOutputCommand: audioOutputCommandOverride ?? generatedCommands.outputCommand,
audioInputCommandOverride,
audioOutputCommandOverride,
bargeInInputCommand: normalizeOptionalTrimmedStringList(chrome.bargeInInputCommand),
bargeInRmsThreshold: resolvePositiveNumber(
chrome.bargeInRmsThreshold,
@@ -219,6 +235,10 @@ export function createMeetingPluginConfigSchema(options: MeetingPluginConfigOpti
parse: resolveConfig,
uiHints: {
defaultMode: { label: "Default Mode", help: DEFAULT_MODE_HELP },
"chrome.audioBackend": {
label: "Chrome Audio Backend",
help: "Auto selects BlackHole 2ch on macOS or PipeWire-Pulse on Linux.",
},
"chrome.browserProfile": { label: "Chrome Profile", advanced: true },
"chrome.guestName": { label: "Guest Name" },
"chrome.waitForInCallMs": { label: "Wait For In-Call (ms)", advanced: true },
@@ -236,8 +256,8 @@ export function createMeetingPluginConfigSchema(options: MeetingPluginConfigOpti
} satisfies OpenClawPluginConfigSchema;
return {
configSchema,
defaultAudioInputCommand: defaultSoxCommands.inputCommand,
defaultAudioOutputCommand: defaultSoxCommands.outputCommand,
defaultAudioInputCommand: defaultAudioRuntime.inputCommand,
defaultAudioOutputCommand: defaultAudioRuntime.outputCommand,
resolveConfig,
resolveGatewayOperationTimeoutMs: (config: MeetingPluginConfig) =>
options.resolveGatewayOperationTimeoutMs(config),
src/meeting-bot/plugin-shell.ts
+6 -12
View File
@@ -34,8 +34,6 @@ import type {
MeetingTranscriptSnapshot,
} from "./session-types.js";
const SYSTEM_PROFILER_COMMAND = "/usr/sbin/system_profiler";
type MeetingPluginShellPlatform = {
browserLabel: string;
displayName: string;
@@ -67,10 +65,13 @@ export function createMeetingPluginNodeHostHandler(options: MeetingPluginNodeHos
bridgeIdPrefix: `${platform.session.idPrefix}_node_`,
talkBackModes: new Set(["agent", "bidi"]),
agentMode: "agent",
defaultAudio: {
backend: "auto",
bufferBytes: 4_096,
format: "pcm16-24khz",
},
normalizeUrl: (value) => platform.urls.validateAndNormalize(value),
normalizeMeetingKey: (value) => platform.urls.normalizeForReuse(value),
outputMentionsAudioDevice: (output) => /\bBlackHole\s+2ch\b/i.test(output),
systemProfilerCommand: SYSTEM_PROFILER_COMMAND,
browser: {
application: "Google Chrome",
buildProfileArgs: (profile) => ["--args", `--profile-directory=${profile}`],
@@ -143,12 +144,7 @@ export function createMeetingPluginChromeTransport<
>(
options: Omit<
MeetingChromeTransportOptions<Mode, Health, Transcript>,
| "browserNodeAdapter"
| "isRealtimeRouteReady"
| "isTalkBackMode"
| "nodeCommandName"
| "outputMentionsAudioDevice"
| "systemProfilerCommand"
"browserNodeAdapter" | "isRealtimeRouteReady" | "isTalkBackMode" | "nodeCommandName"
>,
) {
return createMeetingChromeTransport<MeetingChromeTransportConfig, Mode, Health, Transcript>({
@@ -157,8 +153,6 @@ export function createMeetingPluginChromeTransport<
isRealtimeRouteReady: isMeetingRealtimeRouteReady,
isTalkBackMode: isMeetingTalkBackMode,
nodeCommandName: options.platform.nodeCommandName,
outputMentionsAudioDevice: (output) => /\bBlackHole\s+2ch\b/i.test(output),
systemProfilerCommand: SYSTEM_PROFILER_COMMAND,
});
}
src/meeting-bot/runtime-facade-types.ts
+2
View File
@@ -1,5 +1,6 @@
import type { OpenClawConfig } from "../config/types.openclaw.js";
import type { PluginRuntime, RuntimeLogger } from "../plugins/runtime/types.js";
import type { MeetingAudioBackend } from "./audio-backend.js";
import type {
MeetingBrowserJoinSession,
MeetingPlatformAdapter,
@@ -49,6 +50,7 @@ export type MeetingRuntimeAudioBridge<Health extends MeetingBrowserHealth> =
type MeetingRuntimeLaunchResult<Health extends MeetingBrowserHealth> = {
launched: boolean;
audioBackend?: MeetingAudioBackend;
audioBridge?: MeetingRuntimeAudioBridge<Health>;
browser?: Health;
nodeId?: string;
src/meeting-bot/runtime-facade.ts
+1 -1
View File
@@ -273,7 +273,7 @@ export function createMeetingRuntimeFacade<
tab: result.tab,
});
session.chrome = {
audioBackend: "blackhole-2ch",
audioBackend: result.audioBackend,
launched: result.launched,
nodeId,
browserProfile: this.params.config.chrome.browserProfile,
src/meeting-bot/runtime-setup.ts
+33 -11
View File
@@ -2,6 +2,7 @@ import { uniqueStrings } from "@openclaw/normalization-core/string-normalization
import type { OpenClawConfig } from "../config/types.openclaw.js";
import { formatErrorMessage } from "../infra/errors.js";
import type { PluginRuntime } from "../plugins/runtime/types.js";
import { resolveMeetingAudioRuntimeForFormat, type MeetingAudioRuntime } from "./audio-backend.js";
import { resolveMeetingBrowserNodeInfo } from "./browser-node.js";
import { isMeetingTalkBackMode } from "./meeting-modes.js";
import type { MeetingPluginConfig } from "./plugin-config.js";
@@ -14,7 +15,11 @@ type MeetingSetupNodeAdapter = {
};
type MeetingRuntimeSetupOptions<Config extends MeetingPluginConfig, Mode extends string> = {
assertAudioDeviceAvailable(params: { runtime: PluginRuntime; timeoutMs: number }): Promise<void>;
assertAudioDeviceAvailable(params: {
config: Config;
runtime: PluginRuntime;
timeoutMs: number;
}): Promise<void>;
captionsMessage(mode: Mode): string;
connectedNodeMessage(nodeLabel: string | undefined): string;
guestJoinCheck(config: Config): { message: string; ok: boolean };
@@ -77,8 +82,15 @@ export function createMeetingRuntimeSetup<Config extends MeetingPluginConfig, Mo
command: options.nodeAdapter.nodeCommandName,
params: {
action: "setup",
audioInputCommand: params.config.chrome.audioInputCommand,
audioOutputCommand: params.config.chrome.audioOutputCommand,
audioBackend: params.config.chrome.audioBackend,
audioFormat: params.config.chrome.audioFormat,
audioBufferBytes: params.config.chrome.audioBufferBytes,
...(params.config.chrome.audioInputCommandOverride
? { audioInputCommand: params.config.chrome.audioInputCommandOverride }
: {}),
...(params.config.chrome.audioOutputCommandOverride
? { audioOutputCommand: params.config.chrome.audioOutputCommandOverride }
: {}),
...(params.config.chrome.bargeInInputCommand
? { bargeInInputCommand: params.config.chrome.bargeInInputCommand }
: {}),
@@ -88,7 +100,7 @@ export function createMeetingRuntimeSetup<Config extends MeetingPluginConfig, Mo
status = addMeetingSetupCheck(status, {
id: "chrome-node-audio-prerequisites",
ok: true,
message: "Remote macOS, BlackHole 2ch, and SoX prerequisites are ready",
message: "Remote virtual audio backend and command-pair prerequisites are ready",
});
}
} catch (error) {
@@ -108,23 +120,30 @@ export function createMeetingRuntimeSetup<Config extends MeetingPluginConfig, Mo
}
status = addMeetingSetupCheck(status, {
id: "audio-bridge",
ok:
params.config.chrome.audioInputCommand.length > 0 &&
params.config.chrome.audioOutputCommand.length > 0,
message: `SoX command-pair audio bridge configured (${params.config.chrome.audioFormat})`,
ok: true,
message: `Command-pair audio bridge configured (${params.config.chrome.audioFormat})`,
});
if (transport === "chrome-node") {
return status;
}
let audio: MeetingAudioRuntime | undefined;
try {
audio = resolveMeetingAudioRuntimeForFormat({
backend: params.config.chrome.audioBackend,
bufferBytes: params.config.chrome.audioBufferBytes,
format: params.config.chrome.audioFormat,
inputCommand: params.config.chrome.audioInputCommandOverride,
outputCommand: params.config.chrome.audioOutputCommandOverride,
});
await options.assertAudioDeviceAvailable({
config: params.config,
runtime: params.runtime,
timeoutMs: Math.min(params.config.chrome.joinTimeoutMs, 10_000),
});
status = addMeetingSetupCheck(status, {
id: "chrome-local-audio-device",
ok: true,
message: "BlackHole 2ch audio device found",
message: "Virtual meeting audio backend is ready",
});
} catch (error) {
status = addMeetingSetupCheck(status, {
@@ -132,11 +151,14 @@ export function createMeetingRuntimeSetup<Config extends MeetingPluginConfig, Mo
ok: false,
message: formatErrorMessage(error),
});
if (!audio) {
return status;
}
}
const commands = uniqueStrings(
[
params.config.chrome.audioInputCommand[0],
params.config.chrome.audioOutputCommand[0],
audio.inputCommand[0],
audio.outputCommand[0],
params.config.chrome.bargeInInputCommand?.[0],
].filter((value): value is string => Boolean(value?.trim())),
);
src/meeting-bot/session-types.ts
+3 -1
View File
@@ -1,3 +1,5 @@
import type { MeetingAudioBackend } from "./audio-backend.js";
/** Generic lifecycle state shared by browser and dial-in meeting sessions. */
export type MeetingSessionState = "active" | "ended";
@@ -140,7 +142,7 @@ export type MeetingPluginSession<
THealth extends MeetingBrowserHealth,
> = MeetingSessionRecord<TTransport, TMode> & {
chrome?: {
audioBackend: "blackhole-2ch";
audioBackend?: MeetingAudioBackend;
launched: boolean;
nodeId?: string;
browserProfile?: string;
src/meeting-bot/status-call-source.ts
+16 -10
View File
@@ -34,7 +34,9 @@ export function createMeetingStatusCallSource(options: MeetingStatusCallSourceOp
if (media.length > 0) {
try {
const devices = await navigator.mediaDevices.enumerateDevices();
const output = devices.find((device) => device.kind === "audiooutput" && isBlackHole(device.label));
const output = devices.find(
(device) => device.kind === "audiooutput" && isVirtualAudioDevice(device.label)
);
if (output?.deviceId) {
const routeErrors = [];
const liveStream = (element) =>
@@ -68,7 +70,7 @@ export function createMeetingStatusCallSource(options: MeetingStatusCallSourceOp
originalMuteBySource.set(element, Boolean(element.muted));
}
// Sink changes are asynchronous. Silence the physical output until either
// the source or its fallback bridge is confirmed on BlackHole.
// the source or its fallback bridge is confirmed on the virtual device.
element.muted = true;
}
}
@@ -237,9 +239,13 @@ export function createMeetingStatusCallSource(options: MeetingStatusCallSourceOp
audioOutputRouted = routed.length > 0 && routed.every(Boolean);
if (canMutateSession && !audioOutputRouted) suspendOwnedAudioBridges();
if (audioOutputRouted && bridgeEntries.length > 0) {
notes.push("Routed ${options.platform.displayName} remote audio to BlackHole 2ch through MediaStream bridges.");
notes.push(
"Routed ${options.platform.displayName} remote audio to " +
(output.label || "the virtual audio device") +
" through MediaStream bridges."
);
}
audioOutputDeviceLabel = output.label || "BlackHole 2ch";
audioOutputDeviceLabel = output.label || "Virtual audio device";
// An unloaded Teams media element can reject setSinkId before its stream
// arrives. Keep that state retryable; loaded-source failures are terminal.
if (!audioOutputRouted && routed.length > 0 && routeErrors.length > 0) {
@@ -249,7 +255,7 @@ export function createMeetingStatusCallSource(options: MeetingStatusCallSourceOp
} else {
audioOutputRouted = false;
if (canMutateSession) suspendOwnedAudioBridges();
notes.push("BlackHole 2ch speaker output was not visible to ${options.platform.displayName}.");
notes.push("The OpenClaw virtual audio speaker output was not visible to ${options.platform.displayName}.");
}
} catch (error) {
audioOutputRouted = false;
@@ -257,22 +263,22 @@ export function createMeetingStatusCallSource(options: MeetingStatusCallSourceOp
if (canMutateSession) suspendOwnedAudioBridges();
}
if (!audioOutputRouted && audioOutputRouteError) {
notes.push("Could not route ${options.platform.displayName} speaker output to BlackHole 2ch: " + audioOutputRouteError);
notes.push("Could not route ${options.platform.displayName} speaker output to the OpenClaw virtual audio device: " + audioOutputRouteError);
}
} else {
audioOutputRouted = false;
try {
const devices = await navigator.mediaDevices.enumerateDevices();
const output = devices.find(
(device) => device.kind === "audiooutput" && isBlackHole(device.label)
(device) => device.kind === "audiooutput" && isVirtualAudioDevice(device.label)
);
if (output?.deviceId) {
// Teams can briefly remove every media element during an in-call rerender.
// Retry only after proving the required output still exists.
audioOutputRouteRetryable = true;
audioOutputDeviceLabel = output.label || "BlackHole 2ch";
audioOutputDeviceLabel = output.label || "Virtual audio device";
} else {
notes.push("BlackHole 2ch speaker output was not visible to ${options.platform.displayName}.");
notes.push("The OpenClaw virtual audio speaker output was not visible to ${options.platform.displayName}.");
}
} catch (error) {
audioOutputRouteError = error?.message || String(error);
@@ -611,7 +617,7 @@ export function createMeetingStatusCallSource(options: MeetingStatusCallSourceOp
}
if (inCall && allowMicrophone && !manualAction) {
if (audioInputRouted !== true || audioOutputRouted !== true) {
manualAction = manualActionFor("${options.platform.manualActionReasonPrefix}-audio-choice-required", "Verify BlackHole 2ch is selected as both the ${options.platform.displayName} microphone and speaker before starting talk-back.");
manualAction = manualActionFor("${options.platform.manualActionReasonPrefix}-audio-choice-required", "Verify the OpenClaw virtual audio device is selected as both the ${options.platform.displayName} microphone and speaker before starting talk-back.");
} else if (micMuted !== false) {
manualAction = manualActionFor("${options.platform.manualActionReasonPrefix}-microphone-required", "Unmute the ${options.platform.displayName} microphone and verify the microphone control shows it is on before starting talk-back.");
}