Commit Graph

44 Commits

Author SHA1 Message Date
Peter Steinberger 7dcb4fb760 feat(talk): broker realtime sideband control (#121054)
* feat(talk): broker OpenAI realtime sideband

* test(openai): prove audio-only sideband offer

* fix(talk): harden realtime sideband control

* fix(talk): restore sideband session update fence

* fix(talk): recognize GA function output events

* refactor(talk): unify realtime broker session leases

* fix(talk): use PCM audio for WebRTC sideband

* chore(plugin-sdk): refresh API baseline
2026-08-09 05:14:20 -07:00
Vincent Koc c30322418e fix(openai): share GPT-Live browser broker ownership 2026-07-31 16:00:31 +08:00
synthclaw 2f93db39eb fix(openai): return undefined from cleanup on non-disable reasons
Fixes TS7030: Not all code paths return a value.
2026-07-31 00:31:30 -04:00
synthclaw b9e2d9e81b fix(openai): restrict GPT-Live broker cleanup to plugin disable only
Root cause: the OpenAI plugin registered the process-wide Quicksilver
browser session broker cleanup as an unconditional runtime lifecycle
callback. Session reset/delete/restart cleanup would permanently stop
the shared broker, while talk.catalog continued to report the provider
as ready. All later GPT-Live browser session reservations failed until
process restart.

Fix: the cleanup callback now checks ctx.reason and only tears down
the broker when reason === "disable". Session reset/delete/restart
cleanup leaves the broker running.

Regression test: verifies cleanup is a no-op for reset/delete/restart
and returns a promise for disable.

Fixes openclaw/openclaw#116525
2026-07-31 00:04:45 -04:00
Peter Steinberger dc60a9442d feat(talk): support GPT-Live over gateway relay (#115831)
* feat(talk): add GPT-Live gateway relay peer

* fix(talk): ignore duplicate GPT-Live sideband audio

* fix(openai): harden GPT-Live relay startup

* fix(deps): replace vulnerable werift ip helper

* fix(deps): scope werift ip override version

* fix(openai): normalize relay startup errors

* test(ci): register GPT-Live gateway live shard

* fix(talk): tighten GPT-Live relay readiness

* fix(talk): keep relay override helper private

* fix(talk): resolve relay readiness with launch model

* fix(openai): preserve GPT-Live media clock

* fix(talk): align GPT-Live relay readiness and framing

* fix(openai): expire pending GPT-Live reservations

* fix(openai): harden GPT-Live delegation audio

* style(openai): satisfy RTP reorder lint
2026-07-29 12:54:23 -04:00
Peter Steinberger f78ba09120 feat(talk): support GPT-Live realtime voice over ChatGPT OAuth (#115226)
* feat(gateway): share the canonical browser-origin policy with plugins

Export resolveAcceptedBrowserOrigin through openclaw/plugin-sdk/webhook-request-guards
so browser-facing plugin routes reuse the Gateway's real origin contract instead of a
narrow allowedOrigins array check. Private LAN/Tailnet Control UI loads and the
Host-header fallback were previously rejected with 403 by plugin offer routes while the
Control UI itself worked.

Moves the loopback/forwarded-header helpers to net.ts (re-exported from auth.ts) so the
guard can delegate without importing gateway auth, and migrates the Codex realtime
broker onto the shared seam.

* feat(talk): let providers own agent delegation for realtime voice

Adds an optional runAgentConsult callback to the browser-session create request and
injects the existing embedded consult runtime from talk-client, bound to the same agent
and session key the GA tool path uses. Providers whose realtime protocol delegates work
through their own control channel (rather than GA function calls) can now reach the
OpenClaw agent without a client round-trip.

Threads the effective per-session model into browser-session capability resolution so a
request-level model override selects the right capability set, and propagates a caller
abort signal into consultRealtimeVoiceAgent so a superseded delegation stops its run.

* feat(openai): support GPT-Live realtime voice over ChatGPT OAuth

Implements OpenAI's quicksilver/frameless session natively for Talk browser sessions.
The Gateway creates the WebRTC call (multipart sdp+session to https://api.openai.com/v1/live)
and owns the sideband control socket, so the browser never holds upstream credentials;
delegation.created events run through the OpenClaw agent and stream back as speakable
context appends.

Verified end-to-end on 2026-07-28 against a ChatGPT Pro OAuth profile: call create 201
with an rtc_* id and answer SDP, sideband session.started, session.close teardown.
ChatGPT OAuth is preferred over a Platform API key because /v1/live access for platform
keys is waitlist-gated; the legacy chatgpt.com backend route returns 403 for every model
and protocol version and is not used.

Accepted models are gpt-live-1-codex and gpt-live-1-boulder-alpha; the voice allowlist is
the ten values the route actually accepts, since an invalid voice is rejected at call
creation and cannot be repaired afterwards (session.update reports immutable_field_update).

* docs: document GPT-Live Talk support and its route gotchas

Records the working route and auth, the accepted models and voices, the browser-only
scope, and the two traps that cost the most time: the chatgpt.com backend route returns
403 Voice session access denied for every model, and that same 403 is also what an
invalid voice returns, so it must not be read as an account entitlement block.

* fix(openai): resolve GPT-Live CI failures

* refactor(openai): own zod runtime dependency

* fix(openai): satisfy lint and live-shard gates for GPT-Live

Types the retry-delay finish callback as Error so the rejection reason is provably an
Error at the call site; the abort path already normalized a non-Error AbortSignal reason,
but the unknown parameter type hid that from static analysis.

Registers the new GPT-Live live test in the native-live-extensions-openai shard
expectation. The shard selector already picked the file up from the real tree; only the
hardcoded list in the tooling test lagged.

* fix(openai): clean up post-rebase capability resolver

* fix(openai): preserve GPT-Live delegation fragments

* fix(openai): close GPT-Live sideband handoff race

* fix(openai): accept UUID GPT-Live call ids

* style: apply oxfmt to GPT-Live sources and Talk docs

* style: format Talk docs after rebase

* fix(openai): keep GPT-Live transcript context across ignored delegations
2026-07-28 15:05:55 -04:00
Peter Steinberger 96e5812426 docs: document medium extension sources 2026-06-04 21:33:54 -04:00
Peter Steinberger d92b3b5cc2 refactor: unify OpenAI provider identity
Refactor OpenAI provider identity so OpenAI remains the canonical provider for API-key and OAuth-backed flows while legacy openai-codex state is doctor/migration-only.

Keeps OpenAI Codex Responses as an API/transport class rather than a provider identity, moves auth aliases through providerAuthAliases, updates doctor repair sequencing for old auth/profile state, and refreshes tests/docs around the canonical OpenAI behavior.
2026-05-30 11:48:41 +02:00
Peter Steinberger a0f35574d0 Remove codex-cli backend and migrate to Codex runtime
Remove the bundled codex-cli backend, migrate legacy codex-cli refs and runtime pins to the Codex app-server runtime, and update live/backend workflow coverage for the supported CLI lanes.
2026-05-14 10:07:18 +01:00
pashpashpash 5bcc321343 Scope Codex heartbeat guidance to heartbeat turns (#76788)
* fix(codex): scope heartbeat guidance to collaboration mode

* fix heartbeat tool direct context

* test prompt heartbeat collaboration snapshots

* fix heartbeat changelog credit
2026-05-04 01:58:39 +09:00
Peter Steinberger 4336a7f3a9 refactor(plugin-sdk): narrow config runtime imports 2026-04-27 14:58:32 +01:00
Peter Steinberger ddcc39de91 fix(openai): reuse Codex OAuth for OpenAI images 2026-04-23 22:06:36 +01:00
Peter Steinberger c84a2f5244 feat(openai): add codex oauth image generation 2026-04-23 21:34:24 +01:00
Vincent Koc abb32e39b5 fix(openai): refresh live prompt overlay config 2026-04-22 23:37:16 -07:00
Peter Steinberger 2cd3164a0f feat(providers): share GPT-5 prompt overlay 2026-04-22 23:36:06 +01:00
Peter Steinberger 77e6e4cf87 refactor: move memory embeddings into provider plugins 2026-04-17 02:57:18 +01:00
Eva 6aa63b4fdd agents: add openai provider-owned tool compat 2026-04-11 01:27:31 +01:00
Peter Steinberger af62a2c2e4 style: fix extension lint violations 2026-04-06 14:53:55 +01:00
Peter Steinberger c39f061003 Revert "refactor(cli): remove bundled cli text providers"
This reverts commit 05d351c430.
2026-04-06 13:40:41 +01:00
Peter Steinberger 932194b7d5 feat(video): add provider support and discord fallback 2026-04-05 22:06:56 +01:00
Peter Steinberger 05d351c430 refactor(cli): remove bundled cli text providers 2026-04-05 18:46:36 +01:00
Peter Steinberger 760c4be438 feat(agents): add provider-owned system prompt contributions 2026-04-05 14:05:41 +01:00
Peter Steinberger e3ac0f43df feat(qwen): add qwen provider and video generation 2026-04-04 19:34:56 +01:00
Peter Steinberger 0b1c9c7057 fix: stabilize codex auth ownership and ws fallback cache 2026-04-04 20:03:15 +09:00
Vincent Koc 0099c309c9 refactor(openai): lazy-load provider registration 2026-04-04 14:53:02 +09:00
Peter Steinberger a23ab9b906 refactor: move voice-call realtime providers into extensions 2026-04-04 12:07:23 +09:00
Peter Steinberger a38cb20177 feat(openai): add default prompt overlay 2026-04-04 09:27:07 +09:00
Peter Steinberger a4a00aa1da feat: pluginize cli inference backends 2026-03-26 15:11:15 +00:00
Neerav Makwana 6fd9d2ff38 fix: support OpenAI Codex media understanding (#54829) (thanks @neeravmakwana)
* OpenAI: register Codex media understanding provider

* fix: route codex image prompts through system instructions

* fix: add changelog for codex image tool fix (#54829) (thanks @neeravmakwana)

* fix: remove any from provider registration tests (#54829) (thanks @neeravmakwana)

---------

Co-authored-by: Ayaan Zaidi <hi@obviy.us>
2026-03-26 10:10:11 +05:30
Vincent Koc 645c9210b3 style(format): fix provider test formatting drift 2026-03-22 18:25:55 -07:00
Vincent Koc 0a329b2c9f Revert "style(format): fix check drift in provider tests"
This reverts commit 2619f5fe55.
2026-03-22 18:25:07 -07:00
Vincent Koc 2619f5fe55 style(format): fix check drift in provider tests 2026-03-22 18:24:40 -07:00
Vincent Koc 0f54ca20aa refactor(image-generation): move provider builders into plugins 2026-03-22 17:57:31 -07:00
Vincent Koc de6bf58e79 refactor(tts): move speech providers into plugins 2026-03-22 17:48:59 -07:00
Vincent Koc 6ebcd853be fix(plugin-sdk): isolate provider entry surfaces 2026-03-18 13:20:46 -07:00
Peter Steinberger 6f795fd60e refactor: dedupe bundled plugin entrypoints 2026-03-17 00:14:12 -07:00
Peter Steinberger aa2d5aaa0c feat(plugins): add image generation capability 2026-03-16 22:58:55 -07:00
Peter Steinberger 50c3321d2e feat(media): route image tool through media providers 2026-03-16 22:00:39 -07:00
Peter Steinberger 9ebe38b6e3 refactor: untangle remaining plugin sdk boundaries 2026-03-16 21:16:32 -07:00
Peter Steinberger c081dc52b7 feat(plugins): move media understanding into vendor plugins 2026-03-16 20:58:34 -07:00
Peter Steinberger 3e010e280a feat(plugins): add media understanding provider registration 2026-03-16 20:42:00 -07:00
Peter Steinberger 662031a88e feat(plugins): add speech provider registration 2026-03-16 18:50:09 -07:00
Peter Steinberger b54e37c71f feat(plugins): merge openai vendor seams into one plugin 2026-03-15 18:20:52 -07:00
Peter Steinberger ee7ecb2dd4 feat(plugins): move anthropic and openai vendors to plugins 2026-03-15 17:07:28 -07:00