diff --git a/docs/gateway/external-apps.md b/docs/gateway/external-apps.md index 115419b3250b..5dbc2c767d9c 100644 --- a/docs/gateway/external-apps.md +++ b/docs/gateway/external-apps.md @@ -123,9 +123,11 @@ authenticated readiness responses include `gateway-draining`; unauthenticated remote probes receive only `{ "ready": false }`. The HTTP health probe, suspension methods on existing WebSocket connections, and an already-enabled Admin HTTP RPC route remain available. Other RPCs return retryable -`UNAVAILABLE`. Built-in HTTP user-work routes, including OpenAI-compatible -APIs, tool/session operations, node watches, and configured hooks, return -`503` with `error.code: "gateway_unavailable"`. +`UNAVAILABLE`. Built-in HTTP user-work routes and ordinary plugin HTTP routes, +including OpenAI-compatible APIs, tool/session operations, node watches, and +configured hooks, return `503` with `error.code: "gateway_unavailable"`. New +plugin-owned WebSocket upgrades also return `503`; this covers upgrade +ownership, not work performed later over an established plugin socket. This handshake does not persist incoming messages, stop third-party channel transports, or control the hosting platform. The host must fence its ingress @@ -133,7 +135,8 @@ before preparation and remains responsible for wake, snapshot/freeze, and stop. `activeCount` is the aggregate tracked-work count, while `blockers` contains the non-zero category counts and bounded task details. This is not a general process-quiescence barrier. Channel health, maintenance, cache refresh, -plugin-owned HTTP routes, and plugin-owned background work can remain active. +established plugin WebSocket sessions, and plugin-owned background work can +remain active. The hosting platform must freeze or snapshot the full process tree and its filesystem consistently; unregistered work cannot be proven idle by this first contract. diff --git a/docs/plugins/admin-http-rpc.md b/docs/plugins/admin-http-rpc.md index ab82eac297f1..80d5b44e26b2 100644 --- a/docs/plugins/admin-http-rpc.md +++ b/docs/plugins/admin-http-rpc.md @@ -102,6 +102,7 @@ Treat this plugin as a full Gateway operator surface. - Trusted identity-bearing HTTP auth (`trusted-proxy` mode) honors `x-openclaw-scopes` when present. - `gateway.auth.mode="none"` means this route is unauthenticated if the plugin is enabled. Use that only behind a private ingress you fully trust. - Requests dispatch through the same Gateway method handlers and scope checks as WebSocket RPC, after the plugin route auth passes. +- The route remains reachable during a prepared suspension lease. Bounded request validation and the local `commands.list` discovery response remain available. Of the methods dispatched into the Gateway, only `gateway.suspend.prepare`, `gateway.suspend.status`, and `gateway.suspend.resume` may run while admission is closed; other allowlisted methods return the normal retryable Gateway `UNAVAILABLE` response. - Keep this route on loopback, tailnet, or a private trusted ingress. Do not expose it directly to the public internet. Use separate gateways when callers cross trust boundaries. ## Request diff --git a/docs/plugins/architecture-internals.md b/docs/plugins/architecture-internals.md index 9e1fd8afb621..071f6566573a 100644 --- a/docs/plugins/architecture-internals.md +++ b/docs/plugins/architecture-internals.md @@ -674,6 +674,7 @@ Notes: - `trusted-proxy` callers that do send `x-openclaw-scopes` get the declared scopes instead - a route can opt into `gatewayRuntimeScopeSurface: "trusted-operator"` to always honor `x-openclaw-scopes` for identity-bearing auth modes (falling back to the full CLI default scope set when the header is absent) - Practical rule: do not assume a gateway-auth plugin route is an implicit admin surface. If your route needs admin-only behavior, opt into `trusted-operator` scope surface, require an identity-bearing auth mode, and document the explicit `x-openclaw-scopes` header contract. +- After route matching and authentication, ordinary handlers participate in Gateway root-work admission. A prepared or restarting Gateway returns `503` before invoking the handler. The narrow exception is a manifest-entitled `auth: "gateway"` route that also opts into the route-specific `trusted-operator` surface; it remains reachable so suspension control dispatch cannot be stranded, while ordinary sibling routes from the same plugin remain behind the admission boundary. WebSocket `handleUpgrade` ownership uses the same atomic admission boundary; once the handler accepts a socket, the socket's later lifetime is plugin-owned and is not tracked by this boundary. ## Plugin SDK import paths diff --git a/docs/plugins/manifest.md b/docs/plugins/manifest.md index 88246887421e..d7d6c1f7aee5 100644 --- a/docs/plugins/manifest.md +++ b/docs/plugins/manifest.md @@ -626,7 +626,7 @@ Provider plugins that implement both `resolveUsageAuth` and `fetchUsageSnapshot` General embedding providers should declare `contracts.embeddingProviders` for each adapter registered with `api.registerEmbeddingProvider(...)`. Use the general contract for reusable vector generation, including providers consumed by memory search. `contracts.memoryEmbeddingProviders` is deprecated memory-specific compatibility and remains only while existing providers migrate to the generic embedding provider seam. -`contracts.gatewayMethodDispatch` currently accepts `"authenticated-request"`. It is an API hygiene gate for native plugin HTTP routes that intentionally dispatch Gateway control-plane methods in-process, not a sandbox against malicious native plugins. Use it only for tightly reviewed bundled/operator surfaces that already require Gateway HTTP auth. +`contracts.gatewayMethodDispatch` currently accepts `"authenticated-request"`. It is an API hygiene gate for native plugin HTTP routes that intentionally dispatch Gateway control-plane methods in-process, not a sandbox against malicious native plugins. Use it only for tightly reviewed bundled/operator surfaces that already require Gateway HTTP auth. An entitled route remains reachable while Gateway root-work admission is closed only when it also declares `auth: "gateway"` and the route-specific `gatewayRuntimeScopeSurface: "trusted-operator"`; ordinary sibling routes from the same plugin remain behind the admission boundary. This keeps suspension status and resume reachable without granting the whole plugin an admission bypass. Keep parsing and response shaping bounded outside dispatch; substantive or mutating work must go through Gateway method dispatch, which owns admission and scope enforcement. ## configContracts reference diff --git a/src/gateway/server-http.ts b/src/gateway/server-http.ts index 8e9faac17478..3938d22e7050 100644 --- a/src/gateway/server-http.ts +++ b/src/gateway/server-http.ts @@ -16,10 +16,7 @@ import { createDiagnosticTraceContext, runWithDiagnosticTraceContext, } from "../infra/diagnostic-trace-context.js"; -import { - isGatewayWorkAdmissionClosed, - tryBeginGatewayRootWorkAdmission, -} from "../process/gateway-work-admission.js"; +import { isGatewayWorkAdmissionClosed } from "../process/gateway-work-admission.js"; import { createLazyRuntimeModule } from "../shared/lazy-runtime.js"; import { resolveAssistantIdentity } from "./assistant-identity.js"; import type { AuthRateLimiter } from "./auth-rate-limit.js"; @@ -39,6 +36,10 @@ import { type PluginNodeCapabilitySurface, } from "./plugin-node-capability.js"; import type { HooksRequestHandler } from "./server/hooks-request-handler.js"; +import { + runWithGatewayHttpWorkAdmission, + writeGatewayUpgradeServiceUnavailable, +} from "./server/http-work-admission.js"; import { isProtectedPluginRoutePathFromContext, resolvePluginRoutePathContext, @@ -308,17 +309,6 @@ function writeUpgradeAuthFailure( socket.write("HTTP/1.1 401 Unauthorized\r\nConnection: close\r\n\r\n"); } -function writeUpgradeServiceUnavailable(socket: { write: (chunk: string) => void }, body: string) { - socket.write( - "HTTP/1.1 503 Service Unavailable\r\n" + - "Connection: close\r\n" + - "Content-Type: text/plain; charset=utf-8\r\n" + - `Content-Length: ${Buffer.byteLength(body, "utf8")}\r\n` + - "\r\n" + - body, - ); -} - function parseGatewayRequestPath(rawUrl: string | undefined): string | undefined { try { return new URL(rawUrl ?? "/", "http://localhost").pathname; @@ -354,35 +344,6 @@ export async function runGatewayHttpRequestStages( return false; } -/** Runs one core HTTP user-work route under the same root fence as Gateway RPCs. */ -export async function runWithGatewayHttpWorkAdmission( - res: ServerResponse, - run: () => Promise | boolean, -): Promise { - const admission = tryBeginGatewayRootWorkAdmission(); - if (!admission) { - res.statusCode = 503; - res.setHeader("Content-Type", "application/json; charset=utf-8"); - res.setHeader("Cache-Control", "no-store"); - res.setHeader("Retry-After", "1"); - res.end( - JSON.stringify({ - error: { - message: "Gateway is temporarily unavailable while suspending or restarting", - type: "service_unavailable", - code: "gateway_unavailable", - }, - }), - ); - return true; - } - try { - return await admission.run(async () => await run()); - } finally { - admission.release(); - } -} - function buildPluginRequestStages(params: { req: IncomingMessage; res: ServerResponse; @@ -986,18 +947,18 @@ export function attachGatewayUpgradeHandler(opts: { // Core Gateway upgrades must stop at the HTTP boundary so a client cannot hold an // untracked pre-connect socket after suspension or restart admission closes. if (isGatewayWorkAdmissionClosed()) { - writeUpgradeServiceUnavailable(socket, "Gateway websocket admission closed"); + writeGatewayUpgradeServiceUnavailable(socket, "Gateway websocket admission closed"); socket.destroy(); return; } const preauthBudgetKey = requestClientIp; if (wss.listenerCount("connection") === 0) { - writeUpgradeServiceUnavailable(socket, "Gateway websocket handlers unavailable"); + writeGatewayUpgradeServiceUnavailable(socket, "Gateway websocket handlers unavailable"); socket.destroy(); return; } if (!preauthConnectionBudget.acquire(preauthBudgetKey)) { - writeUpgradeServiceUnavailable(socket, "Too many unauthenticated sockets"); + writeGatewayUpgradeServiceUnavailable(socket, "Too many unauthenticated sockets"); socket.destroy(); return; } diff --git a/src/gateway/server-runtime-state.ts b/src/gateway/server-runtime-state.ts index e30f9cde2780..aaa7a4a6aded 100644 --- a/src/gateway/server-runtime-state.ts +++ b/src/gateway/server-runtime-state.ts @@ -30,15 +30,12 @@ import { createToolEventRecipientRegistry, } from "./server-chat-state.js"; import { MAX_PREAUTH_PAYLOAD_BYTES } from "./server-constants.js"; -import { - attachGatewayUpgradeHandler, - createGatewayHttpServer, - runWithGatewayHttpWorkAdmission, -} from "./server-http.js"; +import { attachGatewayUpgradeHandler, createGatewayHttpServer } from "./server-http.js"; import type { GatewayRequestContext } from "./server-methods/types.js"; import type { DedupeEntry } from "./server-shared.js"; import type { HookClientIpConfig, HooksRequestHandler } from "./server/hooks-request-handler.js"; import { listenGatewayHttpServer } from "./server/http-listen.js"; +import { runWithGatewayHttpWorkAdmission } from "./server/http-work-admission.js"; import type { PluginRoutePathContext } from "./server/plugins-http/path-context.js"; import { shouldEnforceGatewayAuthForPluginPath } from "./server/plugins-http/route-auth.js"; import { findMatchingPluginNodeCapabilityRoute } from "./server/plugins-http/route-capability.js"; diff --git a/src/gateway/server/http-work-admission.ts b/src/gateway/server/http-work-admission.ts new file mode 100644 index 000000000000..a51425a888a5 --- /dev/null +++ b/src/gateway/server/http-work-admission.ts @@ -0,0 +1,69 @@ +// Gateway HTTP boundary helpers coordinate request and upgrade work with host suspension. +import type { ServerResponse } from "node:http"; +import type { Duplex } from "node:stream"; +import { tryBeginGatewayRootWorkAdmission } from "../../process/gateway-work-admission.js"; + +type GatewayBoundaryHandler = () => Promise | boolean; + +async function runWithGatewayBoundaryWorkAdmission( + reject: () => void, + run: GatewayBoundaryHandler, +): Promise { + const admission = tryBeginGatewayRootWorkAdmission(); + if (!admission) { + reject(); + return true; + } + try { + return await admission.run(async () => await run()); + } finally { + admission.release(); + } +} + +/** Runs one HTTP user-work route under the same root fence as Gateway RPCs. */ +export async function runWithGatewayHttpWorkAdmission( + res: ServerResponse, + run: GatewayBoundaryHandler, +): Promise { + return await runWithGatewayBoundaryWorkAdmission(() => { + res.statusCode = 503; + res.setHeader("Content-Type", "application/json; charset=utf-8"); + res.setHeader("Cache-Control", "no-store"); + res.setHeader("Retry-After", "1"); + res.end( + JSON.stringify({ + error: { + message: "Gateway is temporarily unavailable while suspending or restarting", + type: "service_unavailable", + code: "gateway_unavailable", + }, + }), + ); + }, run); +} + +export function writeGatewayUpgradeServiceUnavailable( + socket: Pick, + body: string, +): void { + socket.write( + "HTTP/1.1 503 Service Unavailable\r\n" + + "Connection: close\r\n" + + "Content-Type: text/plain; charset=utf-8\r\n" + + `Content-Length: ${Buffer.byteLength(body, "utf8")}\r\n` + + "\r\n" + + body, + ); +} + +/** Holds upgrade admission until one plugin handler owns or declines the socket. */ +export async function runWithGatewayUpgradeWorkAdmission( + socket: Duplex, + run: GatewayBoundaryHandler, +): Promise { + return await runWithGatewayBoundaryWorkAdmission(() => { + writeGatewayUpgradeServiceUnavailable(socket, "Gateway websocket admission closed"); + socket.destroy(); + }, run); +} diff --git a/src/gateway/server/plugins-http.suspension-admission.test.ts b/src/gateway/server/plugins-http.suspension-admission.test.ts new file mode 100644 index 000000000000..ab52491f1643 --- /dev/null +++ b/src/gateway/server/plugins-http.suspension-admission.test.ts @@ -0,0 +1,407 @@ +// Proves plugin HTTP and upgrade handlers participate in Gateway suspension admission. +import type { IncomingMessage } from "node:http"; +import type { Duplex } from "node:stream"; +import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; +import type { GatewayActiveWorkInspectors } from "../../infra/gateway-active-work.js"; +import { + prepareGatewaySuspend, + resetGatewaySuspendCoordinatorForTest, +} from "../../infra/gateway-suspend-coordinator.js"; +import { dispatchGatewayMethod } from "../../plugin-sdk/gateway-method-runtime.js"; +import type { PluginHttpRouteRegistration } from "../../plugins/registry.js"; +import { + getActiveGatewayRootWorkCount, + resetGatewayWorkAdmission, + tryBeginGatewaySuspendAdmission, +} from "../../process/gateway-work-admission.js"; +import { __testing as controlPlaneRateLimitTesting } from "../control-plane-rate-limit.js"; +import type { GatewayRequestContext } from "../server-methods/types.js"; +import { makeMockHttpResponse } from "../test-http-response.js"; +import { createTestRegistry } from "./__tests__/test-utils.js"; +import { + createGatewayPluginRequestHandler, + createGatewayPluginUpgradeHandler, +} from "./plugins-http.js"; + +const ROUTE_PATH = "/plugin/suspension-proof"; + +function deferred() { + let resolve = () => {}; + const promise = new Promise((done) => { + resolve = done; + }); + return { promise, resolve }; +} + +function createRoute( + params: Partial & Pick, +): PluginHttpRouteRegistration { + return { + pluginId: "suspension-proof", + path: ROUTE_PATH, + auth: "plugin", + match: "exact", + source: "suspension-proof", + ...params, + }; +} + +function createLog() { + return { warn: vi.fn() } as unknown as Parameters< + typeof createGatewayPluginRequestHandler + >[0]["log"]; +} + +function createRootOnlyInspectors(): GatewayActiveWorkInspectors { + return { + getQueueSize: () => 0, + getPendingReplies: () => 0, + getEmbeddedRuns: () => 0, + getCronRuns: () => 0, + getActiveTasks: () => 0, + getTaskBlockers: () => [], + getRootRequests: () => getActiveGatewayRootWorkCount({ excludeCurrent: true }), + getSessionAdmissions: () => 0, + getSessionMutations: () => 0, + getChatRuns: () => 0, + getQueuedTurns: () => 0, + getTerminalPersistence: () => 0, + getTerminalSessions: () => 0, + }; +} + +function prepareWithRootOnly(requestId: string) { + return prepareGatewaySuspend({ + requestId, + pauseScheduling: vi.fn(), + resumeScheduling: vi.fn(), + inspect: createRootOnlyInspectors(), + }); +} + +function createMockUpgradeSocket() { + const socket = { + chunks: [] as string[], + destroyed: false, + write(chunk: string) { + socket.chunks.push(chunk); + }, + destroy() { + socket.destroyed = true; + }, + } as unknown as Duplex & { chunks: string[]; destroyed: boolean }; + return socket; +} + +function createRequestHandler( + routes: PluginHttpRouteRegistration[], + getGatewayRequestContext?: () => GatewayRequestContext, +) { + return createGatewayPluginRequestHandler({ + registry: createTestRegistry({ httpRoutes: routes }), + log: createLog(), + ...(getGatewayRequestContext ? { getGatewayRequestContext } : {}), + }); +} + +function createUpgradeHandler(routes: PluginHttpRouteRegistration[]) { + return createGatewayPluginUpgradeHandler({ + registry: createTestRegistry({ httpRoutes: routes }), + log: createLog(), + }); +} + +beforeEach(() => { + controlPlaneRateLimitTesting.resetControlPlaneRateLimitState(); + resetGatewaySuspendCoordinatorForTest(); + resetGatewayWorkAdmission(); +}); + +afterEach(() => { + controlPlaneRateLimitTesting.resetControlPlaneRateLimitState(); + resetGatewaySuspendCoordinatorForTest(); + resetGatewayWorkAdmission(); +}); + +describe("plugin HTTP suspension admission", () => { + it("keeps an in-flight ordinary route visible to suspension preparation", async () => { + const started = deferred(); + const finish = deferred(); + const handler = createRequestHandler([ + createRoute({ + handler: async () => { + started.resolve(); + await finish.promise; + return true; + }, + }), + ]); + const response = makeMockHttpResponse(); + const pending = handler({ url: ROUTE_PATH } as IncomingMessage, response.res); + await started.promise; + + expect(getActiveGatewayRootWorkCount()).toBe(1); + expect(prepareWithRootOnly("plugin-http-active")).toMatchObject({ + status: "busy", + reason: "active-work", + activeCount: 1, + blockers: [expect.objectContaining({ kind: "root-request", count: 1 })], + }); + + finish.resolve(); + await expect(pending).resolves.toBe(true); + expect(getActiveGatewayRootWorkCount()).toBe(0); + }); + + it("rejects an ordinary route with the canonical HTTP response after admission closes", async () => { + const routeHandler = vi.fn(() => true); + const handler = createRequestHandler([createRoute({ handler: routeHandler })]); + const suspension = tryBeginGatewaySuspendAdmission(() => {}); + expect(suspension?.commit()).toBe(true); + const response = makeMockHttpResponse(); + + await expect(handler({ url: ROUTE_PATH } as IncomingMessage, response.res)).resolves.toBe(true); + + expect(routeHandler).not.toHaveBeenCalled(); + expect(response.res.statusCode).toBe(503); + expect(response.setHeader).toHaveBeenCalledWith("Retry-After", "1"); + expect(JSON.parse(String(response.end.mock.calls[0]?.[0]))).toMatchObject({ + error: { code: "gateway_unavailable" }, + }); + expect(getActiveGatewayRootWorkCount()).toBe(0); + expect(suspension?.release()).toBe(true); + }); + + it("releases ordinary route admission after fallthrough and failure", async () => { + const fallthrough = vi.fn(() => false); + const handled = vi.fn(() => true); + const fallthroughHandler = createRequestHandler([ + createRoute({ handler: fallthrough }), + createRoute({ path: "/plugin", match: "prefix", handler: handled }), + ]); + const fallthroughResponse = makeMockHttpResponse(); + + await expect( + fallthroughHandler({ url: ROUTE_PATH } as IncomingMessage, fallthroughResponse.res), + ).resolves.toBe(true); + expect(fallthrough).toHaveBeenCalledOnce(); + expect(handled).toHaveBeenCalledOnce(); + expect(getActiveGatewayRootWorkCount()).toBe(0); + + const failingHandler = createRequestHandler([ + createRoute({ + handler: () => { + throw new Error("route failed"); + }, + }), + ]); + const failureResponse = makeMockHttpResponse(); + await expect( + failingHandler({ url: ROUTE_PATH } as IncomingMessage, failureResponse.res), + ).resolves.toBe(true); + expect(failureResponse.res.statusCode).toBe(500); + expect(getActiveGatewayRootWorkCount()).toBe(0); + }); + + it("keeps an ordinary sibling from an entitled plugin behind admission", async () => { + const ordinaryHandler = vi.fn(() => true); + const handler = createRequestHandler([ + createRoute({ + auth: "gateway", + gatewayMethodDispatchAllowed: true, + handler: ordinaryHandler, + }), + createRoute({ + path: `${ROUTE_PATH}/control`, + auth: "gateway", + gatewayRuntimeScopeSurface: "trusted-operator", + gatewayMethodDispatchAllowed: true, + handler: () => true, + }), + ]); + const suspension = tryBeginGatewaySuspendAdmission(() => {}); + expect(suspension?.commit()).toBe(true); + const response = makeMockHttpResponse(); + + await expect( + handler({ url: ROUTE_PATH, headers: {} } as IncomingMessage, response.res, undefined, { + gatewayAuthSatisfied: true, + gatewayRequestAuth: { authMethod: "token", trustDeclaredOperatorScopes: false }, + gatewayRequestOperatorScopes: ["operator.write"], + }), + ).resolves.toBe(true); + + expect(ordinaryHandler).not.toHaveBeenCalled(); + expect(response.res.statusCode).toBe(503); + expect(getActiveGatewayRootWorkCount()).toBe(0); + expect(suspension?.release()).toBe(true); + }); + + it("keeps entitled Gateway suspension dispatch outside the plugin route root", async () => { + const cron = { + pauseScheduling: vi.fn(), + resumeScheduling: vi.fn(), + getSuspensionBlockerCount: vi.fn(() => 0), + }; + const context = { + cron, + logGateway: { warn: vi.fn() }, + chatAbortControllers: new Map(), + chatQueuedTurns: new Map(), + terminalSessions: new Map(), + } as unknown as GatewayRequestContext; + let requestedMethod = "gateway.suspend.prepare"; + let requestedParams: Record = { requestId: "admin-http-suspension" }; + let dispatchResponse: Awaited> | undefined; + const handler = createRequestHandler( + [ + createRoute({ + auth: "gateway", + gatewayRuntimeScopeSurface: "trusted-operator", + gatewayMethodDispatchAllowed: true, + handler: async () => { + expect(getActiveGatewayRootWorkCount()).toBe(0); + dispatchResponse = await dispatchGatewayMethod(requestedMethod, requestedParams); + return true; + }, + }), + ], + () => context, + ); + const invoke = async (method: string, params: Record) => { + requestedMethod = method; + requestedParams = params; + dispatchResponse = undefined; + const response = makeMockHttpResponse(); + const handled = await handler( + { url: ROUTE_PATH, headers: {} } as IncomingMessage, + response.res, + undefined, + { + gatewayAuthSatisfied: true, + gatewayRequestAuth: { authMethod: "token", trustDeclaredOperatorScopes: false }, + gatewayRequestOperatorScopes: ["operator.admin"], + }, + ); + expect(handled).toBe(true); + expect(dispatchResponse).toBeDefined(); + return dispatchResponse!; + }; + + const prepared = await invoke("gateway.suspend.prepare", { + requestId: "admin-http-suspension", + }); + expect(prepared).toMatchObject({ + ok: true, + payload: { status: "ready", activeCount: 0, blockers: [] }, + }); + const suspensionId = (prepared.payload as { suspensionId: string }).suspensionId; + + await expect(invoke("gateway.suspend.status", { suspensionId })).resolves.toMatchObject({ + ok: true, + payload: { status: "ready" }, + }); + await expect(invoke("gateway.suspend.resume", { suspensionId })).resolves.toMatchObject({ + ok: true, + payload: { status: "running", resumed: true }, + }); + expect(cron.pauseScheduling).toHaveBeenCalledOnce(); + expect(cron.resumeScheduling).toHaveBeenCalledOnce(); + expect(getActiveGatewayRootWorkCount()).toBe(0); + }); +}); + +describe("plugin upgrade suspension admission", () => { + it("keeps an in-flight upgrade visible to suspension preparation", async () => { + const started = deferred(); + const finish = deferred(); + const handler = createUpgradeHandler([ + createRoute({ + handler: () => false, + handleUpgrade: async () => { + started.resolve(); + await finish.promise; + return true; + }, + }), + ]); + const socket = createMockUpgradeSocket(); + const pending = handler({ url: ROUTE_PATH } as IncomingMessage, socket, Buffer.alloc(0)); + await started.promise; + + expect(getActiveGatewayRootWorkCount()).toBe(1); + expect(prepareWithRootOnly("plugin-upgrade-active")).toMatchObject({ + status: "busy", + reason: "active-work", + activeCount: 1, + blockers: [expect.objectContaining({ kind: "root-request", count: 1 })], + }); + + finish.resolve(); + await expect(pending).resolves.toBe(true); + expect(getActiveGatewayRootWorkCount()).toBe(0); + }); + + it("rejects a new upgrade with HTTP 503 after admission closes", async () => { + const upgrade = vi.fn(() => true); + const handler = createUpgradeHandler([ + createRoute({ handler: () => false, handleUpgrade: upgrade }), + ]); + const suspension = tryBeginGatewaySuspendAdmission(() => {}); + expect(suspension?.commit()).toBe(true); + const socket = createMockUpgradeSocket(); + + await expect( + handler({ url: ROUTE_PATH } as IncomingMessage, socket, Buffer.alloc(0)), + ).resolves.toBe(true); + + expect(upgrade).not.toHaveBeenCalled(); + expect(socket.destroyed).toBe(true); + expect(socket.chunks.join("")).toContain("HTTP/1.1 503 Service Unavailable"); + expect(socket.chunks.join("")).toContain("Gateway websocket admission closed"); + expect(getActiveGatewayRootWorkCount()).toBe(0); + expect(suspension?.release()).toBe(true); + }); + + it("releases upgrade admission after fallthrough and failure", async () => { + const fallthrough = vi.fn(() => false); + const handled = vi.fn(() => true); + const fallthroughHandler = createUpgradeHandler([ + createRoute({ handler: () => false, handleUpgrade: fallthrough }), + createRoute({ + path: "/plugin", + match: "prefix", + handler: () => false, + handleUpgrade: handled, + }), + ]); + const fallthroughSocket = createMockUpgradeSocket(); + + await expect( + fallthroughHandler( + { url: ROUTE_PATH } as IncomingMessage, + fallthroughSocket, + Buffer.alloc(0), + ), + ).resolves.toBe(true); + expect(fallthrough).toHaveBeenCalledOnce(); + expect(handled).toHaveBeenCalledOnce(); + expect(fallthroughSocket.destroyed).toBe(false); + expect(getActiveGatewayRootWorkCount()).toBe(0); + + const failingHandler = createUpgradeHandler([ + createRoute({ + handler: () => false, + handleUpgrade: () => { + throw new Error("upgrade failed"); + }, + }), + ]); + const failureSocket = createMockUpgradeSocket(); + await expect( + failingHandler({ url: ROUTE_PATH } as IncomingMessage, failureSocket, Buffer.alloc(0)), + ).resolves.toBe(true); + expect(failureSocket.destroyed).toBe(true); + expect(getActiveGatewayRootWorkCount()).toBe(0); + }); +}); diff --git a/src/gateway/server/plugins-http.ts b/src/gateway/server/plugins-http.ts index 06969dc81fa3..fcd6c8fbaedc 100644 --- a/src/gateway/server/plugins-http.ts +++ b/src/gateway/server/plugins-http.ts @@ -11,6 +11,10 @@ import type { PluginHttpRouteRegistration, PluginRegistry } from "../../plugins/ import { withPluginRuntimeGatewayRequestScope } from "../../plugins/runtime/gateway-request-scope.js"; import type { AuthorizedGatewayHttpRequest } from "../http-utils.js"; import type { GatewayRequestContext, GatewayRequestOptions } from "../server-methods/types.js"; +import { + runWithGatewayHttpWorkAdmission, + runWithGatewayUpgradeWorkAdmission, +} from "./http-work-admission.js"; import { resolvePluginRouteRuntimeOperatorScopes } from "./plugin-route-runtime-scopes.js"; import { resolvePluginRoutePathContext, @@ -90,6 +94,16 @@ function getMissingPluginRouteRuntimeContext( return context.gatewayRequestOperatorScopes === undefined ? "caller scope context" : undefined; } +function canRunPluginHttpRouteWithoutAdmission(route: PluginHttpRouteRegistration): boolean { + // The manifest entitlement is plugin-wide; require the route-specific trusted operator + // surface so an ordinary sibling cannot start work after suspension reports ready. + return ( + route.auth === "gateway" && + route.gatewayRuntimeScopeSurface === "trusted-operator" && + route.gatewayMethodDispatchAllowed === true + ); +} + function createPluginRouteRuntimeScope(params: { route: PluginHttpRouteRegistration; req: IncomingMessage; @@ -191,18 +205,24 @@ export function createGatewayPluginRequestHandler(params: { for (const route of matchedRoutes) { try { - const handled = await withPluginRuntimeGatewayRequestScope( - createPluginRouteRuntimeScope({ - route, - req, - gatewayRequestContext, - gatewayRequestAuth, - gatewayRequestOperatorScopes, - gatewayRequestClientIp: dispatchContext?.gatewayRequestClientIp, - }), - async () => route.handler(req, res), - ); - if (handled !== false) { + const runRoute = async () => + (await withPluginRuntimeGatewayRequestScope( + createPluginRouteRuntimeScope({ + route, + req, + gatewayRequestContext, + gatewayRequestAuth, + gatewayRequestOperatorScopes, + gatewayRequestClientIp: dispatchContext?.gatewayRequestClientIp, + }), + async () => route.handler(req, res), + )) !== false; + // Entitled trusted-operator routes delegate substantive work through Gateway dispatch. + // An outer root would make gateway.suspend.prepare nested and permanently unreachable. + const handled = canRunPluginHttpRouteWithoutAdmission(route) + ? await runRoute() + : await runWithGatewayHttpWorkAdmission(res, runRoute); + if (handled) { return true; } } catch (err) { @@ -268,18 +288,22 @@ export function createGatewayPluginUpgradeHandler(params: { for (const route of matchedRoutes) { try { - const handled = await withPluginRuntimeGatewayRequestScope( - createPluginRouteRuntimeScope({ - route, - req, - gatewayRequestContext, - gatewayRequestAuth, - gatewayRequestOperatorScopes, - gatewayRequestClientIp: dispatchContext?.gatewayRequestClientIp, - }), - async () => route.handleUpgrade?.(req, socket, head), + const handled = await runWithGatewayUpgradeWorkAdmission( + socket, + async () => + (await withPluginRuntimeGatewayRequestScope( + createPluginRouteRuntimeScope({ + route, + req, + gatewayRequestContext, + gatewayRequestAuth, + gatewayRequestOperatorScopes, + gatewayRequestClientIp: dispatchContext?.gatewayRequestClientIp, + }), + async () => route.handleUpgrade?.(req, socket, head), + )) !== false, ); - if (handled !== false) { + if (handled) { return true; } } catch (err) {