mirror of
https://github.com/openclaw/openclaw.git
synced 2026-08-12 21:53:00 -06:00
fix(gateway): admit plugin HTTP work during suspension (#104112)
* fix(gateway): admit plugin HTTP work during suspension * test(gateway): remove unused suspension import * docs: keep release notes in PR metadata * fix(gateway): scope suspension route bypass
This commit is contained in:
committed by
GitHub
parent
1b3af45657
commit
35d6077db8
@@ -123,9 +123,11 @@ authenticated readiness responses include `gateway-draining`; unauthenticated
|
||||
remote probes receive only `{ "ready": false }`. The HTTP health probe,
|
||||
suspension methods on existing WebSocket connections, and an already-enabled
|
||||
Admin HTTP RPC route remain available. Other RPCs return retryable
|
||||
`UNAVAILABLE`. Built-in HTTP user-work routes, including OpenAI-compatible
|
||||
APIs, tool/session operations, node watches, and configured hooks, return
|
||||
`503` with `error.code: "gateway_unavailable"`.
|
||||
`UNAVAILABLE`. Built-in HTTP user-work routes and ordinary plugin HTTP routes,
|
||||
including OpenAI-compatible APIs, tool/session operations, node watches, and
|
||||
configured hooks, return `503` with `error.code: "gateway_unavailable"`. New
|
||||
plugin-owned WebSocket upgrades also return `503`; this covers upgrade
|
||||
ownership, not work performed later over an established plugin socket.
|
||||
|
||||
This handshake does not persist incoming messages, stop third-party channel
|
||||
transports, or control the hosting platform. The host must fence its ingress
|
||||
@@ -133,7 +135,8 @@ before preparation and remains responsible for wake, snapshot/freeze, and
|
||||
stop. `activeCount` is the aggregate tracked-work count, while `blockers`
|
||||
contains the non-zero category counts and bounded task details. This is not a
|
||||
general process-quiescence barrier. Channel health, maintenance, cache refresh,
|
||||
plugin-owned HTTP routes, and plugin-owned background work can remain active.
|
||||
established plugin WebSocket sessions, and plugin-owned background work can
|
||||
remain active.
|
||||
The hosting platform must freeze or snapshot the full process tree and its
|
||||
filesystem consistently; unregistered work cannot be proven idle by this first
|
||||
contract.
|
||||
|
||||
@@ -102,6 +102,7 @@ Treat this plugin as a full Gateway operator surface.
|
||||
- Trusted identity-bearing HTTP auth (`trusted-proxy` mode) honors `x-openclaw-scopes` when present.
|
||||
- `gateway.auth.mode="none"` means this route is unauthenticated if the plugin is enabled. Use that only behind a private ingress you fully trust.
|
||||
- Requests dispatch through the same Gateway method handlers and scope checks as WebSocket RPC, after the plugin route auth passes.
|
||||
- The route remains reachable during a prepared suspension lease. Bounded request validation and the local `commands.list` discovery response remain available. Of the methods dispatched into the Gateway, only `gateway.suspend.prepare`, `gateway.suspend.status`, and `gateway.suspend.resume` may run while admission is closed; other allowlisted methods return the normal retryable Gateway `UNAVAILABLE` response.
|
||||
- Keep this route on loopback, tailnet, or a private trusted ingress. Do not expose it directly to the public internet. Use separate gateways when callers cross trust boundaries.
|
||||
|
||||
## Request
|
||||
|
||||
@@ -674,6 +674,7 @@ Notes:
|
||||
- `trusted-proxy` callers that do send `x-openclaw-scopes` get the declared scopes instead
|
||||
- a route can opt into `gatewayRuntimeScopeSurface: "trusted-operator"` to always honor `x-openclaw-scopes` for identity-bearing auth modes (falling back to the full CLI default scope set when the header is absent)
|
||||
- Practical rule: do not assume a gateway-auth plugin route is an implicit admin surface. If your route needs admin-only behavior, opt into `trusted-operator` scope surface, require an identity-bearing auth mode, and document the explicit `x-openclaw-scopes` header contract.
|
||||
- After route matching and authentication, ordinary handlers participate in Gateway root-work admission. A prepared or restarting Gateway returns `503` before invoking the handler. The narrow exception is a manifest-entitled `auth: "gateway"` route that also opts into the route-specific `trusted-operator` surface; it remains reachable so suspension control dispatch cannot be stranded, while ordinary sibling routes from the same plugin remain behind the admission boundary. WebSocket `handleUpgrade` ownership uses the same atomic admission boundary; once the handler accepts a socket, the socket's later lifetime is plugin-owned and is not tracked by this boundary.
|
||||
|
||||
## Plugin SDK import paths
|
||||
|
||||
|
||||
@@ -626,7 +626,7 @@ Provider plugins that implement both `resolveUsageAuth` and `fetchUsageSnapshot`
|
||||
|
||||
General embedding providers should declare `contracts.embeddingProviders` for each adapter registered with `api.registerEmbeddingProvider(...)`. Use the general contract for reusable vector generation, including providers consumed by memory search. `contracts.memoryEmbeddingProviders` is deprecated memory-specific compatibility and remains only while existing providers migrate to the generic embedding provider seam.
|
||||
|
||||
`contracts.gatewayMethodDispatch` currently accepts `"authenticated-request"`. It is an API hygiene gate for native plugin HTTP routes that intentionally dispatch Gateway control-plane methods in-process, not a sandbox against malicious native plugins. Use it only for tightly reviewed bundled/operator surfaces that already require Gateway HTTP auth.
|
||||
`contracts.gatewayMethodDispatch` currently accepts `"authenticated-request"`. It is an API hygiene gate for native plugin HTTP routes that intentionally dispatch Gateway control-plane methods in-process, not a sandbox against malicious native plugins. Use it only for tightly reviewed bundled/operator surfaces that already require Gateway HTTP auth. An entitled route remains reachable while Gateway root-work admission is closed only when it also declares `auth: "gateway"` and the route-specific `gatewayRuntimeScopeSurface: "trusted-operator"`; ordinary sibling routes from the same plugin remain behind the admission boundary. This keeps suspension status and resume reachable without granting the whole plugin an admission bypass. Keep parsing and response shaping bounded outside dispatch; substantive or mutating work must go through Gateway method dispatch, which owns admission and scope enforcement.
|
||||
|
||||
## configContracts reference
|
||||
|
||||
|
||||
@@ -16,10 +16,7 @@ import {
|
||||
createDiagnosticTraceContext,
|
||||
runWithDiagnosticTraceContext,
|
||||
} from "../infra/diagnostic-trace-context.js";
|
||||
import {
|
||||
isGatewayWorkAdmissionClosed,
|
||||
tryBeginGatewayRootWorkAdmission,
|
||||
} from "../process/gateway-work-admission.js";
|
||||
import { isGatewayWorkAdmissionClosed } from "../process/gateway-work-admission.js";
|
||||
import { createLazyRuntimeModule } from "../shared/lazy-runtime.js";
|
||||
import { resolveAssistantIdentity } from "./assistant-identity.js";
|
||||
import type { AuthRateLimiter } from "./auth-rate-limit.js";
|
||||
@@ -39,6 +36,10 @@ import {
|
||||
type PluginNodeCapabilitySurface,
|
||||
} from "./plugin-node-capability.js";
|
||||
import type { HooksRequestHandler } from "./server/hooks-request-handler.js";
|
||||
import {
|
||||
runWithGatewayHttpWorkAdmission,
|
||||
writeGatewayUpgradeServiceUnavailable,
|
||||
} from "./server/http-work-admission.js";
|
||||
import {
|
||||
isProtectedPluginRoutePathFromContext,
|
||||
resolvePluginRoutePathContext,
|
||||
@@ -308,17 +309,6 @@ function writeUpgradeAuthFailure(
|
||||
socket.write("HTTP/1.1 401 Unauthorized\r\nConnection: close\r\n\r\n");
|
||||
}
|
||||
|
||||
function writeUpgradeServiceUnavailable(socket: { write: (chunk: string) => void }, body: string) {
|
||||
socket.write(
|
||||
"HTTP/1.1 503 Service Unavailable\r\n" +
|
||||
"Connection: close\r\n" +
|
||||
"Content-Type: text/plain; charset=utf-8\r\n" +
|
||||
`Content-Length: ${Buffer.byteLength(body, "utf8")}\r\n` +
|
||||
"\r\n" +
|
||||
body,
|
||||
);
|
||||
}
|
||||
|
||||
function parseGatewayRequestPath(rawUrl: string | undefined): string | undefined {
|
||||
try {
|
||||
return new URL(rawUrl ?? "/", "http://localhost").pathname;
|
||||
@@ -354,35 +344,6 @@ export async function runGatewayHttpRequestStages(
|
||||
return false;
|
||||
}
|
||||
|
||||
/** Runs one core HTTP user-work route under the same root fence as Gateway RPCs. */
|
||||
export async function runWithGatewayHttpWorkAdmission(
|
||||
res: ServerResponse,
|
||||
run: () => Promise<boolean> | boolean,
|
||||
): Promise<boolean> {
|
||||
const admission = tryBeginGatewayRootWorkAdmission();
|
||||
if (!admission) {
|
||||
res.statusCode = 503;
|
||||
res.setHeader("Content-Type", "application/json; charset=utf-8");
|
||||
res.setHeader("Cache-Control", "no-store");
|
||||
res.setHeader("Retry-After", "1");
|
||||
res.end(
|
||||
JSON.stringify({
|
||||
error: {
|
||||
message: "Gateway is temporarily unavailable while suspending or restarting",
|
||||
type: "service_unavailable",
|
||||
code: "gateway_unavailable",
|
||||
},
|
||||
}),
|
||||
);
|
||||
return true;
|
||||
}
|
||||
try {
|
||||
return await admission.run(async () => await run());
|
||||
} finally {
|
||||
admission.release();
|
||||
}
|
||||
}
|
||||
|
||||
function buildPluginRequestStages(params: {
|
||||
req: IncomingMessage;
|
||||
res: ServerResponse;
|
||||
@@ -986,18 +947,18 @@ export function attachGatewayUpgradeHandler(opts: {
|
||||
// Core Gateway upgrades must stop at the HTTP boundary so a client cannot hold an
|
||||
// untracked pre-connect socket after suspension or restart admission closes.
|
||||
if (isGatewayWorkAdmissionClosed()) {
|
||||
writeUpgradeServiceUnavailable(socket, "Gateway websocket admission closed");
|
||||
writeGatewayUpgradeServiceUnavailable(socket, "Gateway websocket admission closed");
|
||||
socket.destroy();
|
||||
return;
|
||||
}
|
||||
const preauthBudgetKey = requestClientIp;
|
||||
if (wss.listenerCount("connection") === 0) {
|
||||
writeUpgradeServiceUnavailable(socket, "Gateway websocket handlers unavailable");
|
||||
writeGatewayUpgradeServiceUnavailable(socket, "Gateway websocket handlers unavailable");
|
||||
socket.destroy();
|
||||
return;
|
||||
}
|
||||
if (!preauthConnectionBudget.acquire(preauthBudgetKey)) {
|
||||
writeUpgradeServiceUnavailable(socket, "Too many unauthenticated sockets");
|
||||
writeGatewayUpgradeServiceUnavailable(socket, "Too many unauthenticated sockets");
|
||||
socket.destroy();
|
||||
return;
|
||||
}
|
||||
|
||||
@@ -30,15 +30,12 @@ import {
|
||||
createToolEventRecipientRegistry,
|
||||
} from "./server-chat-state.js";
|
||||
import { MAX_PREAUTH_PAYLOAD_BYTES } from "./server-constants.js";
|
||||
import {
|
||||
attachGatewayUpgradeHandler,
|
||||
createGatewayHttpServer,
|
||||
runWithGatewayHttpWorkAdmission,
|
||||
} from "./server-http.js";
|
||||
import { attachGatewayUpgradeHandler, createGatewayHttpServer } from "./server-http.js";
|
||||
import type { GatewayRequestContext } from "./server-methods/types.js";
|
||||
import type { DedupeEntry } from "./server-shared.js";
|
||||
import type { HookClientIpConfig, HooksRequestHandler } from "./server/hooks-request-handler.js";
|
||||
import { listenGatewayHttpServer } from "./server/http-listen.js";
|
||||
import { runWithGatewayHttpWorkAdmission } from "./server/http-work-admission.js";
|
||||
import type { PluginRoutePathContext } from "./server/plugins-http/path-context.js";
|
||||
import { shouldEnforceGatewayAuthForPluginPath } from "./server/plugins-http/route-auth.js";
|
||||
import { findMatchingPluginNodeCapabilityRoute } from "./server/plugins-http/route-capability.js";
|
||||
|
||||
@@ -0,0 +1,69 @@
|
||||
// Gateway HTTP boundary helpers coordinate request and upgrade work with host suspension.
|
||||
import type { ServerResponse } from "node:http";
|
||||
import type { Duplex } from "node:stream";
|
||||
import { tryBeginGatewayRootWorkAdmission } from "../../process/gateway-work-admission.js";
|
||||
|
||||
type GatewayBoundaryHandler = () => Promise<boolean> | boolean;
|
||||
|
||||
async function runWithGatewayBoundaryWorkAdmission(
|
||||
reject: () => void,
|
||||
run: GatewayBoundaryHandler,
|
||||
): Promise<boolean> {
|
||||
const admission = tryBeginGatewayRootWorkAdmission();
|
||||
if (!admission) {
|
||||
reject();
|
||||
return true;
|
||||
}
|
||||
try {
|
||||
return await admission.run(async () => await run());
|
||||
} finally {
|
||||
admission.release();
|
||||
}
|
||||
}
|
||||
|
||||
/** Runs one HTTP user-work route under the same root fence as Gateway RPCs. */
|
||||
export async function runWithGatewayHttpWorkAdmission(
|
||||
res: ServerResponse,
|
||||
run: GatewayBoundaryHandler,
|
||||
): Promise<boolean> {
|
||||
return await runWithGatewayBoundaryWorkAdmission(() => {
|
||||
res.statusCode = 503;
|
||||
res.setHeader("Content-Type", "application/json; charset=utf-8");
|
||||
res.setHeader("Cache-Control", "no-store");
|
||||
res.setHeader("Retry-After", "1");
|
||||
res.end(
|
||||
JSON.stringify({
|
||||
error: {
|
||||
message: "Gateway is temporarily unavailable while suspending or restarting",
|
||||
type: "service_unavailable",
|
||||
code: "gateway_unavailable",
|
||||
},
|
||||
}),
|
||||
);
|
||||
}, run);
|
||||
}
|
||||
|
||||
export function writeGatewayUpgradeServiceUnavailable(
|
||||
socket: Pick<Duplex, "write">,
|
||||
body: string,
|
||||
): void {
|
||||
socket.write(
|
||||
"HTTP/1.1 503 Service Unavailable\r\n" +
|
||||
"Connection: close\r\n" +
|
||||
"Content-Type: text/plain; charset=utf-8\r\n" +
|
||||
`Content-Length: ${Buffer.byteLength(body, "utf8")}\r\n` +
|
||||
"\r\n" +
|
||||
body,
|
||||
);
|
||||
}
|
||||
|
||||
/** Holds upgrade admission until one plugin handler owns or declines the socket. */
|
||||
export async function runWithGatewayUpgradeWorkAdmission(
|
||||
socket: Duplex,
|
||||
run: GatewayBoundaryHandler,
|
||||
): Promise<boolean> {
|
||||
return await runWithGatewayBoundaryWorkAdmission(() => {
|
||||
writeGatewayUpgradeServiceUnavailable(socket, "Gateway websocket admission closed");
|
||||
socket.destroy();
|
||||
}, run);
|
||||
}
|
||||
@@ -0,0 +1,407 @@
|
||||
// Proves plugin HTTP and upgrade handlers participate in Gateway suspension admission.
|
||||
import type { IncomingMessage } from "node:http";
|
||||
import type { Duplex } from "node:stream";
|
||||
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import type { GatewayActiveWorkInspectors } from "../../infra/gateway-active-work.js";
|
||||
import {
|
||||
prepareGatewaySuspend,
|
||||
resetGatewaySuspendCoordinatorForTest,
|
||||
} from "../../infra/gateway-suspend-coordinator.js";
|
||||
import { dispatchGatewayMethod } from "../../plugin-sdk/gateway-method-runtime.js";
|
||||
import type { PluginHttpRouteRegistration } from "../../plugins/registry.js";
|
||||
import {
|
||||
getActiveGatewayRootWorkCount,
|
||||
resetGatewayWorkAdmission,
|
||||
tryBeginGatewaySuspendAdmission,
|
||||
} from "../../process/gateway-work-admission.js";
|
||||
import { __testing as controlPlaneRateLimitTesting } from "../control-plane-rate-limit.js";
|
||||
import type { GatewayRequestContext } from "../server-methods/types.js";
|
||||
import { makeMockHttpResponse } from "../test-http-response.js";
|
||||
import { createTestRegistry } from "./__tests__/test-utils.js";
|
||||
import {
|
||||
createGatewayPluginRequestHandler,
|
||||
createGatewayPluginUpgradeHandler,
|
||||
} from "./plugins-http.js";
|
||||
|
||||
const ROUTE_PATH = "/plugin/suspension-proof";
|
||||
|
||||
function deferred() {
|
||||
let resolve = () => {};
|
||||
const promise = new Promise<void>((done) => {
|
||||
resolve = done;
|
||||
});
|
||||
return { promise, resolve };
|
||||
}
|
||||
|
||||
function createRoute(
|
||||
params: Partial<PluginHttpRouteRegistration> & Pick<PluginHttpRouteRegistration, "handler">,
|
||||
): PluginHttpRouteRegistration {
|
||||
return {
|
||||
pluginId: "suspension-proof",
|
||||
path: ROUTE_PATH,
|
||||
auth: "plugin",
|
||||
match: "exact",
|
||||
source: "suspension-proof",
|
||||
...params,
|
||||
};
|
||||
}
|
||||
|
||||
function createLog() {
|
||||
return { warn: vi.fn() } as unknown as Parameters<
|
||||
typeof createGatewayPluginRequestHandler
|
||||
>[0]["log"];
|
||||
}
|
||||
|
||||
function createRootOnlyInspectors(): GatewayActiveWorkInspectors {
|
||||
return {
|
||||
getQueueSize: () => 0,
|
||||
getPendingReplies: () => 0,
|
||||
getEmbeddedRuns: () => 0,
|
||||
getCronRuns: () => 0,
|
||||
getActiveTasks: () => 0,
|
||||
getTaskBlockers: () => [],
|
||||
getRootRequests: () => getActiveGatewayRootWorkCount({ excludeCurrent: true }),
|
||||
getSessionAdmissions: () => 0,
|
||||
getSessionMutations: () => 0,
|
||||
getChatRuns: () => 0,
|
||||
getQueuedTurns: () => 0,
|
||||
getTerminalPersistence: () => 0,
|
||||
getTerminalSessions: () => 0,
|
||||
};
|
||||
}
|
||||
|
||||
function prepareWithRootOnly(requestId: string) {
|
||||
return prepareGatewaySuspend({
|
||||
requestId,
|
||||
pauseScheduling: vi.fn(),
|
||||
resumeScheduling: vi.fn(),
|
||||
inspect: createRootOnlyInspectors(),
|
||||
});
|
||||
}
|
||||
|
||||
function createMockUpgradeSocket() {
|
||||
const socket = {
|
||||
chunks: [] as string[],
|
||||
destroyed: false,
|
||||
write(chunk: string) {
|
||||
socket.chunks.push(chunk);
|
||||
},
|
||||
destroy() {
|
||||
socket.destroyed = true;
|
||||
},
|
||||
} as unknown as Duplex & { chunks: string[]; destroyed: boolean };
|
||||
return socket;
|
||||
}
|
||||
|
||||
function createRequestHandler(
|
||||
routes: PluginHttpRouteRegistration[],
|
||||
getGatewayRequestContext?: () => GatewayRequestContext,
|
||||
) {
|
||||
return createGatewayPluginRequestHandler({
|
||||
registry: createTestRegistry({ httpRoutes: routes }),
|
||||
log: createLog(),
|
||||
...(getGatewayRequestContext ? { getGatewayRequestContext } : {}),
|
||||
});
|
||||
}
|
||||
|
||||
function createUpgradeHandler(routes: PluginHttpRouteRegistration[]) {
|
||||
return createGatewayPluginUpgradeHandler({
|
||||
registry: createTestRegistry({ httpRoutes: routes }),
|
||||
log: createLog(),
|
||||
});
|
||||
}
|
||||
|
||||
beforeEach(() => {
|
||||
controlPlaneRateLimitTesting.resetControlPlaneRateLimitState();
|
||||
resetGatewaySuspendCoordinatorForTest();
|
||||
resetGatewayWorkAdmission();
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
controlPlaneRateLimitTesting.resetControlPlaneRateLimitState();
|
||||
resetGatewaySuspendCoordinatorForTest();
|
||||
resetGatewayWorkAdmission();
|
||||
});
|
||||
|
||||
describe("plugin HTTP suspension admission", () => {
|
||||
it("keeps an in-flight ordinary route visible to suspension preparation", async () => {
|
||||
const started = deferred();
|
||||
const finish = deferred();
|
||||
const handler = createRequestHandler([
|
||||
createRoute({
|
||||
handler: async () => {
|
||||
started.resolve();
|
||||
await finish.promise;
|
||||
return true;
|
||||
},
|
||||
}),
|
||||
]);
|
||||
const response = makeMockHttpResponse();
|
||||
const pending = handler({ url: ROUTE_PATH } as IncomingMessage, response.res);
|
||||
await started.promise;
|
||||
|
||||
expect(getActiveGatewayRootWorkCount()).toBe(1);
|
||||
expect(prepareWithRootOnly("plugin-http-active")).toMatchObject({
|
||||
status: "busy",
|
||||
reason: "active-work",
|
||||
activeCount: 1,
|
||||
blockers: [expect.objectContaining({ kind: "root-request", count: 1 })],
|
||||
});
|
||||
|
||||
finish.resolve();
|
||||
await expect(pending).resolves.toBe(true);
|
||||
expect(getActiveGatewayRootWorkCount()).toBe(0);
|
||||
});
|
||||
|
||||
it("rejects an ordinary route with the canonical HTTP response after admission closes", async () => {
|
||||
const routeHandler = vi.fn(() => true);
|
||||
const handler = createRequestHandler([createRoute({ handler: routeHandler })]);
|
||||
const suspension = tryBeginGatewaySuspendAdmission(() => {});
|
||||
expect(suspension?.commit()).toBe(true);
|
||||
const response = makeMockHttpResponse();
|
||||
|
||||
await expect(handler({ url: ROUTE_PATH } as IncomingMessage, response.res)).resolves.toBe(true);
|
||||
|
||||
expect(routeHandler).not.toHaveBeenCalled();
|
||||
expect(response.res.statusCode).toBe(503);
|
||||
expect(response.setHeader).toHaveBeenCalledWith("Retry-After", "1");
|
||||
expect(JSON.parse(String(response.end.mock.calls[0]?.[0]))).toMatchObject({
|
||||
error: { code: "gateway_unavailable" },
|
||||
});
|
||||
expect(getActiveGatewayRootWorkCount()).toBe(0);
|
||||
expect(suspension?.release()).toBe(true);
|
||||
});
|
||||
|
||||
it("releases ordinary route admission after fallthrough and failure", async () => {
|
||||
const fallthrough = vi.fn(() => false);
|
||||
const handled = vi.fn(() => true);
|
||||
const fallthroughHandler = createRequestHandler([
|
||||
createRoute({ handler: fallthrough }),
|
||||
createRoute({ path: "/plugin", match: "prefix", handler: handled }),
|
||||
]);
|
||||
const fallthroughResponse = makeMockHttpResponse();
|
||||
|
||||
await expect(
|
||||
fallthroughHandler({ url: ROUTE_PATH } as IncomingMessage, fallthroughResponse.res),
|
||||
).resolves.toBe(true);
|
||||
expect(fallthrough).toHaveBeenCalledOnce();
|
||||
expect(handled).toHaveBeenCalledOnce();
|
||||
expect(getActiveGatewayRootWorkCount()).toBe(0);
|
||||
|
||||
const failingHandler = createRequestHandler([
|
||||
createRoute({
|
||||
handler: () => {
|
||||
throw new Error("route failed");
|
||||
},
|
||||
}),
|
||||
]);
|
||||
const failureResponse = makeMockHttpResponse();
|
||||
await expect(
|
||||
failingHandler({ url: ROUTE_PATH } as IncomingMessage, failureResponse.res),
|
||||
).resolves.toBe(true);
|
||||
expect(failureResponse.res.statusCode).toBe(500);
|
||||
expect(getActiveGatewayRootWorkCount()).toBe(0);
|
||||
});
|
||||
|
||||
it("keeps an ordinary sibling from an entitled plugin behind admission", async () => {
|
||||
const ordinaryHandler = vi.fn(() => true);
|
||||
const handler = createRequestHandler([
|
||||
createRoute({
|
||||
auth: "gateway",
|
||||
gatewayMethodDispatchAllowed: true,
|
||||
handler: ordinaryHandler,
|
||||
}),
|
||||
createRoute({
|
||||
path: `${ROUTE_PATH}/control`,
|
||||
auth: "gateway",
|
||||
gatewayRuntimeScopeSurface: "trusted-operator",
|
||||
gatewayMethodDispatchAllowed: true,
|
||||
handler: () => true,
|
||||
}),
|
||||
]);
|
||||
const suspension = tryBeginGatewaySuspendAdmission(() => {});
|
||||
expect(suspension?.commit()).toBe(true);
|
||||
const response = makeMockHttpResponse();
|
||||
|
||||
await expect(
|
||||
handler({ url: ROUTE_PATH, headers: {} } as IncomingMessage, response.res, undefined, {
|
||||
gatewayAuthSatisfied: true,
|
||||
gatewayRequestAuth: { authMethod: "token", trustDeclaredOperatorScopes: false },
|
||||
gatewayRequestOperatorScopes: ["operator.write"],
|
||||
}),
|
||||
).resolves.toBe(true);
|
||||
|
||||
expect(ordinaryHandler).not.toHaveBeenCalled();
|
||||
expect(response.res.statusCode).toBe(503);
|
||||
expect(getActiveGatewayRootWorkCount()).toBe(0);
|
||||
expect(suspension?.release()).toBe(true);
|
||||
});
|
||||
|
||||
it("keeps entitled Gateway suspension dispatch outside the plugin route root", async () => {
|
||||
const cron = {
|
||||
pauseScheduling: vi.fn(),
|
||||
resumeScheduling: vi.fn(),
|
||||
getSuspensionBlockerCount: vi.fn(() => 0),
|
||||
};
|
||||
const context = {
|
||||
cron,
|
||||
logGateway: { warn: vi.fn() },
|
||||
chatAbortControllers: new Map(),
|
||||
chatQueuedTurns: new Map(),
|
||||
terminalSessions: new Map(),
|
||||
} as unknown as GatewayRequestContext;
|
||||
let requestedMethod = "gateway.suspend.prepare";
|
||||
let requestedParams: Record<string, unknown> = { requestId: "admin-http-suspension" };
|
||||
let dispatchResponse: Awaited<ReturnType<typeof dispatchGatewayMethod>> | undefined;
|
||||
const handler = createRequestHandler(
|
||||
[
|
||||
createRoute({
|
||||
auth: "gateway",
|
||||
gatewayRuntimeScopeSurface: "trusted-operator",
|
||||
gatewayMethodDispatchAllowed: true,
|
||||
handler: async () => {
|
||||
expect(getActiveGatewayRootWorkCount()).toBe(0);
|
||||
dispatchResponse = await dispatchGatewayMethod(requestedMethod, requestedParams);
|
||||
return true;
|
||||
},
|
||||
}),
|
||||
],
|
||||
() => context,
|
||||
);
|
||||
const invoke = async (method: string, params: Record<string, unknown>) => {
|
||||
requestedMethod = method;
|
||||
requestedParams = params;
|
||||
dispatchResponse = undefined;
|
||||
const response = makeMockHttpResponse();
|
||||
const handled = await handler(
|
||||
{ url: ROUTE_PATH, headers: {} } as IncomingMessage,
|
||||
response.res,
|
||||
undefined,
|
||||
{
|
||||
gatewayAuthSatisfied: true,
|
||||
gatewayRequestAuth: { authMethod: "token", trustDeclaredOperatorScopes: false },
|
||||
gatewayRequestOperatorScopes: ["operator.admin"],
|
||||
},
|
||||
);
|
||||
expect(handled).toBe(true);
|
||||
expect(dispatchResponse).toBeDefined();
|
||||
return dispatchResponse!;
|
||||
};
|
||||
|
||||
const prepared = await invoke("gateway.suspend.prepare", {
|
||||
requestId: "admin-http-suspension",
|
||||
});
|
||||
expect(prepared).toMatchObject({
|
||||
ok: true,
|
||||
payload: { status: "ready", activeCount: 0, blockers: [] },
|
||||
});
|
||||
const suspensionId = (prepared.payload as { suspensionId: string }).suspensionId;
|
||||
|
||||
await expect(invoke("gateway.suspend.status", { suspensionId })).resolves.toMatchObject({
|
||||
ok: true,
|
||||
payload: { status: "ready" },
|
||||
});
|
||||
await expect(invoke("gateway.suspend.resume", { suspensionId })).resolves.toMatchObject({
|
||||
ok: true,
|
||||
payload: { status: "running", resumed: true },
|
||||
});
|
||||
expect(cron.pauseScheduling).toHaveBeenCalledOnce();
|
||||
expect(cron.resumeScheduling).toHaveBeenCalledOnce();
|
||||
expect(getActiveGatewayRootWorkCount()).toBe(0);
|
||||
});
|
||||
});
|
||||
|
||||
describe("plugin upgrade suspension admission", () => {
|
||||
it("keeps an in-flight upgrade visible to suspension preparation", async () => {
|
||||
const started = deferred();
|
||||
const finish = deferred();
|
||||
const handler = createUpgradeHandler([
|
||||
createRoute({
|
||||
handler: () => false,
|
||||
handleUpgrade: async () => {
|
||||
started.resolve();
|
||||
await finish.promise;
|
||||
return true;
|
||||
},
|
||||
}),
|
||||
]);
|
||||
const socket = createMockUpgradeSocket();
|
||||
const pending = handler({ url: ROUTE_PATH } as IncomingMessage, socket, Buffer.alloc(0));
|
||||
await started.promise;
|
||||
|
||||
expect(getActiveGatewayRootWorkCount()).toBe(1);
|
||||
expect(prepareWithRootOnly("plugin-upgrade-active")).toMatchObject({
|
||||
status: "busy",
|
||||
reason: "active-work",
|
||||
activeCount: 1,
|
||||
blockers: [expect.objectContaining({ kind: "root-request", count: 1 })],
|
||||
});
|
||||
|
||||
finish.resolve();
|
||||
await expect(pending).resolves.toBe(true);
|
||||
expect(getActiveGatewayRootWorkCount()).toBe(0);
|
||||
});
|
||||
|
||||
it("rejects a new upgrade with HTTP 503 after admission closes", async () => {
|
||||
const upgrade = vi.fn(() => true);
|
||||
const handler = createUpgradeHandler([
|
||||
createRoute({ handler: () => false, handleUpgrade: upgrade }),
|
||||
]);
|
||||
const suspension = tryBeginGatewaySuspendAdmission(() => {});
|
||||
expect(suspension?.commit()).toBe(true);
|
||||
const socket = createMockUpgradeSocket();
|
||||
|
||||
await expect(
|
||||
handler({ url: ROUTE_PATH } as IncomingMessage, socket, Buffer.alloc(0)),
|
||||
).resolves.toBe(true);
|
||||
|
||||
expect(upgrade).not.toHaveBeenCalled();
|
||||
expect(socket.destroyed).toBe(true);
|
||||
expect(socket.chunks.join("")).toContain("HTTP/1.1 503 Service Unavailable");
|
||||
expect(socket.chunks.join("")).toContain("Gateway websocket admission closed");
|
||||
expect(getActiveGatewayRootWorkCount()).toBe(0);
|
||||
expect(suspension?.release()).toBe(true);
|
||||
});
|
||||
|
||||
it("releases upgrade admission after fallthrough and failure", async () => {
|
||||
const fallthrough = vi.fn(() => false);
|
||||
const handled = vi.fn(() => true);
|
||||
const fallthroughHandler = createUpgradeHandler([
|
||||
createRoute({ handler: () => false, handleUpgrade: fallthrough }),
|
||||
createRoute({
|
||||
path: "/plugin",
|
||||
match: "prefix",
|
||||
handler: () => false,
|
||||
handleUpgrade: handled,
|
||||
}),
|
||||
]);
|
||||
const fallthroughSocket = createMockUpgradeSocket();
|
||||
|
||||
await expect(
|
||||
fallthroughHandler(
|
||||
{ url: ROUTE_PATH } as IncomingMessage,
|
||||
fallthroughSocket,
|
||||
Buffer.alloc(0),
|
||||
),
|
||||
).resolves.toBe(true);
|
||||
expect(fallthrough).toHaveBeenCalledOnce();
|
||||
expect(handled).toHaveBeenCalledOnce();
|
||||
expect(fallthroughSocket.destroyed).toBe(false);
|
||||
expect(getActiveGatewayRootWorkCount()).toBe(0);
|
||||
|
||||
const failingHandler = createUpgradeHandler([
|
||||
createRoute({
|
||||
handler: () => false,
|
||||
handleUpgrade: () => {
|
||||
throw new Error("upgrade failed");
|
||||
},
|
||||
}),
|
||||
]);
|
||||
const failureSocket = createMockUpgradeSocket();
|
||||
await expect(
|
||||
failingHandler({ url: ROUTE_PATH } as IncomingMessage, failureSocket, Buffer.alloc(0)),
|
||||
).resolves.toBe(true);
|
||||
expect(failureSocket.destroyed).toBe(true);
|
||||
expect(getActiveGatewayRootWorkCount()).toBe(0);
|
||||
});
|
||||
});
|
||||
@@ -11,6 +11,10 @@ import type { PluginHttpRouteRegistration, PluginRegistry } from "../../plugins/
|
||||
import { withPluginRuntimeGatewayRequestScope } from "../../plugins/runtime/gateway-request-scope.js";
|
||||
import type { AuthorizedGatewayHttpRequest } from "../http-utils.js";
|
||||
import type { GatewayRequestContext, GatewayRequestOptions } from "../server-methods/types.js";
|
||||
import {
|
||||
runWithGatewayHttpWorkAdmission,
|
||||
runWithGatewayUpgradeWorkAdmission,
|
||||
} from "./http-work-admission.js";
|
||||
import { resolvePluginRouteRuntimeOperatorScopes } from "./plugin-route-runtime-scopes.js";
|
||||
import {
|
||||
resolvePluginRoutePathContext,
|
||||
@@ -90,6 +94,16 @@ function getMissingPluginRouteRuntimeContext(
|
||||
return context.gatewayRequestOperatorScopes === undefined ? "caller scope context" : undefined;
|
||||
}
|
||||
|
||||
function canRunPluginHttpRouteWithoutAdmission(route: PluginHttpRouteRegistration): boolean {
|
||||
// The manifest entitlement is plugin-wide; require the route-specific trusted operator
|
||||
// surface so an ordinary sibling cannot start work after suspension reports ready.
|
||||
return (
|
||||
route.auth === "gateway" &&
|
||||
route.gatewayRuntimeScopeSurface === "trusted-operator" &&
|
||||
route.gatewayMethodDispatchAllowed === true
|
||||
);
|
||||
}
|
||||
|
||||
function createPluginRouteRuntimeScope(params: {
|
||||
route: PluginHttpRouteRegistration;
|
||||
req: IncomingMessage;
|
||||
@@ -191,18 +205,24 @@ export function createGatewayPluginRequestHandler(params: {
|
||||
|
||||
for (const route of matchedRoutes) {
|
||||
try {
|
||||
const handled = await withPluginRuntimeGatewayRequestScope(
|
||||
createPluginRouteRuntimeScope({
|
||||
route,
|
||||
req,
|
||||
gatewayRequestContext,
|
||||
gatewayRequestAuth,
|
||||
gatewayRequestOperatorScopes,
|
||||
gatewayRequestClientIp: dispatchContext?.gatewayRequestClientIp,
|
||||
}),
|
||||
async () => route.handler(req, res),
|
||||
);
|
||||
if (handled !== false) {
|
||||
const runRoute = async () =>
|
||||
(await withPluginRuntimeGatewayRequestScope(
|
||||
createPluginRouteRuntimeScope({
|
||||
route,
|
||||
req,
|
||||
gatewayRequestContext,
|
||||
gatewayRequestAuth,
|
||||
gatewayRequestOperatorScopes,
|
||||
gatewayRequestClientIp: dispatchContext?.gatewayRequestClientIp,
|
||||
}),
|
||||
async () => route.handler(req, res),
|
||||
)) !== false;
|
||||
// Entitled trusted-operator routes delegate substantive work through Gateway dispatch.
|
||||
// An outer root would make gateway.suspend.prepare nested and permanently unreachable.
|
||||
const handled = canRunPluginHttpRouteWithoutAdmission(route)
|
||||
? await runRoute()
|
||||
: await runWithGatewayHttpWorkAdmission(res, runRoute);
|
||||
if (handled) {
|
||||
return true;
|
||||
}
|
||||
} catch (err) {
|
||||
@@ -268,18 +288,22 @@ export function createGatewayPluginUpgradeHandler(params: {
|
||||
|
||||
for (const route of matchedRoutes) {
|
||||
try {
|
||||
const handled = await withPluginRuntimeGatewayRequestScope(
|
||||
createPluginRouteRuntimeScope({
|
||||
route,
|
||||
req,
|
||||
gatewayRequestContext,
|
||||
gatewayRequestAuth,
|
||||
gatewayRequestOperatorScopes,
|
||||
gatewayRequestClientIp: dispatchContext?.gatewayRequestClientIp,
|
||||
}),
|
||||
async () => route.handleUpgrade?.(req, socket, head),
|
||||
const handled = await runWithGatewayUpgradeWorkAdmission(
|
||||
socket,
|
||||
async () =>
|
||||
(await withPluginRuntimeGatewayRequestScope(
|
||||
createPluginRouteRuntimeScope({
|
||||
route,
|
||||
req,
|
||||
gatewayRequestContext,
|
||||
gatewayRequestAuth,
|
||||
gatewayRequestOperatorScopes,
|
||||
gatewayRequestClientIp: dispatchContext?.gatewayRequestClientIp,
|
||||
}),
|
||||
async () => route.handleUpgrade?.(req, socket, head),
|
||||
)) !== false,
|
||||
);
|
||||
if (handled !== false) {
|
||||
if (handled) {
|
||||
return true;
|
||||
}
|
||||
} catch (err) {
|
||||
|
||||
Reference in New Issue
Block a user