mirror of
https://github.com/openclaw/openclaw.git
synced 2026-08-25 03:45:46 -06:00
ee6e0251b4
* fix(scripts): bound tsgo runs with the managed-command watchdog run-tsgo bypassed the repo's managed-command seam and called spawnSync directly, so a wedged tsgo blocked its caller indefinitely: no timeout, no process-group cleanup, and no SIGKILL escalation. Observed in the wild as a tsgo holding 2.85 GB for 90+ minutes on 41s of total CPU with RSS frozen to the byte, ignoring SIGTERM, with its wrapper reparented to init. Because shouldReclaimLock() treats a live PID as a valid lock owner, that orphan also held the heavy-check lock until every other invocation hit the 10-minute lock timeout and threw. Route the run through runManagedCommand, which already owns process-group termination and SIGKILL escalation on timeout, and bound it with OPENCLAW_TSGO_TIMEOUT_MS (default 45m) through the shared readPositiveEnvInt helper, mirroring OPENCLAW_CLI_STARTUP_BUILD_TIMEOUT_MS in ensure-cli-startup-build.mts. * fix(scripts): saturate the tsgo watchdog at Node's timer ceiling An OPENCLAW_TSGO_TIMEOUT_MS above 2147483647 reached setTimeout unchanged, where Node collapses it to a 1ms delay, so raising the override killed healthy typechecks immediately instead of loosening the bound. * fix(scripts): make the tsgo watchdog opt-in and stop the harness leaking ClawSweeper review on6ba02c9d0araised two findings. [P1] The 45-minute default applied an unproven deadline to every tsgo invocation. No supported duration contract covers every host and project, and CI already bounds its own tsgo jobs at 15-20 minutes, so the default could only ever fire outside CI where it was least validated. Drop it: an unset OPENCLAW_TSGO_TIMEOUT_MS keeps the pre-existing unbounded wait, so no existing run changes behavior, and operators opt in per host. Documented in docs/help/testing.md beside the sibling Vitest watchdog. [P2] The regression harness could leak its wedged child. The fake compiler ignores SIGTERM by design, so a pre-fix or otherwise failing run left the tree running after spawnSync gave up. Bound the fixture's loop as a backstop. * fix(scripts): set the tsgo watchdog default from measured lane duration ClawSweeper onc7a699ee82reversed its earlier guidance: the opt-in default adopted last iteration "deliberately preserves the indefinite tsgo hang that this PR is meant to fix". Its objection was never that a default existed, only that 45 minutes was unmeasured. Measured instead of guessed: hosted tsgo lanes (check-test-types, and its core stripes) complete in 1-2 minutes across recent successful main runs, against CI job caps of 15-20 minutes. 30 minutes is 15-30x the observed duration, leaves room for a far slower local host, and still bounds the 90-minute and multi-hour wedges that motivated this PR. OPENCLAW_TSGO_TIMEOUT_MS remains the documented override for hosts that need longer. * test(scripts): reap the wedged fake tsgo tree on the harness outer timeout ClawSweeper on0d9f3604e8flagged that the harness can still leave a detached pre-fix process tree alive after its outer timeout. The bounded fixture loop added earlier only capped the leak; it did not terminate the tree. spawnSync's killSignal reaches the direct child only. runManagedCommand spawns the compiler detached into its own process group, so the fake tsgo is a grandchild that never receives that signal. The fixture now records its pid and the harness reaps that group in a finally, with the bounded loop kept as a last-resort backstop. Verified: pid file written with the live pid, and killing that group terminates the tree; focused suite 16/16 with no surviving fake-tsgo processes. * fix(scripts): harden the tsgo watchdog after two-phase code review Review fixes on top of the watchdog change, from one native pass and six cold passes: - A rejected OPENCLAW_TSGO_TIMEOUT_MS escaped main() as a raw module rejection. It now reports one actionable line and exits 1. Strict validation was kept rather than switching to coercion, so a typo cannot silently fall back to the 30-minute default. - The rejection message named a numeric range while the parser enforces plain decimal digits, so 1e5 and 007 were refused by a message saying they qualified. It now names the real format and states that the watchdog cannot be disabled. - The timer ceiling is declared locally rather than imported from packages/. A static import there resolves before the sparse-checkout guard runs, which turned a clean sparse skip into ERR_MODULE_NOT_FOUND and flipped check-changed's typecheck lane from exit 0 to exit 1. - The wedge test asserted the kill message but not the outcome; it now captures the wedged pid and asserts the process group is gone. - Three near-duplicate "not killed" cases are table-driven. - Doc bullet corrected: values ABOVE the ceiling saturate at it, and the rejected-value list now includes non-decimal input. Deferred follow-up, not fixed here: scripts/lib/tsx-cli-shim.mjs shares a 5000ms force-kill delay with managed-child-process, so Ctrl-C can still orphan a wedged compiler about one run in three. Measured base 4/4 orphaned versus 4/10 here, so this change improves it; the fix is out of diff and shared with four other wrappers. * fix(scripts): close tsgo signal cleanup race * fix(scripts): make tsgo watchdog opt-in --------- Co-authored-by: ClawSweeper <steipete+clawsweeper@gmail.com>
413 lines
15 KiB
TypeScript
413 lines
15 KiB
TypeScript
// Run Tsgo tests cover run tsgo script behavior.
|
|
import { spawn, spawnSync } from "node:child_process";
|
|
import fs from "node:fs";
|
|
import path from "node:path";
|
|
import { describe, expect, it } from "vitest";
|
|
import {
|
|
createSparseTsgoSkipEnv,
|
|
getSparseTsgoGuardError,
|
|
shouldSkipSparseTsgoGuardError,
|
|
} from "../../scripts/lib/tsgo-sparse-guard.mts";
|
|
import { resolveTsgoTimeoutMs } from "../../scripts/run-tsgo.mts";
|
|
import { waitForChildClose, waitForDead, waitForPidFile } from "../helpers/process-wait.js";
|
|
import { createScriptTestHarness } from "./test-helpers.js";
|
|
|
|
const { createTempDir } = createScriptTestHarness();
|
|
|
|
describe("run-tsgo sparse guard", () => {
|
|
it("ends sparse-checkout failures with the stable failure trailer", () => {
|
|
const cwd = createTempDir("openclaw-run-tsgo-");
|
|
spawnSync("git", ["init", "-q"], { cwd });
|
|
spawnSync("git", ["config", "core.sparseCheckout", "true"], { cwd });
|
|
|
|
const result = spawnSync(
|
|
process.execPath,
|
|
[path.resolve("scripts/run-tsgo.mjs"), "-p", "test/tsconfig/tsconfig.core.test.json"],
|
|
{
|
|
cwd,
|
|
encoding: "utf8",
|
|
env: process.env,
|
|
},
|
|
);
|
|
|
|
expect(result.status).toBe(1);
|
|
expect(result.stderr.trim().split("\n").at(-1)).toBe("[tsgo] FAILED (exit 1)");
|
|
});
|
|
|
|
it("ignores non-core projects", () => {
|
|
const cwd = createTempDir("openclaw-run-tsgo-");
|
|
|
|
expect(
|
|
getSparseTsgoGuardError(["-p", "tsconfig.extensions.json"], {
|
|
cwd,
|
|
isSparseCheckoutEnabled: () => true,
|
|
}),
|
|
).toBeNull();
|
|
});
|
|
|
|
it("ignores full worktrees", () => {
|
|
const cwd = createTempDir("openclaw-run-tsgo-");
|
|
|
|
expect(
|
|
getSparseTsgoGuardError(["-p", "test/tsconfig/tsconfig.core.test.json"], {
|
|
cwd,
|
|
isSparseCheckoutEnabled: () => false,
|
|
}),
|
|
).toBeNull();
|
|
});
|
|
|
|
it("ignores metadata-only commands", () => {
|
|
const cwd = createTempDir("openclaw-run-tsgo-");
|
|
|
|
expect(
|
|
getSparseTsgoGuardError(["-p", "test/tsconfig/tsconfig.core.test.json", "--showConfig"], {
|
|
cwd,
|
|
isSparseCheckoutEnabled: () => true,
|
|
}),
|
|
).toBeNull();
|
|
});
|
|
|
|
it("ignores sparse worktrees when the required files are present", () => {
|
|
const cwd = createTempDir("openclaw-run-tsgo-");
|
|
const requiredPaths = [
|
|
"packages/plugin-package-contract/src/index.ts",
|
|
"ui/config/control-ui-chunking.ts",
|
|
"ui/src/i18n/lib/registry.ts",
|
|
"ui/src/i18n/lib/types.ts",
|
|
"ui/src/app/settings.ts",
|
|
"ui/src/api/gateway.ts",
|
|
];
|
|
|
|
for (const relativePath of requiredPaths) {
|
|
const absolutePath = path.join(cwd, relativePath);
|
|
const dir = path.dirname(absolutePath);
|
|
fs.mkdirSync(dir, { recursive: true });
|
|
fs.writeFileSync(absolutePath, "", "utf8");
|
|
}
|
|
|
|
expect(
|
|
getSparseTsgoGuardError(["-p", "test/tsconfig/tsconfig.core.test.other.json"], {
|
|
cwd,
|
|
isSparseCheckoutEnabled: () => true,
|
|
sparseCheckoutPatterns: ["/packages/", "/ui/config/", "/ui/src/"],
|
|
}),
|
|
).toBeNull();
|
|
});
|
|
|
|
it("rejects package-test sparse worktrees missing inherited declaration roots", () => {
|
|
const cwd = createTempDir("openclaw-run-tsgo-");
|
|
|
|
expect(
|
|
getSparseTsgoGuardError(["-p", "test/tsconfig/tsconfig.test.packages.json"], {
|
|
cwd,
|
|
fileExists: () => true,
|
|
isSparseCheckoutEnabled: () => true,
|
|
sparseCheckoutPatterns: ["/packages/"],
|
|
}),
|
|
).toMatchInlineSnapshot(`
|
|
"tsconfig.test.packages.json cannot be typechecked from this sparse checkout because tracked project inputs are missing or only partially included:
|
|
- src
|
|
- ui/src
|
|
Expand this worktree's sparse checkout to include those paths, or rerun in a full worktree."
|
|
`);
|
|
});
|
|
|
|
it("rejects declaration-shard sparse worktrees missing inherited roots", () => {
|
|
const cwd = createTempDir("openclaw-run-tsgo-");
|
|
|
|
expect(
|
|
getSparseTsgoGuardError(["-p", "test/tsconfig/tsconfig.test.extension-declarations.json"], {
|
|
cwd,
|
|
fileExists: () => true,
|
|
isSparseCheckoutEnabled: () => true,
|
|
sparseCheckoutPatterns: ["/extensions/"],
|
|
}),
|
|
).toMatchInlineSnapshot(`
|
|
"tsconfig.test.extension-declarations.json cannot be typechecked from this sparse checkout because tracked project inputs are missing or only partially included:
|
|
- src
|
|
- ui/src
|
|
Expand this worktree's sparse checkout to include those paths, or rerun in a full worktree."
|
|
`);
|
|
});
|
|
|
|
it("rejects sparse core worktrees that include only selected ui and package files", () => {
|
|
const cwd = createTempDir("openclaw-run-tsgo-");
|
|
const requiredPaths = [
|
|
"packages/plugin-package-contract/src/index.ts",
|
|
"ui/config/control-ui-chunking.ts",
|
|
"ui/src/i18n/lib/registry.ts",
|
|
"ui/src/i18n/lib/types.ts",
|
|
"ui/src/app/settings.ts",
|
|
"ui/src/api/gateway.ts",
|
|
];
|
|
|
|
for (const relativePath of requiredPaths) {
|
|
const absolutePath = path.join(cwd, relativePath);
|
|
fs.mkdirSync(path.dirname(absolutePath), { recursive: true });
|
|
fs.writeFileSync(absolutePath, "", "utf8");
|
|
}
|
|
|
|
expect(
|
|
getSparseTsgoGuardError(["-p", "test/tsconfig/tsconfig.core.test.json"], {
|
|
cwd,
|
|
isSparseCheckoutEnabled: () => true,
|
|
sparseCheckoutPatterns: [
|
|
"/packages/plugin-package-contract/src/index.ts",
|
|
"/ui/config/control-ui-chunking.ts",
|
|
"/ui/src/i18n/lib/registry.ts",
|
|
"/ui/src/i18n/lib/types.ts",
|
|
"/ui/src/app/settings.ts",
|
|
"/ui/src/api/gateway.ts",
|
|
],
|
|
}),
|
|
).toMatchInlineSnapshot(`
|
|
"tsconfig.core.test.json cannot be typechecked from this sparse checkout because tracked project inputs are missing or only partially included:
|
|
- packages
|
|
- ui/config
|
|
- ui/src
|
|
Expand this worktree's sparse checkout to include those paths, or rerun in a full worktree."
|
|
`);
|
|
});
|
|
|
|
it("returns a helpful message for sparse UI worktrees missing transitive project files", () => {
|
|
const cwd = createTempDir("openclaw-run-tsgo-");
|
|
const uiToolDisplay = path.join(cwd, "ui/src/lib/chat/tool-display.ts");
|
|
fs.mkdirSync(path.dirname(uiToolDisplay), { recursive: true });
|
|
fs.writeFileSync(uiToolDisplay, "", "utf8");
|
|
|
|
expect(
|
|
getSparseTsgoGuardError(["-p", "tsconfig.ui.json"], {
|
|
cwd,
|
|
isSparseCheckoutEnabled: () => true,
|
|
}),
|
|
).toMatchInlineSnapshot(`
|
|
"tsconfig.ui.json cannot be typechecked from this sparse checkout because tracked project inputs are missing or only partially included:
|
|
- apps/shared/OpenClawKit/Sources/OpenClawKit/Resources/tool-display.json
|
|
Expand this worktree's sparse checkout to include those paths, or rerun in a full worktree."
|
|
`);
|
|
});
|
|
|
|
it("rejects sparse UI worktrees missing the transitive src root", () => {
|
|
const cwd = createTempDir("openclaw-run-tsgo-");
|
|
|
|
expect(
|
|
getSparseTsgoGuardError(["-p", "tsconfig.ui.json"], {
|
|
cwd,
|
|
fileExists: () => true,
|
|
isSparseCheckoutEnabled: () => true,
|
|
sparseCheckoutPatterns: ["/packages/", "/ui/config/", "/ui/src/"],
|
|
}),
|
|
).toMatchInlineSnapshot(`
|
|
"tsconfig.ui.json cannot be typechecked from this sparse checkout because tracked project inputs are missing or only partially included:
|
|
- src
|
|
Expand this worktree's sparse checkout to include those paths, or rerun in a full worktree."
|
|
`);
|
|
});
|
|
|
|
it("returns a helpful message for sparse core-test worktrees missing ui and packages files", () => {
|
|
const cwd = createTempDir("openclaw-run-tsgo-");
|
|
|
|
expect(
|
|
getSparseTsgoGuardError(["-p", "test/tsconfig/tsconfig.core.test.json"], {
|
|
cwd,
|
|
isSparseCheckoutEnabled: () => true,
|
|
}),
|
|
).toMatchInlineSnapshot(`
|
|
"tsconfig.core.test.json cannot be typechecked from this sparse checkout because tracked project inputs are missing or only partially included:
|
|
- packages/plugin-package-contract/src/index.ts
|
|
- ui/config/control-ui-chunking.ts
|
|
- ui/src/api/gateway.ts
|
|
- ui/src/app/settings.ts
|
|
- ui/src/i18n/lib/registry.ts
|
|
- ui/src/i18n/lib/types.ts
|
|
Expand this worktree's sparse checkout to include those paths, or rerun in a full worktree."
|
|
`);
|
|
});
|
|
|
|
it("recognizes the check:changed sparse-skip env", () => {
|
|
expect(shouldSkipSparseTsgoGuardError({ OPENCLAW_TSGO_SPARSE_SKIP: "1" })).toBe(true);
|
|
expect(shouldSkipSparseTsgoGuardError({ OPENCLAW_TSGO_SPARSE_SKIP: "true" })).toBe(true);
|
|
expect(shouldSkipSparseTsgoGuardError({ OPENCLAW_TSGO_SPARSE_SKIP: "0" })).toBe(false);
|
|
expect(createSparseTsgoSkipEnv({ PATH: "/usr/bin" })).toStrictEqual({
|
|
PATH: "/usr/bin",
|
|
OPENCLAW_TSGO_SPARSE_SKIP: "1",
|
|
});
|
|
});
|
|
});
|
|
|
|
describe.skipIf(process.platform === "win32")("run-tsgo watchdog", () => {
|
|
it("keeps the watchdog opt-in", () => {
|
|
expect(resolveTsgoTimeoutMs({})).toBeUndefined();
|
|
expect(resolveTsgoTimeoutMs({ OPENCLAW_TSGO_TIMEOUT_MS: " " })).toBeUndefined();
|
|
expect(resolveTsgoTimeoutMs({ OPENCLAW_TSGO_TIMEOUT_MS: "30000" })).toBe(30_000);
|
|
});
|
|
|
|
function writeFakeTsgo(cwd: string, body: string) {
|
|
const binDir = path.join(cwd, "node_modules", ".bin");
|
|
fs.mkdirSync(binDir, { recursive: true });
|
|
const fakeTsgo = path.join(binDir, "tsgo");
|
|
fs.writeFileSync(fakeTsgo, body, "utf8");
|
|
fs.chmodSync(fakeTsgo, 0o755);
|
|
}
|
|
|
|
// The fake compiler is a grandchild in its own process group, so spawnSync's
|
|
// killSignal never reaches it. Its recorded pid is the only handle the harness
|
|
// has to tear the tree down when the outer timeout fires on a pre-fix run.
|
|
function readFakeTsgoPid(cwd: string) {
|
|
const pidFile = path.join(cwd, "fake-tsgo.pid");
|
|
if (!fs.existsSync(pidFile)) {
|
|
return undefined;
|
|
}
|
|
const pid = Number.parseInt(fs.readFileSync(pidFile, "utf8").trim(), 10);
|
|
return Number.isInteger(pid) && pid > 1 ? pid : undefined;
|
|
}
|
|
|
|
function reapFakeTsgo(cwd: string) {
|
|
const pid = readFakeTsgoPid(cwd);
|
|
if (pid === undefined) {
|
|
return;
|
|
}
|
|
for (const target of [-pid, pid]) {
|
|
try {
|
|
process.kill(target, "SIGKILL");
|
|
} catch {
|
|
// Already reaped by the watchdog under test.
|
|
}
|
|
}
|
|
}
|
|
|
|
function runFakeTsgo(
|
|
cwd: string,
|
|
timeoutMs: string | undefined,
|
|
onBeforeReap?: (pid: number | undefined) => void,
|
|
) {
|
|
const { OPENCLAW_TSGO_TIMEOUT_MS: _unset, ...baseEnv } = process.env;
|
|
try {
|
|
return spawnSync(
|
|
process.execPath,
|
|
[path.resolve("scripts/run-tsgo.mjs"), "-p", "tsconfig.extensions.json"],
|
|
{
|
|
cwd,
|
|
encoding: "utf8",
|
|
env:
|
|
timeoutMs === undefined ? baseEnv : { ...baseEnv, OPENCLAW_TSGO_TIMEOUT_MS: timeoutMs },
|
|
// spawnSync blocks this thread, so vitest's own per-test budget can never
|
|
// fire; a regression here would hang the worker instead of failing.
|
|
timeout: 25_000,
|
|
killSignal: "SIGKILL",
|
|
},
|
|
);
|
|
} finally {
|
|
onBeforeReap?.(readFakeTsgoPid(cwd));
|
|
reapFakeTsgo(cwd);
|
|
}
|
|
}
|
|
|
|
it.each([{ bound: "0" }, { bound: "abc" }])(
|
|
"explains a rejected OPENCLAW_TSGO_TIMEOUT_MS of $bound instead of crashing",
|
|
({ bound }) => {
|
|
const cwd = createTempDir("openclaw-run-tsgo-watchdog-");
|
|
writeFakeTsgo(cwd, "#!/bin/sh\nexit 0\n");
|
|
|
|
const result = runFakeTsgo(cwd, bound);
|
|
|
|
expect(result.status).toBe(1);
|
|
expect(result.stderr).toContain("must be plain decimal digits");
|
|
expect(result.stderr).toContain("Unset it to disable the watchdog");
|
|
expect(result.stderr).not.toContain("at readPositiveEnvInt");
|
|
expect(result.stderr.trim().split("\n").at(-1)).toBe("[tsgo] FAILED (exit 1)");
|
|
},
|
|
30_000,
|
|
);
|
|
|
|
it("kills a wedged tsgo that ignores SIGTERM instead of blocking its caller forever", () => {
|
|
const cwd = createTempDir("openclaw-run-tsgo-watchdog-");
|
|
// Mirrors the observed wedge: the checker refuses SIGTERM and never reports,
|
|
// so only a process-group SIGKILL frees the caller. It records its pid so the
|
|
// harness can reap the tree, and self-exits as a last-resort backstop.
|
|
writeFakeTsgo(
|
|
cwd,
|
|
'#!/bin/sh\necho $$ > "$(dirname "$0")/../../fake-tsgo.pid"\ntrap \'\' TERM\ni=0\nwhile [ $i -lt 60 ]; do sleep 1; i=$((i+1)); done\n',
|
|
);
|
|
|
|
const observedBeforeReap = {
|
|
error: undefined as unknown,
|
|
pid: undefined as number | undefined,
|
|
};
|
|
const result = runFakeTsgo(cwd, "2000", (pid) => {
|
|
observedBeforeReap.pid = pid;
|
|
if (pid === undefined) {
|
|
return;
|
|
}
|
|
try {
|
|
process.kill(pid, 0);
|
|
} catch (error) {
|
|
observedBeforeReap.error = error;
|
|
}
|
|
});
|
|
|
|
expect(result.status).toBe(1);
|
|
expect(result.stderr).toContain("killed the tsgo process tree");
|
|
// Printing the message is not the contract; the tree actually being gone is.
|
|
expect(observedBeforeReap.pid).toBeDefined();
|
|
expect(observedBeforeReap.error).toMatchObject({ code: "ESRCH" });
|
|
expect(result.stderr.trim().split("\n").at(-1)).toBe("[tsgo] FAILED (exit 1)");
|
|
}, 30_000);
|
|
|
|
it("lets the inner supervisor reap a wedged compiler before the wrapper exits on SIGTERM", async () => {
|
|
const cwd = createTempDir("openclaw-run-tsgo-signal-");
|
|
const pidFile = path.join(cwd, "fake-tsgo.pid");
|
|
fs.writeFileSync(path.join(cwd, "tsconfig.extensions.json"), "{}\n");
|
|
writeFakeTsgo(
|
|
cwd,
|
|
'#!/bin/sh\necho $$ > "$(dirname "$0")/../../fake-tsgo.pid"\ntrap \'\' TERM HUP INT\nwhile true; do sleep 1; done\n',
|
|
);
|
|
const wrapper = spawn(
|
|
process.execPath,
|
|
[path.resolve("scripts/run-tsgo.mjs"), "-p", "tsconfig.extensions.json"],
|
|
{ cwd, stdio: "ignore" },
|
|
);
|
|
|
|
try {
|
|
const compilerPid = await waitForPidFile(pidFile, 10_000);
|
|
wrapper.kill("SIGTERM");
|
|
|
|
await expect(waitForChildClose(wrapper, 15_000)).resolves.toEqual({
|
|
code: 143,
|
|
signal: null,
|
|
});
|
|
await expect(waitForDead(compilerPid, 2_000)).resolves.toBeUndefined();
|
|
} finally {
|
|
if (wrapper.exitCode === null && wrapper.signalCode === null) {
|
|
wrapper.kill("SIGKILL");
|
|
}
|
|
reapFakeTsgo(cwd);
|
|
}
|
|
}, 20_000);
|
|
|
|
// Every bound that must leave a completing compiler alone. The ceiling case is the
|
|
// regression that matters: without saturation Node collapses the delay to 1ms and
|
|
// would kill this sleeping child immediately.
|
|
it.each([
|
|
{ bound: undefined, name: "the disabled watchdog", body: "#!/bin/sh\nsleep 2\nexit 0\n" },
|
|
{ bound: "30000", name: "an explicit bound", body: "#!/bin/sh\nexit 0\n" },
|
|
{
|
|
bound: "2147483648",
|
|
name: "an override past Node's timer ceiling",
|
|
body: "#!/bin/sh\nsleep 1\nexit 0\n",
|
|
},
|
|
])(
|
|
"leaves a completing tsgo alone under $name",
|
|
({ bound, body }) => {
|
|
const cwd = createTempDir("openclaw-run-tsgo-watchdog-");
|
|
writeFakeTsgo(cwd, body);
|
|
|
|
const result = runFakeTsgo(cwd, bound);
|
|
|
|
expect(result.status).toBe(0);
|
|
expect(result.stderr).not.toContain("killed the tsgo process tree");
|
|
},
|
|
30_000,
|
|
);
|
|
});
|