Commit Graph

32 Commits

Author SHA1 Message Date
Ayaan Zaidi 10c774cf38 feat(mantis): keep QA leases alive and stage fixture plugins (#127804)
* feat(mantis): keep leases alive and stage fixture plugins

* refactor(mantis): deduplicate lease keepalive guard

* fix(mantis): retry keepalive on unexpected broker errors

* fix(mantis): fence active proof on terminal lease loss
2026-08-22 14:09:28 +05:30
Ayaan Zaidi 460f5b6c61 perf(mantis): parallelize proof builds and warm caches (#127798)
* perf(mantis): parallelize proof builds and warm caches

* fix(mantis): size build image for the copied pnpm store

* chore(ui): refresh startup JS gzip baseline after streamed-markdown perf work

Identical source measures 345034-345058 B across builds while the
committed baseline left only a 9 B margin under the 512 B ratchet
tolerance, so build-artifacts flips on gzip nondeterminism (green on
main run 32559609413, red on PR run 32559442295, red locally).
Regenerated with scripts/check-control-ui-performance.mts
--update-baseline; the 350 KiB hard ceiling still bounds creep.
2026-08-22 07:53:36 +00:00
Ayaan Zaidi 0933263f76 fix(mantis): publish proof comments on auto-triggered runs (#127787)
## What Problem This Solves

Auto-triggered Mantis proof runs (label/`clawsweeper_label` and other non-comment request sources) end with no PR comment at all. The durable evidence publisher runs with `--create-missing false` and only edits an existing marker comment, but the inline status comment carrying that marker was only created when `request_source == 'issue_comment'`. Label-triggered runs therefore published nothing and logged the misleading "Skipped stale Mantis QA evidence comment because its status is no longer active" — observed on PR #127735. This is the silent-failure class: a Mantis run completes and the PR shows no visible outcome.

## Why This Change Was Made

- `.github/workflows/mantis-telegram-desktop-proof.yml`: the status ack comment (👀 + active-job link + run-scoped marker) is now created for every request source that resolves to a PR (`pr_number != ''`), not only `issue_comment`. The 👀 *reaction* stays `issue_comment`-only (it lives in `mantis-resolve-request.yml`, untouched — there is no triggering comment to react to on label runs).
- The start-failure fallback comment and the existing-artifact republish path now use the same run-scoped marker `<!-- mantis-telegram-desktop-proof:${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT} -->` as the status comment and the main publisher, so every publisher edits the single run-owned comment (ack → progress → final proof; no comment spam). The republish path gets an explicit `--create-missing false` to match. The design invariant that makes `false` safe: the fallback status-comment step is not `continue-on-error`, so a run in which no marker comment could be created fails `resolve_request` and never reaches publish.
- `scripts/mantis/publish-pr-evidence.mjs`: the two skip cases now log honestly — "no existing comment found" vs "could not update existing comment" — instead of one misleading stale-status message.

## User Impact

Operators triggering Mantis via labels (ClawSweeper flows) now get the same single evolving PR comment as comment-triggered runs: an immediate 👀 ack with the running job link, edited in place into the final proof evidence. No more runs that finish invisibly.

## Evidence

- Focused suite: `node scripts/run-vitest.mjs test/scripts/mantis-telegram-desktop-proof-workflow.test.ts` — 28/28, including new assertions that the status/failure comment gates use `pr_number != ''` (and not `request_source`) and that both publishers pass the run-scoped marker with `--create-missing false`.
- `node scripts/check-changed.mjs -- <touched files>` green; `git diff --check` clean.
- Marker alignment verified across all five sites in the workflow (status comment, prior-attempt cleanup regex, fallback comment, failure report, both publisher invocations): all use `mantis-telegram-desktop-proof:${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}`.
- Live-run proof of the label-triggered path requires a merged workflow (GitHub runs the workflow from the default branch for these triggers), so the first post-merge label-triggered Mantis run is the live verification; stated here as the known evidence gap.

Production LOC delta: −4 (workflow/tooling); tests +13.
2026-08-22 11:10:11 +05:30
Ayaan Zaidi 10e0e690df feat(mantis): programmable proof boundaries with recipe library (#127782)
Makes the Mantis Telegram Desktop proof agent programmable at its trust boundaries: declarative Bot API fault rules (drop/status per method), per-request scripted mock-provider responses, observe-until predicates (post-cursor events/text, cumulative provider count), and bounded proxy-side recording of outbound Bot API requests as trusted lane facts. Adds a reusable recipe library under .github/codex/prompts/mantis-recipes/ and raises the proof agent's reasoning effort to high.

Security: the SUT container shadows proxy-control with an inaccessible tmpfs so candidate PR code sharing the mantis-sut uid cannot read or rewrite the proxy's recorded evidence; unmount is blocked by cap-drop/no-new-privileges. Proof doctrine now treats proxy-recorded Bot API facts as trusted comparison evidence and provider request logs as diagnostics.

Follow-up named in PR: move the mock OpenAI server out of the SUT container so provider request facts also become candidate-tamper-proof.
2026-08-22 11:06:06 +05:30
Ayaan Zaidi e8d2b8cb18 fix(mantis): keep Telegram proof running and honest (#127108)
Make long, free-form Telegram proof runs truthful and resilient. Keep the trusted mock harness current across historical SUTs, preserve intentional silence and blocked outcomes, remove fixed attempt/lifetime caps, and export cropped motion proof without the prior memory spike.

Co-authored-by: Ayaan Zaidi <hi@obviy.us>
2026-08-21 17:53:54 +05:30
Ayaan Zaidi 1aa211be4a fix(qa): make Mantis backfills reliable (#126980)
Preserve honest blocked proof outcomes and publish visible stop-reports without marking them passed. Serialize burst runs through the authoritative Telegram-user lease while reserving time for proof and cleanup.
2026-08-21 10:01:58 +05:30
Ayaan Zaidi 4768ac53c5 feat(qa): acknowledge Mantis PR requests (#126702)
Give maintainers immediate visibility when Mantis is requested. Bare mentions now react, link the active run, and keep one run-owned status comment through proof, short-circuit, or failure.

Co-authored-by: Ayaan Zaidi <hi@obviy.us>
2026-08-20 21:19:40 +05:30
Ayaan Zaidi b7b7eec5ee feat(qa): run Mantis Telegram proof on local desktop (#126220)
Move Mantis Telegram Desktop proof from the remote AWS/Crabbox lane to a recorder-driven local Docker desktop. Keep proof scenarios agent-authored, cache trusted build outputs, and publish exact visible Telegram evidence without writing the QA bot token to artifacts.

Co-authored-by: Ayaan Zaidi <hi@obviy.us>
2026-08-20 11:12:51 +00:00
Ayaan Zaidi 97e3136b9e feat(qa): add a standalone Telegram Desktop recorder with a prebaked desktop image (#125186)
Records native Telegram Desktop from a digest-pinned prebaked image: a ready desktop in ~12s with no per-lease apt installs or downloads, which also removes the dpkg-lock failures that killed recent runs.

The recorder only records; callers drive the turn and supply the TDLib driver used for QR authorization. Nothing in this repository invokes it yet - routing the Mantis Telegram Desktop Proof workflow through it is a follow-up.
2026-08-19 11:23:48 +05:30
Peter Steinberger c70aee247e refactor(scripts): migrate JavaScript tools to TypeScript (#121005)
* refactor(scripts): migrate JavaScript tools to TypeScript

* fix(ci): keep changed-scope preflight zero-install

* fix(ci): preserve zero-install script owners

* fix(ci): complete script migration follow-through

* fix(release): keep stable closeout zero-install

* fix(scripts): preserve standalone execution boundaries

* fix(scripts): repair standalone loader boundaries

* fix(scripts): normalize gateway observation ids

* fix(scripts): keep Docker packager standalone

* test(scripts): preserve rebase cleanup helpers

* test(sessions): use tracked temp directory
2026-08-09 07:21:35 -07:00
Peter Steinberger 3224005c82 fix(ci): bound Telegram Mantis proof cleanup (#115253)
* test(mantis): bound Telegram proof cleanup

* test(mantis): enforce cleanup deadline in wrapper

* test(mantis): supervise cleanup through bash
2026-07-28 11:44:28 -04:00
Peter Steinberger 1695854877 fix(qa): isolate Mantis candidate execution (#114465)
* fix(qa): allow restricted Mantis worktree traversal

* fix(qa): pin Mantis harness checkout revision

* fix(qa): isolate Mantis candidate execution

* fix(qa): harden Mantis teardown
2026-07-27 12:11:34 -04:00
NIO bde17f99e9 fix(mantis): bound upload error response bodies (#109044)
* fix(mantis): bound upload error response bodies

Replace unbounded response.text() on non-ok upload error paths with
readBoundedResponseText (64 KiB cap). Only swallow the size-exceeded
diagnostic; propagate signal aborts and other failures so timeouts
during body reading are not masked as generic upload failures.

- Import shared readBoundedResponseText from scripts/lib/bounded-response.mjs
- Add MANTIS_UPLOAD_ERROR_BODY_MAX_BYTES constant (64 KiB)
- Catch only size-exceeded errors; re-throw timeouts and stream errors
- Add test: oversized body bounded at 64 KiB
- Add test: signal abort during body read propagates correctly
- Add test: small error body within bound reads normally

* fix(mantis): contextualize bounded upload failures

Co-authored-by: 胡根深 0668000903 <hu.genshen@xydigit.com>

---------

Co-authored-by: Peter Steinberger <steipete@gmail.com>
2026-07-18 22:00:15 +01:00
Peter Steinberger 3d1b771035 fix(ci): catch script declaration drift before merge (#110248)
* fix(ci): verify script declaration contracts

* fix(ci): harden declaration export analysis

* fix(ci): cover opaque script module exports

* test(ci): cover cyclic script declaration barrels
2026-07-18 02:54:26 +01:00
Alix-007 8781ba78de fix(mantis): bound R2 evidence uploads (#108616) 2026-07-15 22:57:36 -07:00
Peter Steinberger bae9752c5a refactor(deadcode): enforce repository hard zero (#108641) 2026-07-15 22:40:00 -07:00
Dallin Romney 77d07dc3e9 refactor(qa): migrate Telegram scenarios into QA Lab (#108430)
* refactor(qa): migrate Telegram scenarios into QA Lab

* refactor(qa): remove retired Telegram runner exports
2026-07-15 12:55:00 -07:00
Peter Steinberger fe261b0f59 chore(tooling): typecheck root test/** with a dedicated tsgo lane (#104475)
* chore(types): add declaration files for scripts/lib and scripts/e2e modules

* chore(types): add declaration files for top-level script modules (a-m)

* chore(types): add declaration files for top-level script modules (n-z)

* test: use a non-secret-shaped gateway token fixture

* test: type ci workflow guard helpers for the root test lane

* chore(tooling): typecheck root test/** with a dedicated tsgo lane

- test/tsconfig/tsconfig.test.root.json: root-test program (strict unused checks,
  fixtures excluded; two Docker E2E clients that import built dist/** stay out,
  same rationale as the scripts/e2e exclusion in tsconfig.scripts.json)
- tsgo:test:root wired into tsgo:test, check:test-types, scripts/check.mjs, and
  the ci.yml test-types shard, mirroring the tsgo:scripts lane (#104348)
- changed-lane routing: test/**/*.ts (excluding fixtures) and the lane tsconfig
  now trigger 'typecheck test root' in check:changed; previously test/ paths ran
  lint only, so harness type errors surfaced first in CI (#104287 envDir case)
- burn down all 1071 latent type errors in the program: precise param/local
  types across test/scripts, test/vitest, test/e2e, and transitive scripts/e2e
  program members; 205 sibling .d.mts declaration files for imported .mjs
  modules (committed separately); zero any, zero ts-expect-error
- resolve the pre-existing testing star-export ambiguity in
  scripts/e2e/parallels/common.ts with an explicit re-export

Closes #104388

* chore(types): correct declaration fidelity per structured review

- re-derive 51 .d.mts files from implementation data flow instead of
  initializers: fix a wrong never return (runTestProjectsDelegation returns
  the child), add encoding-sensitive exec/spawn overloads (plain-gh), restore
  the full release profile union, make parsed paths string | null, add missing
  parseArgs fields via help/non-help unions, add a missing sibling declaration
  (budget-number-args), drop 15 unused lint directives
- precise install-record/tuple typing removes the type-aware oxlint
  regressions the first declarations caused in scripts/e2e implementations
- route .mts declaration edits under test/ to the testRoot lane and reference
  the test-root project from tsconfig.projects.json so tsgo:all covers it
  (closes both review findings against the lane wiring)

* chore(scripts): keep telegram runner dist typing structural for the boundary guard

* chore(types): declare runtime pack and gateway readiness exports added on main

* test: pin the importTargetPlan form of the plugin-contract plan import

The guard expectation still referenced the raw await import( form that
7ae5996bb3 (#103975) replaced with the importTargetPlan fallback helper;
the assertion fails on current main.
2026-07-11 06:15:41 -07:00
brokemac79 037412ec7d ci(mantis): add web UI chat proof lane (#100472)
* ci(mantis): add web ui chat proof lane

* ci(mantis): tighten web ui proof candidate parsing

* ci: tighten Mantis Web UI proof lane

---------

Co-authored-by: Peter Steinberger <steipete@gmail.com>
2026-07-07 10:54:35 +01:00
Dallin Romney 1affe4fcdf Fold Telegram RTT sampling into live QA evidence (#92550)
* refactor(qa): fold telegram rtt into live evidence

* test: default package telegram rtt samples

* refactor(qa-lab): fold telegram rtt into live evidence

* fix(qa-lab): keep package telegram rtt optional for focused runs

* fix(qa-lab): avoid stale rtt evidence on failed samples

* fix(qa-lab): pass telegram live env into credential leasing

* fix(qa-lab): update telegram canary remediation artifacts

* docs(qa): remove stale telegram observed artifact guidance

* fix(qa-lab): clarify telegram empty-reply remediation

* fix(qa-lab): honor telegram rtt timeout

* ci(qa): drop stale telegram capture env

* refactor: align telegram evidence coverage fields

* fix: ignore stale telegram observed artifacts

* fix: preserve telegram rtt coverage mapping

* fix: omit unused telegram rtt catch binding

* docs: document telegram rtt check selector
2026-06-14 17:02:33 +08:00
Dallin Romney 4809ac70fa Add QA evidence artifact output (#91484)
* feat: add qa evidence summary normalization

* chore: rename qa evidence target environment

* chore: align qa evidence profile terminology

* chore: align qa evidence summary fields

* chore: add qa evidence taxonomy ref

* test: remove stale multipass evidence example

* test(qa): normalize vitest and playwright evidence

* test(qa): slim evidence summary metadata

* test(qa): clarify evidence summary inputs

* test(qa): rename scenario specs in evidence flow

* test(qa): treat evidence profiles as mapping strings

* test(qa): use neutral evidence test identity

* test(qa): nest evidence summary joins

* refactor(qa): normalize live evidence summaries

* fix(qa): accept normalized telegram rtt summaries

* fix(qa): normalize evidence lane summaries

* fix(qa): align evidence summaries with requirements

* refactor(qa): tighten evidence summary builders

* refactor(qa): restore standard evidence ids

* fix(qa): keep legacy summaries out of rtt evidence

* refactor(qa): make package evidence provenance explicit

* test(qa): keep script tests out of qa lab internals

* refactor(qa): rename scenario evidence definitions

* refactor(qa): clean evidence summary wording

* test(qa): fix evidence summary test inputs

* refactor(qa): simplify evidence identity fields

* refactor(qa): tighten evidence summary inputs

* refactor(qa): rename evidence artifact
2026-06-12 16:12:58 -07:00
Vincent Koc cc2ed8dbf6 fix(test): require passing Telegram evidence reports 2026-06-06 21:14:51 +02:00
Peter Steinberger 613a2835cb docs: document scoped script helpers 2026-06-04 23:57:22 -04:00
Ayaan Zaidi 98256b192b fix(mantis): suppress auto no-proof comments 2026-05-18 21:11:21 +05:30
Ayaan Zaidi 6baa2b38b2 ci(mantis): make telegram proof skips public-safe 2026-05-18 07:54:11 +05:30
Ayaan Zaidi f1b92c8885 fix(mantis): publish evidence to r2 (#81845)
* fix(mantis): publish evidence to r2

* ci(mantis): pass r2 artifact credentials

* ci(mantis): pin artifact bucket config

* fix(mantis): link raw evidence index
2026-05-15 16:23:53 +05:30
Peter Steinberger c9b6b0be0e fix(mantis): stop git-backed evidence publishing 2026-05-14 18:34:31 +01:00
Ayaan Zaidi adc3ba45ca ci(mantis): render proof media responsively 2026-05-11 20:08:02 +05:30
Ayaan Zaidi 1ff833d66d ci(mantis): fit telegram proof gifs in comments 2026-05-11 19:59:54 +05:30
Ayaan Zaidi 892f611065 test(mantis): package telegram desktop proof evidence 2026-05-11 08:15:34 +05:30
Peter Steinberger 378da8b9d3 feat: add telegram mantis evidence builder 2026-05-10 06:39:54 +01:00
Peter Steinberger f3d531439b feat: add reusable Mantis evidence publishing 2026-05-05 09:02:04 +01:00