db533799d5
fix(qa): reject incomplete evidence runs ( #126407 )
...
* fix(qa): enforce terminal evidence lifecycle
* test(qa): mark CPU fixture summary completed
* fix(qa): fence calls after flow timeout
2026-08-19 13:30:02 -07:00
08142099da
refactor(plugins): remove test-only facades and adapter ( #122532 )
...
* refactor(plugins): trim test-only facades
* refactor(whatsapp): remove legacy outbound adapter
2026-08-12 01:20:09 -07:00
b080dd1e76
refactor: consolidate coercion contracts ( #122458 )
...
* refactor: consolidate coercion contracts
Centralize exact string, record, numeric, date, Boolean, argument, and structured-error coercions while preserving call-site semantics.
Migrate canonical-name collisions and deprecated internal SDK bypasses, deleting 55 net production/tooling lines. Expand declaration ownership enforcement to 101 allowed helpers and add a narrow export-completeness audit.
* fix: preserve standalone script coercions
Keep copied Control UI tooling self-contained and retain the trusted release harness module-relative source seam when the harness runs against an old target cwd.
2026-08-11 23:26:37 -07:00
126b5a08e7
fix(qa): resolve live model pairs after auth selection ( #121065 )
...
Centralize QA model-pair construction after resolving the actual primary,
preserve explicit and manual single-model selections, and keep omitted
live-frontier alternates on the provider-owned OpenAI Sol/Luna policy.
Punchcard-Session: amber-workshop-workshop-36
Co-authored-by: Dallin Romney <6581799+RomneyDa@users.noreply.github.com >
2026-08-10 02:59:12 +08:00
d6fc4a2a95
refactor(qa): make module flows implementation-aware ( #118069 )
2026-08-06 19:51:28 +08:00
7dcb1ad5ce
fix(qa): attest realized profile executions ( #119761 )
...
* fix(qa): attest profile execution plans
Persist the canonical profile membership partition and scheduler-owned expected and observed execution cells in QA evidence. Validate successful evidence completeness and bind the canonical plan digest into workflow manifests.
Punchcard-Session: silver-meadow-lantern-vx
* test(ci): exercise QA evidence artifact round trips
Punchcard-Session: silver-meadow-lantern-vx
* fix(qa): record only realized profile executions
Co-authored-by: Dallin Romney <dallinromney@gmail.com >
Co-authored-by: ruel225 <ruel225@users.noreply.github.com >
Punchcard-Session: silver-meadow-lantern-vx
* test(qa): type realized channel evidence
Punchcard-Session: silver-meadow-lantern-vx
---------
Co-authored-by: Dallin Romney <dallinromney@gmail.com >
Co-authored-by: ruel225 <ruel225@users.noreply.github.com >
2026-08-06 08:33:02 +08:00
ae54355087
fix(qa): preserve profile execution contracts ( #119519 )
...
Punchcard-Session: silver-valley-valley-dt
2026-08-05 15:07:40 +08:00
2970ccb53f
fix(qa): report failed evaluations and scenario runs accurately ( #118120 )
2026-08-02 12:38:35 -07:00
7ea4129227
fix(qa): authenticate summaries and bound native scenario commands ( #116748 )
...
Co-authored-by: Peter Steinberger <steipete@macos.shared >
2026-07-31 03:12:48 -07:00
61cf4b9246
test(buzz): add live QA Lab channel lane ( #116298 )
...
* feat(buzz): add live QA runner
* ci(qa): add gated Buzz live lane
* docs(qa): document Buzz live testing
* fix(buzz): keep QA relay type private
* test(buzz): type QA suite mock
* fix(buzz): preserve racing QA reply relations
* fix(qa): harden Buzz credential redaction
* fix(buzz): ignore retained QA relay events
* fix(buzz): clean up observer setup failures
* test(buzz): return complete racing messages
* fix(buzz): default QA credentials to file
* fix(qa): validate pooled Buzz secrets
* fix(qa): redact structured Buzz auth tags
* fix(buzz): preserve QA credential env override
* fix(qa): redact inspected Buzz auth tags
* test(buzz): limit live QA to supported scenarios
* ci(qa): run Buzz lane in release checks
* fix(qa): require secure remote Buzz relays
* fix(qa): satisfy Buzz QA validation gates
* fix(qa): inject credential leasing into transport adapters
2026-07-30 17:44:27 -07:00
d16e33e08e
fix: restore trustworthy end-to-end QA and cross-channel delivery ( #115404 )
...
* fix(qa): repair verified end-to-end and channel regressions
* fix(gateway): make interrupted restart recovery lifecycle-safe
* test(heartbeat): target the canonical recovery session store
* fix(gateway): prioritize durable restart recovery before heartbeat
* fix(qa): preserve safe restart recovery and channel expiry
* fix(qa): fail closed and fence restart recovery
* test(agents): isolate restart recovery timing
* test(agents): prove actual restart retry timing
* fix(qa): report incompatible profile scenarios
* fix(scripts): resolve symlinked docker scheduler entrypoints
* fix(qa): require fresh native test evidence
* fix(heartbeat): fence active restart recovery delivery
* fix(gateway): consume untargeted restart acknowledgements
* fix(qa): satisfy exhaustive hosted validation gates
* fix(agents): fence stopped restart recovery dispatch
* style(agents): format restart recovery lifecycle regression
* test(gateway): isolate context prewarm sidecar lifecycle
* test(qa): make scenario process timeout cleanup deterministic
* fix(qa): stamp synthetic gateway configs with current version
* fix(openai): preserve vision capabilities in stale model catalogs
* test(qa): align profile channel rejection with current main
* fix(openai): forward supported moderation for image edits
* fix: restore latest-main CI and image edit documentation
* fix(qa): retain relocated code-mode evidence validation
* fix(openai): expose GPT-5.4 vision in static catalog
* fix(pricing): honor explicit model cost overrides
* test(pricing): keep isolated provider regressions deterministic
* fix(openai): inherit transport for discovered static models
* fix(gateway): honor agent-owned static image capabilities
* test(gateway): preserve prepared-snapshot attachment races
* test(gateway): isolate subagent persistence failure injection
* test(gateway): exercise concurrent voice replay admission
* fix(gateway): restore stale model image capabilities
* fix(agents): publish configured model vision capabilities
* fix(agents): isolate detached media transcript ownership
* test(agents): preserve generic transcript lock regression
* fix(gateway): require proven static model route identity
* fix(qa): accept bounded full-size generated image attachments
* fix(qa): require fresh script producer evidence
* test(qa): prove native E2E scenario execution
2026-07-29 12:45:27 -04:00
6090386dc7
fix: make Codex harness caching and benchmark parity exact ( #115663 )
2026-07-29 03:33:36 -04:00
781744e125
fix(qa): run profile scenarios on every eligible channel ( #115540 )
...
* fix(qa): resolve channels for portable live scenarios
* fix(qa): expand profile scenarios across eligible channels
2026-07-29 13:14:37 +08:00
641c82b5bd
fix(qa): restore full-stack scenarios and truthful suite exits ( #115290 )
...
* fix(qa): restore full-stack scenarios and truthful suite exits
* test(qa): respect plugin boundaries and atomic media delivery
2026-07-28 13:00:22 -04:00
9feb6ad161
refactor(qa): make scenario selection taxonomy-owned ( #114055 )
...
* refactor(qa): remove scenario execution profiles
* fix(qa): preserve smoke CI membership
* test(qa): drop redundant smoke pack order check
* refactor(qa): remove Matrix runner profiles
* refactor(qa): simplify Matrix CI orchestration
* refactor(qa): derive execution sets from taxonomy
* fix(qa): keep Matrix shard helper private
* refactor(qa): derive profile scenarios from taxonomy
* refactor(qa): reconcile taxonomy selection
* refactor(qa): use canonical profile planning
* refactor(qa): select canonical coverage owners
* fix(qa): keep profile selector internal
* refactor(qa): remove representative coverage owners
2026-07-28 22:23:41 +08:00
90564722ec
fix(qa-lab): retry failed suite partitions ( #115046 )
2026-07-28 16:13:22 +08:00
d2d6108d2c
fix(qa-lab): filter unsupported profile channels ( #115026 )
2026-07-28 15:25:36 +08:00
37cf0c6064
improve: reduce GPT-5.6 coding harness overhead ( #114574 )
...
* improve: reduce GPT-5.6 coding harness overhead
* refactor: extract runtime parity timing helpers
* fix(qa): verify native apply-patch runtime parity
* fix(qa): use mock patch execution evidence
* fix(qa): preserve workspace templates in full syncs
* fix(qa): make rejected patch fixtures deterministic
* fix(qa): prove apply-patch workspace containment
* fix(qa): expose runtime prompt-cache usage
* fix(codex): confine forced private QA patch turns
* fix(qa): verify native patch execution and sandbox precedence
* fix(qa): execute native patches and prove disk mutation
* fix(codex): avoid duplicate native patch registration
* perf(agents): keep code mode catalog guidance compact
* fix(qa): make native Codex patch evidence deterministic
* fix(qa): recognize native sandbox denial errors
* fix(qa): advertise native Codex mock model capabilities
* fix(qa): seed native Codex mock model catalog at startup
* fix(qa): recognize native Codex patch denial results
* fix(ci): repair Codex harness parity checks
* fix(qa): distinguish unknown prompt-cache measurements
* fix(qa): accept native Codex project patch denials
* fix(ci): isolate runtime parity report contracts
* fix(codex): eliminate native runtime transport overhead
* fix(qa): consume runtime parity report contract
* fix(qa): compare matched runtime timing captures
* fix(qa): type native response delta context
* fix(qa): satisfy native codex websocket lint
2026-07-27 19:24:52 -04:00
9310586b09
fix(qa-lab): unify runner control plane ( #112838 )
2026-07-27 14:05:14 +08:00
d8f099ebed
fix(qa): honor fail-fast across unified suite runners ( #113843 )
2026-07-25 13:26:51 -07:00
19d17bc3c4
refactor(qa): canonicalize runtime-pair lanes ( #112603 )
2026-07-23 10:06:12 +09:00
82fa956d5a
fix(qa): resolve profile channels through selected driver ( #109350 )
...
* fix(qa): keep generic profiles off channel-specific lanes
* fix(qa): resolve profile channels through selected driver
2026-07-17 16:42:18 -07:00
9a2f25c82b
refactor(qa): make channel drivers run-level ( #108602 )
2026-07-16 00:59:03 -07:00
2d9b78fff7
refactor(qa): canonicalize live scenario coverage ( #108464 )
2026-07-15 16:50:41 -07:00
af71400c8e
refactor(qa): route live channel selectors through suite host ( #108429 )
...
* refactor(qa): route live channel selectors through suite host
* fix(qa): repair live selector CI
* fix(qa): remove retired runner exports
* fix(qa): remove obsolete live evidence
2026-07-15 13:35:34 -07:00
ad34552473
refactor(qa): migrate Matrix scenarios into QA Lab ( #103589 )
...
* refactor(qa): migrate Matrix scenarios into QA Lab
* fix(qa): build Matrix boundary declarations
* fix(qa): preserve Matrix preview boundaries
* fix(qa): preserve Matrix hot reload semantics
* fix(qa): harden Matrix destructive scenario failures
* fix(qa): harden Matrix scenario isolation
* fix(qa): close Matrix negative scenario blind spots
* fix(qa): isolate Matrix substrate state
* fix(qa): harden Matrix transport substrate
* fix(qa): preserve Matrix profile and event parity
* fix(qa): preserve explicit scenario models
* fix(qa): align Matrix scenario coverage taxonomy
* fix(qa): format Matrix allowlist cleanup
* fix(qa): satisfy migrated Matrix CI contracts
* fix(qa): reconcile Matrix migration with current main
* fix(qa): break scenario flow import cycle
* fix(qa): reconcile Matrix max-lines ownership
* fix(qa): address Matrix review boundaries
* fix(qa): remove stale Matrix lint suppression
* fix(qa): adopt split Matrix E2EE flows
* fix(qa): export Matrix scenario record guard
* fix(qa): align Matrix migration with privatized helpers
* refactor(qa): finish Matrix QA Lab ownership
* fix(qa): preserve Matrix suite defaults
* fix(qa): reconcile Matrix cleanup with current main
* test(qa): follow canonical Matrix profile size
* fix(qa): guard stale Matrix QA package output
* docs(qa): redirect retired Matrix QA pages
* refactor(qa): finish Matrix runner rename
* test(qa): assert Matrix defaults through profile resolver
* docs: refresh QA cleanup map
* fix(qa): privatize Matrix storage discovery
2026-07-15 01:22:20 -07:00
f81f9d8570
chore: enforce max-lines suppression ratchet ( #107315 )
...
* ci: enforce max-lines suppression ratchet
* chore: prune stale max-lines suppression
* fix: close max-lines ratchet enforcement gaps
* fix: harden max-lines ratchet checks
* fix(ci): satisfy max-lines ratchet checks
* style: format max-lines declarations
* fix(ci): match oxlint suppression grammar
* test: isolate max-lines git fixtures
* chore: prune resolved max-lines debt
* test: skip newline path fixture on Windows
* fix: harden max-lines suppression ratchet
* chore: refresh max-lines baseline
* fix: close max-lines ratchet bypasses
* fix: derive ratchet base from PR merge tree
* fix: support older Git in staged ratchet
* fix: align max-lines declarations and baseline
* chore: refresh max-lines baseline for current main
* fix: exclude generated wizard locales from max-lines
* chore: prune resolved max-lines debt
2026-07-14 09:27:02 -07:00
54d31a1306
refactor(qa): share Matrix flows across drivers ( #101253 )
2026-07-14 01:48:30 -07:00
4d15c47e9f
fix(qa): keep runtime parity lanes reproducible
2026-07-12 15:10:38 +08:00
4040267af9
fix(qa): run mixed channel suites
2026-07-12 06:51:59 +02:00
4bc300843d
fix(qa): keep runtime parity tiers flow-compatible ( #103609 )
...
* fix(qa): keep runtime parity tiers flow-compatible
Refs #103588
* test(qa): follow canonical frontier defaults
2026-07-10 10:51:13 +01:00
19491fe34c
Revert "fix(qa): keep smoke profile on one channel ( #101173 )" ( #101184 )
...
This reverts commit b773f7981a
2026-07-06 13:29:07 -07:00
b773f7981a
fix(qa): keep smoke profile on one channel ( #101173 )
2026-07-06 13:26:58 -07:00
b29d472e41
refactor(qa): add canonical live channel adapters ( #99707 )
2026-07-06 09:24:34 -07:00
677c8ff846
improve: enforce canonical QA scenario ownership ( #99628 )
...
* improve: enforce canonical QA scenario ownership
* docs: refresh QA documentation map
* docs: condense QA ownership guidance
* docs: remove QA migration guidance
* refactor: simplify QA scenario ownership checks
* docs: align QA coverage authoring guidance
2026-07-03 15:51:18 -07:00
c313642ae2
fix(qa-lab): use scoped crabline package
2026-06-23 17:47:07 +08:00
5738cfb6df
fix(qa): avoid default artifact directory collisions
2026-06-23 10:53:46 +02:00
def4b51485
fix(qa): gate smoke profile scenarios by channel driver
2026-06-23 09:34:52 +08:00
de9c94cbbb
feat(qa): forward shared suite flags to multipass runner ( #91506 )
2026-06-22 15:48:05 -07:00
63b13ea837
feat(qa): crabline channel driver ( #91502 )
...
* feat(qa): add crabline channel driver seam
* feat: run crabline channel driver smoke
* chore: keep crabline qa dependency dev-only
* refactor(qa): keep crabline driver details opaque
* chore(qa): pin crabline to merged driver API
* feat(qa): drive channel driver from profiles
* fix(qa): declare crabline runtime peer
* feat(qa): resolve crabline channel from scenarios
* feat(qa): treat unsupported profile channels as coverage gaps
* Revert "feat(qa): treat unsupported profile channels as coverage gaps"
This reverts commit 65a97016558705514a1c1ec74870de9b0a6e7886.
* fix(qa): adapt crabline driver to chat sdk cli
* refactor(qa): pass channel driver metadata directly
* chore(qa): update crabline provider pin
* chore(qa): default channel scenarios to driver
* chore: repair qa dependency lockfile
* chore: allow native qa dependency builds
* fix(qa): satisfy crabline driver lint
* fix(qa): satisfy crabline ci gates
* Use crabline transport for smoke QA profile
* fix(qa): keep crabline driver opt-in
* fix(qa): reuse crabline telegram driver token
* fix(qa): route smoke profile through crabline
* fix(qa): run full smoke profile lane
* fix(qa): remove smoke scenario workflow filter
* fix: stabilize crabline smoke qa profile
* fix: pin crabline qa dependency
* test: keep crabline smoke credential-free
* fix: skip visible reasoning lane for crabline smoke
* fix: unblock crabline qa ci
* Update crabline dependency
* Pin crabline to merged main
* Use Crabline fake provider servers
2026-06-22 15:24:59 -07:00
e2b6753b87
fix(qa-lab): bound credential payload reads
2026-06-17 11:59:55 +02:00
c32ba171db
fix(qa): fail unsuccessful self-checks
2026-06-16 01:32:47 +02:00
e32929e12c
Add slim evidence mode for QA profile evidence ( #93179 )
...
* test(qa): compact profile evidence execution metadata
* docs(qa): document compact profile evidence
* test(qa): support compact evidence mode
* test(qa): rename compact evidence mode to slim
* docs(qa): trim slim evidence wording
* fix(qa): avoid commander runtime import
2026-06-15 14:50:40 -07:00
1f8c4d3958
simplify QA evidence profile and mappings/coverage shape ( #93153 )
...
* test(qa): simplify evidence coverage shape
* test(qa): collapse evidence scorecard metadata
* test(qa): document evidence schema version
2026-06-14 22:26:58 -07:00
3d38c9a633
test(qa): embed profile scorecard evidence ( #93109 )
...
* test(qa): embed profile scorecard evidence
* test(qa): fix profile runner return lint
* test(qa): satisfy suite command lint return
2026-06-14 20:51:38 -07:00
e8db9c3bc0
test(qa): add qa run --profile and unified output summary/evidence ( #91587 )
...
* test(qa): add mapped qa run profiles
* test(qa): document mapped profile runner
* test(qa): validate run profiles from mapping
* test(qa): preserve root profile parsing
* test(qa): simplify taxonomy profile dispatch
* test(qa): align tool coverage CLI expectation
* test(qa): fix profile dispatch fixture type
* test(qa): share profile runner option types
* test(qa): split shared cli runner options
* test(qa): unify profile suite artifacts
* fix(qa): filter profile scenarios by provider lane
* test(qa): drop native scenario subreports
* fix(qa): keep native log refs repo-relative
* fix(cli): preserve qa run root profile parsing
* fix(qa): avoid qa profile flag collision
* fix(qa): reject profile flags without qa profile
2026-06-14 18:08:42 -07:00
561b293c7a
Run Vitest and Playwright scenarios from qa suite ( #92606 )
...
* test(qa): run vitest and playwright scenarios from qa suite
* fix(qa): harden scenario suite dispatch
* refactor(qa): share scenario path utilities
* refactor(qa): share test file scenario runner
* refactor(qa): route test file scenarios through suite runtime
* refactor(qa): use explicit suite runtime result kind
* test(qa): write suite evidence artifact
* refactor(qa): clarify suite execution dispatch
* fix(qa): keep test-file scenarios out of flow-only runners
* refactor(qa): export mixed scenario suite runner
2026-06-13 01:06:10 -07:00
4809ac70fa
Add QA evidence artifact output ( #91484 )
...
* feat: add qa evidence summary normalization
* chore: rename qa evidence target environment
* chore: align qa evidence profile terminology
* chore: align qa evidence summary fields
* chore: add qa evidence taxonomy ref
* test: remove stale multipass evidence example
* test(qa): normalize vitest and playwright evidence
* test(qa): slim evidence summary metadata
* test(qa): clarify evidence summary inputs
* test(qa): rename scenario specs in evidence flow
* test(qa): treat evidence profiles as mapping strings
* test(qa): use neutral evidence test identity
* test(qa): nest evidence summary joins
* refactor(qa): normalize live evidence summaries
* fix(qa): accept normalized telegram rtt summaries
* fix(qa): normalize evidence lane summaries
* fix(qa): align evidence summaries with requirements
* refactor(qa): tighten evidence summary builders
* refactor(qa): restore standard evidence ids
* fix(qa): keep legacy summaries out of rtt evidence
* refactor(qa): make package evidence provenance explicit
* test(qa): keep script tests out of qa lab internals
* refactor(qa): rename scenario evidence definitions
* refactor(qa): clean evidence summary wording
* test(qa): fix evidence summary test inputs
* refactor(qa): simplify evidence identity fields
* refactor(qa): tighten evidence summary inputs
* refactor(qa): rename evidence artifact
2026-06-12 16:12:58 -07:00
3dc6ac3802
fix(qa): fail closed on skipped suite summaries
2026-06-07 11:20:25 +02:00
e12141fa9f
fix(qa): gate live transport exits on summaries
2026-06-07 02:53:50 +02:00
Peter Steinberger