mirror of
https://github.com/openclaw/openclaw.git
synced 2026-08-12 21:53:00 -06:00
chore: sync canonical autoreview skill (#100210)
* chore: sync canonical autoreview skill * fix: preserve autoreview Cursor aliases * fix: harden autoreview isolation * fix: require explicit trust for Cursor reviews * fix: preserve autoreview closeout integrity * fix: close autoreview isolation gaps * fix: reject incomplete autoreview inputs * fix: isolate tool-less autoreview engines * fix: fail closed for unsupported Droid isolation * fix: fail closed for unsupported Droid isolation * fix: fail closed for unsupported Droid isolation * test: route package git fixture through temp dir tests * fix: fail closed for unsupported Droid isolation * fix: fail closed for unsupported Droid isolation * fix: fail closed for unsupported Droid isolation
This commit is contained in:
committed by
GitHub
parent
fd2e4da006
commit
1a7f1133db
@@ -1,17 +1,19 @@
|
||||
---
|
||||
name: autoreview
|
||||
description: "Auto Review closeout. Codex review is the default when no engine is set and is the recommended reviewer."
|
||||
description: "Pre-commit/ship code review: Codex default; optional Claude, Pi, Droid, Copilot, Cursor, or OpenCode."
|
||||
---
|
||||
|
||||
# Auto Review
|
||||
|
||||
Run the bundled structured review helper as a closeout check. This is code review, not Guardian `auto_review` approval routing.
|
||||
|
||||
Codex review is the default when no engine is set. It usually delivers the best review results and should remain the normal final closeout engine.
|
||||
Codex review is the default when no engine is set. It uses `gpt-5.5` by default, usually delivers the best review results, and should remain the normal final closeout engine. Claude review is optional and uses `claude-fable-5` by default.
|
||||
|
||||
For user-visible behavior, pair autoreview with `behavior-validator`. Autoreview is source-aware and judges the change bundle; behavior validation is source-blind and judges the running product or tool against a behavior contract. A clean autoreview is not proof that a UI, CLI, API, or generated artifact works from the user's perspective.
|
||||
|
||||
Use when:
|
||||
|
||||
- user asks for Codex review / Claude review / autoreview / second-model review
|
||||
- user asks for Codex review / Claude review / Pi review / Droid review / Cursor review / OpenCode review / autoreview / second-model review
|
||||
- after non-trivial code edits, before final/commit/ship
|
||||
- reviewing a local branch or PR branch after fixes
|
||||
|
||||
@@ -29,11 +31,12 @@ Use when:
|
||||
- For security-audit suppression changes, verify accepted findings remain auditable: suppressed findings stay in structured output, active output keeps an unsuppressible suppression notice, and aggregate findings cannot hide unrelated active risk.
|
||||
- Never switch or override the requested review engine/model. If the review hits model capacity, retry the same command a few times with the same engine/model.
|
||||
- Be patient with large bundles. Structured review can take up to 30 minutes while the model call is active, especially with Codex tools or web search.
|
||||
- Treat heartbeat lines like `review still running: ... elapsed=... pid=...` as healthy progress, not a hang. Let the helper continue while heartbeats are advancing. Pass `--stream-engine-output` when live engine text is useful; Codex, Claude, and cursor-agent filter tool/file chatter, other engines pass raw output through.
|
||||
- Treat heartbeat lines like `review still running: ... elapsed=... pid=...` as healthy progress, not a hang. Let the helper continue while heartbeats are advancing. Pass `--stream-engine-output` when live engine text is useful; Codex, Claude, and Cursor filter tool/file chatter, other engines pass raw output through.
|
||||
- Do not kill a review just because it has been quiet for 2-5 minutes, or because it is still running under the 30-minute window. Inspect the process only after missing multiple expected heartbeats, after 30 minutes, or after an obviously failed subprocess; prefer letting the same helper command finish.
|
||||
- Tools are useful in review mode. The helper allows read-only inspection tools and web search by default so reviewers can check dependency contracts, upstream docs, and current behavior.
|
||||
- Security perspective is always included, but it should not cripple legitimate functionality. Report security findings only when the change creates a concrete, actionable risk or removes an important safety check.
|
||||
- For regression provenance, if no blamed PR is traceable, use the blamed commit as the provenance: commit SHA, date, and author username. Do not guess a merger or frame missing PR metadata as a separate finding.
|
||||
- For regression provenance, keep roles separate: blamed code author, blamed PR author, PR merger/committer, current PR author, and PR/date. If no blamed PR is traceable, use the blamed commit as the provenance: commit SHA, date, and author username. Do not guess a merger or frame missing PR metadata as a separate finding.
|
||||
- If the blamed PR was merged by `clawsweeper[bot]` or another automation, identify the human trigger when practical. Check timeline/comments first; if rate-limited, use gitcrawl/cache or public PR HTML. Look for maintainer commands such as `@clawsweeper automerge`, `/landpr`, or labels/status comments that armed automerge. Report `automerge triggered by @login`; if not found, say trigger unknown.
|
||||
- Do not invoke built-in `codex review`, nested reviewers, or reviewer panels from inside the review. The helper builds one bundle, calls one selected engine, validates one structured result, and stops.
|
||||
- Stop as soon as the helper exits 0 with no accepted/actionable findings. Do not run an extra review just to get a nicer "clean" line, a second opinion, or clearer closeout wording.
|
||||
- Treat the helper's successful exit plus absence of actionable findings as the clean review result, even if the underlying Codex CLI output is terse.
|
||||
@@ -79,12 +82,39 @@ On release, beta, stable, hotfix, signing, notarization, appcast, package-publis
|
||||
- Keep proof tied to the release target: exact branch/ref, failing check or shipped-risk reason, smallest command/proof, and whether the fix must also forward-port to `main`.
|
||||
- If review discovers a real but non-critical design problem during release closeout, stop with a follow-up issue/PR plan; do not use the release branch as the refactor lane.
|
||||
|
||||
## Skill Path (set once)
|
||||
|
||||
Set the skill script paths once, then use `"$AUTOREVIEW"` and `"$AUTOREVIEW_HARNESS"` in the examples below.
|
||||
|
||||
Choose one:
|
||||
|
||||
```bash
|
||||
# Project-local skill in the current repo:
|
||||
export AUTOREVIEW=".agents/skills/autoreview/scripts/autoreview"
|
||||
export AUTOREVIEW_HARNESS=".agents/skills/autoreview/scripts/test-review-harness"
|
||||
```
|
||||
|
||||
```bash
|
||||
# Source checkout of openclaw/agent-skills:
|
||||
export AUTOREVIEW="skills/autoreview/scripts/autoreview"
|
||||
export AUTOREVIEW_HARNESS="skills/autoreview/scripts/test-review-harness"
|
||||
```
|
||||
|
||||
```bash
|
||||
# Global skill:
|
||||
export AGENTS_HOME="${AGENTS_HOME:-$HOME/.agents}"
|
||||
export AUTOREVIEW="$AGENTS_HOME/skills/autoreview/scripts/autoreview"
|
||||
export AUTOREVIEW_HARNESS="$AGENTS_HOME/skills/autoreview/scripts/test-review-harness"
|
||||
```
|
||||
|
||||
When using Claude Code, set `AGENTS_HOME="$HOME/.claude"` for global skills. Project-local skills live under `.claude/skills/` in the current repo.
|
||||
|
||||
## Pick Target
|
||||
|
||||
Dirty local work:
|
||||
|
||||
```bash
|
||||
<autoreview-helper> --mode local
|
||||
"$AUTOREVIEW" --mode local
|
||||
```
|
||||
|
||||
Use this only when the patch is actually unstaged/staged/untracked in the
|
||||
@@ -97,32 +127,26 @@ only proves there is no local patch.
|
||||
Branch/PR work:
|
||||
|
||||
```bash
|
||||
<autoreview-helper> --mode branch --base origin/main
|
||||
"$AUTOREVIEW" --mode branch --base origin/main
|
||||
```
|
||||
|
||||
Optional review context is first-class:
|
||||
Optional review context is first-class. Prompt files and datasets must be repo-relative so review bundles cannot pull arbitrary host files:
|
||||
|
||||
```bash
|
||||
<autoreview-helper> --mode branch --base origin/main --prompt-file /tmp/review-notes.md --dataset /tmp/evidence.json
|
||||
"$AUTOREVIEW" --mode branch --base origin/main --prompt-file review-notes.md --dataset evidence.json
|
||||
```
|
||||
|
||||
If an open PR exists, use its actual base:
|
||||
|
||||
```bash
|
||||
base=$(gh pr view --json baseRefName --jq .baseRefName)
|
||||
<autoreview-helper> --mode branch --base "origin/$base"
|
||||
"$AUTOREVIEW" --mode branch --base "origin/$base"
|
||||
```
|
||||
|
||||
Committed single change:
|
||||
|
||||
```bash
|
||||
<autoreview-helper> --mode commit --commit HEAD
|
||||
```
|
||||
|
||||
or with the helper:
|
||||
|
||||
```bash
|
||||
/Users/steipete/Projects/agent-scripts/skills/autoreview/scripts/autoreview --mode commit --commit HEAD
|
||||
"$AUTOREVIEW" --mode commit --commit HEAD
|
||||
```
|
||||
|
||||
Use commit review for already-landed or already-pushed work on `main`. Reviewing
|
||||
@@ -135,7 +159,7 @@ with `--base`.
|
||||
Format first if formatting can change line locations. Then it is OK to run tests and review in parallel:
|
||||
|
||||
```bash
|
||||
scripts/autoreview --parallel-tests "<focused test command>"
|
||||
"$AUTOREVIEW" --parallel-tests "<focused test command>"
|
||||
```
|
||||
|
||||
On Windows, the default `--parallel-tests` shell preserves the platform `cmd.exe`
|
||||
@@ -149,30 +173,133 @@ Tradeoff: tests may force code changes that stale the review. If tests or review
|
||||
Run multiple reviewers against one frozen bundle:
|
||||
|
||||
```bash
|
||||
<autoreview-helper> --reviewers codex,claude
|
||||
"$AUTOREVIEW" --reviewers codex,claude,pi,opencode
|
||||
```
|
||||
|
||||
`--panel` is shorthand for Codex plus Claude unless `--engine` changes the first reviewer:
|
||||
|
||||
```bash
|
||||
<autoreview-helper> --panel
|
||||
"$AUTOREVIEW" --panel
|
||||
```
|
||||
|
||||
Set reviewer models and thinking/effort explicitly:
|
||||
|
||||
```bash
|
||||
<autoreview-helper> --reviewers codex,claude --model codex=gpt-5.1 --thinking codex=high --model claude=sonnet --thinking claude=max
|
||||
"$AUTOREVIEW" --reviewers codex,claude --model codex=gpt-5.5 --thinking codex=high --model claude=claude-fable-5 --thinking claude=max
|
||||
```
|
||||
|
||||
Inline syntax is also supported:
|
||||
Inline syntax is also supported for simple model IDs:
|
||||
|
||||
```bash
|
||||
<autoreview-helper> --reviewers codex:gpt-5.1:high,claude:sonnet:max
|
||||
"$AUTOREVIEW" --reviewers codex:gpt-5.5:high,claude:claude-fable-5:max
|
||||
```
|
||||
|
||||
Codex maps thinking to `model_reasoning_effort` and accepts `low`, `medium`,
|
||||
`high`, or `xhigh`. Claude maps thinking to `--effort` and also accepts `max`.
|
||||
Engines without a real thinking knob reject `--thinking`.
|
||||
For models with slashes or extra colons, prefer keyed form:
|
||||
|
||||
```bash
|
||||
"$AUTOREVIEW" --engine pi --model anthropic/claude-sonnet-4 --thinking high
|
||||
"$AUTOREVIEW" --engine opencode --model opencode/north-mini-code-free --thinking high
|
||||
"$AUTOREVIEW" --engine cursor --model auto --cursor-allow-workspace-instructions
|
||||
"$AUTOREVIEW" --reviewers codex,pi --model codex=gpt-5.5 --model pi=anthropic/claude-sonnet-4
|
||||
"$AUTOREVIEW" --reviewers codex,opencode --model codex=gpt-5.5 --model opencode=opencode/north-mini-code-free
|
||||
"$AUTOREVIEW" --reviewers codex,cursor --model codex=gpt-5.5 --model cursor=auto --cursor-allow-workspace-instructions
|
||||
```
|
||||
|
||||
`--reviewers all` covers Codex, Claude, Copilot, Pi, and OpenCode. Cursor requires both explicit selection (`--engine cursor` or named in `--reviewers`) and `--cursor-allow-workspace-instructions` because the current Cursor CLI does not document a per-run flag that ignores project-local instructions/config. Droid selection currently fails closed because its CLI cannot disable both project instructions and all tools.
|
||||
|
||||
## Models and thinking
|
||||
|
||||
The helper accepts `--model` globally or per engine (`engine=model`) and `--thinking` globally or per engine (`engine=level`). Repeat either flag for multiple reviewers.
|
||||
|
||||
Recommended model defaults:
|
||||
|
||||
| Engine | Default model | Source note |
|
||||
|--------|---------------|-------------|
|
||||
| **codex** (default) | `gpt-5.5` | OpenAI's current GPT-5.5 alias |
|
||||
| **claude** | `claude-fable-5` | Anthropic's most capable widely released Claude model |
|
||||
|
||||
CLI flags and environment variables override these defaults. Droid, Copilot, Pi, Cursor, and OpenCode do not get built-in model defaults here because their provider catalogs are external to the Codex/Claude closeout path and may vary by installation.
|
||||
|
||||
| Engine | Model flag | Example model IDs | Thinking flag | Accepted levels |
|
||||
|--------|------------|-------------------|---------------|-----------------|
|
||||
| **codex** (default) | `codex --model X exec ...` | `gpt-5.5`, `gpt-5.5-2026-04-23` | `-c model_reasoning_effort=Y` | `none`, `minimal`, `low`, `medium`, `high`, `xhigh` |
|
||||
| **claude** | `claude --model X` | `claude-fable-5`, `claude-opus-4-8`, `claude-sonnet-4-6`, `claude-haiku-4-5` | `--effort Y` | `low`, `medium`, `high`, `xhigh`, `max` |
|
||||
| **droid** | currently refused | Factory model IDs | `-r, --reasoning-effort Y` | `off`, `none`, `low`, `medium`, `high`, `xhigh`, `max` |
|
||||
| **copilot** | `copilot --model X` | `gpt-5.2`, Copilot model aliases | not supported | n/a |
|
||||
| **pi** | `pi --model X` | `anthropic/claude-sonnet-4`, `openai/gpt-4o` | `--thinking Y` | `off`, `minimal`, `low`, `medium`, `high`, `xhigh` |
|
||||
| **cursor** | `cursor-agent --model X` | `auto`, Cursor model aliases | not supported | n/a |
|
||||
| **opencode** | `opencode run -m X` | `opencode/north-mini-code-free`, OpenCode provider/model IDs | `--variant Y` | `minimal`, `low`, `medium`, `high`, `max` |
|
||||
|
||||
Claude also supports `--fallback-model a,b` for availability-based fallback chains ([model-config](https://code.claude.com/docs/en/model-config)). Current Claude docs note that auth, billing, rate-limit, request-size, and transport errors do not trigger fallback, and the changelog documents interactive-session support in `v2.1.166`.
|
||||
|
||||
Examples matching current `main` behavior:
|
||||
|
||||
```bash
|
||||
# Codex with explicit model and reasoning
|
||||
"$AUTOREVIEW" --engine codex --model gpt-5.5 --thinking high
|
||||
|
||||
# Codex fast mode (priority service tier); needs a model whose catalog lists the tier, silently standard otherwise
|
||||
"$AUTOREVIEW" --engine codex --codex-speed fast
|
||||
|
||||
# Arbitrary Codex config overrides (isolation flags still win; --codex-speed wins over a service_tier here)
|
||||
"$AUTOREVIEW" --engine codex --codex-config 'service_tier="fast"'
|
||||
|
||||
# Claude Code aliases or full model names, with optional availability fallback
|
||||
"$AUTOREVIEW" --engine claude --model claude-fable-5 --thinking max
|
||||
"$AUTOREVIEW" --engine claude --model claude-fable-5 --fallback-model claude-opus-4-8,claude-sonnet-4-6
|
||||
|
||||
# GitHub Copilot (model only; no thinking knob)
|
||||
"$AUTOREVIEW" --engine copilot --model gpt-5.2
|
||||
|
||||
# Pi with explicit model and thinking level
|
||||
"$AUTOREVIEW" --engine pi --model anthropic/claude-sonnet-4 --thinking high --pi-bin pi
|
||||
|
||||
# Cursor print-mode review (`cursor-agent` remains a compatibility alias)
|
||||
"$AUTOREVIEW" --engine cursor --model auto --cursor-bin cursor-agent --cursor-allow-workspace-instructions
|
||||
|
||||
# OpenCode with explicit provider/model and variant
|
||||
"$AUTOREVIEW" --engine opencode --model opencode/north-mini-code-free --thinking high
|
||||
```
|
||||
|
||||
`--cursor-agent-bin` and `CURSOR_AGENT_BIN` remain compatibility aliases for
|
||||
`--cursor-bin` and `CURSOR_BIN`.
|
||||
|
||||
### Environment defaults
|
||||
|
||||
CLI flags take precedence over environment variables.
|
||||
|
||||
Store persistent personal defaults in your shell startup file or launcher
|
||||
environment. For repository-local defaults, use an existing local environment
|
||||
loader such as an untracked `.envrc`; the helper does not write a config file.
|
||||
|
||||
| Variable | Purpose |
|
||||
|----------|---------|
|
||||
| `AUTOREVIEW_MODEL` | Override the built-in default `--model` for all engines |
|
||||
| `AUTOREVIEW_THINKING` | Default `--thinking` for all engines |
|
||||
| `AUTOREVIEW_FALLBACK_MODEL` | Default Claude `--fallback-model` chain |
|
||||
| `AUTOREVIEW_<ENGINE>_MODEL` | Per-engine model override, for example `AUTOREVIEW_CODEX_MODEL=gpt-5.5` |
|
||||
| `AUTOREVIEW_<ENGINE>_THINKING` | Per-engine thinking override |
|
||||
| `AUTOREVIEW_CODEX_CONFIG` | Default Codex `-c key=value` overrides, semicolon-separated, e.g. `service_tier="fast"`; isolation flags still win |
|
||||
| `AUTOREVIEW_CODEX_SPEED` | Default Codex service tier: `fast` (priority), `flex`, or `default`; silently standard when the model does not list the tier |
|
||||
| `AUTOREVIEW_CLAUDE_FALLBACK_MODEL` | Claude-only fallback chain |
|
||||
| `AUTOREVIEW_CURSOR_ALLOW_WORKSPACE_INSTRUCTIONS` | Required `1`/true opt-in for Cursor reviews of trusted repositories |
|
||||
|
||||
Codex maps thinking to `model_reasoning_effort`. Claude maps thinking to `--effort`. Droid maps thinking to `-r, --reasoning-effort`. Pi maps thinking to `--thinking`. OpenCode maps thinking to `--variant`. Copilot and Cursor reject `--thinking`. Only Claude accepts `--fallback-model`; global CLI/env fallback requires at least one Claude reviewer, and engine-specific fallback overrides require that reviewer to be selected. Non-Claude fallback overrides, including `AUTOREVIEW_<NONCLAUDE>_FALLBACK_MODEL`, fail closed instead of being silently ignored.
|
||||
|
||||
## Review engine isolation
|
||||
|
||||
When autoreview runs inside the repository under review, external reviewer CLIs must not load project-local trust or configuration that the branch controls.
|
||||
|
||||
| Engine | Isolation flags | Reference |
|
||||
|--------|-----------------|-----------|
|
||||
| **codex** | Auth-only config overrides, `-c project_doc_max_bytes=0`, repo `trust_level="untrusted"`, `exec --ignore-user-config --ignore-rules`, plus read-only sandbox | Codex CLI `exec --help` |
|
||||
| **claude** | `--safe-mode --setting-sources user --strict-mcp-config --disallowedTools mcp__*` plus explicit `--allowedTools` (`--safe-mode` requires Claude Code `v2.1.169+`) | Claude Code [CLI reference](https://code.claude.com/docs/en/cli-reference) |
|
||||
| **droid** | Fails closed: current CLI cannot disable both project instructions and all tools | Droid CLI `exec --help` and `--list-tools` |
|
||||
| **pi** | `--no-approve --no-session --no-context-files --no-extensions --no-skills --no-prompt-templates --no-themes --no-tools` | Pi CLI `--help`; requires Pi `v0.79.0+` |
|
||||
| **opencode** | `opencode run --dir <repo> --pure --format json`, prompt over stdin, neutral subprocess cwd, injected deny-by-default permissions, project config disabled | OpenCode CLI `--help` |
|
||||
| **cursor** | `cursor-agent --print --mode ask --sandbox enabled --output-format json|stream-json`, prompt over stdin, temporary read-only permission config, help-probed flags, and mandatory explicit trusted-workspace opt-in | Cursor CLI [headless mode](https://cursor.com/docs/cli/headless), [output format](https://cursor.com/docs/cli/reference/output-format), [permissions](https://cursor.com/docs/cli/reference/permissions), [configuration](https://cursor.com/docs/cli/reference/configuration) |
|
||||
|
||||
Codex `--ignore-user-config` skips config loading for the exec run. Autoreview reconstructs only the documented `cli_auth_credentials_store`, `forced_login_method`, and `forced_chatgpt_workspace_id` settings from `CODEX_HOME/config.toml`, keeping authentication and workspace restrictions usable without forwarding unrelated user configuration. The explicit repo trust override and zero project-doc budget keep reviewed-repo `AGENTS.md` and `.codex/` trust surfaces out of the review prompt. `--ignore-rules` skips user/project execpolicy rules. Claude `--safe-mode` disables project hooks, skills, plugins, MCP servers, and CLAUDE.md while preserving normal authentication, model selection, built-in tools, and permissions; managed settings policy can still apply. `--setting-sources user` avoids project/local settings from the reviewed checkout, and current Claude Code docs note the project-skill blocking behavior was fixed in `v2.1.69`. `--strict-mcp-config` and `--disallowedTools mcp__*` keep MCP unavailable to the review run. `--bare` is not used here because Claude's headless docs say it skips OAuth and keychain reads. Droid fails closed because its CLI cannot disable reviewed-repository `AGENTS.md` loading and all tools in the same run. Pi `--no-approve` ignores project-local files for one run; the helper requires Pi `v0.79.0+` plus help output that advertises every required isolation flag because older legacy binaries can ignore unknown flags. The current package is `@earendil-works/pi-coding-agent`; deprecated `@mariozechner/pi-coding-agent` `0.73.x` is intentionally rejected. Pi version/help probes and the review command run from neutral temporary directories, not the reviewed repo. Pi `--no-context-files` removes `AGENTS.md`/`CLAUDE.md`, the resource-disable flags keep `.pi` extensions, skills, prompts, and themes out of the run, `--no-session` avoids writing review sessions, and `--no-tools` prevents built-in read tools from escaping the repository through absolute paths. OpenCode starts from a neutral temporary directory, points at the reviewed repo with `--dir`, disables project config through `OPENCODE_DISABLE_PROJECT_CONFIG=1`, and injects `OPENCODE_CONFIG_CONTENT`; permissions default to deny, allow read/grep/glob, preserve OpenCode's `.env` ask rules, and gate `websearch`/`webfetch` with `--no-web-search`. The injected config also clears command/instruction/plugin arrays and disables write/edit/bash/task/skill/todowrite tools without changing user auth storage. Cursor's documented headless path is print mode with JSON output and workspace-relative project-resource discovery. Because the CLI exposes no per-run flag that disables every current and future project instruction surface, autoreview requires `--cursor-allow-workspace-instructions` (or its environment equivalent) for every Cursor run. Project-local Cursor/Claude hook settings, project MCP config, and global Cursor MCP config remain hard refusals because hooks execute host commands and MCP tools cannot be constrained to read-only review access. Cursor capability probes run from neutral temporary directories with the sanitized engine environment. Review runs set documented `CURSOR_CONFIG_DIR` to an ephemeral configuration that allows workspace reads while denying shell commands and relative or absolute writes. The helper sends review prompts to OpenCode and Cursor over stdin rather than argv and extracts final structured JSON from terminal result/text events. OpenCode and Cursor reject `--no-tools`; Cursor also rejects `--no-web-search` because the CLI does not expose a documented per-run web-search disable flag.
|
||||
|
||||
## Context Efficiency
|
||||
|
||||
@@ -180,44 +307,28 @@ Run the helper directly so target selection, engine choice, structured validatio
|
||||
|
||||
## Helper
|
||||
|
||||
OpenClaw repo-local helper:
|
||||
After setting `AUTOREVIEW` and `AUTOREVIEW_HARNESS` above:
|
||||
|
||||
```bash
|
||||
.agents/skills/autoreview/scripts/autoreview --help
|
||||
```
|
||||
|
||||
On native Windows, invoke the extensionless Python helper through Python:
|
||||
|
||||
```powershell
|
||||
python .agents\skills\autoreview\scripts\autoreview --help
|
||||
"$AUTOREVIEW" --help
|
||||
```
|
||||
|
||||
The smoke harness has thin shell wrappers over a shared Python implementation:
|
||||
|
||||
```bash
|
||||
.agents/skills/autoreview/scripts/test-review-harness --fixture benign --engine codex
|
||||
"$AUTOREVIEW_HARNESS" --fixture benign --engine codex
|
||||
```
|
||||
|
||||
On native Windows, invoke the extensionless Python helper through Python:
|
||||
|
||||
```powershell
|
||||
.agents\skills\autoreview\scripts\test-review-harness.ps1 -Fixture benign -Engine codex
|
||||
python skills\autoreview\scripts\autoreview --help
|
||||
```
|
||||
|
||||
`agent-scripts` checkout helper:
|
||||
and the smoke harness:
|
||||
|
||||
```bash
|
||||
skills/autoreview/scripts/autoreview --help
|
||||
```
|
||||
|
||||
Global helper from `agent-scripts`:
|
||||
|
||||
```bash
|
||||
~/.codex/skills/agent-scripts/autoreview/scripts/autoreview --help
|
||||
```
|
||||
|
||||
If installed from `agent-scripts`, path is:
|
||||
|
||||
```bash
|
||||
/Users/steipete/Projects/agent-scripts/skills/autoreview/scripts/autoreview --help
|
||||
```powershell
|
||||
skills\autoreview\scripts\test-review-harness.ps1 -Fixture benign -Engine codex
|
||||
```
|
||||
|
||||
The helper:
|
||||
@@ -226,16 +337,22 @@ The helper:
|
||||
- accepts `--mode uncommitted` as an alias for `--mode local`
|
||||
- otherwise uses current PR base if `gh pr view` works
|
||||
- otherwise uses `origin/main` for non-main branches
|
||||
- supports `--engine codex`, `claude`, `droid`, `copilot`, and `cursor-agent`; default is `AUTOREVIEW_ENGINE` or `codex`; Codex should remain the default when nothing is set
|
||||
- does not fetch automatically during branch review; the selected base ref must already resolve locally
|
||||
- recognizes `--engine droid` only to fail closed with an isolation error; runnable engines are `codex`, `claude`, `copilot`, `pi`, `opencode`, and `cursor`; default is `AUTOREVIEW_ENGINE` or `codex`
|
||||
- resolves bare `git`, `gh`, reviewer, and PowerShell shell commands from absolute `PATH` entries only, never from the reviewed checkout; explicit relative `--*-bin` paths are resolved from the reviewed repository root
|
||||
- use `--mode commit --commit <ref>` for already-committed work, especially clean `main` after landing
|
||||
- should be left in `--mode auto` or forced to `--mode branch` for PR/branch work; do not force `--mode local` after committing
|
||||
- writes only to stdout unless `--output`, `--json-output`, or live streamed engine stderr is set
|
||||
- supports `--dry-run`, `--parallel-tests`, `--parallel-tests-shell`, `--prompt`, `--prompt-file`, `--dataset`, `--no-tools`, `--no-web-search`, and commit refs
|
||||
- supports `--stream-engine-output` or `AUTOREVIEW_STREAM_ENGINE_OUTPUT=1` for live engine text while preserving structured validation; Codex, Claude, and cursor-agent hide tool/file event details, emit compact activity summaries, and report usage at turn completion
|
||||
- supports opt-in review panels with `--panel` / `--reviewers`, plus per-engine `--model` and `--thinking`
|
||||
- allows read-only tools and web search by default where the selected CLI supports them; forbids nested review in the prompt; Codex is run through `codex exec` with read-only sandbox and structured output; cursor-agent is run through headless `--print` in ask mode with sandboxing enabled from a helper-owned temporary workspace
|
||||
- rejects `--no-web-search` for cursor-agent because the Cursor CLI does not expose a CLI-level web-search disable switch
|
||||
- supports `--dry-run`, `--parallel-tests`, `--parallel-tests-shell`, `--prompt`, repo-relative `--prompt-file`, repo-relative `--dataset`, `--no-tools`, `--no-web-search`, repeatable Codex-only `--codex-config key=value`, Codex-only `--codex-speed fast|flex|default`, and commit refs
|
||||
- supports `--stream-engine-output` or `AUTOREVIEW_STREAM_ENGINE_OUTPUT=1` for live engine text while preserving structured validation; Codex, Claude, and Cursor hide tool/file event details, emit compact activity summaries, and report usage at turn completion
|
||||
- supports opt-in review panels with `--panel` / `--reviewers`, plus per-engine `--model`, `--thinking`, and Claude `--fallback-model`
|
||||
- uses built-in model defaults `codex=gpt-5.5` and `claude=claude-fable-5`; honors `AUTOREVIEW_MODEL`, `AUTOREVIEW_THINKING`, `AUTOREVIEW_FALLBACK_MODEL`, and per-engine `AUTOREVIEW_<ENGINE>_MODEL` / `AUTOREVIEW_<ENGINE>_THINKING` environment overrides when CLI flags are omitted
|
||||
- allows read-only tools and web search by default where the selected CLI supports them; forbids nested review in the prompt; Codex is run through `codex exec` with auth-only user settings, read-only sandbox, reviewed-repo instruction/config/rule isolation flags, and structured output
|
||||
- runs Claude with `--safe-mode` (`v2.1.169+`), `--setting-sources user`, MCP disabled, explicit allowed tools, and `--fallback-model` when set, so reviewed-repo hooks/skills/MCP do not affect the review run while normal auth still works; managed settings policy can still apply
|
||||
- refuses Droid reviews until the CLI exposes a complete project-instruction and tool-isolation contract
|
||||
- runs Pi `v0.79.0+` from neutral temporary directories with `--no-approve`, `--no-session`, disabled Pi context/resource loading, and `--no-tools` because its built-in read tools are not repository-confined
|
||||
- runs OpenCode with `opencode run --dir <repo> --pure --format json` from a neutral temporary directory, forwards `--model` and `--variant`, injects deny-by-default permissions, disables project config loading, and passes the review prompt over stdin
|
||||
- runs Cursor only with mandatory trusted-workspace opt-in, uses `cursor-agent --print --mode ask --sandbox enabled --output-format json`, forwards `--model`, passes the review prompt over stdin, and always refuses project-local hooks/MCP
|
||||
- prints `review still running: <engine> elapsed=<seconds>s pid=<pid>` to stderr at long-running intervals while waiting for the selected review engine, unless streamed output or compact Codex activity has been visible recently
|
||||
- prints `autoreview clean: no accepted/actionable findings reported` when the selected review command exits 0
|
||||
- exits nonzero when accepted/actionable findings are present
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,410 @@
|
||||
#!/usr/bin/env python3
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import importlib.util
|
||||
import json
|
||||
import os
|
||||
import runpy
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import unittest
|
||||
from importlib.machinery import SourceFileLoader
|
||||
from pathlib import Path
|
||||
from unittest import mock
|
||||
|
||||
|
||||
SCRIPT_PATH = Path(__file__).with_name("autoreview")
|
||||
LOADER = SourceFileLoader("autoreview_module", str(SCRIPT_PATH))
|
||||
SPEC = importlib.util.spec_from_loader(LOADER.name, LOADER)
|
||||
assert SPEC is not None
|
||||
AUTOREVIEW = importlib.util.module_from_spec(SPEC)
|
||||
LOADER.exec_module(AUTOREVIEW)
|
||||
|
||||
|
||||
FINAL_REPORT = {
|
||||
"findings": [],
|
||||
"overall_correctness": "patch is correct",
|
||||
"overall_explanation": "clean",
|
||||
"overall_confidence": 0.9,
|
||||
}
|
||||
|
||||
DRAFT_REPORT = {
|
||||
"findings": [
|
||||
{
|
||||
"title": "Draft finding",
|
||||
"body": "draft",
|
||||
"priority": "P3",
|
||||
"confidence": 0.2,
|
||||
"category": "maintainability",
|
||||
"code_location": {"file_path": "draft.js", "line": 1},
|
||||
}
|
||||
],
|
||||
"overall_correctness": "patch is incorrect",
|
||||
"overall_explanation": "draft",
|
||||
"overall_confidence": 0.2,
|
||||
}
|
||||
|
||||
|
||||
class AutoreviewCursorTests(unittest.TestCase):
|
||||
def test_extract_json_prefers_terminal_result_event(self) -> None:
|
||||
stream = "\n".join(
|
||||
[
|
||||
json.dumps(
|
||||
{
|
||||
"type": "assistant",
|
||||
"message": {"role": "assistant", "content": [{"type": "text", "text": json.dumps(DRAFT_REPORT)}]},
|
||||
}
|
||||
),
|
||||
json.dumps(
|
||||
{
|
||||
"type": "result",
|
||||
"subtype": "success",
|
||||
"result": json.dumps(FINAL_REPORT),
|
||||
"session_id": "session-id",
|
||||
"request_id": "request-id",
|
||||
}
|
||||
),
|
||||
]
|
||||
)
|
||||
self.assertEqual(AUTOREVIEW.extract_json(stream), FINAL_REPORT)
|
||||
|
||||
def test_extract_json_can_fallback_to_assistant_message(self) -> None:
|
||||
stream = json.dumps(
|
||||
{
|
||||
"type": "assistant",
|
||||
"message": {"role": "assistant", "content": [{"type": "text", "text": json.dumps(FINAL_REPORT)}]},
|
||||
}
|
||||
)
|
||||
self.assertEqual(AUTOREVIEW.extract_json(stream), FINAL_REPORT)
|
||||
|
||||
def test_extract_json_does_not_fallback_past_bad_terminal_result(self) -> None:
|
||||
stream = "\n".join(
|
||||
[
|
||||
json.dumps(
|
||||
{
|
||||
"type": "assistant",
|
||||
"message": {"role": "assistant", "content": [{"type": "text", "text": json.dumps(FINAL_REPORT)}]},
|
||||
}
|
||||
),
|
||||
json.dumps(
|
||||
{
|
||||
"type": "result",
|
||||
"subtype": "success",
|
||||
"result": "not json",
|
||||
}
|
||||
),
|
||||
]
|
||||
)
|
||||
with self.assertRaises(SystemExit) as exc_info:
|
||||
AUTOREVIEW.extract_json(stream)
|
||||
self.assertIn("review engine result was not structured JSON", str(exc_info.exception))
|
||||
|
||||
|
||||
class AutoreviewCompatibilityTests(unittest.TestCase):
|
||||
@classmethod
|
||||
def setUpClass(cls) -> None:
|
||||
cls.home_dir = tempfile.TemporaryDirectory(prefix="autoreview-test-home.")
|
||||
cls.home_patch = mock.patch.object(Path, "home", return_value=Path(cls.home_dir.name))
|
||||
cls.home_patch.start()
|
||||
cls.home_keys = ("HOME", "USERPROFILE", "HOMEDRIVE", "HOMEPATH")
|
||||
cls.old_home_env = {key: os.environ.get(key) for key in cls.home_keys}
|
||||
os.environ["HOME"] = cls.home_dir.name
|
||||
os.environ["USERPROFILE"] = cls.home_dir.name
|
||||
os.environ.pop("HOMEDRIVE", None)
|
||||
os.environ.pop("HOMEPATH", None)
|
||||
|
||||
@classmethod
|
||||
def tearDownClass(cls) -> None:
|
||||
cls.home_patch.stop()
|
||||
for key, value in cls.old_home_env.items():
|
||||
if value is None:
|
||||
os.environ.pop(key, None)
|
||||
else:
|
||||
os.environ[key] = value
|
||||
cls.home_dir.cleanup()
|
||||
|
||||
def test_harness_opts_both_cursor_aliases_into_trusted_fixture(self) -> None:
|
||||
harness_path = SCRIPT_PATH.with_name("test-review-harness.py")
|
||||
namespace = runpy.run_path(str(harness_path))
|
||||
commands: list[list[str]] = []
|
||||
run_reviews = namespace["run_reviews"]
|
||||
with mock.patch.dict(
|
||||
run_reviews.__globals__,
|
||||
{
|
||||
"run": lambda command, _cwd: commands.append(command),
|
||||
"validate_prompt_policy": lambda _repo, _autoreview: None,
|
||||
},
|
||||
), tempfile.TemporaryDirectory(prefix="autoreview-harness-test.") as tmpdir:
|
||||
run_reviews(Path(tmpdir), SCRIPT_PATH.parent, "benign", ["cursor", "cursor-agent"])
|
||||
self.assertEqual(len(commands), 2)
|
||||
for command in commands:
|
||||
self.assertIn("--cursor-allow-workspace-instructions", command)
|
||||
|
||||
def test_cursor_agent_bin_cli_alias(self) -> None:
|
||||
with mock.patch.object(
|
||||
sys,
|
||||
"argv",
|
||||
["autoreview", "--cursor-agent-bin", "/tmp/legacy-cursor"],
|
||||
):
|
||||
args = AUTOREVIEW.parse_args()
|
||||
self.assertEqual(args.cursor_bin, "/tmp/legacy-cursor")
|
||||
|
||||
def test_cursor_agent_bin_env_alias(self) -> None:
|
||||
with mock.patch.dict(
|
||||
os.environ,
|
||||
{"CURSOR_AGENT_BIN": "/tmp/legacy-cursor"},
|
||||
clear=False,
|
||||
):
|
||||
os.environ.pop("CURSOR_BIN", None)
|
||||
with mock.patch.object(sys, "argv", ["autoreview"]):
|
||||
args = AUTOREVIEW.parse_args()
|
||||
self.assertEqual(args.cursor_bin, "/tmp/legacy-cursor")
|
||||
|
||||
def test_cursor_agent_reviewer_alias_normalizes_to_cursor(self) -> None:
|
||||
self.assertEqual(
|
||||
AUTOREVIEW.parse_reviewer_token("cursor-agent:auto"),
|
||||
("cursor", "auto", None),
|
||||
)
|
||||
|
||||
def test_cursor_agent_keyed_option_normalizes_to_cursor(self) -> None:
|
||||
self.assertEqual(
|
||||
AUTOREVIEW.parse_keyed_options(["cursor-agent=auto"], "model"),
|
||||
(None, {"cursor": "auto"}),
|
||||
)
|
||||
|
||||
def test_codex_config_status_exposes_keys_only(self) -> None:
|
||||
args = argparse.Namespace(codex_config=['model_provider="private-value"'])
|
||||
self.assertEqual(AUTOREVIEW.codex_config_keys(args), ["model_provider"])
|
||||
|
||||
def test_extract_json_accepts_dict_result_payload(self) -> None:
|
||||
payload = {
|
||||
"type": "result",
|
||||
"subtype": "success",
|
||||
"result": FINAL_REPORT,
|
||||
"session_id": "session-id",
|
||||
"request_id": "request-id",
|
||||
}
|
||||
self.assertEqual(AUTOREVIEW.extract_json(json.dumps(payload)), FINAL_REPORT)
|
||||
|
||||
def test_extract_json_accepts_result_string_with_preamble(self) -> None:
|
||||
payload = {
|
||||
"type": "result",
|
||||
"subtype": "success",
|
||||
"result": "Inspecting the diff first.\n" + json.dumps(FINAL_REPORT),
|
||||
}
|
||||
self.assertEqual(AUTOREVIEW.extract_json(json.dumps(payload)), FINAL_REPORT)
|
||||
|
||||
def test_extract_findings_json_from_text_prefers_last_findings_object(self) -> None:
|
||||
later_report = {
|
||||
"findings": [
|
||||
{
|
||||
"title": "Later finding",
|
||||
"body": "later",
|
||||
"priority": "P2",
|
||||
"confidence": 0.8,
|
||||
"category": "bug",
|
||||
"code_location": {"file_path": "later.js", "line": 2},
|
||||
}
|
||||
],
|
||||
"overall_correctness": "patch is incorrect",
|
||||
"overall_explanation": "later",
|
||||
"overall_confidence": 0.8,
|
||||
}
|
||||
text = f"{json.dumps(FINAL_REPORT)} separator {json.dumps(later_report)}"
|
||||
self.assertEqual(AUTOREVIEW.extract_findings_json_from_text(text), later_report)
|
||||
|
||||
def test_retry_filter_only_matches_parse_failures(self) -> None:
|
||||
self.assertTrue(AUTOREVIEW.is_structured_output_failure("review engine returned non-JSON output: nope"))
|
||||
self.assertTrue(AUTOREVIEW.is_structured_output_failure("review engine result was not structured JSON:\nnope"))
|
||||
self.assertFalse(AUTOREVIEW.is_structured_output_failure("review JSON missing required key: findings"))
|
||||
self.assertFalse(AUTOREVIEW.is_structured_output_failure("finding 0 has invalid priority"))
|
||||
|
||||
def test_cursor_workspace_instructions_fail_closed(self) -> None:
|
||||
with tempfile.TemporaryDirectory(prefix="autoreview-cursor-test.") as tmpdir:
|
||||
repo = Path(tmpdir)
|
||||
args = argparse.Namespace(
|
||||
thinking=None,
|
||||
tools=True,
|
||||
web_search=True,
|
||||
cursor_allow_workspace_instructions=False,
|
||||
cursor_bin="cursor-agent",
|
||||
model="auto",
|
||||
stream_engine_output=False,
|
||||
)
|
||||
with self.assertRaises(SystemExit) as exc_info:
|
||||
AUTOREVIEW.run_cursor(args, repo, "prompt")
|
||||
self.assertIn("requires --cursor-allow-workspace-instructions", str(exc_info.exception))
|
||||
|
||||
def test_cursor_local_mcp_requires_explicit_approval(self) -> None:
|
||||
with tempfile.TemporaryDirectory(prefix="autoreview-cursor-test.") as tmpdir:
|
||||
repo = Path(tmpdir)
|
||||
(repo / ".cursor").mkdir()
|
||||
(repo / ".cursor" / "mcp.json").write_text("{}\n")
|
||||
args = argparse.Namespace(
|
||||
thinking=None,
|
||||
tools=True,
|
||||
web_search=True,
|
||||
cursor_allow_workspace_instructions=True,
|
||||
cursor_bin="cursor-agent",
|
||||
model="auto",
|
||||
stream_engine_output=False,
|
||||
)
|
||||
with self.assertRaises(SystemExit) as exc_info:
|
||||
AUTOREVIEW.run_cursor(args, repo, "prompt")
|
||||
self.assertIn("cursor engine refused project-local MCP config", str(exc_info.exception))
|
||||
|
||||
def test_cursor_local_hooks_are_always_refused(self) -> None:
|
||||
with tempfile.TemporaryDirectory(prefix="autoreview-cursor-test.") as tmpdir:
|
||||
repo = Path(tmpdir)
|
||||
(repo / ".cursor").mkdir()
|
||||
(repo / ".cursor" / "hooks.json").write_text("{}\n")
|
||||
args = argparse.Namespace(
|
||||
thinking=None,
|
||||
tools=True,
|
||||
web_search=True,
|
||||
cursor_allow_workspace_instructions=True,
|
||||
cursor_bin="cursor-agent",
|
||||
model="auto",
|
||||
stream_engine_output=False,
|
||||
)
|
||||
with self.assertRaises(SystemExit) as exc_info:
|
||||
AUTOREVIEW.run_cursor(args, repo, "prompt")
|
||||
self.assertIn("cursor engine refused project-local hooks", str(exc_info.exception))
|
||||
|
||||
def test_cursor_local_permissions_are_always_refused(self) -> None:
|
||||
with tempfile.TemporaryDirectory(prefix="autoreview-cursor-test.") as tmpdir:
|
||||
repo = Path(tmpdir)
|
||||
(repo / ".cursor").mkdir()
|
||||
(repo / ".cursor" / "cli.json").write_text("{}\n")
|
||||
args = argparse.Namespace(
|
||||
thinking=None,
|
||||
tools=True,
|
||||
web_search=True,
|
||||
cursor_allow_workspace_instructions=True,
|
||||
cursor_bin="cursor-agent",
|
||||
model="auto",
|
||||
stream_engine_output=False,
|
||||
)
|
||||
with self.assertRaises(SystemExit) as exc_info:
|
||||
AUTOREVIEW.run_cursor(args, repo, "prompt")
|
||||
self.assertIn("cursor engine refused project-local permission config", str(exc_info.exception))
|
||||
|
||||
def test_cursor_command_uses_current_print_contract(self) -> None:
|
||||
with tempfile.TemporaryDirectory(prefix="autoreview-cursor-test.") as tmpdir:
|
||||
root = Path(tmpdir)
|
||||
repo = root / "repo"
|
||||
repo.mkdir()
|
||||
cursor_bin = root / "cursor-agent"
|
||||
record_path = root / "record.json"
|
||||
AUTOREVIEW.write_executable(cursor_bin, AUTOREVIEW.fake_cursor_script())
|
||||
args = argparse.Namespace(
|
||||
thinking=None,
|
||||
tools=True,
|
||||
web_search=True,
|
||||
cursor_allow_workspace_instructions=True,
|
||||
cursor_bin=str(cursor_bin),
|
||||
model=None,
|
||||
stream_engine_output=False,
|
||||
)
|
||||
old_record = os.environ.get("AUTOREVIEW_FAKE_RECORD")
|
||||
try:
|
||||
os.environ["AUTOREVIEW_FAKE_RECORD"] = str(record_path)
|
||||
AUTOREVIEW.run_cursor(args, repo, "prompt")
|
||||
finally:
|
||||
if old_record is None:
|
||||
os.environ.pop("AUTOREVIEW_FAKE_RECORD", None)
|
||||
else:
|
||||
os.environ["AUTOREVIEW_FAKE_RECORD"] = old_record
|
||||
record = json.loads(record_path.read_text())
|
||||
self.assertEqual(Path(record["cwd"]).resolve(), repo.resolve())
|
||||
self.assertEqual(record["stdin"], "prompt")
|
||||
self.assertIn("--print", record["argv"])
|
||||
self.assertIn("--output-format", record["argv"])
|
||||
self.assertIn("json", record["argv"])
|
||||
self.assertIn("--mode", record["argv"])
|
||||
self.assertIn("ask", record["argv"])
|
||||
self.assertIn("--sandbox", record["argv"])
|
||||
self.assertIn("enabled", record["argv"])
|
||||
for unsupported in ("--workspace", "--trust"):
|
||||
self.assertNotIn(unsupported, record["argv"])
|
||||
|
||||
def test_cursor_engine_runs_end_to_end_with_sanitized_environment(self) -> None:
|
||||
with tempfile.TemporaryDirectory(prefix="autoreview-cursor-e2e.") as tmpdir:
|
||||
root = Path(tmpdir)
|
||||
repo = root / "repo"
|
||||
repo.mkdir()
|
||||
subprocess.run(["git", "init", "--quiet"], cwd=repo, check=True)
|
||||
subprocess.run(["git", "config", "user.name", "AutoReview Test"], cwd=repo, check=True)
|
||||
subprocess.run(["git", "config", "user.email", "autoreview@example.invalid"], cwd=repo, check=True)
|
||||
source = repo / "example.txt"
|
||||
source.write_text("before\n")
|
||||
subprocess.run(["git", "add", "example.txt"], cwd=repo, check=True)
|
||||
subprocess.run(["git", "commit", "--quiet", "-m", "test: seed fixture"], cwd=repo, check=True)
|
||||
source.write_text("after\n")
|
||||
|
||||
cursor_bin = root / "cursor-agent"
|
||||
record_path = root / "record.json"
|
||||
AUTOREVIEW.write_executable(cursor_bin, AUTOREVIEW.fake_cursor_script())
|
||||
env = os.environ.copy()
|
||||
env.update(
|
||||
{
|
||||
"AUTOREVIEW_FAKE_RECORD": str(record_path),
|
||||
"AUTOREVIEW_FAKE_CURSOR_INVOCATIONS": str(root / "cursor-invocations.jsonl"),
|
||||
"GIT_CONFIG_GLOBAL": str(root / "hostile-gitconfig"),
|
||||
"NODE_OPTIONS": "--require=hostile.js",
|
||||
"PYTHONPATH": str(root / "hostile-python"),
|
||||
"PATH": f"{repo}{os.pathsep}{env.get('PATH', '')}",
|
||||
}
|
||||
)
|
||||
result = subprocess.run(
|
||||
[
|
||||
sys.executable,
|
||||
str(SCRIPT_PATH),
|
||||
"--mode",
|
||||
"local",
|
||||
"--engine",
|
||||
"cursor",
|
||||
"--cursor-bin",
|
||||
str(cursor_bin),
|
||||
"--cursor-allow-workspace-instructions",
|
||||
],
|
||||
cwd=repo,
|
||||
env=env,
|
||||
text=True,
|
||||
capture_output=True,
|
||||
check=False,
|
||||
)
|
||||
|
||||
self.assertEqual(result.returncode, 0, result.stderr)
|
||||
self.assertIn("autoreview clean: no accepted/actionable findings reported", result.stdout)
|
||||
record = json.loads(record_path.read_text())
|
||||
self.assertEqual(Path(record["cwd"]).resolve(), repo.resolve())
|
||||
self.assertIn("diff --git a/example.txt b/example.txt", record["stdin"])
|
||||
self.assertIn("-before", record["stdin"])
|
||||
self.assertIn("+after", record["stdin"])
|
||||
self.assertEqual(record["environment"]["GIT_CONFIG_GLOBAL"], None)
|
||||
self.assertEqual(record["environment"]["NODE_OPTIONS"], None)
|
||||
self.assertEqual(record["environment"]["PYTHONPATH"], None)
|
||||
self.assertNotIn(str(repo), record["environment"]["PATH"].split(os.pathsep))
|
||||
cursor_config_dir = Path(record["environment"]["CURSOR_CONFIG_DIR"])
|
||||
self.assertFalse(cursor_config_dir.exists())
|
||||
cursor_config = json.loads(record["cursor_config"])
|
||||
self.assertEqual(cursor_config["permissions"]["allow"], ["Read(**)"])
|
||||
self.assertEqual(
|
||||
cursor_config["permissions"]["deny"],
|
||||
["Shell(*)", "Write(**)", "Write(/**)"],
|
||||
)
|
||||
|
||||
invocations = [json.loads(line) for line in (root / "cursor-invocations.jsonl").read_text().splitlines()]
|
||||
help_invocation = next(invocation for invocation in invocations if "--help" in invocation["argv"])
|
||||
self.assertNotEqual(Path(help_invocation["cwd"]).resolve(), repo.resolve())
|
||||
self.assertEqual(help_invocation["environment"]["GIT_CONFIG_GLOBAL"], None)
|
||||
self.assertEqual(help_invocation["environment"]["NODE_OPTIONS"], None)
|
||||
self.assertEqual(help_invocation["environment"]["PYTHONPATH"], None)
|
||||
self.assertNotIn(str(repo), help_invocation["environment"]["PATH"].split(os.pathsep))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -3,7 +3,7 @@ param(
|
||||
[ValidateSet('malicious', 'benign')]
|
||||
[string] $Fixture,
|
||||
|
||||
[ValidateSet('codex', 'claude', 'droid', 'copilot', 'cursor-agent')]
|
||||
[ValidateSet('codex', 'claude', 'droid', 'copilot', 'pi', 'opencode', 'cursor', 'cursor-agent')]
|
||||
[string[]] $Engine,
|
||||
|
||||
[Alias('h')]
|
||||
|
||||
@@ -13,7 +13,7 @@ from collections.abc import Callable
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
ENGINES = ("codex", "claude", "droid", "copilot", "cursor-agent")
|
||||
ENGINES = ("codex", "claude", "droid", "copilot", "pi", "opencode", "cursor", "cursor-agent")
|
||||
DEFAULT_ENGINES = ("codex", "claude")
|
||||
|
||||
MALICIOUS_INITIAL = """export function uploadPath(name) {
|
||||
@@ -175,6 +175,10 @@ def run_reviews(repo: Path, script_dir: Path, fixture: str, engines: list[str])
|
||||
"--prompt",
|
||||
MALICIOUS_PROMPT if fixture == "malicious" else BENIGN_PROMPT,
|
||||
]
|
||||
if engine in {"cursor", "cursor-agent"}:
|
||||
# The harness owns this temporary fixture, so it can make the
|
||||
# trusted-workspace assertion required by Cursor reviews.
|
||||
command.append("--cursor-allow-workspace-instructions")
|
||||
if fixture == "malicious":
|
||||
command.extend(["--require-finding", "command", "--expect-findings"])
|
||||
run(command, repo)
|
||||
|
||||
@@ -0,0 +1,369 @@
|
||||
#!/usr/bin/env python3
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import os
|
||||
import runpy
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import unittest
|
||||
from unittest import mock
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
SCRIPT = Path(__file__).resolve().parents[1] / "scripts" / "autoreview"
|
||||
|
||||
|
||||
def load_helper() -> dict[str, object]:
|
||||
return runpy.run_path(str(SCRIPT), run_name="autoreview_under_test")
|
||||
|
||||
|
||||
def git(repo: Path, *args: str) -> str:
|
||||
env = os.environ.copy()
|
||||
env.update(
|
||||
{
|
||||
"GIT_AUTHOR_NAME": "Autoreview Test",
|
||||
"GIT_AUTHOR_EMAIL": "autoreview@example.invalid",
|
||||
"GIT_COMMITTER_NAME": "Autoreview Test",
|
||||
"GIT_COMMITTER_EMAIL": "autoreview@example.invalid",
|
||||
}
|
||||
)
|
||||
result = subprocess.run(
|
||||
["git", *args],
|
||||
cwd=repo,
|
||||
env=env,
|
||||
check=True,
|
||||
text=True,
|
||||
stdout=subprocess.PIPE,
|
||||
stderr=subprocess.PIPE,
|
||||
)
|
||||
return result.stdout
|
||||
|
||||
|
||||
def init_repo(tempdir: Path) -> Path:
|
||||
repo = tempdir / "repo"
|
||||
repo.mkdir()
|
||||
git(repo, "init", "-q")
|
||||
git(repo, "config", "user.name", "Autoreview Test")
|
||||
git(repo, "config", "user.email", "autoreview@example.invalid")
|
||||
return repo
|
||||
|
||||
|
||||
class AutoreviewHardeningTests(unittest.TestCase):
|
||||
def setUp(self) -> None:
|
||||
self.helper = load_helper()
|
||||
|
||||
def test_local_bundle_blocks_sensitive_untracked_file(self) -> None:
|
||||
with tempfile.TemporaryDirectory() as tempdir:
|
||||
repo = init_repo(Path(tempdir))
|
||||
(repo / ".env").write_text("placeholder=true\n", encoding="utf-8")
|
||||
|
||||
with self.assertRaisesRegex(SystemExit, "untracked sensitive files"):
|
||||
self.helper["local_bundle"](repo)
|
||||
|
||||
def test_local_bundle_omits_safe_untracked_binary_content(self) -> None:
|
||||
with tempfile.TemporaryDirectory() as tempdir:
|
||||
repo = init_repo(Path(tempdir))
|
||||
(repo / "image.bin").write_bytes(b"\x89PNG\r\n\0binary-content")
|
||||
|
||||
bundle, truncated = self.helper["local_bundle"](repo)
|
||||
|
||||
self.assertIn("## image.bin\n[binary file omitted]", bundle)
|
||||
self.assertFalse(truncated)
|
||||
|
||||
def test_branch_bundle_rejects_unsafe_or_unknown_base_before_diff(self) -> None:
|
||||
with tempfile.TemporaryDirectory() as tempdir:
|
||||
repo = init_repo(Path(tempdir))
|
||||
(repo / "tracked.txt").write_text("base\n", encoding="utf-8")
|
||||
git(repo, "add", "tracked.txt")
|
||||
git(repo, "commit", "-q", "-m", "base")
|
||||
|
||||
with self.assertRaisesRegex(SystemExit, "unsafe base ref"):
|
||||
self.helper["branch_bundle"](repo, "--help")
|
||||
with self.assertRaisesRegex(SystemExit, "unknown base ref"):
|
||||
self.helper["branch_bundle"](repo, "origin/main")
|
||||
|
||||
def test_git_path_list_preserves_newline_filenames(self) -> None:
|
||||
if os.name == "nt":
|
||||
self.skipTest("Windows filesystems do not support newline path components")
|
||||
with tempfile.TemporaryDirectory() as tempdir:
|
||||
repo = init_repo(Path(tempdir))
|
||||
rel = "line\nbreak.txt"
|
||||
(repo / rel).write_text("content\n", encoding="utf-8")
|
||||
git(repo, "add", rel)
|
||||
|
||||
paths = self.helper["git_path_list"](repo, "ls-files", "-z")
|
||||
|
||||
self.assertIn(rel, paths)
|
||||
|
||||
def test_bounded_truncates_large_bundle_component(self) -> None:
|
||||
bounded = self.helper["bounded"]("x" * 25, 10)
|
||||
|
||||
self.assertEqual(bounded, "x" * 10 + "\n\n[truncated at 10 characters]\n")
|
||||
|
||||
def test_pi_refuses_truncated_review_input(self) -> None:
|
||||
reviewer = argparse.Namespace(engine="pi", tools=True)
|
||||
|
||||
with self.assertRaisesRegex(SystemExit, "pi engine refused truncated review input"):
|
||||
self.helper["ensure_reviewer_input_complete"](
|
||||
reviewer,
|
||||
True,
|
||||
)
|
||||
|
||||
self.helper["ensure_reviewer_input_complete"](
|
||||
reviewer,
|
||||
False,
|
||||
)
|
||||
self.helper["ensure_reviewer_input_complete"](
|
||||
argparse.Namespace(engine="codex", tools=True),
|
||||
True,
|
||||
)
|
||||
with self.assertRaisesRegex(SystemExit, "claude engine refused truncated review input"):
|
||||
self.helper["ensure_reviewer_input_complete"](
|
||||
argparse.Namespace(engine="claude", tools=True),
|
||||
True,
|
||||
)
|
||||
with self.assertRaisesRegex(SystemExit, "droid engine refused truncated review input"):
|
||||
self.helper["ensure_reviewer_input_complete"](
|
||||
argparse.Namespace(engine="droid", tools=False),
|
||||
True,
|
||||
)
|
||||
|
||||
def test_safe_git_env_preserves_trusted_platform_and_helper_paths(self) -> None:
|
||||
with tempfile.TemporaryDirectory() as tempdir:
|
||||
root = Path(tempdir)
|
||||
repo = init_repo(root)
|
||||
repo_bin = repo / "bin"
|
||||
trusted_bin = root / "trusted-bin"
|
||||
repo_bin.mkdir()
|
||||
trusted_bin.mkdir()
|
||||
with mock.patch.dict(
|
||||
os.environ,
|
||||
{
|
||||
"PATH": os.pathsep.join((str(repo_bin), str(trusted_bin))),
|
||||
"SYSTEMROOT": "C:\\Windows",
|
||||
"GIT_DIR": str(repo / ".git"),
|
||||
"OPENAI_API_KEY": "must-not-reach-git",
|
||||
},
|
||||
clear=False,
|
||||
):
|
||||
env = self.helper["safe_git_env"](repo)
|
||||
|
||||
self.assertNotIn(str(repo_bin.resolve()), env["PATH"].split(os.pathsep))
|
||||
self.assertIn(str(trusted_bin.resolve()), env["PATH"].split(os.pathsep))
|
||||
self.assertEqual(env["SYSTEMROOT"], "C:\\Windows")
|
||||
self.assertNotIn("GIT_DIR", env)
|
||||
self.assertNotIn("OPENAI_API_KEY", env)
|
||||
|
||||
def test_boolean_environment_values_fail_closed(self) -> None:
|
||||
with mock.patch.dict(os.environ, {"AUTOREVIEW_TEST_BOOL": "flase"}):
|
||||
with self.assertRaisesRegex(SystemExit, "invalid boolean environment value"):
|
||||
self.helper["env_truthy"]("AUTOREVIEW_TEST_BOOL")
|
||||
|
||||
def test_droid_fails_closed_without_complete_isolation(self) -> None:
|
||||
with tempfile.TemporaryDirectory() as tempdir:
|
||||
repo = init_repo(Path(tempdir))
|
||||
(repo / "AGENTS.md").write_text("hostile instructions\n", encoding="utf-8")
|
||||
|
||||
with self.assertRaisesRegex(SystemExit, "droid engine is unavailable"):
|
||||
self.helper["run_droid"](argparse.Namespace(), repo, "prompt")
|
||||
|
||||
def test_prompt_file_keeps_recoverable_repo_path(self) -> None:
|
||||
with tempfile.TemporaryDirectory() as tempdir:
|
||||
repo = init_repo(Path(tempdir))
|
||||
(repo / "review.md").write_text("review context\n", encoding="utf-8")
|
||||
args = argparse.Namespace(prompt=[], prompt_file=["review.md"])
|
||||
|
||||
prompt, truncated = self.helper["load_extra_prompt"](args, repo)
|
||||
|
||||
self.assertIn("# Prompt file: review.md", prompt)
|
||||
self.assertFalse(truncated)
|
||||
|
||||
def test_cursor_refuses_global_mcp_config(self) -> None:
|
||||
with tempfile.TemporaryDirectory() as tempdir:
|
||||
root = Path(tempdir)
|
||||
repo = init_repo(root)
|
||||
global_mcp = root / ".cursor" / "mcp.json"
|
||||
global_mcp.parent.mkdir()
|
||||
global_mcp.write_text("{}\n", encoding="utf-8")
|
||||
args = argparse.Namespace(
|
||||
thinking=None,
|
||||
tools=True,
|
||||
web_search=True,
|
||||
cursor_allow_workspace_instructions=True,
|
||||
)
|
||||
|
||||
with mock.patch.object(Path, "home", return_value=root):
|
||||
with self.assertRaisesRegex(SystemExit, "cursor engine refused global MCP config"):
|
||||
self.helper["run_cursor"](args, repo, "prompt")
|
||||
|
||||
def test_read_text_truncates_without_scanning_tail(self) -> None:
|
||||
with tempfile.TemporaryDirectory() as tempdir:
|
||||
path = Path(tempdir) / "large.txt"
|
||||
path.write_bytes(b"x" * 200_000 + b"\0tail")
|
||||
|
||||
text = self.helper["read_text"](path)
|
||||
|
||||
self.assertIn("[truncated at 180000 characters]", text)
|
||||
self.assertNotEqual(text, "[binary file omitted]")
|
||||
|
||||
def test_evidence_file_must_be_repo_relative_and_not_symlinked(self) -> None:
|
||||
with tempfile.TemporaryDirectory() as tempdir:
|
||||
root = Path(tempdir)
|
||||
repo = init_repo(root)
|
||||
outside = root / "outside.md"
|
||||
outside.write_text("outside\n", encoding="utf-8")
|
||||
|
||||
with self.assertRaisesRegex(SystemExit, "repo-relative"):
|
||||
self.helper["validate_evidence_file"](repo, str(outside), "--prompt-file")
|
||||
|
||||
target = repo / "notes.md"
|
||||
target.write_text("notes\n", encoding="utf-8")
|
||||
link = repo / "link.md"
|
||||
try:
|
||||
link.symlink_to(target)
|
||||
except OSError as exc:
|
||||
if os.name == "nt" and getattr(exc, "winerror", None) == 1314:
|
||||
self.skipTest("Windows symlink privilege is not available")
|
||||
raise
|
||||
with self.assertRaisesRegex(SystemExit, "symlinked"):
|
||||
self.helper["validate_evidence_file"](repo, "link.md", "--dataset")
|
||||
|
||||
def test_safe_engine_env_strips_process_injection_variables(self) -> None:
|
||||
old = os.environ.copy()
|
||||
with tempfile.TemporaryDirectory() as tempdir:
|
||||
repo = init_repo(Path(tempdir))
|
||||
try:
|
||||
os.environ["GIT_DIR"] = "/tmp/unsafe-git-dir"
|
||||
os.environ["GIT_CONFIG_COUNT"] = "99"
|
||||
os.environ["DYLD_INSERT_LIBRARIES"] = "/tmp/unsafe.dylib"
|
||||
os.environ["NODE_OPTIONS"] = "--require=/tmp/unsafe.js"
|
||||
|
||||
env = self.helper["safe_engine_env"](repo)
|
||||
|
||||
self.assertNotEqual(env.get("GIT_DIR"), "/tmp/unsafe-git-dir")
|
||||
self.assertEqual(
|
||||
env["GIT_CONFIG_COUNT"],
|
||||
str(len(self.helper["ENGINE_GIT_CONFIG_OVERRIDES"])),
|
||||
)
|
||||
self.assertNotIn("DYLD_INSERT_LIBRARIES", env)
|
||||
self.assertNotIn("NODE_OPTIONS", env)
|
||||
finally:
|
||||
os.environ.clear()
|
||||
os.environ.update(old)
|
||||
|
||||
def test_safe_engine_env_excludes_repo_local_path_entries(self) -> None:
|
||||
old_path = os.environ.get("PATH", "")
|
||||
with tempfile.TemporaryDirectory() as tempdir:
|
||||
repo = init_repo(Path(tempdir))
|
||||
os.environ["PATH"] = f"{repo}{os.pathsep}{old_path}"
|
||||
try:
|
||||
env = self.helper["safe_engine_env"](repo)
|
||||
finally:
|
||||
os.environ["PATH"] = old_path
|
||||
|
||||
self.assertNotIn(str(repo.resolve()), env["PATH"].split(os.pathsep))
|
||||
|
||||
def test_safe_engine_env_ignores_inaccessible_path_entries(self) -> None:
|
||||
old_path = os.environ.get("PATH", "")
|
||||
with tempfile.TemporaryDirectory() as tempdir:
|
||||
root = Path(tempdir)
|
||||
repo = init_repo(root)
|
||||
blocked = root / "blocked"
|
||||
os.environ["PATH"] = f"{blocked}{os.pathsep}{old_path}"
|
||||
original_exists = Path.exists
|
||||
|
||||
def fake_exists(path: Path) -> bool:
|
||||
if str(path) == str(blocked):
|
||||
raise PermissionError("access denied")
|
||||
return original_exists(path)
|
||||
|
||||
try:
|
||||
with mock.patch.object(Path, "exists", fake_exists):
|
||||
env = self.helper["safe_engine_env"](repo)
|
||||
finally:
|
||||
os.environ["PATH"] = old_path
|
||||
|
||||
self.assertNotIn(str(blocked), env["PATH"].split(os.pathsep))
|
||||
|
||||
def test_run_with_heartbeat_replaces_undecodable_engine_output(self) -> None:
|
||||
with tempfile.TemporaryDirectory() as tempdir:
|
||||
result = self.helper["run_with_heartbeat"](
|
||||
[
|
||||
sys.executable,
|
||||
"-c",
|
||||
"import sys; sys.stdout.buffer.write(b'\\x90\\n')",
|
||||
],
|
||||
Path(tempdir),
|
||||
label="decode-test",
|
||||
heartbeat_seconds=1,
|
||||
)
|
||||
|
||||
self.assertEqual(result.returncode, 0, result.stderr)
|
||||
self.assertIn("\ufffd", result.stdout)
|
||||
|
||||
def test_large_repo_relative_evidence_file_is_truncated(self) -> None:
|
||||
with tempfile.TemporaryDirectory() as tempdir:
|
||||
repo = init_repo(Path(tempdir))
|
||||
evidence = repo / "evidence.txt"
|
||||
evidence.write_text("x" * 600_000, encoding="utf-8")
|
||||
|
||||
_, content, truncated = self.helper["validate_evidence_file"](repo, "evidence.txt", "--dataset")
|
||||
|
||||
self.assertIn("[truncated at 180000 characters]", content)
|
||||
self.assertTrue(truncated)
|
||||
|
||||
def test_copilot_allows_web_fetch_only_when_web_search_is_enabled(self) -> None:
|
||||
captured: list[list[str]] = []
|
||||
|
||||
def fake_run_with_heartbeat(
|
||||
cmd: list[str],
|
||||
cwd: Path,
|
||||
**kwargs: object,
|
||||
) -> subprocess.CompletedProcess[str]:
|
||||
captured.append(cmd)
|
||||
return subprocess.CompletedProcess(cmd, 0, '{"findings":[]}', "")
|
||||
|
||||
self.helper["run_copilot"].__globals__["run_with_heartbeat"] = fake_run_with_heartbeat
|
||||
self.helper["run_copilot"].__globals__["resolve_command"] = (
|
||||
lambda command, repo: f"/resolved/{command}"
|
||||
)
|
||||
args = argparse.Namespace(
|
||||
copilot_bin="copilot",
|
||||
thinking=None,
|
||||
tools=True,
|
||||
model=None,
|
||||
web_search=False,
|
||||
stream_engine_output=False,
|
||||
)
|
||||
|
||||
self.helper["run_copilot"](args, Path("/repo"), "prompt")
|
||||
|
||||
self.assertNotIn("--allow-tool=web_fetch", captured[-1])
|
||||
self.assertFalse(any(arg == "--allow-all-urls" for arg in captured[-1]))
|
||||
|
||||
args.web_search = True
|
||||
self.helper["run_copilot"](args, Path("/repo"), "prompt")
|
||||
|
||||
self.assertIn("--allow-tool=web_fetch", captured[-1])
|
||||
self.assertIn("--allow-all-urls", captured[-1])
|
||||
|
||||
def test_self_test_shortcut_runs_deterministic_checks(self) -> None:
|
||||
command = [str(SCRIPT), "--self-test"]
|
||||
if os.name == "nt":
|
||||
command = [sys.executable, str(SCRIPT), "--self-test"]
|
||||
result = subprocess.run(
|
||||
command,
|
||||
check=False,
|
||||
text=True,
|
||||
stdout=subprocess.PIPE,
|
||||
stderr=subprocess.PIPE,
|
||||
)
|
||||
|
||||
self.assertEqual(result.returncode, 0, result.stderr)
|
||||
self.assertIn("autoreview engine isolation self-test: ok", result.stdout)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
Reference in New Issue
Block a user