Refine and regression-test the unified benchmark prompt

[isPANN]

Background

PR #72 converges the repository on one benchmark protocol: source-only Top50 triage followed by 50 isolated rule episodes. The model-facing contract now lives in benchmark/agent_config.yaml, and its bytes are frozen into each result through prompt_id.

The next quality lever is the prompt itself. It should make the intended workflow obvious and reduce wasted turns and protocol mistakes without reintroducing the removed whole-repo/single-rule, coding-agent, CLI/API, or historical-run modes.

Objective

Refine and regression-test the unified triage and episode prompts so an agent can use its limited budget efficiently while preserving the benchmark's frozen protocol.

Use existing non-ranking trajectories to identify concrete prompt-related failure modes first; every material wording change should map to an observed ambiguity, wasted action, or invalid action rather than adding generic advice.

Interface (Input → Output)

• Triage input: repository path, canonical rule inventory, and authoritative budget status.
• Triage output: exactly 50 unique canonical rules, optionally with concise hypotheses, frozen once with commit-top50 shortlist.json. No pred or submit use.
• Episode input: one frozen rule, its triage hypothesis, position, source path, and authoritative budget status.
• Episode output: shell actions investigating only that rule; if a bug is reproduced, a certificate under $PRB_ARTIFACT_DIR submitted with submit <certificate.json> within the two-attempt limit.
• Every turn: exactly one bash command. The runner's counters remain authoritative.

The prompt may change. Runtime budgets, tool permissions, shortlist size, certificate schema, scoring, and rankability rules may not.

Deliverables

• Revise benchmark/agent_config.yaml for clarity, concision, and phase-specific guidance.
• Add render-level prompt contract tests that exercise both branches through the real config/template path.
• Record a short before/after table in the PR description: observed trajectory failure → prompt change → expected behavior.
• Run a matched, non-ranking baseline/candidate smoke comparison using the same model, library commit, inference settings, and benchmark budgets. Report protocol-error and wasted-action counts separately for triage and episodes; do not claim a scoring improvement from an unmatched run.

Verification

Run:

pytest -q benchmark/tests/test_prompt_contract.py benchmark/tests/test_top50_runner.py benchmark/tests/test_run_top50.py benchmark/tests/test_top50_submission.py

Expected result: all tests pass, and test_prompt_contract.py demonstrates that:

1. both phase prompts render without unresolved template variables;
2. triage exposes the inventory and exact shortlist/commit contract but no episode-only instructions;
3. an episode exposes only its assigned rule, hypothesis, certificate/submit workflow, and two-attempt limit but no triage-only instructions;
4. the shared one-command and authoritative-budget rules appear in both rendered prompts;
5. no removed execution mode or deprecated skill name returns.

Review the attached matched baseline/candidate artifacts and confirm the candidate introduces no new protocol-error category and does not increase the total number of prompt-attributable invalid or wasted actions.

Negative control: temporarily remove the triage commit-top50 contract or the episode rule-isolation/submit contract from the candidate prompt. The new prompt contract test must fail with a phase-specific assertion.

Dependency

Depends on PR #72.

Out of scope

• Changing benchmark budgets, scoring, rankability, or submission verification.
• Adding another runner or execution mode.
• Renaming or splitting $run-benchmark again.