Skip to content

Standardize the Self-selected Top50 benchmark#72

Merged
isPANN merged 9 commits into
mainfrom
codex/issue-68-calibration-release
Jul 20, 2026
Merged

Standardize the Self-selected Top50 benchmark#72
isPANN merged 9 commits into
mainfrom
codex/issue-68-calibration-release

Conversation

@isPANN

@isPANN isPANN commented Jul 20, 2026

Copy link
Copy Markdown
Collaborator

Closes #65
Closes #66
Closes #67
Closes #68
Closes #73

Summary

This single PR replaces the previous four-PR stack and delivers the complete Standardized Model API / Self-selected Top50 benchmark:

  • introduce the problem-reductions repository and round-trip bug semantics directly in the triage prompt, then freeze a source-only Top50 with concrete hypotheses
  • run 50 fresh sequential isolated rule episodes with fixed logical limits and exactly two submit attempts per selected rule
  • define the benchmark limits once in code; remove the external budget file and redundant contract/policy version identifiers
  • keep configuration claims out of submissions while validating actual private ledgers against the built-in limits
  • package every shell and pred result with deterministic ANSI/progress/repetition cleanup, diagnostic-first previews capped at 10,000 characters, and read-only bounded raw logs
  • bind exact observation metadata bidirectionally to shell and pred actions and reject outcome, command, size, or record tampering
  • independently re-score accepted certificates and publish aggregate-only results
  • isolate historical whole-repository results from the standardized Top50 table using the existing agent mode
  • expose no logical-budget configuration and reject custom prompt, strategy, backend, model-kwargs, or pin overrides before scored model access
  • verify baked source bytes, target marker, and trusted pred path
  • retain the internal non-ranking calibration record and offline consistency checker
  • migrate runner, schemas, CI, docs, skills, Docker metadata, and leaderboard presentation

Calibration scope

The checked-in grid is a human-reviewed bounded-prefix replay record from privately retained internal pilot trajectories. The offline command verifies the checked-in record, complete grid, selected knee, report, and built-in parameters; it deliberately does not claim to reconstruct private raw trajectories in CI.

Verification

  • pytest -q — 333 passed, 5 skipped
  • python3 -m benchmark.calibrate_budget --check benchmark/docs/budget-calibration.json
  • changed-file Ruff checks
  • git diff --check
  • deterministic common-command demonstration recorded under runscribe/A-observation-packaging/A1-policy-shortens-noisy-common-commands/runs/2026-07-20T17-24-32__final-policy-clean

Do not merge until CI is green and the user explicitly approves.

@isPANN isPANN changed the title Freeze the first Top50 evidence contract Standardize and freeze the Self-selected Top50 benchmark Jul 20, 2026
@isPANN
isPANN changed the base branch from codex/issue-67-top50-contract to main July 20, 2026 06:39
@isPANN isPANN changed the title Standardize and freeze the Self-selected Top50 benchmark Standardize the Self-selected Top50 benchmark Jul 20, 2026
@isPANN
isPANN merged commit 672b214 into main Jul 20, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant