Add contamination file-path identification (SWE-Bench Illusion methodology)#738
Open
Groenbech96 wants to merge 15 commits into
Open
Add contamination file-path identification (SWE-Bench Illusion methodology)#738Groenbech96 wants to merge 15 commits into
Groenbech96 wants to merge 15 commits into
Conversation
Implements the SWE-Bench Illusion (arXiv:2506.12286) context-free bug localization probe as a black-box memorization detector that needs no access to model training data. The model is given ONLY the bug report (no repository, no code) and asked which file(s) contain the bug; predictions are scored against the files the gold fix patch touches. Because AL has rigid file-naming conventions, absolute accuracy is convention-inflated, so the probe reports the pre-cutoff vs control delta as the actual contamination signal. - Pure, tested core: prompt builder, gold-file extraction, prediction parsing, exact/basename scoring, aggregation, and temporal cutoff split. - Runner: single-shot Copilot CLI in an isolated empty dir (repo withheld). - CLI: `bcbench contamination filepath-probe` and `... summarize --cutoff`. - Workflow: contamination-probe.yml (dispatch + weekly), no BC container/build. - 25 unit tests for the pure core. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
Review fixes for the file-path identification probe: - Isolation (Important): restrict the Copilot CLI to the `write` tool only (--silent --available-tools=write), removing shell/read/fetch so the model cannot reach the checked-out repo or the gold patch. Parse the answer from stdout instead of a written file. Verified live end-to-end. - Scheduled cutoff (Important): weekly runs fall back to the CONTAMINATION_CUTOFF repo variable so the pre-vs-control delta is still reported. - Timeout (Important): add PROBE_TIMEOUT_SECONDS (15m) below the 20m workflow step timeout so a hung probe records an error result before the step is killed. - Add missing contamination/__init__.py with the package public API. - Validate top_k >= 1 in the CLI. - Replace object-typed lambdas + type: ignore with Callable in aggregate_results. - Add runner unit tests: save/load round-trip and mocked success/error paths. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Groenbech96
commented
Jul 8, 2026
- Collapse implicitly-concatenated multi-line strings in _build_markdown_report into single literals (CodeQL: implicit string concatenation in a list). - Use a single import style for bcbench.contamination.runner in the probe tests (CodeQL: module imported with both 'import' and 'import from'). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Cast the JSON dict to dict[str, object] in _candidate_path so ty accepts the string key on .get() (narrowed-from-object dict had key type Never). - Remove a now-unused `# ty: ignore[invalid-argument-type]` in the nl2al pipeline test that ty 0.0.33 flags as unused, unblocking the lint check. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The SWE-Bench Illusion paper (arXiv:2506.12286) calls these "diagnostic tasks" and names this specific one "file-path identification"; "probe" was ad-hoc. Rename throughout to match the paper: - CLI: `contamination filepath-probe` -> `contamination file-path-identification` - module filepath_probe.py -> filepath_identification.py - FilePathProbe* -> FilePathIdentification*, ProbeAggregate -> IdentificationAggregate - run/save/load/build helpers and PROBE_* constants renamed accordingly - result suffix *.filepath-probe.jsonl -> *.file-path-identification.jsonl - workflow renamed to contamination-file-path-identification.yml - tests renamed to test_filepath_identification.py No behavioural change. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Groenbech96
force-pushed
the
contamination-filepath-probe
branch
from
cf37596 to
c5cef89
Compare
Groenbech96
changed the title
Groenbech96
force-pushed
the
contamination-filepath-probe
branch
from
137ea40 to
e3452a9
Compare
Two gratuitous divergences from house patterns, surfaced in review: - Move the hardcoded IDENTIFICATION_TIMEOUT_SECONDS into TimeoutConfig (`filepath_identification`), where every other timeout lives; the runner now reads `_config.timeout.filepath_identification`. - De-duplicate the triplicated Copilot-CLI lookup into a shared `bcbench.copilot_cli.find_copilot`, now used by the identification runner, the code-review judge, and the copilot agent (the judge keeps a `_find_copilot` alias so existing test patch points still work). No behavioural change. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Groenbech96
force-pushed
the
contamination-filepath-probe
branch
from
e3452a9 to
a8595e8
Compare
Reverted immediately after the run. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…utput - Add docs/contamination-file-path-identification.md explaining gold-file extraction, path/basename normalization, the exact/basename hit & recall formulas, aggregation, and the cutoff delta, with direct citations from "The SWE-Bench Illusion" (arXiv:2506.12286). - Link the doc + paper from the summarize step-summary report footer. - Add a Diagnostics entry on the docs home page. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Reverted immediately after the run. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…s-4.8" This reverts commit 095fcab.
…on CLI Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…on logs Reverted immediately after. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…escription logs" This reverts commit 93aeece.
gggdttt
reviewed
Jul 9, 2026
gggdttt
left a comment
gggdttt
left a comment
Collaborator
There was a problem hiding this comment.
A few methodology / design questions from a close read — none blocking. Details inline.
| lines.append(_aggregate_row(pre_agg)) | ||
| lines.append(_aggregate_row(post_agg)) | ||
|
|
||
| delta = pre_agg.basename_hit_rate - post_agg.basename_hit_rate |
Collaborator
There was a problem hiding this comment.
The headline delta uses basename_hit, but the docs argue exact_hit is the stronger (convention-resistant) memorization signal — should we headline the exact-path delta instead of (or alongside) basename?
| [ | ||
| copilot_cmd, | ||
| "--silent", # emit only the agent response on stdout | ||
| "--available-tools=write", # the model's ONLY tool: no shell/read/fetch to reach the repo or gold patch |
Collaborator
There was a problem hiding this comment.
Correct me if I misunderstand - this task should only need a JSON text answer, allowing the write tool risks the model writing to a file instead of stdout right?
| data); on/after-cutoff entries act as the clean control. Entries with an | ||
| unparseable date are conservatively grouped with the pre-cutoff set. | ||
| """ | ||
| boundary = date.fromisoformat(cutoff) |
Collaborator
There was a problem hiding this comment.
Minor : An invalid --cutoff raises a raw ValueError here — could we pre-validate in the summarize command and surface a typer.BadParameter instead?
gggdttt
approved these changes
Jul 9, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds a file-path identification contamination probe — a black-box memorization detector that requires no access to model training data.
Motivated by the discussion on work item 632829 (dataset refresh / contamination). Per the SWE-Bench Illusion methodology Haoran cited, contamination detection (not exact quantification) is feasible black-box: give the model only the bug report — no repository, no code — and ask which file(s) contain the bug. A model that localizes bugs it can't see is exhibiting memorization.
How it maps onto BC-Bench
patchtouches (reusesextract_file_paths_from_patch).entry.get_task()(the problem statement) only.AL has rigid naming conventions (
<ObjectName>.<ObjectType>.al), so a model can often guessSalesHeader.Table.alfrom a description without contamination. A raw accuracy number is therefore convention-inflated. The probe reports the pre-cutoff (suspect) vs on/after-cutoff (control) delta as the actual contamination signal — runsummarize --cutoff YYYY-MM-DD. The dataset-refresh work provides the fresh control set for free.Changes
src/bcbench/contamination/— pure core (filepath_probe.py) + runner (runner.py).bcbench contamination filepath-probe <id>andbcbench contamination summarize --cutoffCLI commands..github/workflows/contamination-probe.yml—workflow_dispatch+ weekly schedule, runs onubuntu-latest(no BC container/build), matrixed, then summarizes to the step summary.tests/test_filepath_probe.py— 25 unit tests for the pure core.Testing
pytest tests/test_filepath_probe.py→ 25 passed; broader CLI/type tests green (57 total).ruff checkclean; workflow YAML validated.