Fix last-screenshot selection: sort trajectory images by step index#2
Open
jianganghan wants to merge 1 commit into
Open
Fix last-screenshot selection: sort trajectory images by step index#2jianganghan wants to merge 1 commit into
jianganghan wants to merge 1 commit into
Conversation
os.listdir returns entries in arbitrary order, so taking [-1] as the
"last" screenshot silently picks a wrong step on some filesystems
(observed 0.png, the initial page, on macOS/APFS), causing the judge
to fail tasks for "no visual evidence in final screenshot". Even
lexicographic order breaks past step 9 ('9.png' > '15.png'). Sort by
numeric step index instead.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
os.listdirorder is filesystem-dependent; taking[-1]as the last screenshot picks a wrong step on macOS/APFS (observed0.png, the initial page), causing systematic false FAILs in NavEval ("final screenshot shows only the homepage"). Sort by numeric step index instead — plain lexicographic sorting would also break beyond step 9 ('9.png' > '15.png').Verified on a 10-task Protocol-III run (judge: claude-sonnet-4-5): before the fix all 6 evaluable tasks were judged FAIL (0.15); after, 3 flipped to PASS (0.95), matching manual review of the trajectories.
Fixes #1.
🤖 Generated with Claude Code