diff --git a/.agents/skills/submit-benchmark-result/SKILL.md b/.agents/skills/submit-benchmark-result/SKILL.md index 290c564..f4ee906 100644 --- a/.agents/skills/submit-benchmark-result/SKILL.md +++ b/.agents/skills/submit-benchmark-result/SKILL.md @@ -1,101 +1,136 @@ --- name: submit-benchmark-result -description: Validate and upload an existing problem-reductions benchmark submission through the private intake. Use when a user wants to submit, upload, publish, dry-run, or test a submission.json they already produced, especially when the user is not a repository maintainer. Handles local validation, test-versus-official intent, intake authentication, upload confirmation, and submission ID reporting; routes missing results to run-benchmark instead of running a model itself. +description: Validate and officially upload an existing problem-reductions benchmark submission. Use when the user wants to submit a submission.json. If no result exists yet, route the request to $run-benchmark. --- # Submit a benchmark result -Submit an existing `submission.json` without requiring GitHub repository, R2, Worker, or -Actions access. Treat the file as confidential because it contains certificates and the -submit ledger. +Follow this workflow for an existing `submission.json`. Do not run a model or create a +submission in this skill. -## 1. Locate the authoritative result +## 1. Find and validate the file -Ask for the submission path only when it was not supplied. Accept the authoritative output -from either benchmark backend; do not reconstruct it from logs or edit its metrics, -certificates, `library_commit`, or ledger. +Ask for the path only if the user did not provide it. If no result exists, use +`$run-benchmark` instead. -If no result exists and the user wants to run the benchmark, invoke `$run-benchmark`. Do not -choose a model/backend or start a paid run inside this skill. +Treat the file as confidential. Do not print certificates, trajectories, source instances, +submit-log contents, or credentials. Do not edit the submission. -Read the current round from `README.md` and the current structure from -`benchmark/submission.schema.json`. Inspect only summary fields; never print certificates, -source instances, submit-log contents, trajectories, or credentials. Report: - -- model and `library_commit`; -- claimed bugs, token total, and submit-attempt count; -- `run_error`, when present; -- absolute submission path. - -Run the repository client as a local courtesy check: +Run: ```bash python3 -m benchmark.submit --predictions --dry-run ``` -Stop on failure. Do not repair an invalid result by hand; send it back to the producing run. +Stop if validation fails. Otherwise report only: -## 2. Choose the outcome +- absolute path; +- `model` and `library_commit`; +- claimed bugs, `total_tokens_k`, and number of submit attempts; +- `run_error`, if present. -Ask only when the user has not already chosen: +Do not submit a result containing `run_error`. Report the error and stop. -> What should happen to this result? -> -> 1. Keep it local after validation. -> 2. Upload an intake test that is scored privately and never reaches the leaderboard. -> 3. Upload an official submission. +## 2. Confirm the upload -For option 1, report validation and stop. For option 2, use `--test`. For option 3, never -use `--test`. A non-test result carrying `run_error` is not a clean official submission; -offer local-only or test upload instead. +Immediately before uploading, show: -## 3. Authenticate without exposing credentials +- endpoint hostname; +- absolute file path; +- model and claimed bug count; -Require `PRB_SUBMIT_URL` from the repository documentation or maintainer. Never ask the user -to paste any token into chat, print an environment variable, or commit credentials. +State that this is an official submission and that the private file will leave the machine, +then obtain explicit confirmation. -Use GitHub-backed Cloudflare Access through `PRB_ACCESS_TOKEN`: +## 3. Prepare authentication -1. Require `cloudflared` locally. -2. For a new session, obtain the application-scoped token only inside the confirmed upload - command: `PRB_ACCESS_TOKEN="$(cloudflared access login --no-verbose --auto-close )" `. - `cloudflared` opens the configured GitHub login. -3. A later upload may use `cloudflared access token -app=` inside the same - command substitution while the local Access session remains valid. -4. Never run either token-producing command standalone. Pass its stdout directly through - the client-supported environment variable; do not display it, save it yourself, or put it - in a command-line flag. +Check whether the Cloudflare Access client is already available: -Do not substitute `gh auth token`, a GitHub personal access token, or `GITHUB_TOKEN`; the -intake must never receive the user's general GitHub credential. +```bash +command -v cloudflared >/dev/null && cloudflared --version +``` -If Cloudflare Access is not deployed or the user is not allowed by its policy, stop and ask -the maintainer to fix the Access application. There is no shared-key fallback. Never request -an intake key in chat. +If it is missing, identify the operating system and tell the user that `cloudflared` must +be installed before submission. Give the installation step from Cloudflare's official +[downloads page](https://developers.cloudflare.com/cloudflare-one/networks/connectors/cloudflare-tunnel/downloads/) +for that system. Obtain confirmation before running an installer or package-manager +command. On macOS with Homebrew, the official command is: + +```bash +brew install cloudflared +``` -## 4. Confirm and upload once +Do not improvise a download URL, use `sudo`, or change the system without confirmation. +After installation, run `cloudflared --version`. If installation cannot be completed, +report the prerequisite and stop before uploading. -Before the external write, show the endpoint hostname, absolute file path, model, claimed -bug count, and test/official mode. State that the private certificate payload will leave the -machine, then obtain explicit confirmation. +## 4. Log in and upload once -Use the repository client so validation and test marking stay consistent: +Use the official intake: ```bash -python3 -m benchmark.submit --predictions --test # private test -python3 -m benchmark.submit --predictions # official +export PRB_SUBMIT_URL=https://intake.prb-bench.workers.dev/submit +PRB_ACCESS_APP="${PRB_SUBMIT_URL%/submit}" +PRB_ACCESS_TOKEN="$(cloudflared access login --no-verbose --auto-close "$PRB_ACCESS_APP")" +test -n "$PRB_ACCESS_TOKEN" +PRB_ACCESS_TOKEN="$PRB_ACCESS_TOKEN" \ + python3 -m benchmark.submit --predictions +unset PRB_ACCESS_TOKEN ``` -Do not retry an ambiguous timeout automatically: the first request may already have reached -R2 and a retry can create a duplicate. On HTTP 401/403, re-authenticate or report the missing -Access deployment. On HTTP 413, do not trim the evidence; report the size limit. On HTTP 429, -stop and report the rate limit. +The login opens GitHub in the user's browser. If no browser opens, give the user the login +URL printed by `cloudflared` and wait for them to finish. Keep token acquisition inside +command substitution. Stop if login fails or returns an empty token; do not run the upload +command. Never print or ask for the token. Do not substitute a GitHub PAT, `gh auth token`, +or `GITHUB_TOKEN`. -## 5. Report the handoff +Upload only once. Do not retry a timeout because the first request may already be queued. +For HTTP 401, re-authenticate. If the login page says access is denied or the upload returns +HTTP 403, explain that the GitHub account is not authorized. Tell the user to ensure their +GitHub primary email has been added to the intake authorization list by a maintainer, then +stop. For 413 or 429, stop and report the error. -On success, report the returned `submission_id`, model, mode, and endpoint hostname. Make it -clear that `accepted` means privately queued, not scored or published. The submitter does not -trigger Actions, inspect R2, or open the leaderboard PR; those are maintainer responsibilities. +## 5. Report the result + +On success, report the `submission_id`, model, and endpoint. Say that `accepted` means +queued privately, not scored or published. Keep the local submission until scoring is +confirmed. + +## 6. Trigger scoring + +After reporting an accepted submission, trigger the scoring workflow once. If GitHub CLI +is installed and authenticated, run: + +```bash +SUBMISSION_ID="" +RUN_URL="$(gh workflow run score-from-r2.yml \ + --repo CodingThrust/problem-reductions-benchmark \ + --ref main)" +RUN_ID="${RUN_URL##*/}" +gh run watch "$RUN_ID" \ + --repo CodingThrust/problem-reductions-benchmark \ + --exit-status --compact + +SHORT_ID="${SUBMISSION_ID:0:8}" +gh pr list \ + --repo CodingThrust/problem-reductions-benchmark \ + --state all --limit 100 \ + --json headRefName,url \ + --jq ".[] | select(.headRefName | endswith(\"--${SHORT_ID}\")) | .url" +``` -Never claim a score until the maintainer confirms the private scorer finished. Preserve the -local submission and logs until that confirmation. +If `gh` is unavailable, give the user the +[workflow page](https://github.com/CodingThrust/problem-reductions-benchmark/actions/workflows/score-from-r2.yml) +and ask them to select **Run workflow** on `main`. + +The intake authorization list and GitHub Actions permissions are separate. If GitHub hides +the **Run workflow** button or returns 403, explain that a repository maintainer or a +collaborator with Actions write permission must trigger it. Do not upload again: the +accepted submission remains privately queued and the daily scheduled workflow will process +it if nobody triggers a run manually. + +When the triggered run succeeds, return the matching PR URL. Match it using the first eight +characters of this upload's `submission_id`, which appear at the end of the leaderboard +branch name; do not return an unrelated latest PR. If the run fails, return its URL and the +failure. If it succeeds but no matching PR exists, return the run URL and explain that no PR +was created instead of guessing a link. diff --git a/.github/workflows/score-from-r2.yml b/.github/workflows/score-from-r2.yml index e7cbcd1..7527d0e 100644 --- a/.github/workflows/score-from-r2.yml +++ b/.github/workflows/score-from-r2.yml @@ -16,11 +16,6 @@ name: Score submissions from R2 (private → aggregate) on: workflow_dispatch: # manual trigger (maintainer) - inputs: - reset_results: - description: "Wipe R2 results/ before scoring (clear stale scored files)" - type: boolean - default: false schedule: - cron: "0 16 * * *" # daily sweep of R2 for new submissions (16:00 UTC). @@ -69,12 +64,6 @@ jobs: AWS_DEFAULT_REGION: auto run: | mkdir -p incoming results/scored - # reset_results (manual): wipe prior scored detail so the board is rebuilt from - # scratch — clears stale/test entries that predate a rule change. - if [ "${{ github.event.inputs.reset_results }}" = "true" ]; then - echo "reset_results: wiping s3://$BUCKET/results" - aws s3 rm "s3://$BUCKET/results" --recursive --endpoint-url "$R2_ENDPOINT" || true - fi # Snapshot the exact queue keys before downloading. Only this immutable manifest # may be archived later; an object arriving while scoring remains in incoming/. aws s3api list-objects-v2 --bucket "$BUCKET" --prefix incoming/ \ diff --git a/benchmark/tests/test_docs.py b/benchmark/tests/test_docs.py index 6e2d3a5..56dcc2b 100644 --- a/benchmark/tests/test_docs.py +++ b/benchmark/tests/test_docs.py @@ -105,15 +105,47 @@ def test_submit_skill_validates_before_upload(self): t = _text(SUBMIT_SKILL) assert "python3 -m benchmark.submit" in t assert "--dry-run" in t - assert "--test" in t assert "explicit confirmation" in t - def test_submit_skill_keeps_github_credentials_out_of_intake(self): + def test_submit_skill_has_no_intake_test_mode(self): t = _text(SUBMIT_SKILL) - assert "github-backed cloudflare access" in t + assert "--test" not in t + assert "smoke test" not in t + + def test_submit_skill_uses_access_without_github_credentials(self): + t = _text(SUBMIT_SKILL) + assert "cloudflared access login" in t + assert "prb_access_token" in t + assert 'test -n "$prb_access_token"' in t + assert "unset prb_access_token" in t assert "gh auth token" in t - assert "personal access token" in t - assert "cloudflare access is not deployed" in t + assert "github pat" in t + + def test_submit_skill_prepares_cloudflared(self): + t = _text(SUBMIT_SKILL) + assert "command -v cloudflared" in t + assert "cloudflared --version" in t + assert "brew install cloudflared" in t + assert "obtain confirmation before running an installer" in t + + def test_submit_skill_explains_access_denial(self): + t = _text(SUBMIT_SKILL) + assert "http 403" in t + assert "github primary email" in t + assert "authorization list by a maintainer" in t + + def test_submit_skill_triggers_scoring_without_reset(self): + t = _text(SUBMIT_SKILL) + assert "gh workflow run score-from-r2.yml" in t + assert "--repo codingthrust/problem-reductions-benchmark" in t + assert "reset_results" not in t + assert "actions write permission" in t + assert 'run_id="${run_url##*/}"' in t + assert "gh run watch" in t + assert 'short_id="${submission_id:0:8}"' in t + assert "--json headrefname,url" in t + assert "return the matching pr url" in t + assert "do not return an unrelated latest pr" in t @pytest.mark.parametrize( "path", [GUIDE, SUBMIT_SKILL, SUBMISSIONS_README, SITE_INDEX, INTAKE_README]) @@ -125,3 +157,6 @@ class TestScorerWorkflow: def test_empty_r2_queue_is_valid(self): text = SCORER_WORKFLOW.read_text(encoding="utf-8") assert "jq -r '.[]? | select(endswith(\".json\"))'" in text + + def test_workflow_exposes_no_result_reset(self): + assert "reset_results" not in _text(SCORER_WORKFLOW)