Remove legacy intake authentication

.agents/skills/submit-benchmark-result/SKILL.md

@@ -54,25 +54,24 @@ offer local-only or test upload instead.
 Require `PRB_SUBMIT_URL` from the repository documentation or maintainer. Never ask the user
 to paste any token into chat, print an environment variable, or commit credentials.
 
-Prefer GitHub-backed Cloudflare Access when the deployed client documents
-`PRB_ACCESS_TOKEN`:
+Use GitHub-backed Cloudflare Access through `PRB_ACCESS_TOKEN`:
 
 1. Require `cloudflared` locally.
-2. Obtain the application-scoped token only inside the confirmed upload command, for example
-   `PRB_ACCESS_TOKEN="$(cloudflared access token -app=<application-url>)" <submit-command>`.
-   `cloudflared` opens the configured GitHub login in a browser when needed.
-3. Never run `cloudflared access login` or `cloudflared access token` standalone because
-   some versions print the JWT. Pass it through the client-supported environment variable
-   for that one process only; do not display, persist, or put it in a command-line flag.
+2. For a new session, obtain the application-scoped token only inside the confirmed upload
+   command: `PRB_ACCESS_TOKEN="$(cloudflared access login --no-verbose --auto-close <application-url>)" <submit-command>`.
+   `cloudflared` opens the configured GitHub login.
+3. A later upload may use `cloudflared access token -app=<application-url>` inside the same
+   command substitution while the local Access session remains valid.
+4. Never run either token-producing command standalone. Pass its stdout directly through
+   the client-supported environment variable; do not display it, save it yourself, or put it
+   in a command-line flag.
 
 Do not substitute `gh auth token`, a GitHub personal access token, or `GITHUB_TOKEN`; the
 intake must never receive the user's general GitHub credential.
 
-If the deployed client does not yet support Access, use the legacy `PRB_API_KEY` path only
-when a credential is already configured locally and the maintainer confirms that endpoint
-mode is enabled. If it is absent, stop and report that self-service authentication is not
-deployed. Ask the maintainer to enable GitHub Access or issue an out-of-band per-user or
-one-time intake credential; never request a shared long-lived key in chat.
+If Cloudflare Access is not deployed or the user is not allowed by its policy, stop and ask
+the maintainer to fix the Access application. There is no shared-key fallback. Never request
+an intake key in chat.
 
 ## 4. Confirm and upload once
 

.github/workflows/score-from-r2.yml

@@ -79,7 +79,7 @@ jobs:
           # may be archived later; an object arriving while scoring remains in incoming/.
           aws s3api list-objects-v2 --bucket "$BUCKET" --prefix incoming/ \
             --endpoint-url "$R2_ENDPOINT" --query 'Contents[].Key' --output json \
-            | jq -r '.[] | select(endswith(".json"))' > queue-manifest.txt
+            | jq -r '.[]? | select(endswith(".json"))' > queue-manifest.txt
           while IFS= read -r key; do
             [ -n "$key" ] || continue
             rel="${key#incoming/}"

CONTRIBUTING.md

@@ -195,16 +195,17 @@ GitHub-backed Cloudflare Access application:
 ```bash
 export PRB_SUBMIT_URL=<intake endpoint>   # from the maintainer
 PRB_ACCESS_APP="${PRB_SUBMIT_URL%/submit}"
-PRB_ACCESS_TOKEN="$(cloudflared access token -app="$PRB_ACCESS_APP")" \
+PRB_ACCESS_TOKEN="$(cloudflared access login --no-verbose --auto-close "$PRB_ACCESS_APP")" \
   python -m benchmark.submit --predictions out/submission.json
 #   --dry-run   validate locally, don't send
 #   --test      scored + stored privately, but excluded from the public leaderboard
 ```
 
-The token is short-lived and scoped to this Access application. Keep token acquisition
-inside command substitution because a standalone `cloudflared access login` or
-`cloudflared access token` may print it. Never substitute `gh auth token`, a GitHub PAT, or
-`GITHUB_TOKEN`.
+The token is short-lived and scoped to this Access application. Keep the login inside
+command substitution so its JWT goes directly into the one upload process instead of the
+terminal. A later upload may replace `access login --no-verbose --auto-close` with
+`access token -app` while the local Access session is still valid. Never substitute
+`gh auth token`, a GitHub PAT, or `GITHUB_TOKEN`.
 
 The CLI validates the file against `submission.schema.json`, then uploads it to a private
 store (Cloudflare R2). The maintainer's scorer re-verifies it with `pred` (see §3) and opens

benchmark/submit.py

@@ -4,11 +4,11 @@
 Uploads a run's ``submission.json`` (produced by ``make run``) to the benchmark's
 serverless intake endpoint, which deposits it into the PRIVATE submission store. The
 submission body carries the answer key (certificates + submit ledger), so it never goes to a
-public repo — it travels over HTTPS to the endpoint, which holds the write token.
+public repo — it travels over HTTPS through GitHub-backed Cloudflare Access.
 
     export PRB_SUBMIT_URL=https://<your-worker>/submit
-    export PRB_ACCESS_TOKEN="$(cloudflared access token -app=https://<your-worker>)"
-    python -m benchmark.submit --predictions out/submission.json
+    PRB_ACCESS_TOKEN="$(cloudflared access login --no-verbose --auto-close https://<your-worker>)" \
+      python -m benchmark.submit --predictions out/submission.json
     # → prints the submission id returned by the endpoint
 
 The client validates the file locally FIRST (valid JSON, required envelope fields, and —
@@ -19,7 +19,6 @@
 
 HTTP contract (endpoint side):
     POST <url>   Cf-Access-Token: <app-scoped JWT>   Content-Type: application/json
-    legacy:      Authorization: Bearer <PRB_API_KEY>
     body    = the submission.json object
     200/201 → {"submission_id": "...", "status": "accepted", ...}
     4xx/5xx → {"error": "..."}  (or a non-JSON body, surfaced as text)
@@ -121,15 +120,6 @@ def validate_submission(sub: dict) -> list[str]:
     return problems
 
 
-def _auth_headers(api_key: str | None, access_token: str | None) -> dict[str, str]:
-    """Build one authentication header, preferring the scoped Access credential."""
-    if access_token:
-        return {"Cf-Access-Token": access_token}
-    if api_key:
-        return {"Authorization": f"Bearer {api_key}"}
-    return {}
-
-
 def _post(url: str, payload: bytes, auth_headers: dict[str, str],
           timeout: float = 60.0) -> tuple[int, dict | str]:
     """POST raw JSON bytes. Returns (status_code, parsed-or-text body).
@@ -158,8 +148,8 @@ def _maybe_json(text: str):
         return text
 
 
-def submit(path: Path, url: str | None, api_key: str | None = None, *,
-           access_token: str | None = None, dry_run: bool = False,
+def submit(path: Path, url: str | None, *, access_token: str | None = None,
+           dry_run: bool = False,
            timeout: float = 60.0, mark_test: bool = False) -> dict:
     """Validate and (unless dry_run) upload a submission. Returns a result dict.
 
@@ -181,13 +171,12 @@ def submit(path: Path, url: str | None, api_key: str | None = None, *,
 
     if not url:
         raise ValueError("no endpoint URL — set PRB_SUBMIT_URL or pass --url")
-    auth_headers = _auth_headers(api_key, access_token)
-    if not auth_headers:
-        raise ValueError("no intake credential — set PRB_ACCESS_TOKEN (preferred) or "
-                         "PRB_API_KEY (legacy)")
+    if not access_token:
+        raise ValueError("no intake credential — set PRB_ACCESS_TOKEN")
 
     payload = json.dumps(sub).encode("utf-8")
-    status, body = _post(url, payload, auth_headers, timeout=timeout)
+    status, body = _post(
+        url, payload, {"Cf-Access-Token": access_token}, timeout=timeout)
     if not (200 <= status < 300):
         reason = body.get("error") if isinstance(body, dict) else body
         raise ValueError(f"endpoint returned HTTP {status}: {reason}")
@@ -206,8 +195,6 @@ def main() -> None:
                         help="Path to submission.json (from `make run`)")
     parser.add_argument("--url", default=os.environ.get("PRB_SUBMIT_URL"),
                         help="Intake endpoint URL (env PRB_SUBMIT_URL)")
-    parser.add_argument("--api-key", default=os.environ.get("PRB_API_KEY"),
-                        help="Legacy bearer token (env PRB_API_KEY); prefer PRB_ACCESS_TOKEN")
     parser.add_argument("--dry-run", action="store_true",
                         help="Validate the file locally and report what would be sent; no upload")
     parser.add_argument("--test", action="store_true",
@@ -221,7 +208,7 @@ def main() -> None:
         parser.error(f"no such file: {path}")
 
     try:
-        result = submit(path, args.url, args.api_key,
+        result = submit(path, args.url,
                         access_token=os.environ.get("PRB_ACCESS_TOKEN"), dry_run=args.dry_run,
                         timeout=args.timeout, mark_test=args.test)
     except (ValueError, json.JSONDecodeError, OSError) as e:

benchmark/tests/test_docs.py

@@ -17,6 +17,10 @@
 API_SKILL = REPO_ROOT / ".agents/skills/run-api-benchmark/SKILL.md"
 CLI_SKILL = REPO_ROOT / ".agents/skills/run-cli-benchmark/SKILL.md"
 SUBMIT_SKILL = REPO_ROOT / ".agents/skills/submit-benchmark-result/SKILL.md"
+SCORER_WORKFLOW = REPO_ROOT / ".github/workflows/score-from-r2.yml"
+SUBMISSIONS_README = REPO_ROOT / "submissions/README.md"
+SITE_INDEX = REPO_ROOT / "site/index.html"
+INTAKE_README = REPO_ROOT / "intake/cloudflare-worker/README.md"
 
 
 def _text(path: Path) -> str:
@@ -106,8 +110,18 @@ def test_submit_skill_validates_before_upload(self):
 
     def test_submit_skill_keeps_github_credentials_out_of_intake(self):
         t = _text(SUBMIT_SKILL)
-        prose = " ".join(t.split())
         assert "github-backed cloudflare access" in t
         assert "gh auth token" in t
         assert "personal access token" in t
-        assert "self-service authentication is not deployed" in prose
+        assert "cloudflare access is not deployed" in t
+
+    @pytest.mark.parametrize(
+        "path", [GUIDE, SUBMIT_SKILL, SUBMISSIONS_README, SITE_INDEX, INTAKE_README])
+    def test_public_submission_docs_have_no_shared_intake_key(self, path):
+        assert "PRB_API_KEY" not in path.read_text(encoding="utf-8")
+
+
+class TestScorerWorkflow:
+    def test_empty_r2_queue_is_valid(self):
+        text = SCORER_WORKFLOW.read_text(encoding="utf-8")
+        assert "jq -r '.[]? | select(endswith(\".json\"))'" in text

benchmark/tests/test_submit.py

@@ -67,51 +67,43 @@ def test_valid_submit_ledger_passes(self, tmp_path):
 class TestSubmit:
     def test_dry_run_does_not_send(self, tmp_path, monkeypatch):
         monkeypatch.setattr(sub, "_post", lambda *a, **k: pytest.fail("must not POST on dry-run"))
-        out = sub.submit(_valid(tmp_path, results=[_bug_row()]), "https://x/submit", "k", dry_run=True)
+        out = sub.submit(_valid(tmp_path, results=[_bug_row()]),
+                         "https://x/submit", dry_run=True)
         assert out["claimed_bugs"] == 1 and "dry-run" in out["status"]
 
     def test_invalid_submission_raises_before_network(self, tmp_path, monkeypatch):
         monkeypatch.setattr(sub, "_post", lambda *a, **k: pytest.fail("must not POST invalid"))
         p = _valid(tmp_path, results=[_bug_row(with_cert=False, with_traj=False)])
         with pytest.raises(ValueError, match="certificate"):
-            sub.submit(p, "https://x/submit", "k")
+            sub.submit(p, "https://x/submit")
 
     def test_success_returns_body(self, tmp_path, monkeypatch):
         monkeypatch.setattr(sub, "_post",
                             lambda url, payload, headers, timeout=60.0: (
                                 201, {"submission_id": "abc", "status": "accepted"}))
-        out = sub.submit(_valid(tmp_path), "https://x/submit", "k")
+        out = sub.submit(_valid(tmp_path), "https://x/submit", access_token="access-jwt")
         assert out["submission_id"] == "abc"
 
-    def test_access_token_is_preferred_over_legacy_key(self, tmp_path, monkeypatch):
+    def test_access_token_uses_access_header(self, tmp_path, monkeypatch):
         def post(url, payload, headers, timeout=60.0):
             assert headers == {"Cf-Access-Token": "access-jwt"}
             return 201, {"submission_id": "abc", "status": "accepted"}
 
         monkeypatch.setattr(sub, "_post", post)
-        out = sub.submit(_valid(tmp_path), "https://x/submit", "legacy-key",
-                         access_token="access-jwt")
+        out = sub.submit(_valid(tmp_path), "https://x/submit", access_token="access-jwt")
         assert out["submission_id"] == "abc"
 
-    def test_legacy_key_uses_authorization_header(self, tmp_path, monkeypatch):
-        def post(url, payload, headers, timeout=60.0):
-            assert headers == {"Authorization": "Bearer legacy-key"}
-            return 201, {"submission_id": "abc", "status": "accepted"}
-
-        monkeypatch.setattr(sub, "_post", post)
-        sub.submit(_valid(tmp_path), "https://x/submit", "legacy-key")
-
     def test_non_2xx_raises(self, tmp_path, monkeypatch):
         monkeypatch.setattr(sub, "_post",
                             lambda *a, **k: (429, {"error": "quota exceeded"}))
         with pytest.raises(ValueError, match="429.*quota"):
-            sub.submit(_valid(tmp_path), "https://x/submit", "k")
+            sub.submit(_valid(tmp_path), "https://x/submit", access_token="access-jwt")
 
     def test_missing_url_or_key_raises(self, tmp_path):
         with pytest.raises(ValueError, match="URL"):
-            sub.submit(_valid(tmp_path), "", "k")
+            sub.submit(_valid(tmp_path), "", access_token="access-jwt")
         with pytest.raises(ValueError, match="intake credential"):
-            sub.submit(_valid(tmp_path), "https://x/submit", "")
+            sub.submit(_valid(tmp_path), "https://x/submit")
 
     def test_non_json_success_is_not_reported_as_accepted(self, tmp_path, monkeypatch):
         monkeypatch.setattr(sub, "_post", lambda *a, **k: (200, "Access login page"))

intake/cloudflare-worker/README.md

@@ -21,10 +21,7 @@ npx wrangler login
 # 1. private bucket for raw submissions
 npx wrangler r2 bucket create prb-submissions
 
-# 2. migration-only bearer token; remove it after the Access test succeeds
-npx wrangler secret put PRB_API_KEY
-
-# 3. deploy the legacy-compatible Worker once
+# 2. deploy the Access-only Worker
 npm run deploy
 # → registers your workers.dev subdomain (e.g. prb-bench) and prints the endpoint,
 #   e.g. https://intake.prb-bench.workers.dev
@@ -33,10 +30,9 @@ npm run deploy
 ### GitHub-backed Cloudflare Access
 
 Create a GitHub identity provider under **Zero Trust → Integrations → Identity providers**.
-Then create a self-hosted Access application for the `intake` Worker. During migration,
-protect the preview hostname first. Its Allow policy should select the intended GitHub
-organization/team, or exact GitHub-verified emails together with `Login Methods: GitHub`;
-never use `Everyone`.
+Then create a self-hosted Access application for the `intake` Worker. Its Allow policy
+should select the intended GitHub organization/team, or exact GitHub-verified emails
+together with `Login Methods: GitHub`; never use `Everyone`.
 
 The checked-in `TEAM_DOMAIN` and `POLICY_AUD` in `wrangler.toml` identify the configured
 Access organization and application. They are public identifiers, not credentials. The
@@ -51,8 +47,8 @@ npm run preview
 
 The Worker validates the `Cf-Access-Jwt-Assertion` signature, issuer, audience, and expiry
 against Cloudflare's rotating JWKS. It records the verified Access subject/email separately
-from the submitter-claimed `submitted_by` field. If an assertion is present but invalid, the
-request is rejected and cannot fall back to `PRB_API_KEY`.
+from the submitter-claimed `submitted_by` field. A missing or invalid assertion is rejected;
+there is no shared intake credential.
 
 The Worker only writes to R2 — no GitHub token needed. Scoring is picked up by
 `.github/workflows/score-from-r2.yml` on its **daily cron** (or trigger it manually via
@@ -69,25 +65,16 @@ Submitters authenticate in their own browser and obtain a short-lived, applicati
 token. No maintainer-issued secret or GitHub personal access token is involved:
 
 ```bash
-export PRB_SUBMIT_URL=https://access-auth-intake.prb-bench.workers.dev/submit
+export PRB_SUBMIT_URL=https://intake.prb-bench.workers.dev/submit
 PRB_ACCESS_APP="${PRB_SUBMIT_URL%/submit}"
-PRB_ACCESS_TOKEN="$(cloudflared access token -app="$PRB_ACCESS_APP")" \
+PRB_ACCESS_TOKEN="$(cloudflared access login --no-verbose --auto-close "$PRB_ACCESS_APP")" \
   python3 -m benchmark.submit --predictions out/submission.json --test
 ```
 
 Remove `--test` only when the run is ready to become an official leaderboard submission.
-Keep the `cloudflared access token` call inside command substitution: running
-`cloudflared access login` or `cloudflared access token` by itself may print the JWT. Do not
-print, persist, or pass `PRB_ACCESS_TOKEN` as a command-line argument. The client prefers it
-over an ambient legacy `PRB_API_KEY` and never sends both.
-
-After an end-to-end preview test succeeds, change the Access destination from Preview Worker
-to the production `intake` Worker, deploy the tested version with `npm run deploy`, repeat the
-`--test` upload against the production URL, and remove the migration secret:
-
-```bash
-npx wrangler secret delete PRB_API_KEY
-```
+Keep token acquisition inside command substitution so the JWT goes directly into the upload
+process instead of the terminal. While the local Access session is valid, a later upload may
+use `cloudflared access token -app="$PRB_ACCESS_APP"` in the same position.
 
 ## R2 credentials for the scoring worker
 
@@ -98,7 +85,6 @@ R2 → *Manage API Tokens*, create an **Object Read & Write** token, then add th
 See `.github/workflows/score-from-r2.yml`.
 
 ## Notes
-- `PRB_API_KEY` is only a migration fallback for requests with no Access assertion.
 - `gh auth token`, GitHub PATs, and `GITHUB_TOKEN` are never intake credentials.
 - Max body 25 MB. For larger submissions, hand out an R2 presigned PUT URL
   instead of POSTing the body — not needed at current scale.