Add GitHub-backed intake authentication

.github/workflows/ci.yml

@@ -24,6 +24,24 @@ jobs:
       - name: Run unit tests (judgment + non-integration)
         run: pytest -v -m "not integration"
 
+      - uses: actions/setup-node@v6
+        with:
+          node-version: "24"
+          cache: npm
+          cache-dependency-path: intake/cloudflare-worker/package-lock.json
+
+      - name: Install intake Worker dependencies
+        working-directory: intake/cloudflare-worker
+        run: npm ci
+
+      - name: Test intake Worker
+        working-directory: intake/cloudflare-worker
+        run: npm test
+
+      - name: Bundle intake Worker
+        working-directory: intake/cloudflare-worker
+        run: npm run check
+
       # Calibration needs the real `pred` binary (Rust, built from problem-reductions at
       # PR_REF). The bare runner has no pred, so run it inside the scoring image — the same
       # pinned-pred environment used to score submissions. The image's default CMD is the

CONTRIBUTING.md

@@ -187,18 +187,23 @@ file, preserves test/official intent, handles the authentication mode actually d
 the intake, asks before the external write, and reports the opaque submission ID. Submitters
 do not need repository write access, R2 access, or permission to run the scoring workflow.
 
-The current legacy deployment uses an endpoint URL plus an intake credential configured
-locally. Do not paste the credential into chat or commit it:
+For either upload, get the endpoint URL from the maintainer and authenticate through the
+GitHub-backed Cloudflare Access application:
 
 ```bash
 export PRB_SUBMIT_URL=<intake endpoint>   # from the maintainer
-export PRB_API_KEY=<token>                 # from the maintainer
-
-python -m benchmark.submit --predictions out/submission.json
+PRB_ACCESS_APP="${PRB_SUBMIT_URL%/submit}"
+PRB_ACCESS_TOKEN="$(cloudflared access token -app="$PRB_ACCESS_APP")" \
+  python -m benchmark.submit --predictions out/submission.json
 #   --dry-run   validate locally, don't send
 #   --test      scored + stored privately, but excluded from the public leaderboard
 ```
 
+The token is short-lived and scoped to this Access application. Keep token acquisition
+inside command substitution because a standalone `cloudflared access login` or
+`cloudflared access token` may print it. Never substitute `gh auth token`, a GitHub PAT, or
+`GITHUB_TOKEN`.
+
 The CLI validates the file against `submission.schema.json`, then uploads it to a private
 store (Cloudflare R2). The maintainer's scorer re-verifies it with `pred` (see §3) and opens
 a PR that updates the aggregate `site/results.json`; merging deploys to **GitHub Pages**.

benchmark/submit.py

@@ -7,7 +7,7 @@
 public repo — it travels over HTTPS to the endpoint, which holds the write token.
 
     export PRB_SUBMIT_URL=https://<your-worker>/submit
-    export PRB_API_KEY=...
+    export PRB_ACCESS_TOKEN="$(cloudflared access token -app=https://<your-worker>)"
     python -m benchmark.submit --predictions out/submission.json
     # → prints the submission id returned by the endpoint
 
@@ -18,7 +18,8 @@
 courtesy gate. Use ``--dry-run`` to validate without sending.
 
 HTTP contract (endpoint side):
-    POST <url>   Authorization: Bearer <key>   Content-Type: application/json
+    POST <url>   Cf-Access-Token: <app-scoped JWT>   Content-Type: application/json
+    legacy:      Authorization: Bearer <PRB_API_KEY>
     body    = the submission.json object
     200/201 → {"submission_id": "...", "status": "accepted", ...}
     4xx/5xx → {"error": "..."}  (or a non-JSON body, surfaced as text)
@@ -120,15 +121,25 @@ def validate_submission(sub: dict) -> list[str]:
     return problems
 
 
-def _post(url: str, api_key: str, payload: bytes, timeout: float = 60.0) -> tuple[int, dict | str]:
-    """POST raw JSON bytes with a bearer token. Returns (status_code, parsed-or-text body).
+def _auth_headers(api_key: str | None, access_token: str | None) -> dict[str, str]:
+    """Build one authentication header, preferring the scoped Access credential."""
+    if access_token:
+        return {"Cf-Access-Token": access_token}
+    if api_key:
+        return {"Authorization": f"Bearer {api_key}"}
+    return {}
+
+
+def _post(url: str, payload: bytes, auth_headers: dict[str, str],
+          timeout: float = 60.0) -> tuple[int, dict | str]:
+    """POST raw JSON bytes. Returns (status_code, parsed-or-text body).
 
     Isolated so tests can monkeypatch the single network call.
     """
     req = urllib.request.Request(
         url, data=payload, method="POST",
         headers={"Content-Type": "application/json",
-                 "Authorization": f"Bearer {api_key}",
+                 **auth_headers,
                  "User-Agent": "prb-submit"},
     )
     try:
@@ -147,7 +158,8 @@ def _maybe_json(text: str):
         return text
 
 
-def submit(path: Path, url: str, api_key: str, *, dry_run: bool = False,
+def submit(path: Path, url: str | None, api_key: str | None = None, *,
+           access_token: str | None = None, dry_run: bool = False,
            timeout: float = 60.0, mark_test: bool = False) -> dict:
     """Validate and (unless dry_run) upload a submission. Returns a result dict.
 
@@ -169,15 +181,22 @@ def submit(path: Path, url: str, api_key: str, *, dry_run: bool = False,
 
     if not url:
         raise ValueError("no endpoint URL — set PRB_SUBMIT_URL or pass --url")
-    if not api_key:
-        raise ValueError("no API key — set PRB_API_KEY or pass --api-key")
+    auth_headers = _auth_headers(api_key, access_token)
+    if not auth_headers:
+        raise ValueError("no intake credential — set PRB_ACCESS_TOKEN (preferred) or "
+                         "PRB_API_KEY (legacy)")
 
     payload = json.dumps(sub).encode("utf-8")
-    status, body = _post(url, api_key, payload, timeout=timeout)
+    status, body = _post(url, payload, auth_headers, timeout=timeout)
     if not (200 <= status < 300):
         reason = body.get("error") if isinstance(body, dict) else body
         raise ValueError(f"endpoint returned HTTP {status}: {reason}")
-    return body if isinstance(body, dict) else {"status": "accepted", "response": body}
+    if not isinstance(body, dict):
+        raise ValueError("endpoint returned a non-JSON response; Cloudflare Access login "
+                         "may be required")
+    if not body.get("submission_id"):
+        raise ValueError("endpoint accepted the request but returned no submission_id")
+    return body
 
 
 def main() -> None:
@@ -188,7 +207,7 @@ def main() -> None:
     parser.add_argument("--url", default=os.environ.get("PRB_SUBMIT_URL"),
                         help="Intake endpoint URL (env PRB_SUBMIT_URL)")
     parser.add_argument("--api-key", default=os.environ.get("PRB_API_KEY"),
-                        help="Bearer token for the endpoint (env PRB_API_KEY)")
+                        help="Legacy bearer token (env PRB_API_KEY); prefer PRB_ACCESS_TOKEN")
     parser.add_argument("--dry-run", action="store_true",
                         help="Validate the file locally and report what would be sent; no upload")
     parser.add_argument("--test", action="store_true",
@@ -202,7 +221,8 @@ def main() -> None:
         parser.error(f"no such file: {path}")
 
     try:
-        result = submit(path, args.url, args.api_key, dry_run=args.dry_run,
+        result = submit(path, args.url, args.api_key,
+                        access_token=os.environ.get("PRB_ACCESS_TOKEN"), dry_run=args.dry_run,
                         timeout=args.timeout, mark_test=args.test)
     except (ValueError, json.JSONDecodeError, OSError) as e:
         print(f"✗ {e}", file=sys.stderr)

benchmark/tests/test_submit.py

@@ -81,10 +81,29 @@ def test_invalid_submission_raises_before_network(self, tmp_path, monkeypatch):
 
     def test_success_returns_body(self, tmp_path, monkeypatch):
         monkeypatch.setattr(sub, "_post",
-                            lambda url, key, payload, timeout=60.0: (201, {"submission_id": "abc", "status": "accepted"}))
+                            lambda url, payload, headers, timeout=60.0: (
+                                201, {"submission_id": "abc", "status": "accepted"}))
         out = sub.submit(_valid(tmp_path), "https://x/submit", "k")
         assert out["submission_id"] == "abc"
 
+    def test_access_token_is_preferred_over_legacy_key(self, tmp_path, monkeypatch):
+        def post(url, payload, headers, timeout=60.0):
+            assert headers == {"Cf-Access-Token": "access-jwt"}
+            return 201, {"submission_id": "abc", "status": "accepted"}
+
+        monkeypatch.setattr(sub, "_post", post)
+        out = sub.submit(_valid(tmp_path), "https://x/submit", "legacy-key",
+                         access_token="access-jwt")
+        assert out["submission_id"] == "abc"
+
+    def test_legacy_key_uses_authorization_header(self, tmp_path, monkeypatch):
+        def post(url, payload, headers, timeout=60.0):
+            assert headers == {"Authorization": "Bearer legacy-key"}
+            return 201, {"submission_id": "abc", "status": "accepted"}
+
+        monkeypatch.setattr(sub, "_post", post)
+        sub.submit(_valid(tmp_path), "https://x/submit", "legacy-key")
+
     def test_non_2xx_raises(self, tmp_path, monkeypatch):
         monkeypatch.setattr(sub, "_post",
                             lambda *a, **k: (429, {"error": "quota exceeded"}))
@@ -94,5 +113,15 @@ def test_non_2xx_raises(self, tmp_path, monkeypatch):
     def test_missing_url_or_key_raises(self, tmp_path):
         with pytest.raises(ValueError, match="URL"):
             sub.submit(_valid(tmp_path), "", "k")
-        with pytest.raises(ValueError, match="API key"):
+        with pytest.raises(ValueError, match="intake credential"):
             sub.submit(_valid(tmp_path), "https://x/submit", "")
+
+    def test_non_json_success_is_not_reported_as_accepted(self, tmp_path, monkeypatch):
+        monkeypatch.setattr(sub, "_post", lambda *a, **k: (200, "Access login page"))
+        with pytest.raises(ValueError, match="non-JSON.*Access login"):
+            sub.submit(_valid(tmp_path), "https://x/submit", access_token="access-jwt")
+
+    def test_success_without_submission_id_is_rejected(self, tmp_path, monkeypatch):
+        monkeypatch.setattr(sub, "_post", lambda *a, **k: (201, {"status": "accepted"}))
+        with pytest.raises(ValueError, match="no submission_id"):
+            sub.submit(_valid(tmp_path), "https://x/submit", access_token="access-jwt")

intake/cloudflare-worker/README.md

@@ -1,34 +1,59 @@
 # prb submission intake (Cloudflare Worker)
 
 The write-only, confidential intake for `prb submit`. Submitters POST their `submission.json`
-over HTTPS; this Worker checks a bearer key and deposits the raw body (certificates plus the
-bounded submit ledger) into a **private R2 bucket**. The public leaderboard only receives the
-aggregate that `score-from-r2.yml` derives.
+over HTTPS; Cloudflare Access authenticates the submitter with GitHub, this Worker verifies
+the application JWT, and then deposits the raw body (certificates plus the bounded submit
+ledger) into a **private R2 bucket**. The public leaderboard only receives the aggregate that
+`score-from-r2.yml` derives.
 
 ```
-prb submit ──POST /submit──▶ Worker ──put──▶ R2 s3://prb-submissions/incoming/<ts>-<uuid>.json
-                              (Bearer PRB_API_KEY)                     (private)
+prb submit ──GitHub Access──▶ Worker ──put──▶ R2 s3://prb-submissions/incoming/<ts>-<uuid>.json
+             (short JWT)      (verify JWT)                              (private)
 ```
 
 ## One-time setup (maintainer)
 
 ```bash
-npm i -g wrangler
-wrangler login
+cd intake/cloudflare-worker
+npm install
+npx wrangler login
 
 # 1. private bucket for raw submissions
-wrangler r2 bucket create prb-submissions
+npx wrangler r2 bucket create prb-submissions
 
-# 2. the bearer token submitters will use (pick a strong value)
-wrangler secret put PRB_API_KEY        # paste the token
+# 2. migration-only bearer token; remove it after the Access test succeeds
+npx wrangler secret put PRB_API_KEY
 
-# 3. deploy
-cd intake/cloudflare-worker
-wrangler deploy
+# 3. deploy the legacy-compatible Worker once
+npm run deploy
 # → registers your workers.dev subdomain (e.g. prb-bench) and prints the endpoint,
 #   e.g. https://intake.prb-bench.workers.dev
 ```
 
+### GitHub-backed Cloudflare Access
+
+Create a GitHub identity provider under **Zero Trust → Integrations → Identity providers**.
+Then create a self-hosted Access application for the `intake` Worker. During migration,
+protect the preview hostname first. Its Allow policy should select the intended GitHub
+organization/team, or exact GitHub-verified emails together with `Login Methods: GitHub`;
+never use `Everyone`.
+
+The checked-in `TEAM_DOMAIN` and `POLICY_AUD` in `wrangler.toml` identify the configured
+Access organization and application. They are public identifiers, not credentials. The
+GitHub OAuth client secret stays in Cloudflare and must not be added to this repository.
+
+Upload an authenticated preview without promoting it to production:
+
+```bash
+npm run preview
+# → https://access-auth-intake.<workers-subdomain>.workers.dev
+```
+
+The Worker validates the `Cf-Access-Jwt-Assertion` signature, issuer, audience, and expiry
+against Cloudflare's rotating JWKS. It records the verified Access subject/email separately
+from the submitter-claimed `submitted_by` field. If an assertion is present but invalid, the
+request is rejected and cannot fall back to `PRB_API_KEY`.
+
 The Worker only writes to R2 — no GitHub token needed. Scoring is picked up by
 `.github/workflows/score-from-r2.yml` on its **daily cron** (or trigger it manually via
 `workflow_dispatch`). It scores privately and opens a PR; the maintainer reviewing +
@@ -40,30 +65,29 @@ submission-format failures go to `failed/` with diagnostics under `failed-status
 retryable verifier/infrastructure failures stay in `incoming/`. An upload that arrives
 during scoring is not in that run's snapshot and remains queued for the next run.
 
-Give submitters the endpoint URL + a key:
+Submitters authenticate in their own browser and obtain a short-lived, application-scoped
+token. No maintainer-issued secret or GitHub personal access token is involved:
 
 ```bash
-export PRB_SUBMIT_URL=https://intake.prb-bench.workers.dev/submit
-export PRB_API_KEY=<token>
-python3 -m benchmark.submit --predictions out/submission.json --test
+export PRB_SUBMIT_URL=https://access-auth-intake.prb-bench.workers.dev/submit
+PRB_ACCESS_APP="${PRB_SUBMIT_URL%/submit}"
+PRB_ACCESS_TOKEN="$(cloudflared access token -app="$PRB_ACCESS_APP")" \
+  python3 -m benchmark.submit --predictions out/submission.json --test
 ```
 
 Remove `--test` only when the run is ready to become an official leaderboard submission.
+Keep the `cloudflared access token` call inside command substitution: running
+`cloudflared access login` or `cloudflared access token` by itself may print the JWT. Do not
+print, persist, or pass `PRB_ACCESS_TOKEN` as a command-line argument. The client prefers it
+over an ambient legacy `PRB_API_KEY` and never sends both.
 
-### Authentication direction
+After an end-to-end preview test succeeds, change the Access destination from Preview Worker
+to the production `intake` Worker, deploy the tested version with `npm run deploy`, repeat the
+`--test` upload against the production URL, and remove the migration secret:
 
-`PRB_API_KEY` is the legacy bootstrap mode. For multiple submitters, prefer placing this
-Worker behind [Cloudflare Access](https://developers.cloudflare.com/cloudflare-one/access-controls/applications/choose-application-type/),
-using [GitHub as the identity provider](https://developers.cloudflare.com/cloudflare-one/integrations/identity-providers/github/),
-and authorizing the intended GitHub organization, team, or accounts. Cloudflare supports
-protecting a Worker directly, including its `workers.dev` route, and end users can obtain an
-application-scoped CLI session with `cloudflared access login`.
-
-Do not accept a user's `gh auth token` or GitHub personal access token. Before removing the
-legacy key, update the submit client to send the application-scoped Access token and make the
-Worker validate the signed `Cf-Access-Jwt-Assertion` issuer and audience as documented by
-Cloudflare. Until both pieces are deployed, the repository skill reports GitHub self-service
-authentication as unavailable rather than silently weakening authentication.
+```bash
+npx wrangler secret delete PRB_API_KEY
+```
 
 ## R2 credentials for the scoring worker
 
@@ -74,8 +98,8 @@ R2 → *Manage API Tokens*, create an **Object Read & Write** token, then add th
 See `.github/workflows/score-from-r2.yml`.
 
 ## Notes
-- Bearer auth is a single shared secret for now; per-submitter keys / quotas can move to a
-  Worker KV lookup later without changing `prb submit`.
+- `PRB_API_KEY` is only a migration fallback for requests with no Access assertion.
+- `gh auth token`, GitHub PATs, and `GITHUB_TOKEN` are never intake credentials.
 - Max body 25 MB. For larger submissions, hand out an R2 presigned PUT URL
   instead of POSTing the body — not needed at current scale.
 - The intake response intentionally exposes only the opaque submission ID, not the internal