Skip to content

Prod Cloud Run candidate smoke tests race the app import (no health wait before tests) #3731

Description

@anth-volk

Failure

The Stage 1 observation deploy (run https://github.com/PolicyEngine/policyengine-api/actions/runs/28811205668) failed at Deploy production Cloud Run candidate → Run Cloud Run candidate smoke tests with httpx.ReadTimeout on test_cloud_run_candidate_health_routes. Everything else in the chain — including the staging Cloud Run leg with the #3730 tag fix — passed. No traffic was affected: the failure precedes the promote step, and App Engine production promoted successfully in the same run.

Cause

The gunicorn early-bind entrypoint (#3720) deliberately decoupled "platform-ready" (port bound, revision Ready, gcloud run deploy returns) from "app-ready" (country-package import finished, ~161s with CPU boost). Correlated timeline from the revision logs (policyengine-api-00035-vud):

  • 18:07:13 — gunicorn binds the port → deploy step succeeds
  • ~18:08:50 — smoke tests fire against the tag URL, mid-import
  • 18:09:55 — Application startup complete (~65s too late for the tests)

The prod candidate job's step order (deploy → resolve URL → smoke tests → promote → health wait) silently relied on the old uvicorn boot order, where the port binding implied the import was done. The staging job is immune because it already runs health_check.sh before any test traffic.

Fix

Insert a Wait for Cloud Run candidate health step (health_check.sh against the candidate tag URL's /readiness-check, 900s budget) between URL resolution and the smoke tests in deploy-cloud-run-candidate, making the prod job symmetric with staging.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions