Failure
The Stage 1 observation deploy (run https://github.com/PolicyEngine/policyengine-api/actions/runs/28811205668) failed at Deploy production Cloud Run candidate → Run Cloud Run candidate smoke tests with httpx.ReadTimeout on test_cloud_run_candidate_health_routes. Everything else in the chain — including the staging Cloud Run leg with the #3730 tag fix — passed. No traffic was affected: the failure precedes the promote step, and App Engine production promoted successfully in the same run.
Cause
The gunicorn early-bind entrypoint (#3720) deliberately decoupled "platform-ready" (port bound, revision Ready, gcloud run deploy returns) from "app-ready" (country-package import finished, ~161s with CPU boost). Correlated timeline from the revision logs (policyengine-api-00035-vud):
- 18:07:13 — gunicorn binds the port → deploy step succeeds
- ~18:08:50 — smoke tests fire against the tag URL, mid-import
- 18:09:55 —
Application startup complete (~65s too late for the tests)
The prod candidate job's step order (deploy → resolve URL → smoke tests → promote → health wait) silently relied on the old uvicorn boot order, where the port binding implied the import was done. The staging job is immune because it already runs health_check.sh before any test traffic.
Fix
Insert a Wait for Cloud Run candidate health step (health_check.sh against the candidate tag URL's /readiness-check, 900s budget) between URL resolution and the smoke tests in deploy-cloud-run-candidate, making the prod job symmetric with staging.
Failure
The Stage 1 observation deploy (run https://github.com/PolicyEngine/policyengine-api/actions/runs/28811205668) failed at Deploy production Cloud Run candidate → Run Cloud Run candidate smoke tests with
httpx.ReadTimeoutontest_cloud_run_candidate_health_routes. Everything else in the chain — including the staging Cloud Run leg with the #3730 tag fix — passed. No traffic was affected: the failure precedes the promote step, and App Engine production promoted successfully in the same run.Cause
The gunicorn early-bind entrypoint (#3720) deliberately decoupled "platform-ready" (port bound, revision
Ready,gcloud run deployreturns) from "app-ready" (country-package import finished, ~161s with CPU boost). Correlated timeline from the revision logs (policyengine-api-00035-vud):Application startup complete(~65s too late for the tests)The prod candidate job's step order (deploy → resolve URL → smoke tests → promote → health wait) silently relied on the old uvicorn boot order, where the port binding implied the import was done. The staging job is immune because it already runs
health_check.shbefore any test traffic.Fix
Insert a
Wait for Cloud Run candidate healthstep (health_check.shagainst the candidate tag URL's/readiness-check, 900s budget) between URL resolution and the smoke tests indeploy-cloud-run-candidate, making the prod job symmetric with staging.