Wait for prod Cloud Run candidate readiness before smoke tests#3732
Merged
Conversation
The gunicorn early-bind entrypoint decouples revision readiness (port bound) from app readiness (country-package import finished, ~160s), so gcloud run deploy now returns while the app is still booting. The prod candidate job ran its smoke tests immediately after deploy and hit the importing instance, failing with read timeouts; the staging job was unaffected because it already gates test traffic on health_check.sh. Add the same readiness wait against the candidate tag URL before the smoke tests. Fixes #3731 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #3732 +/- ##
=======================================
Coverage 79.82% 79.82%
=======================================
Files 70 70
Lines 4336 4336
Branches 808 808
=======================================
Hits 3461 3461
Misses 655 655
Partials 220 220 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #3731
Summary
The Stage 1 observation deploy (run 28811205668) failed at Run Cloud Run candidate smoke tests with
httpx.ReadTimeout. Correlated revision logs show the mechanism: the gunicorn early-bind entrypoint (#3720) makesgcloud run deployreturn the moment the port binds (18:07:13), while the app import completes ~2.7 minutes later (18:09:55,Application startup complete) — and the prod job fired its smoke tests in between (~18:08:50). The step order relied on the old uvicorn boot behavior where port-bound implied app-ready.No traffic was affected: the failure precedes the promote step, and App Engine production promoted successfully in the same run. The staging job already had the correct pattern (health wait before test traffic), which is why it passed.
Change
One step in
push.yml'sdeploy-cloud-run-candidatejob:Wait for Cloud Run candidate health—health_check.shpolling the candidate tag URL's/readiness-check(900s budget, 15s request timeout, 5s interval) — inserted between URL resolution and the smoke tests. The prod job is now symmetric with staging: deploy → resolve → wait until app-ready → smoke test → promote → existing post-promote service-URL health check.Verification
tests/unit/test_cloud_run_deploy_scripts.py: 30/30, including the workflow-structure invariant that promote follows the smoke tests.After merge
This PR's push cycle becomes the Stage 1 exit-gate observation run — third attempt, with both post-merge discoveries (tag length, smoke-test race) fixed.
🤖 Generated with Claude Code