Isolate staging Cloud Run service and add backend response header (migration PR 4, stage 1)#3720
Merged
Merged
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #3720 +/- ##
==========================================
+ Coverage 79.72% 79.82% +0.09%
==========================================
Files 70 70
Lines 4326 4336 +10
Branches 807 808 +1
==========================================
+ Hits 3449 3461 +12
+ Misses 657 655 -2
Partials 220 220 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Migration PR 4 stage 1. The staging Cloud Run track previously promoted staging-configured revisions to 100% of the production policyengine-api service URL on every push; once a load balancer fronts that service this becomes a public incident per deploy. Staging now deploys and promotes its own policyengine-api-staging service, with the image name decoupled from the service name because production reuses the staging-built image. Every API response now carries X-PolicyEngine-Backend (app_engine or cloud_run) so traffic-weight changes and rollbacks during the host cutover can be verified in-band with sampled requests. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Pin CLOUD_RUN_SERVICE explicitly on the production Cloud Run job and add workflow invariants: every service-targeting Cloud Run job must pin a service, and only the production job may promote the production service. The cloud_run_env.sh default silently targeting production was the mechanism behind the original staging-promotes-to-prod bug. Drop the dead try/except around the backend header assignment; get_api_host_backend() is fail-safe by design, so the except could only mask real header-assignment bugs, and the ASGI path never had one. Document the one-time staging service bootstrap (gcloud rejects --no-traffic when creating a new service, and the deploy service account cannot set the public invoker binding) in a Stage 1 runbook. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Cold-start imports (~232s, dominated by country package construction) sit on the edge of Cloud Run's hard 240s startup probe cap, which cannot be extended by any probe configuration. Startup CPU boost shortens the import within that cap; the structural fix (binding the port before app import) follows in the next migration stage. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Cloud Run's startup probe hard-caps total probe time at 240s, and the app import (~232s, dominated by country package construction) races it on every instance start, failing roughly half of first-attempt deploys. gunicorn's master binds the listen socket before forking workers, and without --preload the import happens in the worker post-fork, so the probe passes in milliseconds and readiness is governed by the pipeline's health_check.sh /readiness-check poll (900s budget) instead of the platform cap. --timeout 0 disables the worker heartbeat watchdog that would otherwise kill the worker mid-import; --keep-alive 5 and --forwarded-allow-ips preserve today's uvicorn keep-alive and proxy header behavior (UvicornWorker leaves proxy_headers at uvicorn's default of enabled). health_check.sh curls now carry --max-time (default 10s, overridable via HEALTH_CHECK_CURL_MAX_TIME_SECONDS): with the port open before the app can answer, an unanswered poll would otherwise block until Cloud Run's 300s request timeout. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ba9b32e to
f061823
Compare
docs/ was accumulating per-PR runbooks that interleave one-time records with operational reference, and the PR 3 runbook described the shared single-service behavior this PR removes. Split the two concerns: docs/migration/cloud-run-operations.md is the single living reference (topology, startup behavior, IAM constraints, verification commands), edited in place as behavior changes; per-stage records move to docs/migration/history/ as append-only documents with banners pointing at the living doc. The Stage 1 record now covers the startup de-flake items and carries the full amended exit-gate list. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This was referenced Jul 6, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Stage 1 of the API host cutover ("PR 4" of the unified API v2-alpha migration plan): before a load balancer can front the production Cloud Run service, staging deploys must stop promoting to it.
policyengine-api-staging.deploy-cloud-run-stagingandpromote-cloud-run-stagingpreviously used thecloud_run_env.shdefaultCLOUD_RUN_SERVICE=policyengine-api, so every push promoted a staging-configured revision to 100% of the production service URL for the duration of the pipeline. Both jobs now pinCLOUD_RUN_SERVICE: policyengine-api-stagingat the job level. The first staging deploy after merge auto-creates the new service.CLOUD_RUN_IMAGE_URIembedded${CLOUD_RUN_SERVICE}; production reuses the staging-built image, so the image path now uses a fixedCLOUD_RUN_IMAGE_NAME(defaultpolicyengine-api) regardless of service override.X-PolicyEngine-Backend(app_engine/cloud_run, from the existingAPI_HOST_BACKENDmigration flag). Set in the shared Flaskafter_requesthook (covers App Engine and Cloud Run's Flask-mounted routes) and in the ASGI middleware for native FastAPI routes. This is the in-band signal for verifying traffic-weight changes and rollbacks during the ramp with sampled requests; the helper never raises, falling back to the default backend on an invalid flag.Test plan
pytest tests/unit/test_cloud_run_deploy_scripts.py tests/unit/routes/test_migration_context_logging.py tests/unit/test_migration_flags.py tests/unit/test_asgi_factory.py— 72 passed, including new invariants: staging jobs pin the staging service, production job never references it, image URI is service-independent, deploy/promote dry-runs target the overridden service, and backend-header behavior on Flask, native FastAPI, and hook-less fallback paths.ruff format --checkclean on changed files.gcloud run services describe policyengine-apishows onlystage3-prod-*tags at 100% throughout;policyengine-api-staginghealthy and passing the staging integration suite; header present on GAE prod, CR staging, and CR prod URLs.🤖 Generated with Claude Code