Skip to content

Isolate staging Cloud Run service and add backend response header (migration PR 4, stage 1)#3720

Merged
anth-volk merged 6 commits into
masterfrom
feat/migration-pr4-stage1-staging-isolation
Jul 6, 2026
Merged

Isolate staging Cloud Run service and add backend response header (migration PR 4, stage 1)#3720
anth-volk merged 6 commits into
masterfrom
feat/migration-pr4-stage1-staging-isolation

Conversation

@anth-volk

Copy link
Copy Markdown
Collaborator

Summary

Stage 1 of the API host cutover ("PR 4" of the unified API v2-alpha migration plan): before a load balancer can front the production Cloud Run service, staging deploys must stop promoting to it.

  • Isolate staging Cloud Run onto policyengine-api-staging. deploy-cloud-run-staging and promote-cloud-run-staging previously used the cloud_run_env.sh default CLOUD_RUN_SERVICE=policyengine-api, so every push promoted a staging-configured revision to 100% of the production service URL for the duration of the pipeline. Both jobs now pin CLOUD_RUN_SERVICE: policyengine-api-staging at the job level. The first staging deploy after merge auto-creates the new service.
  • Decouple the image name from the service name. CLOUD_RUN_IMAGE_URI embedded ${CLOUD_RUN_SERVICE}; production reuses the staging-built image, so the image path now uses a fixed CLOUD_RUN_IMAGE_NAME (default policyengine-api) regardless of service override.
  • Tag every API response with X-PolicyEngine-Backend (app_engine/cloud_run, from the existing API_HOST_BACKEND migration flag). Set in the shared Flask after_request hook (covers App Engine and Cloud Run's Flask-mounted routes) and in the ASGI middleware for native FastAPI routes. This is the in-band signal for verifying traffic-weight changes and rollbacks during the ramp with sampled requests; the helper never raises, falling back to the default backend on an invalid flag.

Test plan

  • pytest tests/unit/test_cloud_run_deploy_scripts.py tests/unit/routes/test_migration_context_logging.py tests/unit/test_migration_flags.py tests/unit/test_asgi_factory.py — 72 passed, including new invariants: staging jobs pin the staging service, production job never references it, image URI is service-independent, deploy/promote dry-runs target the overridden service, and backend-header behavior on Flask, native FastAPI, and hook-less fallback paths.
  • ruff format --check clean on changed files.
  • After merge (stage 1 exit gates): observe one full push cycle — gcloud run services describe policyengine-api shows only stage3-prod-* tags at 100% throughout; policyengine-api-staging healthy and passing the staging integration suite; header present on GAE prod, CR staging, and CR prod URLs.

🤖 Generated with Claude Code

@codecov

codecov Bot commented Jul 2, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 79.82%. Comparing base (446d995) to head (0839a96).
⚠️ Report is 9 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #3720      +/-   ##
==========================================
+ Coverage   79.72%   79.82%   +0.09%     
==========================================
  Files          70       70              
  Lines        4326     4336      +10     
  Branches      807      808       +1     
==========================================
+ Hits         3449     3461      +12     
+ Misses        657      655       -2     
  Partials      220      220              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

anth-volk and others added 5 commits July 6, 2026 18:01
Migration PR 4 stage 1. The staging Cloud Run track previously promoted
staging-configured revisions to 100% of the production policyengine-api
service URL on every push; once a load balancer fronts that service this
becomes a public incident per deploy. Staging now deploys and promotes
its own policyengine-api-staging service, with the image name decoupled
from the service name because production reuses the staging-built image.

Every API response now carries X-PolicyEngine-Backend (app_engine or
cloud_run) so traffic-weight changes and rollbacks during the host
cutover can be verified in-band with sampled requests.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Pin CLOUD_RUN_SERVICE explicitly on the production Cloud Run job and add
workflow invariants: every service-targeting Cloud Run job must pin a
service, and only the production job may promote the production service.
The cloud_run_env.sh default silently targeting production was the
mechanism behind the original staging-promotes-to-prod bug.

Drop the dead try/except around the backend header assignment;
get_api_host_backend() is fail-safe by design, so the except could only
mask real header-assignment bugs, and the ASGI path never had one.

Document the one-time staging service bootstrap (gcloud rejects
--no-traffic when creating a new service, and the deploy service account
cannot set the public invoker binding) in a Stage 1 runbook.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Cold-start imports (~232s, dominated by country package construction)
sit on the edge of Cloud Run's hard 240s startup probe cap, which
cannot be extended by any probe configuration. Startup CPU boost
shortens the import within that cap; the structural fix (binding the
port before app import) follows in the next migration stage.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Cloud Run's startup probe hard-caps total probe time at 240s, and the
app import (~232s, dominated by country package construction) races it
on every instance start, failing roughly half of first-attempt deploys.
gunicorn's master binds the listen socket before forking workers, and
without --preload the import happens in the worker post-fork, so the
probe passes in milliseconds and readiness is governed by the
pipeline's health_check.sh /readiness-check poll (900s budget) instead
of the platform cap. --timeout 0 disables the worker heartbeat watchdog
that would otherwise kill the worker mid-import; --keep-alive 5 and
--forwarded-allow-ips preserve today's uvicorn keep-alive and proxy
header behavior (UvicornWorker leaves proxy_headers at uvicorn's
default of enabled).

health_check.sh curls now carry --max-time (default 10s, overridable
via HEALTH_CHECK_CURL_MAX_TIME_SECONDS): with the port open before the
app can answer, an unanswered poll would otherwise block until Cloud
Run's 300s request timeout.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@anth-volk anth-volk force-pushed the feat/migration-pr4-stage1-staging-isolation branch from ba9b32e to f061823 Compare July 6, 2026 16:02
docs/ was accumulating per-PR runbooks that interleave one-time records
with operational reference, and the PR 3 runbook described the shared
single-service behavior this PR removes. Split the two concerns:
docs/migration/cloud-run-operations.md is the single living reference
(topology, startup behavior, IAM constraints, verification commands),
edited in place as behavior changes; per-stage records move to
docs/migration/history/ as append-only documents with banners pointing
at the living doc. The Stage 1 record now covers the startup de-flake
items and carries the full amended exit-gate list.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@anth-volk anth-volk marked this pull request as ready for review July 6, 2026 17:10
@anth-volk anth-volk merged commit 245a4b9 into master Jul 6, 2026
10 checks passed
@anth-volk anth-volk deleted the feat/migration-pr4-stage1-staging-isolation branch July 6, 2026 17:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant