Stop idle App Engine versions after deploy#3733
Draft
anth-volk wants to merge 2 commits into
Draft
Conversation
App Engine Flexible keeps an always-on VM for every SERVING version regardless of traffic, so the staging version and superseded prod versions left SERVING after each deploy keep costing ~$212/mo each. The pipeline promoted traffic but never deactivated old versions. - Stop the staging version after its integration tests pass. - After a production promote, keep the live version plus the newest 2 prod versions warm for rollback and stop every other SERVING version; delete stopped versions beyond the newest 20 to respect App Engine's 210-versions-per-service limit. Stopping (not deleting) preserves rollback via `gcloud app versions start` + `set-traffic`. Adds stop_app_engine_version.sh and cleanup_app_engine_versions.sh, plus two jobs in push.yml. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #3733 +/- ##
=======================================
Coverage 79.82% 79.82%
=======================================
Files 70 70
Lines 4336 4336
Branches 808 808
=======================================
Hits 3461 3461
Misses 655 655
Partials 220 220 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
The stopped-version retention was a single newest-20 pool shared across prod and staging. Since only prod versions are meaningful rollback targets, keep the newest 10 stopped prod versions and only the newest 3 stopped staging versions; delete everything else stopped (older prod, older staging, and legacy timestamp-named versions). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
The v1 API runs on App Engine Flexible, which keeps at least one always-on VM for every
SERVINGversion regardless of traffic (currentapp.yaml: 4 vCPU / 24 GB ≈ ~$212/mo per version). The deploy pipeline promotes traffic to the new version but never deactivates the ones it supersedes, so both staging and old prod versions accumulate as idle-but-billing VMs. A recent snapshot showed ~11–14 VMs running with only one serving traffic — the bulk of a ~$6,050/mo App Engine bill.This adds post-deploy cleanup so idle versions stop costing money, while preserving rollback.
Changes
stop-staging-app-engine-version(afterintegration-tests-staging): stops the staging version once its tests pass. Staging holds 0 VMs between deploys; each deploy makes a fresh one.cleanup-prod-app-engine-versions(afterpromote-production): keeps the live version plus the newest 2 prod versions warm for instant rollback, and stops every otherSERVINGversion (older prod + leftover staging). It then prunes stopped versions to the newest 10 prod and newest 3 staging, deleting everything else stopped (older prod, older staging, and legacy timestamp-named versions) to stay under App Engine's 210-versions-per-service limit. Never touches the version receiving traffic.stop_app_engine_version.shandcleanup_app_engine_versions.sh— idempotent,DRY_RUN=1supported, written for bash 3.2 so they dry-run locally too.Why stop, not delete (for the warm/rollback set)
Stopping deactivates the VM (→ ~$0) but keeps the version deployed and restartable, so rollback still works:
gcloud app versions start <v>thengcloud app services set-traffic default --splits=<v>=1. Keeping the 2 newest prod versions warm gives instant rollback, and the newest 10 stopped prod versions remain as cold rollback targets — the original reason old versions were kept SERVING (a new deploy's simulation-API dependency could fail). Only prod versions are meaningful rollback targets, so staging retention is kept small.Verification
gcloud app versions stop/startconfirmed working live on this project's automatic-scaling Flex versions (stoppedprod-2393→STOPPED→ started →SERVING); it's the sameVersions.UpdateVersioncall already used to trim the fleet on 2026-07-02. Noapp.yamlscaling-mode change needed.policyengine-apiproduced the expected plan: keepprod-2405(live) +prod-2403warm; stop 12 idle versions; keep the newest 10 stopped prod + 3 stopped staging and delete the other 68 stopped versions; traffic-serving version untouched.push.ymlvalidated as YAML; changelog fragment added; no Python files changed (format/lint gate N/A).Scope
Deploy-pipeline / cost only — no API contract or route changes; everything stays on the App Engine v1 path. This is a bridge until v1 finishes migrating to Cloud Run (min=0 = true scale-to-zero, at which point this machinery retires).
Expected impact
Steady-state running VMs drop from ~11–14 to 2 (live + previous prod); staging ~$0 except during a deploy's test window → roughly $6,050/mo → ~$450–500/mo.
🤖 Generated with Claude Code