Speed up docs publishes by reusing the previously built CI image - #70650
Merged
Conversation
potiuk
requested review from
amoghrajesh,
ashb,
bugraoz93,
choo121600,
ephraimbuddy,
gopidesupavan,
jason810496,
jedcunningham,
jscheffl and
vatsrahul1001
as code owners
July 28, 2026 22:43
boring-cyborg
Bot
added
area:dev-tools
backport-to-v3-3-test
Publishing docs for a ref cut before main pays for a from-scratch CI image build. The registry cache the build reads is main's, and the first Dockerfile.ci change made since the ref was cut breaks the cache chain at the third layer - #70285, which bakes Node.js and pnpm into the image, is by itself enough to make the twenty-minute Python build run again. Nothing caches those layers for the ref being documented except the image the previous publish of that same ref already built. That image could not serve as a cache until now. BuildKit imports the cache manifest of an image loaded from a tarball and then discards every record whose layers it cannot pull from a registry, so a docker load contributes nothing at all; serving the image from a registry on localhost is what makes its records usable. Recording those records in the first place needs inline cache metadata, which no image carried before.
potiuk
force-pushed
the
seed-ci-image-cache-from-stashed-image
branch
from
July 29, 2026 08:08
58b4fad to
5269813
Compare
uranusjr
approved these changes
Jul 29, 2026
Contributor
Backport failed to create: v3-3-test. View the failure log Run detailsNote: As of Merging PRs targeted for Airflow 3.X In matter of doubt please ask in #release-management Slack channel.
You can attempt to backport this manually by running: cherry_picker d9d8fdb v3-3-testThis should apply the commit to the v3-3-test branch and leave the commit in conflict state marking After you have resolved the conflicts, you can continue the backport process by running: cherry_picker --continueIf you don't have cherry-picker installed, see the installation guide. |
1 task
potiuk
added a commit
that referenced
this pull request
Jul 29, 2026
Docs for a ref are published more than once - an RC, then the final docs after the vote - and the second publish rebuilds an image the first one already built from the very same commit. Seeding the build cache with that image, as #70650 does, still pays for a docker load, a registry push, a build that hits cache on every layer, an export and two multi-gigabyte stash uploads, none of which can produce anything the first publish did not already produce. Reusing the image outright rather than as a cache turns on knowing it was built from this commit. A ref does not say that: a branch documented twice moves between the two publishes, so its image is a good cache and a wrong answer. The commit each image is stashed with is what tells those apart, and it is stashed on its own so that deciding costs a few bytes rather than the image the decision may make unnecessary. Reusing it also means publishing nothing, which the shared per-branch stash could not support: it is written by every build on this branch, so whoever restores it next would get sources they never asked for. The ref's own stash is the one this run's docs and registry jobs now read, so a run with nothing to build leaves them reading what the previous publish left there. The mount cache moves with it for the same reason - it holds the dependency set the sources resolve to, and a ref's and the branch tip's are exactly what differ.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Docs publishes rebuild the CI image from scratch whenever the ref being documented was cut before the last
Dockerfile.cichange on main. The registry cache the build reads is main's, so #70285 (Node.js/pnpm baked into the image) breaks the cache chain at layer 3 of 26 and the ~20 minute Python build runs again. Measured on run 30398809294: 1 of 26 layersCACHED,install_os_dependencies.sh1173.7s, build step 25m24s. A canary main build, whose sources match its cache, is 6m42s.The image the previous publish of that same ref stashed matches those layers exactly, so this seeds the build from it.
ci-image-build.ymlgainsseed-cache-from-stashed-image(default off). When on, it restores the stashed image, republishes it to aregistry:2on localhost and passes it to the build as an extra--cache-from.docker loaded image and then discards every record whose layers it cannot pull from a registry. Verified locally with the same image, same builder and same args: throughdocker loadevery layer re-ran, throughlocalhost:5000every layer came backCACHED.BUILDKIT_INLINE_CACHE=1so they carry those records at all. Also verified that this arg does not participate in layer cache keys, so it does not invalidate the existing registry cache when it lands.image-stash-suffixkeeps a per-ref copy of the image stash, retained 6 days because the same ref is rebuilt at least three days apart. Seeding prefers that copy and falls back to the shared per-branch stash, which Tests AMD keeps warm for main.Only
publish-docs-to-s3.ymlopts in. Tests AMD is unchanged apart from the inline cache record in the images it produces.Expected effect: re-publishing the same ref should build in a couple of minutes instead of 25; a different ref that shares the image-defining files should skip the 20 minute layer; a genuine miss costs about 6 minutes of seeding once, and that run then writes its own per-ref slot.
Was generative AI tooling used to co-author this PR?
Generated-by: Claude Code (Opus 5) following the guidelines