Skip to content

CI: harden container builds against transient registry and CDN failures#100

Merged
aviator-app[bot] merged 3 commits into
mainfrom
rjh.ci-harden-docker-hub-pulls
Jul 20, 2026
Merged

CI: harden container builds against transient registry and CDN failures#100
aviator-app[bot] merged 3 commits into
mainfrom
rjh.ci-harden-docker-hub-pulls

Conversation

@rjhuijsman

@rjhuijsman rjhuijsman commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

Transient network failures -- 5xx responses and SSL/TCP connection resets from registries and download CDNs -- kept killing CI builds. Any failed network fetch in the container build path would break the whole CI.

This PR prevents these types of failures by hardening some network fetches:

  • Base-image pulls: before each devcontainer build, the base image is pulled into the runner's local image store with retries and backoff via the pre-pull-docker-image composite action from 3rdparty/dev-tools (Add a pre-pull-docker-image composite action 3rdparty/dev-tools#101). The devcontainers/ci build uses the default docker buildx driver, which resolves FROM images from the local store, so a pre-pulled base never touches the registry on the critical path. The development-containers workflow's build-test-push job gets the same pre-pull for reboot/containers/development.
  • Raw downloads in the Dockerfiles: every curl/wget fetch in the repo-root and development container Dockerfiles now retries transient failures via native flags (e.g. curl --retry 5 --retry-all-errors).

And while we're here...

  • Base-image sync: the development container's Dockerfile had gone stale on ubuntu:jammy-20240808; it now shares the repo-root Dockerfile's pinned base.

@aviator-app

aviator-app Bot commented Jul 20, 2026

Copy link
Copy Markdown

Current Aviator status

Aviator will automatically update this comment as the status of the PR changes.
Comment /aviator refresh to force Aviator to re-examine your PR (or learn about other /aviator commands).

This PR was merged using Aviator (commit f781c03).


See the real-time status of this PR on the Aviator webapp.
Use the Aviator Chrome Extension to see the status of your PR within GitHub.

@rjhuijsman
rjhuijsman force-pushed the rjh.ci-harden-docker-hub-pulls branch from 2ffe6c9 to d8ef25a Compare July 20, 2026 15:31
@rjhuijsman
rjhuijsman requested review from Copilot and onelxj and removed request for onelxj July 20, 2026 16:13
@rjhuijsman rjhuijsman self-assigned this Jul 20, 2026
@rjhuijsman
rjhuijsman marked this pull request as ready for review July 20, 2026 16:14

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR hardens CI container builds against transient registry/CDN failures by adding retries to Dockerfile network fetches and pre-pulling base images in GitHub Actions to keep registries off the build critical path.

Changes:

  • Pre-pull devcontainer/development container base images (with retries/backoff via a composite action) before running builds in CI workflows.
  • Add retry/backoff flags to curl/wget downloads in Dockerfiles and avoid piping installer scripts directly into interpreters.
  • Align the development container’s pinned Ubuntu base image with the repo-root Dockerfile.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 8 comments.

File Description
reboot/containers/development/Dockerfile Align base image with repo root and add retrying downloads + safer script execution.
Dockerfile Add retrying downloads and switch several script installs from pipe-to-shell to file-then-run.
.github/workflows/development_containers_build_test_push.yml Add pre-pull step for the development container base image before builds.
.github/workflows/devcontainer.yml Add pre-pull step for the devcontainer base image before devcontainers/ci build.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread reboot/containers/development/Dockerfile Outdated
Comment thread Dockerfile Outdated
Comment thread Dockerfile Outdated
Comment thread Dockerfile Outdated
Comment thread .github/workflows/devcontainer.yml
Comment thread .github/workflows/development_containers_build_test_push.yml
Comment thread .github/workflows/devcontainer.yml
Comment thread .github/workflows/development_containers_build_test_push.yml
Comment thread Dockerfile Outdated
claude added 3 commits July 20, 2026 21:20
Transient registry failures -- 5xx responses from Docker Hub and other registries -- kept aborting `docker buildx` before it could resolve a base image, failing CI jobs before they ran a single test; reruns fixed it, but the flakes kept bouncing PRs and burning hours. Every build job in `build_and_test.yml` builds the devcontainer through the reusable `devcontainer.yml` workflow via `devcontainers/ci`, and the development-containers workflow's `build-test-push` job builds `reboot/containers/development` twice, so any single base-image pull was a job-wide point of failure.

Added a step to both workflows that pre-pulls the base image into the runner's local image store with retries before the build, via the parameterized `pre-pull-docker-image` composite action from dev-tools, taking the registry off the build's critical path.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UPXVU5ytyWJxSL4YChKhm6
The development container's `Dockerfile` had gone stale on `ubuntu:jammy-20240808` while the repo-root `Dockerfile` moved on to `ubuntu:jammy-20251001`; the two are supposed to share the same pinned base. Updated the development container to the same tag (verified multi-arch: amd64 and arm64 manifests exist). No other file pins or echoes either tag.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UPXVU5ytyWJxSL4YChKhm6
CDNs and download hosts intermittently reset connections or return 5xx errors, and a single such failure in any download aborted the whole container build before CI ran a single test -- most recently `curl: (35) OpenSSL SSL_connect: Connection reset by peer` fetching the awscli installer.

Gave every raw `curl` and `wget` download in the repo-root and development container `Dockerfile`s retry semantics using the tools' native flags, and downloads that piped installer scripts straight into an interpreter now fetch to a temporary file and execute it instead, so a torn stream can no longer reach an interpreter. Nothing about what gets downloaded changes; validated by building the `respect-current-minimum` stage and the development image through its `uv` install.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UPXVU5ytyWJxSL4YChKhm6
@aviator-app
aviator-app Bot force-pushed the rjh.ci-harden-docker-hub-pulls branch from ff1857a to 9e35b77 Compare July 20, 2026 21:20
@aviator-app
aviator-app Bot merged commit f781c03 into main Jul 20, 2026
20 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants