Build/test tools: retry incomplete Gutenberg build downloads - #12700
Build/test tools: retry incomplete Gutenberg build downloads#12700lancewillett wants to merge 1 commit into
Conversation
Some PHPUnit jobs fail before tests start when the prebuilt Gutenberg archive
download from the GitHub Container Registry closes early ("Premature close").
The downloader streams directly into extraction, leaving no clean retry
boundary. Download to a temporary file first, validate the manifest-declared
size, and retry interrupted transfers up to two more times. Log each attempt
and clean up temporary files. Non-transient HTTP failures still stop
immediately; workflow structure, caching, and artifact sharing are unchanged.
See #65720.
Props lance.willett@automattic.com
|
The following accounts have interacted with this PR and/or linked issues. I will continue to update these lists as activity occurs. You can also manually ask me to refresh this list by adding the Core Committers: Use this line as a base for the props when committing in SVN: To understand the WordPress project's expectations around crediting contributors, please review the Contributor Attribution page in the Core Handbook. |
There was a problem hiding this comment.
Pull request overview
This PR improves the robustness of the Gutenberg artifact fetch used by the WordPress develop tooling by adding a retryable, size-validated download step that writes to a temporary archive file before extraction, making interrupted transfers recoverable.
Changes:
- Add
downloadArchive()with bounded retries, linear backoff + jitter, and manifest-size validation before extraction. - Update
tools/gutenberg/download.jsto download the blob to a temp archive, then extract from disk (and clean up the temp file afterward). - Add Node test coverage for retry, size mismatch, cleanup, and non-retryable failures, plus a new
npm run test:toolsentry point.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| tools/gutenberg/download.js | Implements retrying, size-checked archive download to a temp file and disk-based extraction. |
| tools/gutenberg/tests/download.test.js | Adds unit + subprocess tests validating retry/backoff, size validation, and cleanup behavior. |
| package.json | Adds test:tools script to run the new Node tests. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| /** @type {import('stream/web').ReadableStream} */ ( response.body ) | ||
| ), | ||
| meter, | ||
| fs.createWriteStream( archivePath ) |
| } catch ( error ) { | ||
| console.error( '❌ Download/extraction failed:', /** @type {Error} */ ( error ).message ); | ||
| process.exit( 1 ); | ||
| process.exitCode = 1; | ||
| return; | ||
| } finally { | ||
| if ( archivePath ) { | ||
| fs.rmSync( archivePath, { force: true } ); | ||
| } | ||
| } |
Test using WordPress PlaygroundThe changes in this pull request can previewed and tested using a WordPress Playground instance. WordPress Playground is an experimental project that creates a full WordPress instance entirely within the browser. Some things to be aware of
For more details about these limitations and more, check out the Limitations page in the WordPress Playground documentation. |
lancewillett
changed the title
|
Closing in favor of #12701 (download the Gutenberg build once per run). That change already ships its own bounded retry with SHA-256 verification for the single remaining fetch, and it removes about 176 of the 177 per-run downloads this PR only hardened. So this retry is redundant once #12701 lands. |
Trac ticket: https://core.trac.wordpress.org/ticket/65720
Summary
Make the Gutenberg build download survive an interrupted transfer.
Non-transient HTTP failures still stop immediately. Workflow structure, caching, and artifact sharing are unchanged.
Why
PHPUnit jobs sometimes fail before a single test runs, with
Download/extraction failed: Premature close. The prebuilt Gutenberg archive is streamed from the GitHub Container Registry directly into extraction, so a stream that closes early corrupts the extraction with no clean boundary to retry from. Writing to a temporary file first creates that boundary, and validating the declared size means a short read is caught before it can look like a successful download.Follow up to the Gutenberg artifact-fetching work in #64393.
Testing
New unit tests cover the retry path, size validation, temp-file cleanup, and the non-transient failure case. Run them with:
or directly:
Related
Interim, focused retry fix. The larger download-once and cache work (Trac https://core.trac.wordpress.org/ticket/65721, PRs #12701 and #12702) reworks the same downloader and reimplements bounded retries more fully, so it can supersede this once it lands. This smaller change is safe to land first for immediate relief; the other then rebases and keeps its own implementation. Both add
tools/gutenberg/tests/download.test.jsand atest:toolsscript.