Build/Test Tools: Trim the PHPUnit matrix to boundary PHP versions (full matrix runs weekly) - #12719
Conversation
Reduce the PHPUnit matrix on the active branch to the highest and lowest supported version of each major PHP branch, matching the boundary testing already used on older branches. Version-specific failures cluster at the boundaries, so the intermediate minor versions add job count without adding coverage. This is especially useful when preparing backports across multiple branches, where the full matrix multiplies queued jobs and floods the shared runner pool during a release. Props lance.willett@automattic.com. See #65736.
|
The following accounts have interacted with this PR and/or linked issues. I will continue to update these lists as activity occurs. You can also manually ask me to refresh this list by adding the Core Committers: Use this line as a base for the props when committing in SVN: To understand the WordPress project's expectations around crediting contributors, please review the Contributor Attribution page in the Core Handbook. |
There was a problem hiding this comment.
Pull request overview
This PR reduces GitHub Actions PHPUnit job counts by trimming the PHP version matrix to boundary versions (lowest/highest supported per major) on the active branch, aiming to preserve meaningful coverage while easing runner load during releases.
Changes:
- Reduce the
phpmatrix intest-with-mysqlfrom seven minors to7.4,8.0, and8.5. - Reduce the
phpmatrix intest-with-mariadbfrom seven minors to7.4,8.0, and8.5. - Reduce the
phpmatrix intest-innovation-releasesfrom seven minors to7.4,8.0, and8.5.
Comments suppressed due to low confidence (1)
.github/workflows/phpunit-tests.yml:157
- The new comment is slightly misleading in this job: while the base
php:axis is trimmed to boundary versions, theinclude:section below still runs additional jobs on intermediate PHP minors (e.g. 8.3). Consider rewording so it’s clear the boundary trim applies to the base matrix only.
# Test highest and lowest supported version of each major.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Push and pull request events test the boundary PHP versions only. The weekly scheduled run tests every supported version, so full coverage is preserved without multiplying job count on every change. Props lance.willett@automattic.com. See #65736.
johnbillion
left a comment
There was a problem hiding this comment.
There is an element of risk with this change, but:
- That can be offset with a new daily or weekly workflow that exercises the full matrix
- The benefit of faster overall feedback as a result of pushes and PRs is worth it, especially shortly prior to and during release when lots of pushes are being made
|
@johnbillion Agree on the tradeoff. If we could make all runs faster, maybe we can keep the full matrix. Another idea is to allow a PR tag Apologies, I made a mistake in the first commit. Can you re-review 2538d74 ? Adds the weekly back. |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.
Comments suppressed due to low confidence (3)
.github/workflows/phpunit-tests.yml:78
workflow_dispatchruns will now use the trimmed boundary PHP list because the condition only treatsscheduleas the full-matrix case. If you want manual runs to retain the ability to exercise the full PHP matrix on demand (without waiting for the weekly cron), includeworkflow_dispatchin the full-matrix branch and update the comment accordingly.
# The scheduled run tests every supported PHP version. Other events test the highest and lowest of each major.
php: ${{ github.event_name == 'schedule' && fromJSON('["7.4","8.0","8.1","8.2","8.3","8.4","8.5"]') || fromJSON('["7.4","8.0","8.5"]') }}
.github/workflows/phpunit-tests.yml:158
workflow_dispatchruns will now use the trimmed boundary PHP list because the condition only treatsscheduleas the full-matrix case. If you want manual runs to retain the ability to exercise the full PHP matrix on demand (without waiting for the weekly cron), includeworkflow_dispatchin the full-matrix branch and update the comment accordingly.
# The scheduled run tests every supported PHP version. Other events test the highest and lowest of each major.
php: ${{ github.event_name == 'schedule' && fromJSON('["7.4","8.0","8.1","8.2","8.3","8.4","8.5"]') || fromJSON('["7.4","8.0","8.5"]') }}
.github/workflows/phpunit-tests.yml:213
workflow_dispatchruns will now use the trimmed boundary PHP list because the condition only treatsscheduleas the full-matrix case. If you want manual runs to retain the ability to exercise the full PHP matrix on demand (without waiting for the weekly cron), includeworkflow_dispatchin the full-matrix branch and update the comment accordingly.
# The scheduled run tests every supported PHP version. Other events test the highest and lowest of each major.
php: ${{ github.event_name == 'schedule' && fromJSON('["7.4","8.0","8.1","8.2","8.3","8.4","8.5"]') || fromJSON('["7.4","8.0","8.5"]') }}
Test only the highest and lowest supported PHP version of each major (7.4, 8.0, 8.5) on push and pull request events; the weekly scheduled run still exercises every supported version. Version-specific failures cluster at the boundaries of each major, so the intermediate minors add job count without adding coverage. Cuts base PHPUnit combinations from roughly 168 to 72 per CI run for both time and cost savings, and a much better developer experience in core GitHub repos. Developed in: #12719 Props johnbillion, adrianmoldovanwp Fixes #65736, see #64083 git-svn-id: https://develop.svn.wordpress.org/trunk@62858 602fd350-edb4-49c9-b593-d223f7449a82
Test only the highest and lowest supported PHP version of each major (7.4, 8.0, 8.5) on push and pull request events; the weekly scheduled run still exercises every supported version. Version-specific failures cluster at the boundaries of each major, so the intermediate minors add job count without adding coverage. Cuts base PHPUnit combinations from roughly 168 to 72 per CI run for both time and cost savings, and a much better developer experience in core GitHub repos. Developed in: WordPress/wordpress-develop#12719 Props johnbillion, adrianmoldovanwp Fixes #65736, see #64083 Built from https://develop.svn.wordpress.org/trunk@62858 git-svn-id: http://core.svn.wordpress.org/trunk@62138 1a063a9b-81f0-0310-95a4-ce76da25c4cd
What
Trims the PHPUnit test matrix to boundary PHP versions on trunk and pull requests: the highest and lowest supported version of each major.
7.4,8.0, and8.5replace the full seven-minor list, intest-with-mysql,test-with-mariadb, andtest-innovation-releases.Full coverage is preserved by the existing weekly scheduled run: on the Sunday cron, every supported PHP version is tested; push and pull request events test the boundary versions only.
Why
This mirrors the trim #64083 applied to older branches, whose title named "(and possibly
trunk)". Version-specific failures cluster at the boundaries of each major, so the intermediate minors add job count without adding coverage. The weekly full run keeps the intermediate versions covered without multiplying job count on every change.Impact
Base PHPUnit combinations drop from ~168 to ~72 on push and pull requests, roughly a 55% cut, before the unchanged
include:jobs. The weekly run is unchanged.How the weekly full matrix works
The
phpmatrix is conditional on the event:Scheduled runs fire from the default branch, so the weekly full sweep covers trunk.
Scope
PHP axis only. Database versions, multisite, memcached, and the single-representative
include:jobs are untouched. A database-axis trim on the same principle is a possible follow-up, filed separately.Trac ticket: Core-65736
Use of AI Tools
AI assistance: Yes
Tool(s): Claude Code
Model(s): Opus 4.8
Used for: Identifying the boundary-trim approach from the existing older-branch precedent, the weekly-full conditional, and drafting the change and ticket text. Verified manually.