[Phase 7] Soft-launch E2E PR checks

[kieran-osgood-shopify]

Stack Context

This is Phase 7 of the BrowserStack Maestro E2E stack for Checkout Kit, stacked on GitHub E2E reporting.

Parent issue: https://github.com/shop/issues-checkout-kit/issues/1101
Epic: https://github.com/shop/issues-checkout-kit/issues/1084

What?

I made a pivot in this branch - I was introducing a flag E2E_STRICT that would surface failures differently depending whether we wanted the pipeline to be "blocking" for PR merges. I realised this wasn't necessary as all we need is to add/remove the status check from the required checks for PR merge.

It also led to problematic bugs in the pipeline where the e2e-report stage (that reports the comments back on the PR / posts the status checks) would be missing the artifacts it needs to report.

I removed all E2E_STRICT usage, accepting the false branch, and we ratify at the e2e-report stage whether we received all the reports we need to post the status.

• adds the "pull_request" trigger to the bitrise config
• Skips draft PR's
• add some resiliency with http retries, and ensuring reporting captures gaps where things fail in the "setup" stage still produce a report
• uses rolling builds to cancel BrowserStack builds on timeout or duplicate runs

Why?

The E2E pipeline should start non-blocking while the infrastructure and suite stabilize. Developers still get commit statuses, Check Runs, and actionable failure comments, but PR progress is not blocked until we are confident that the checks are free of false positives

Results

Topology on Bitrise

When one suite is failing

After fixing all the tests you'll see the workflow update the comment

[kieran-osgood-shopify]

• [Phase 7] Soft-launch E2E PR checks #364 : 2 dependent PRs (#401 , #417 ) 👈 (View in Graphite)
• [Phase 6] Report E2E results to GitHub #363
• [Phase 5] Run BrowserStack Maestro from E2E matrix rows #362
• [Phase 4] Build React Native iOS E2E IPA artifact #361
• [Phase 3] Build React Native Android E2E APK artifact #360
• [Phase 2] Add Bitrise E2E pipeline skeleton #357
• [Phase1] Add E2E matrix launch smoke foundation #356
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[bitrise]

🤖 Bitrise Build Report > Checkout Kit E2E > e2e > #55: ✅ SUCCESS

• Duration: 8m 48s
• Workflows: 6 total / 6 executed / 0 aborted / 0 skipped / 0 failed / 6 successful

[kieran-osgood-shopify]

^I have disabled these bitrise build report comments so its only the one we manually post above in future

[kieran-osgood-shopify]

Merge activity

• Jul 15, 10:38 AM UTC: A user started a stack merge that includes this pull request via Graphite.
• Jul 15, 10:38 AM UTC: @kieran-osgood-shopify merged this pull request with Graphite.