Cucumber: add support for Split by Test Examples (split slow feature files by scenario) - #348
Conversation
Split slow feature files by scenario across parallel CI nodes, mirroring the existing RSpec Split by Test Examples feature. Scenarios are addressed as file:line test example paths (e.g., features/a.feature:12), which Cucumber natively supports on the command line. Scenario Outline example rows are split individually because the JSON formatter expands them to one element per row. Opt in with KNAPSACK_PRO_CUCUMBER_SPLIT_BY_TEST_EXAMPLES=true. Requires Cucumber >= 4.0 (verified against Cucumber 4.1, 7.1, and 11.1). - KnapsackPro::Adapters::CucumberAdapter implements split_by_test_cases_enabled?, calculate_slow_id_paths, and the id-path helpers used by KnapsackPro::TestSuite, so both Regular Mode and Queue Mode pick up the split through the existing generic flow. - The rake task knapsack_pro:cucumber_test_example_detector runs cucumber --dry-run --format json on the slow test files (fetched from the API build distribution, like RSpec) to enumerate scenario paths. The dry run respects the user's cucumber args (e.g. --tags), which are parsed with Shellwords to keep quoted tag expressions intact. - The Around hook records time per scenario path when the scenario's file was split (Tracker#scheduled_test_path?), and per file otherwise. - KnapsackPro::TestCaseMergers::CucumberMerger merges file:line entries fetched from the API back into whole-file times before determining slow test files, mirroring the RSpec merger.
|
Note: converting to draft whilst we test drive this against our cucumber suite for a while, will promote again once we're confident in it |
Queue Mode for Cucumber shells out a brand new cucumber process for every batch pulled from the Queue, paying the full application boot cost (support code, e.g. a Rails app) each time. With Split by Test Examples enabled, batch counts grow substantially and that boot cost dominates the build time. With KNAPSACK_PRO_CUCUMBER_QUEUE_PRELOAD=true the runner boots Cucumber once per CI node via a dry run that loads the support code, then executes each batch in a forked child process that reuses the preloaded Cucumber::Runtime through the public Cucumber::Cli::Main#execute!(existing_runtime) seam (the old spork/DRb integration point, still present in Cucumber 4 through 11). Details: - Cucumber::Runtime memoizes reports, formatters, and parsed features against the first configuration it ran with; the child resets those instance variables so they are rebuilt against the batch's configuration and event bus (otherwise results are published to the parent's event bus and the child exits 0 even on failures). The loaded support code is preserved. - The parent process loads the support code, so the adapter's at_exit hooks (save subset queue report, after-subset-queue hook) are skipped in the parent via KNAPSACK_PRO_CUCUMBER_PRELOAD_PARENT_PID; forked children run them as before, preserving per-batch reporting. - Children exit via Kernel.exit so at_exit reporting runs; the batch exit status is propagated to the runner loop unchanged. - POSIX only: raises a clear error when Process.fork is unavailable. Verified against Cucumber 9.2.1 and 11.1.1 with a simulated queue: single boot, correct per-scenario time tracking through forks, correct non-zero exit status propagation from failing batches.
|
Pushed a second commit (bb8e507) adding an experimental Queue Mode preload mode for Cucumber ( Trial data (Rails app, 12 CI nodes, ~14 min/node baseline): enabling Split by Test Examples split 21 slow feature files into 400+ scenario queue items, which grew batch counts from ~7 to ~50 per node. Because Queue Mode shells out a brand-new The preload mode boots Cucumber once per node (a dry run loads the support code), then runs each batch in a forked child that reuses the preloaded One notable detail: |
…ev/null Cucumber expands cucumber.yml profiles into the CLI args before validating that no two formatters write to the same stream. When a user profile contains e.g. --format SomeFormatter --out /dev/null, the preload dry run's own --out /dev/null triggered: All but one formatter must use --out, only one can print to each stream Write the preload dry run progress output to a unique file under .knapsack_pro instead.
Real-world trial findings: - The preload dry run covered the whole test suite, which can take minutes (step matching runs for every step even in a dry run; ~0.6s per scenario on a large Rails app). The dry run only exists to load the support code, so limit it to a single sample test file path (the first path of the first batch). - Database connections opened while booting the app were inherited by every forked child; concurrent use of an inherited socket corrupts the connection and hangs or fails queries (observed as Capybara rendering error pages instead of the app). Clear ActiveRecord connection pools in the parent after preloading so the parent and each child lazily check out fresh connections. - Add a KnapsackPro::Hooks::Queue.after_preload_fork hook, called in each forked child right after the fork, so apps can re-establish resources that do not survive a fork (e.g. SemanticLogger.reopen for log appender threads, or reconnecting Redis clients).
…rofiles) Second round of real-world trial fixes, found by reproducing the failure locally with per-thread SQL instrumentation: - The preload dry run instantiated formatters from cucumber.yml profiles. A formatter touching Capybara (e.g. one that captures screenshots) boots the Capybara server - and a browser - in the parent process. Forked children then inherit Capybara's session pool and app->port registry, find the parent's server responsive, and send their requests to a server running in the WRONG PROCESS: it cannot see the child's open database transactions (observed as every scenario failing at login because the test-created tenant was invisible), and the parent's lingering browser/server kept the CI step's output pipes open, hanging the job until timeout. The preload dry run now strips -p/--profile and runs with --no-profile: it only exists to load the support code. - Belt and braces for the same class of bug: forked children now clear Capybara's session pool and Capybara::Server.ports before running, so a child always boots its own server even if something did boot one in the parent. - Clear ActiveRecord connection leases (not just pools) in the parent after preloading: leases are keyed by thread identity, which survives a fork, so a child's first checkout could return the parent's connection object and reconnect it in place (undefined behavior on a forked libpq handle).
|
Trial update — the preload mode is now validated end-to-end on a real CI suite (Rails app, 12 nodes, Capybara/Selenium browser tests, ~720 scenarios):
Getting there surfaced three fork-hygiene bugs, all fixed on the branch (commits
The remaining ~4 min over baseline is per-batch browser/Capybara-server startup and the split-detector dry run at queue init — inherent to batch granularity rather than to the preloader. Note the trial deliberately forced aggressive splitting ( |
|
One more data point from the trial suite: preload mode pays off even with scenario splitting disabled. With default slow-file settings (nothing splits in this suite) and only
That's ~15% off the wall clock purely from booting Rails once per node instead of once per batch (~7 batches × ~25s here). So the preload commit is useful to any Cucumber Queue Mode user with a slow-booting app, independent of the Split by Test Examples feature — it may be worth splitting into its own PR for that reason. Happy to do that if preferred. |
1 participant
Summary
This PR adds Split by Test Examples support for Cucumber, so slow
.featurefiles can be split by scenario across parallel CI nodes — mirroring the existing RSpec feature. The docs currently say "Only RSpec is supported. Let us know if you use a different test runner" — we do! 🙂It is opt-in via
KNAPSACK_PRO_CUCUMBER_SPLIT_BY_TEST_EXAMPLES=trueand requires Cucumber >= 4.0.Why this is a natural fit for Cucumber
Cucumber scenarios have a native command-line address:
features/a.feature:12. Unlike RSpec example ids (spec/a_spec.rb[1:2]), scenario line numbers are stable file properties, and Cucumber has supported runningpath:line(including individual Scenario Outline example rows) since forever. I verified the underlying mechanics empirically against Cucumber 4.1, 7.1, and 11.1:cucumber --dry-run --format json --out report.json <files>lists every runnable scenario as an element withuri+line, with Scenario Outline example rows expanded to one element per row — on all three versions, exit code 0.cucumber features/a.feature:15(an outline example row line) runs exactly that pickle, andtest_case.location.file/.linein anAroundhook match the dry-run values — so times can be tracked per scenario path.file:linethat a--tagsfilter excludes exits 0 (even with--strict), so stale schedules can't fail a build.How it works
The generic split machinery already flows through
KnapsackPro::TestSuite#calculate_test_files→adapter_class.split_by_test_cases_enabled?/calculate_slow_id_paths/concat_test_filesfor both Regular Mode and Queue Mode, so this PR implements the adapter contract for Cucumber and mirrors the RSpec architecture piece by piece:CucumberAdapter.split_by_test_cases_enabled?— gated on the new env var, raises for Cucumber < 4.0 (analogous to the RSpec >= 3.3.0 gate).knapsack_pro:cucumber_test_example_detectorrake task (shelled out like the RSpec detector, to not pollute the runner process) fetches slow test files from the API build distribution, then runs an in-processCucumber::Cli::Maindry run with--format jsonto enumerate scenario paths. The user's cucumber args are forwarded (parsed withShellwordsso--tags "not @slow"survives), so tag filters are respected during scenario enumeration. The task clearsKNAPSACK_PRO_REGULAR_MODE_ENABLED/KNAPSACK_PRO_QUEUE_MODE_ENABLEDbefore the dry run so the adapter bound infeatures/support/env.rbdoes not register time tracking or save an (empty) report to the API from inside the detector process.TestCaseMergers::CucumberMergermergesfile:lineentries from the API back into whole-file times before slow-file determination (same max-of-file-vs-sum semantics as the RSpec merger).Aroundhook now records time underfeatures/a.feature:12when that exact test example path is scheduled on the node (newTracker#scheduled_test_path?, which reuses the existing prerun-tests report on disk to work across processes for Regular Mode's forked rake task and Queue Mode's per-batch cucumber processes), and under the file path otherwise.KNAPSACK_PRO_CUCUMBER_OPTIONSfor the detector, mirroringKNAPSACK_PRO_RSPEC_OPTIONS.No behavior changes when the feature flag is off (the default).
Testing
file:line/Around-hook mechanics also verified on 4.1 and 7.1):TestSuite#calculate_test_fileswith the feature enabled replaces a slow feature file with its scenario paths (tag-filtered scenarios excluded) while keeping fast files whole:Notes for reviewers
file:lineas opaque path strings exactly the wayspec/a_spec.rb[1:2]id paths are treated, so I believe no API changes are needed — but you would know best whether anything server-side special-cases path formats.false(opt-in), unlike the RSpec one which now defaults totrue— that seemed the safe choice for a new feature; happy to change.