perf(fuse): bounded spin-wait for metadata request completion (CORE-48) - #17
Conversation
On virtualized ARM64 every reschedule IPI is a trap (no TIF_POLLING_NRFLAG, single LLC domain keeps wake_affine from migrating the waiter onto the IRQ CPU), so a synchronous FUSE workload pays device-IRQ trap + IPI trap + two context switches per operation — the reschedule-IPI count matches the virtio-fs request IRQ count 1:1. Poll FR_FINISHED for up to 40us before sleeping (KVM halt polling at the FUSE wait point), gated to non-READ/WRITE opcodes: bulk replies outlive the budget and an ungated spin cost -54% sequential-read throughput. A/B on the ArcBox bench trio (Apple VZ guest, 18 vCPU): metadata_stat 31.8k -> 50.3k ops/s (+58%, 97% of the hard-pinned ceiling), negative_lookup +40%, create_delete +28%, sequential read -3.5% (noise). Beats Colima's same-context numbers. Full investigation record: Linear CORE-48.
Greptile SummaryThis PR adds bounded polling before FUSE request waits.
Confidence Score: 4/5The PR is not yet safe to merge because directory-read requests still enter the polling path despite the attempted non-metadata exclusion. The revised switch handles several operations named in the earlier review, but directory reads remain admitted through the default branch and can still incur the full spin budget on every request. Files Needing Attention: patches/fuse-spin-wait.patch
|
| Filename | Overview |
|---|---|
| patches/fuse-spin-wait.patch | Adds the bounded FUSE completion spin and expands the opcode exclusion gate. |
Reviews (2): Last reviewed commit: "perf(fuse): gate the spin to virtio-clas..." | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1096af1277
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Acceptance A/B complete (2026-07-31, back-to-back isolated-VM boots, identical bundle except the kernel)
Every row improves or holds; the v1 ungated −54% sequential regression is fully absent with the opcode gate. Boot/lifecycle sanity: 5 green boots of the patched kernel across the lab rounds, docker pull/run normal in each. Real-world surface this accelerates: bind-mounted host volumes ( Remaining before release train (assets.lock bump side): multi-flow iperf sanity ride-along, and the boot-assets bundle rebuild picks the patch up automatically via |
… opcodes Review round: the single READ/WRITE exclusion left slow non-metadata requests (flush/fsync, copy_file_range, blocking locks) burning the budget, and classic /dev/fuse daemons with arbitrary completion latency took the spin path on every build. The spin now requires a kernel-mediated transport (fc->iq.ops != fuse_dev_fiq_ops, i.e. virtio-fs) and skips every legitimately-blocking opcode. Revalidated on hardware: metadata_stat 50.6k (+57% vs stock), negative_lookup 37.6k (+57%), create_delete +17%, find_recursive +11%, rm_rf 14% faster, sequential read -3% (noise).
…513) Bundle v0.6.13 ships kernel v0.0.22 with the opcode- and transport-gated FUSE spin-wait (arcboxlabs/kernel#17): host-volume metadata ops +44..64% (metadata_stat 32.1k -> 52.8k ops/s), bulk I/O unchanged. Full A/B record in Linear CORE-48. Validated: boot_assets e2e green on the new bundle (full Docker lifecycle, 116s).
1 participant
Problem
On virtualized ARM64, every synchronous FUSE operation pays a cross-vCPU round trip: device-IRQ trap → reschedule-IPI trap → two context switches. Two architectural facts make this unavoidable by scheduling alone (all A/B'd in Linear CORE-48):
TIF_POLLING_NRFLAG, sotry_to_wake_upalways sends a physical IPI —idle=polland__wake_up_syncmeasurably change nothing.wake_affinenever migrates the waiter onto the IRQ CPU — the best sched-feature toggle recovered +12% vs the +82% of hard pinning.Measured 1:1 correspondence between virtio-fs request IRQs and reschedule IPIs during a stat loop.
Fix
request_wait_answerpollsFR_FINISHEDfor ≤40µs before sleeping (KVM halt-polling applied at the FUSE wait point), gated to non-READ/WRITE opcodes. The waiter observes fast completions while still runnable: no wakeup IPI, no context switches.need_resched()breaks the spin the moment the CPU has real work.The gate is load-bearing: ungated, bulk READ replies outlive the budget and the burnt CPU cost −54% sequential read; gated, sequential read is unchanged (−3.5%, run noise).
Numbers (Apple VZ guest, 18 vCPU, same-VM A/B, ArcBox bench trio)
Also beats Colima's same-context numbers (51.7k / 30.2k / 6.7k) on the identical Apple virtiofs device.
Notes
fs/fusecode and applies to both arch builds; the win is ARM64-virt-specific, and on x86 the polling flag already avoids most wake IPIs — the gate +need_resched()bound the cost there to the spin budget on slow metadata ops.bench_virtiofse2e trio + sequential (done, table above), docker-build A/B on the ArcBox side (running, will attach), multi-flow iperf sanity.Refs Linear CORE-48.