Skip to content

perf(config): cut container-start latency — HZ=1000, voluntary preemption, PSI default-off (ABX-498) - #16

Merged
AprilNEA merged 3 commits into
masterfrom
perf/container-start-latency
Jul 24, 2026
Merged

perf(config): cut container-start latency — HZ=1000, voluntary preemption, PSI default-off (ABX-498)#16
AprilNEA merged 3 commits into
masterfrom
perf/container-start-latency

Conversation

@AprilNEA

Copy link
Copy Markdown
Member

Supersedes #14 (HZ=1000) and #15 (voluntary-preemption experiment). Consolidates the three validated container-start kernel-config wins from the ABX-498 investigation into one release.

Changes (both arches, identical)

symbol from to why
CONFIG_HZ 100 1000 container start is a chain of jiffy-rounded scheduler waits; 10 ms→1 ms rounding
preemption PREEMPT_LAZY PREEMPT_VOLUNTARY lazy⇒PREEMPT_RCU makes rcu_barrier() on veth/netns teardown wait ~74 ms/veth; +25% ctx-switch
CONFIG_PSI_DEFAULT_DISABLED =y PSI accounting on the wakeup fast path costs ~0.85 s/build; keep compiled in (psi=1 re-enables)

Build-time assertions updated in both the host and docker build paths: HZ_1000 and PSI_DEFAULT_DISABLED join the olddefconfig-survival for sym loop; the PREEMPT_LAZY conditional assertion is replaced by a hard PREEMPT_VOLUNTARY assertion (a future kernel dropping the voluntary choice now fails loudly instead of silently re-selecting lazy).

Validation (ABX-498 four-kernel A/B, same host, docker run --rm alpine true ×10)

kernel veth del same-core switch run e2e
6.18 lazy HZ100 (shipping) 74 ms 8.6 µs 496 ms
6.18 lazy HZ1000 (#14) 10 ms 8.6 µs 378 ms
6.18 voluntary HZ1000 (#15) ~0 ms 6.8 µs 371 ms
+ psi=0 (this PR, compile-time) ~0 ms 3.5 µs

With all three: docker run 496→246 ms, simple cold build 8.1→1.31 s (Colima 173 ms / 0.99 s). Guest-internal container start reaches parity; the residual is a moby exit-lock phase race (upstream, container-lifetime <30 ms only).

Balloon safety (PSI default-off)

CONFIG_PSI=y stays, so the capability is compiled in. The guest agent's balloon pressure watch (PsiTrigger::open) already treats a failed /proc/pressure/memory write (EOPNOTSUPP when PSI is default-disabled) as "stay on the meminfo/refault sampling path" — a designed, tested degrade. The balloon keeps functioning; memory-pressure detection moves from event-driven to 1 s polling while shrunk. psi=1 on the cmdline restores the trigger.

Consumed by arcbox after a kernel release → boot-assets bundle bump → assets.lock.

AprilNEA added 2 commits July 24, 2026 12:03
Container start is a chain of jiffy-rounded scheduler waits (containerd
poll, shim handshake completions, runc, dockerd↔containerd gRPC backoff).
At HZ=100 each rounds up to 10 ms; at HZ=1000 to 1 ms. Measured
docker-run end-to-end 496→378 ms and multi-step builds proportionally,
matching the HZ=1000 guests Colima/OrbStack ship. NO_HZ_IDLE keeps the
tickless-idle CPU target intact. Refs ABX-496/ABX-498.
The 6.18 bump switched arm64/x86 to lazy preemption on the premise of
near-voluntary throughput. That premise fails for this guest's
container-churn workload: lazy pulls in PREEMPT_RCU, so rcu_barrier() on
the veth/netns teardown path (every container stop) waits for
preemptible-RCU callbacks — ~74 ms/veth vs ~0 ms under voluntary — and
same-core context-switch cost rises ~25%. Revert to voluntary; the build
assertion now pins it so a future kernel that drops the choice fails
loudly instead of silently re-selecting lazy. See ABX-498.
@linear-code

linear-code Bot commented Jul 24, 2026

Copy link
Copy Markdown

ABX-498

@greptile-apps

greptile-apps Bot commented Jul 24, 2026

Copy link
Copy Markdown

Greptile Summary

Updates both supported architecture configurations to reduce container-start latency:

  • Selects a 1000 Hz timer frequency and voluntary preemption.
  • Keeps PSI compiled in but disables it by default.
  • Adds matching post-olddefconfig assertions to native and Docker build paths.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains within the scope of the available previous review threads.

T-Rex T-Rex Logs

What T-Rex did

  • Reviewed the pre-change state where PSI support was unresolved due to CONFIG_PSI_DEFAULT_DISABLED not being set.
  • Validated the post-change state where the fragment now resolves all five requested symbols and the build assertions pass with exit code 0.
  • Confirmed shell syntax validation passed for the build script, the injection script, and the generated harness.

View all artifacts

T-Rex Ran code and verified through T-Rex

Important Files Changed

Filename Overview
configs/arcbox-arm64.config Applies the HZ, preemption, and PSI defaults to the arm64 kernel configuration.
configs/arcbox-x86_64.config Applies the same kernel configuration changes to x86_64.
scripts/build-kernel.sh Verifies that the required configuration choices survive olddefconfig in both build paths.

Reviews (2): Last reviewed commit: "perf(config): disable PSI by default (ke..." | Re-trigger Greptile

PSI's per-task-state accounting sits on the scheduler wakeup fast path.
The container-build gRPC orchestration (buildkit/dockerd/containerd,
thousands of wakeups per step) pays ~0.85 s per simple build with it on,
and same-core context switch nearly doubles (ABX-498; psi=0 A/B). Keep
CONFIG_PSI=y so the balloon pressure watch can still be enabled with
psi=1, but default it off. Default-off leaves /proc/pressure uncreated,
so the guest agent's balloon watch gets ENOENT opening it and falls back
to meminfo/refault sampling — verified in-guest: the agent logs 'PSI
unavailable (No such file or directory); staying on the sampling path'
and the balloon keeps functioning (event-driven -> 1 s poll). Refs
ABX-498.
@AprilNEA
AprilNEA force-pushed the perf/container-start-latency branch from 29547df to 2de9578 Compare July 24, 2026 04:17
@AprilNEA
AprilNEA merged commit 921b741 into master Jul 24, 2026
7 checks passed
@AprilNEA
AprilNEA deleted the perf/container-start-latency branch July 24, 2026 19:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant