Skip to content

Commit 5722cdc

Browse files
1stvamp1stvamp
committed
docs(run-engine): benchmark Redis CPU and memory vs concurrency-key cardinality
Adds a resource/cardinality arm to the CK virtual-time benchmark: a plan, a runnable harness (drives a real RunQueue against a dedicated Redis, flag OFF vs ON, and reads server-side INFO memory/cpu, MEMORY USAGE and OBJECT ENCODING; inert without CK_BENCH_REDIS_URL), and results from a local homelab. Findings: the :ckVtime ZSET is essentially a second copy of :ckIndex (~150 bytes per key), so memory is linear in cardinality (about +1.3MB for a 10k-key queue) and TTL-reclaimed; Redis CPU overhead is small and does not scale with cardinality (+5 to +12 percent on an identical workload, per-script cost flat at ~25 usec/call) because the dequeue window is fixed and the ZSET ops are O(log N); and ckVtime membership tracks ckIndex exactly under sustained churn.
1 parent 9d79929 commit 5722cdc

3 files changed

Lines changed: 628 additions & 0 deletions

File tree

‎internal-packages/run-engine/design/benchmarks/2026-07-28-ck-vtime-resource-cardinality-plan.md‎

Lines changed: 156 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,156 @@
1+
# CK virtual-time scheduling: Redis CPU + memory vs cardinality test plan
2+
3+
Answers the review question: how do these changes affect the run-queue Redis
4+
CPU and memory, and how do both react as concurrency-key cardinality grows (e.g.
5+
a base queue that suddenly has 10k distinct concurrency keys)?
6+
7+
This is a cost/scaling plan, separate from the fairness A/B
8+
(`results-2026-07-27.md`). Same environment constraints: run on a single box, all
9+
numbers **relative** (flag OFF vs ON, identical load, same box), against a
10+
dedicated throwaway Redis.
11+
12+
## What the change adds to Redis (grounded in the branch)
13+
14+
Per base queue, flag ON adds:
15+
16+
- `:ckVtime`, a ZSET whose members are the exact same full CK-variant queue-name
17+
strings the existing `:ckIndex` ZSET already holds, each with an 8-byte double
18+
score. So it is effectively a second copy of `ckIndex`'s membership.
19+
- `:ckVtimeFloor`, a STRING holding one number. Negligible.
20+
21+
Both are GC'd from the dequeue path when a variant drains, carry a 24h TTL, and
22+
live under the base queue's `{org}` hash tag. The per-call dequeue scan window is
23+
`maxCount * windowMultiplier` (default 30) and does **not** grow with cardinality;
24+
the cardinality-sensitive operations are the ZSET writes/reads (`ZADD` `NX`,
25+
`ZSCORE`, `ZRANGE` by rank, `ZRANGE 0 0`), which are O(log N) on the `ckVtime`
26+
skiplist.
27+
28+
## Hypotheses
29+
30+
1. **Memory grows linearly with cardinality, adding roughly one `ckIndex`-sized
31+
ZSET per base queue.** Incremental `used_memory` under ON minus OFF should
32+
track `cardinality x per-member cost`, and `MEMORY USAGE :ckVtime` should be
33+
close to `MEMORY USAGE :ckIndex` for the same queue (same members, one extra
34+
double score). At 10k keys on one queue this is a low single-digit MB for that
35+
queue, bounded and TTL-reclaimed. There is a one-step jump at the
36+
listpack->skiplist encoding boundary (128 entries by default).
37+
2. **Redis CPU per operation grows sub-linearly (about O(log cardinality)), not
38+
linearly.** The fixed 30-entry window scan dominates the vtime-specific work
39+
and does not change with N; the ZSET ops add a `log N` term. So dequeue/enqueue
40+
`usec_per_call` should rise only mildly from 100 to 10k keys, and the ON/OFF
41+
`usec_per_call` ratio should stay roughly flat across the sweep.
42+
3. **Tombstone drift stays bounded under high-cardinality churn.** `ack`, TTL
43+
expiry, and DLQ drain a variant without removing it from `ckVtime` (documented
44+
in `CK_VTIME_KNOWN_LIMITATIONS.md`). Under churn where variants drain via those
45+
paths rather than a vtime dequeue, `ckVtime` may transiently exceed `ckIndex`,
46+
but it should self-heal (next vtime pass GCs empties) or expire (24h TTL), so
47+
`size(ckVtime) / size(ckIndex)` stays bounded and does not grow without limit.
48+
49+
## Scenarios
50+
51+
All on a single base queue (worst case for one queue's ZSETs), dedicated Redis,
52+
flag OFF then ON with identical load.
53+
54+
- **Cardinality sweep (memory).** Enqueue N distinct concurrency keys, one
55+
message each, N in {100, 1_000, 10_000, 50_000}. Measure the Redis memory
56+
footprint at rest for each N, OFF vs ON. This isolates the storage cost with no
57+
dequeue activity.
58+
- **Steady-state load (CPU).** At each N, after building cardinality, run a fixed
59+
60s workload of enqueue + batched dequeue (`maxCount 10`) + ack at a capped
60+
concurrency, so keys are continuously served and re-registered. Measure Redis
61+
CPU and per-command time over the window, OFF vs ON.
62+
- **Churn / tombstone (memory under adversarial drain).** Build 10k keys, then
63+
drain them via `ack` (not via vtime dequeue) while enqueuing new keys, for a
64+
fixed duration. Sample `size(ckVtime)` and `size(ckIndex)` over time and confirm
65+
the ratio stays bounded (self-heal + TTL), not monotonically growing.
66+
67+
## Metrics and collection (exact commands)
68+
69+
Use a second plain Redis client for measurement so it does not perturb the
70+
harness. `redis-cli` shown; the harness issues the same via ioredis.
71+
72+
Memory:
73+
74+
- Totals: `INFO memory` -> `used_memory`, `used_memory_dataset`. Delta ON vs OFF
75+
at each N is the incremental footprint.
76+
- Per structure (exact bytes): `MEMORY USAGE {org...}:queue:<base>:ckIndex` and
77+
`MEMORY USAGE {org...}:queue:<base>:ckVtime`. Report both and the ratio.
78+
- Encoding: `OBJECT ENCODING <ckVtime key>` (listpack vs skiplist) at each N, to
79+
mark the transition.
80+
81+
CPU:
82+
83+
- Process CPU over the load window: `INFO cpu` -> `used_cpu_user` +
84+
`used_cpu_sys`, sampled before and after the fixed 60s load; the delta is Redis
85+
CPU-seconds consumed. Divide by op count for CPU-per-op.
86+
- Per-command time: `CONFIG RESETSTAT` before the window, then `INFO commandstats`
87+
after -> `cmdstat_zadd`, `cmdstat_zrange`, `cmdstat_zrangebyscore`,
88+
`cmdstat_zscore`, `cmdstat_get`, `cmdstat_set`, `cmdstat_expire`
89+
(`calls`, `usec`, `usec_per_call`). Compare OFF vs ON.
90+
- Cross-check (optional, OS-level): `pidstat -p <redis-pid> 1` over the window, or
91+
`redis-cli --latency` / `--latency-history` for command latency.
92+
93+
Derived:
94+
95+
- Memory vs N curve (expect linear, slope ~ per-member bytes), OFF and ON.
96+
- CPU-per-op vs N curve (expect flat-ish / log), OFF/ON ratio.
97+
- `size(ckVtime)/size(ckIndex)` over time in the churn scenario (expect bounded).
98+
99+
## A/B procedure
100+
101+
1. Dedicated throwaway Redis; `FLUSHDB` between arms and between cardinality
102+
points. Warm up once.
103+
2. For each N in the sweep, for each arm (OFF, ON):
104+
- Build N keys (enqueue N distinct concurrency keys).
105+
- Snapshot memory (`used_memory`, `MEMORY USAGE` of both ZSETs, `OBJECT
106+
ENCODING`).
107+
- `CONFIG RESETSTAT`; run the fixed 60s steady load; snapshot `INFO cpu` and
108+
`INFO commandstats`.
109+
- Record, then `FLUSHDB`.
110+
3. N trials of the load window per point; report median for CPU (memory at rest is
111+
near-deterministic).
112+
4. Run the churn scenario once per arm at N = 10k.
113+
114+
Only the OFF-vs-ON delta and the shape of the growth curves are reported; absolute
115+
throughput on a single box is not prod scale.
116+
117+
## Results template
118+
119+
### Memory at rest, per cardinality (single base queue)
120+
121+
| keys (N) | used_memory OFF | used_memory ON | delta | MEMORY USAGE ckIndex | MEMORY USAGE ckVtime | ckVtime/ckIndex | ckVtime encoding |
122+
| --- | --- | --- | --- | --- | --- | --- | --- |
123+
| 100 | | | | | | | |
124+
| 1,000 | | | | | | | |
125+
| 10,000 | | | | | | | |
126+
| 50,000 | | | | | | | |
127+
128+
### Redis CPU under 60s steady load, per cardinality
129+
130+
| keys (N) | CPU-sec OFF | CPU-sec ON | delta | dequeue usec/call OFF | dequeue usec/call ON | delta | total redis calls OFF/ON |
131+
| --- | --- | --- | --- | --- | --- | --- | --- |
132+
| 100 | | | | | | | |
133+
| 1,000 | | | | | | | |
134+
| 10,000 | | | | | | | |
135+
136+
### Tombstone drift (N=10k, drain via ack)
137+
138+
| elapsed | size(ckIndex) | size(ckVtime) | ratio |
139+
| --- | --- | --- | --- |
140+
| 0s | | | |
141+
| 30s | | | |
142+
| 60s | | | |
143+
144+
## Harness
145+
146+
Extend the existing micro-benchmark
147+
(`../../src/run-queue/bench/ckMicroBench.bench.test.ts`), which already drives a
148+
real `RunQueue` against an external Redis and reads `INFO commandstats`. Add a
149+
resource/cardinality mode that: builds N variants, snapshots `used_memory` +
150+
`MEMORY USAGE` of the base queue's `ckIndex`/`ckVtime` + `OBJECT ENCODING`, runs a
151+
fixed-duration steady load while sampling `INFO cpu`/`commandstats`, and emits the
152+
tables above. The churn scenario reuses the same enqueue/ack primitives with a
153+
drain-by-ack loop and periodic `ZCARD` sampling of both ZSETs.
154+
155+
Reuse the same dedicated Redis and the OFF-vs-ON constructor-flag pattern, so this
156+
arm, like the micro-benchmark, needs no webapp or redeploy.

‎internal-packages/run-engine/design/benchmarks/results-cardinality-2026-07-28.md‎

Lines changed: 82 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,82 @@
1+
# CK virtual-time scheduling: Redis CPU + memory vs cardinality (2026-07-28)
2+
3+
Answers the review question: how do these changes affect the run-queue Redis CPU
4+
and memory, and how do both react as concurrency-key cardinality grows?
5+
6+
Run on a **local homelab box, not production**, against a dedicated Redis
7+
configured for measurement (no RDB/AOF in the sampling windows, `maxmemory 0`,
8+
zset listpack thresholds at their defaults so the encoding boundary sits at 128).
9+
All numbers are **relative** (flag OFF vs ON, identical load, same box); absolute
10+
throughput is not prod scale. Method: `2026-07-28-ck-vtime-resource-cardinality-plan.md`.
11+
Server-side metrics (`INFO memory`/`cpu`/`commandstats`, `MEMORY USAGE`,
12+
`OBJECT ENCODING`) are RTT-independent.
13+
14+
## Memory at rest (single base queue, one queued message per key)
15+
16+
| keys (N) | used_memory OFF | used_memory ON | ON-OFF delta | ckIndex | ckVtime | ckVtime/ckIndex | ckVtime encoding |
17+
| --- | --- | --- | --- | --- | --- | --- | --- |
18+
| 100 | 1.84 MB | 1.93 MB | +0.09 MB | 6.9 KB | 6.2 KB | 0.89 | listpack |
19+
| 1,000 | 2.42 MB | 2.58 MB | +0.15 MB | 131 KB | 131 KB | 1.00 | skiplist |
20+
| 10,000 | 9.12 MB | 10.4 MB | +1.26 MB | 1.42 MB | 1.42 MB | 1.00 | skiplist |
21+
| 50,000 | 39.1 MB | 45.6 MB | +6.51 MB | 7.55 MB | 7.54 MB | 1.00 | skiplist |
22+
23+
The `:ckVtime` ZSET is the whole added footprint, and it is essentially a second
24+
copy of `:ckIndex`: same members (the full CK-variant queue names), one extra
25+
8-byte score, so `MEMORY USAGE(ckVtime) ~= MEMORY USAGE(ckIndex)` once past the
26+
listpack boundary. Cost is linear in cardinality at roughly **150 bytes per
27+
concurrency key** on top of the index the queue already keeps: about +1.3 MB for
28+
a queue with 10k keys, +6.5 MB at 50k. It is bounded by live cardinality (entries
29+
are GC'd when a variant drains) and expires on the 24h state TTL. The
30+
`used_memory` delta tracks the direct `MEMORY USAGE(ckVtime)` figure to within
31+
allocator noise. The listpack->skiplist transition lands between 100 and 1,000
32+
keys as expected.
33+
34+
## Redis CPU under an identical workload (2,000 rounds, ~42k script calls)
35+
36+
Same logical workload both arms (enqueue + batched dequeue + ack), so the
37+
`evalsha` call count is identical and the difference is pure vtime overhead. Note
38+
the RunQueue Lua runs as `EVALSHA`, so per-`redis.call` costs inside a script are
39+
not separable in `commandstats`; the reportable signals are total Redis CPU and
40+
aggregate `evalsha` time per call.
41+
42+
| keys (N) | CPU-sec OFF | CPU-sec ON | delta | overhead | evalsha usec/call OFF | evalsha usec/call ON |
43+
| --- | --- | --- | --- | --- | --- | --- |
44+
| 100 | 1.34 | 1.49 | +0.15 | +12% | 22.3 | 26.2 |
45+
| 1,000 | 1.34 | 1.44 | +0.10 | +7% | 22.2 | 24.8 |
46+
| 10,000 | 1.38 | 1.45 | +0.07 | +5% | 23.8 | 25.8 |
47+
48+
CPU overhead does not grow with cardinality, it shrinks: +12% at 100 keys down to
49+
+5% at 10k. Per-script cost stays roughly flat (about +2 to +4 usec/call, ~25
50+
usec/call at every cardinality), which is what the design predicts: the pass-1
51+
dequeue window is fixed (`maxCount * multiplier`, default 30) and independent of
52+
N, and the added ZSET ops are O(log N), so `log2(10000) ~= 13` adds a negligible
53+
constant. The overhead falls as a percentage because that fixed per-call cost is
54+
amortised over more work as the queue grows.
55+
56+
## Membership under sustained churn (N = 10,000, flag ON)
57+
58+
60 rounds of continuous registration + drain (fresh keys enqueued while others
59+
are served and acked), holding cardinality at 10k:
60+
61+
| round | ckIndex card | ckVtime card | ratio |
62+
| --- | --- | --- | --- |
63+
| 0 | 10,000 | 10,000 | 1.0 |
64+
| 20 | 10,000 | 10,000 | 1.0 |
65+
| 40 | 10,000 | 10,000 | 1.0 |
66+
| 59 | 10,000 | 10,000 | 1.0 |
67+
68+
`ckVtime` membership tracks `ckIndex` exactly throughout: no tombstone
69+
accumulation, no unbounded growth. The bounded/self-healing drift the limitations
70+
doc calls out (ack/TTL/DLQ draining a variant without a vtime GC) stays reclaimed
71+
by the next vtime pass and the state TTL.
72+
73+
## Bottom line for the cardinality question
74+
75+
- **Memory** grows linearly with concurrency-key cardinality, adding one
76+
`ckIndex`-sized ZSET per base queue (~150 B/key): a low-single-digit MB even at
77+
10k keys on one queue, bounded by live cardinality and TTL-reclaimed. A sudden
78+
10k-key queue costs about +1.3 MB for that queue.
79+
- **CPU** overhead is small and does not scale with cardinality: ~+5 to +12% on
80+
an identical workload, per-script cost flat at ~+2 to +4 usec/call regardless of
81+
N, because the dequeue scan window is fixed and the ZSET ops are O(log N).
82+
- **No runaway state**: `ckVtime` never outgrows `ckIndex` under sustained churn.

0 commit comments

Comments
 (0)