From 652dc164565d8c1b0176b5329dcafd134b098f37 Mon Sep 17 00:00:00 2001 From: Joyce Fee Date: Wed, 22 Jul 2026 13:13:28 -0400 Subject: [PATCH 1/8] docs: add fetch read coalescing page (DOC-2386) New consume-data page for the 26.2 opt-in per-shard fetch read coalescer: concept, enable procedure, hit-ratio evaluation guidance, and limitations. Adds a nav entry and an env-gated cross-link TIP on the follower-fetching page (single-sourced; cloud build unaffected). Sourced from the engineering TOI (cupboard engineering/core/toi/v26.2/2026-07-21-fetch-read-coalescing.md). Co-Authored-By: Claude Fable 5 --- modules/ROOT/nav.adoc | 1 + .../consume-data/fetch-read-coalescing.adoc | 84 +++++++++++++++++++ .../pages/consume-data/follower-fetching.adoc | 4 + 3 files changed, 89 insertions(+) create mode 100644 modules/develop/pages/consume-data/fetch-read-coalescing.adoc diff --git a/modules/ROOT/nav.adoc b/modules/ROOT/nav.adoc index a413d36df3..668928239b 100644 --- a/modules/ROOT/nav.adoc +++ b/modules/ROOT/nav.adoc @@ -48,6 +48,7 @@ ** xref:develop:consume-data/index.adoc[Consume Data] *** xref:develop:consume-data/consumer-offsets.adoc[Consumer Offsets] *** xref:develop:consume-data/follower-fetching.adoc[Follower Fetching] +*** xref:develop:consume-data/fetch-read-coalescing.adoc[Fetch Read Coalescing] *** xref:console:ui/programmable-push-filters.adoc[Filter Messages] *** xref:console:ui/record-deserialization.adoc[Deserialize Messages] *** xref:console:ui/paginate-messages-events.adoc[] diff --git a/modules/develop/pages/consume-data/fetch-read-coalescing.adoc b/modules/develop/pages/consume-data/fetch-read-coalescing.adoc new file mode 100644 index 0000000000..88d43884fe --- /dev/null +++ b/modules/develop/pages/consume-data/fetch-read-coalescing.adoc @@ -0,0 +1,84 @@ += Fetch Read Coalescing +:description: Reduce redundant read CPU and fetch-response memory under high consumer fan-out by sharing one read result across concurrent fetches of the same data. +:page-categories: Clients, Development + +When many consumers fetch the same partition at the same offset concurrently (high read fan-out), the broker does redundant work: each fetch performs its own log read, its own serialization, and allocates its own copy of the response bytes. With a fan-out of N consumers, that is roughly N times the read CPU and N times the fetch-response memory for byte-identical output. + +Fetch read coalescing removes that redundancy. The broker reads and serializes each unique read once, and shares the single result with every concurrent (and eligible back-to-back) consumer of the same data: one read, one serialization, one buffer, fanned out to all requesters. Under aligned fan-out, read CPU and fetch-response memory drop from roughly N times to one. + +Fetch read coalescing is off by default. Enable it when many consumers tail the same partitions with the same fetch settings, for example fan-out delivery of one stream to many downstream applications. Workloads with misaligned or low fan-out see little to no benefit and should leave it disabled. See <> for how to measure whether it helps your workload. + +== How fetch read coalescing works + +The coalescer sits on the fetch read path of each shard and groups reads by the properties that make two reads produce byte-identical output: + +* Partition +* Fetch offset +* Isolation level +* `max_bytes` budget +* Whether the read is obligatory (guarantees at least one batch and ignores the strict byte limit) or strict (honors `max_bytes`). These are grouped apart, so neither serves the other. + +For each fetch read, one of three things happens: + +* *Ready hit*: A previously completed read is still retained and covers what this fetch needs, so its result is reused with no new read. +* *In-flight hit*: A read for the same data is already running, so this fetch awaits that read instead of starting its own. +* *Miss*: The read runs exactly once, and its result is shared with any waiting fetches and retained for later readers. If the read fails, the error is propagated to all waiters. + +The shared result is reference-counted. The coalescer keeps only a weak handle to it, so it never pins memory; the requesting fetches hold the only strong references until their responses are sent. This is where the memory savings come from: coalesced consumers share one buffer instead of each holding its own copy. + +Coalescing is scoped per shard: it collapses only the fan-out that lands on the same shard. It composes with xref:develop:consume-data/follower-fetching.adoc[follower fetching] rather than replacing it: follower fetching spreads consumers across replicas to distribute the read load, and coalescing removes the redundancy within each shard. + +== Enable fetch read coalescing + +Enable coalescing with the `kafka_fetch_read_coalescing_enabled` cluster property: + +[,bash] +---- +rpk cluster config set kafka_fetch_read_coalescing_enabled true +---- + +This is a runtime change; no restart is required. The coalescing cache is fixed-size (about 0.5 MB per shard) and is allocated only while the feature is enabled; there is no separate sizing property. Setting the property back to `false` disables coalescing and clears the per-shard cache immediately. + +== Evaluate coalescing effectiveness + +The coalescer exports four counters on the internal `/metrics` endpoint, one series per shard. The metrics are registered only when internal metrics are enabled (that is, when `disable_metrics` is not set). + +|=== +| Metric | Description + +| `vectorized_kafka_fetch_read_coalescer_insertions_total` +| Fetch reads that missed and performed a new read. + +| `vectorized_kafka_fetch_read_coalescer_reinsertions_total` +| Fetch reads that re-read an existing stale or expired entry. + +| `vectorized_kafka_fetch_read_coalescer_ready_hits_total` +| Fetch reads served from a retained, already-completed read. + +| `vectorized_kafka_fetch_read_coalescer_inflight_hits_total` +| Fetch reads served by awaiting an in-flight read. +|=== + +The effectiveness of coalescing is the ratio of hits (reads avoided) to insertions (reads actually performed): + +[.no-copy] +---- +coalescing hit ratio = + (ready_hits_total + inflight_hits_total) + / (insertions_total + reinsertions_total + ready_hits_total + inflight_hits_total) +---- + +A ratio near 0 means little fan-out is being collapsed: the reads aren't aligning on the same partition, offset, and fetch settings, and the workload is unlikely to benefit from coalescing. A high ratio means the coalescer is absorbing most of the fan-out. + +== Limitations + +* *Only identical reads coalesce.* Consumers reading the same data but with a different `max_bytes`, a different offset, or a different isolation level do not share a read. The benefit scales with how aligned the fan-out is: the ideal case is many consumers tailing the same partition at the same offset with the same fetch sizing. +* *Obligatory and strict reads are grouped apart.* A single partition and offset fetched both ways at the same time incurs one duplicate read by design. +* *Retention is best-effort.* Completed results are only reusable by later readers while a fetch still references them; the coalescer never keeps a result alive on its own. The reliable savings are on genuinely concurrent readers of the same data, and back-to-back reuse is opportunistic. +* *Per-shard scope.* Coalescing collapses only the fan-out that lands on the same shard. Where consumers are spread across replicas and shards, for example with follower fetching, each shard coalesces the fan-out local to it. +* *It is a de-duplication mechanism, not a throughput mechanism.* Coalescing reduces read CPU and fetch-response memory; it does not change where reads land or reduce the total volume of unique reads. + +== Suggested reading + +* xref:develop:consume-data/follower-fetching.adoc[] +* xref:manage:monitoring.adoc[] diff --git a/modules/develop/pages/consume-data/follower-fetching.adoc b/modules/develop/pages/consume-data/follower-fetching.adoc index 1ece89ff49..8592c54b26 100644 --- a/modules/develop/pages/consume-data/follower-fetching.adoc +++ b/modules/develop/pages/consume-data/follower-fetching.adoc @@ -31,6 +31,10 @@ ifdef::env-cloud[] For each consumer, set the `client.rack` property to a rack ID. Rack awareness is pre-enabled for cloud-based clusters in multi-AZ environments. endif::[] +ifndef::env-cloud[] +TIP: Follower fetching spreads consumers across replicas to distribute the read load. If many consumers also fetch the same data concurrently, xref:develop:consume-data/fetch-read-coalescing.adoc[fetch read coalescing] complements it by removing the redundant work within each shard. +endif::[] + include::shared:partial$suggested-video.adoc[] * https://www.youtube.com/watch?v=wV6gH5_yVaw&ab_channel=RedpandaData[YouTube - Redpanda Office Hour: Follower Fetching (52 mins)^] From d4500fe963012865ebd00e020061f17e5ba6afb7 Mon Sep 17 00:00:00 2001 From: Joyce Fee Date: Wed, 22 Jul 2026 13:58:00 -0400 Subject: [PATCH 2/8] docs: apply review feedback on fetch-read-coalescing page Reword the enable guidance (disabled by default, little-to-no benefit) and add an intro sentence to Limitations. Co-Authored-By: Claude Fable 5 --- modules/develop/pages/consume-data/fetch-read-coalescing.adoc | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/modules/develop/pages/consume-data/fetch-read-coalescing.adoc b/modules/develop/pages/consume-data/fetch-read-coalescing.adoc index 88d43884fe..963a5eeafc 100644 --- a/modules/develop/pages/consume-data/fetch-read-coalescing.adoc +++ b/modules/develop/pages/consume-data/fetch-read-coalescing.adoc @@ -6,7 +6,7 @@ When many consumers fetch the same partition at the same offset concurrently (hi Fetch read coalescing removes that redundancy. The broker reads and serializes each unique read once, and shares the single result with every concurrent (and eligible back-to-back) consumer of the same data: one read, one serialization, one buffer, fanned out to all requesters. Under aligned fan-out, read CPU and fetch-response memory drop from roughly N times to one. -Fetch read coalescing is off by default. Enable it when many consumers tail the same partitions with the same fetch settings, for example fan-out delivery of one stream to many downstream applications. Workloads with misaligned or low fan-out see little to no benefit and should leave it disabled. See <> for how to measure whether it helps your workload. +Fetch read coalescing is disabled by default. Enable it when many consumers tail the same partitions with the same fetch settings, for example fan-out delivery of one stream to many downstream applications. Workloads with misaligned or low fan-out see little-to-no benefit and should leave it disabled. See <> for how to measure whether it helps your workload. == How fetch read coalescing works @@ -72,6 +72,8 @@ A ratio near 0 means little fan-out is being collapsed: the reads aren't alignin == Limitations +Before enabling fetch read coalescing, familiarize yourself with the following limitations: + * *Only identical reads coalesce.* Consumers reading the same data but with a different `max_bytes`, a different offset, or a different isolation level do not share a read. The benefit scales with how aligned the fan-out is: the ideal case is many consumers tailing the same partition at the same offset with the same fetch sizing. * *Obligatory and strict reads are grouped apart.* A single partition and offset fetched both ways at the same time incurs one duplicate read by design. * *Retention is best-effort.* Completed results are only reusable by later readers while a fetch still references them; the coalescer never keeps a result alive on its own. The reliable savings are on genuinely concurrent readers of the same data, and back-to-back reuse is opportunistic. From 3f9c0ac6c93715ec9836bfc48c55e7356512d431 Mon Sep 17 00:00:00 2001 From: Joyce Fee Date: Wed, 22 Jul 2026 14:13:29 -0400 Subject: [PATCH 3/8] docs: single-source fetch-read-coalescing for Redpanda Cloud Per SME (Brandon Allard), the feature can be enabled on BYOC and Dedicated clusters. Wrap the page in a single-source tag with env gating: cloud gets an availability line, the enable procedure with a pointer to cloud cluster configuration, and the limitations; the internal-metrics evaluation section is excluded from the cloud build (only public_metrics is customer-visible in Cloud). Companion stub: cloud-docs#645. Co-Authored-By: Claude Fable 5 --- .../consume-data/fetch-read-coalescing.adoc | 21 +++++++++++++++++++ 1 file changed, 21 insertions(+) diff --git a/modules/develop/pages/consume-data/fetch-read-coalescing.adoc b/modules/develop/pages/consume-data/fetch-read-coalescing.adoc index 963a5eeafc..9fd98af879 100644 --- a/modules/develop/pages/consume-data/fetch-read-coalescing.adoc +++ b/modules/develop/pages/consume-data/fetch-read-coalescing.adoc @@ -1,12 +1,20 @@ = Fetch Read Coalescing :description: Reduce redundant read CPU and fetch-response memory under high consumer fan-out by sharing one read result across concurrent fetches of the same data. :page-categories: Clients, Development +// tag::single-source[] When many consumers fetch the same partition at the same offset concurrently (high read fan-out), the broker does redundant work: each fetch performs its own log read, its own serialization, and allocates its own copy of the response bytes. With a fan-out of N consumers, that is roughly N times the read CPU and N times the fetch-response memory for byte-identical output. Fetch read coalescing removes that redundancy. The broker reads and serializes each unique read once, and shares the single result with every concurrent (and eligible back-to-back) consumer of the same data: one read, one serialization, one buffer, fanned out to all requesters. Under aligned fan-out, read CPU and fetch-response memory drop from roughly N times to one. +ifdef::env-cloud[] +Fetch read coalescing is available on BYOC and Dedicated clusters. + +Fetch read coalescing is disabled by default. Enable it when many consumers tail the same partitions with the same fetch settings, for example fan-out delivery of one stream to many downstream applications. Workloads with misaligned or low fan-out see little-to-no benefit and should leave it disabled. +endif::[] +ifndef::env-cloud[] Fetch read coalescing is disabled by default. Enable it when many consumers tail the same partitions with the same fetch settings, for example fan-out delivery of one stream to many downstream applications. Workloads with misaligned or low fan-out see little-to-no benefit and should leave it disabled. See <> for how to measure whether it helps your workload. +endif::[] == How fetch read coalescing works @@ -39,6 +47,11 @@ rpk cluster config set kafka_fetch_read_coalescing_enabled true This is a runtime change; no restart is required. The coalescing cache is fixed-size (about 0.5 MB per shard) and is allocated only while the feature is enabled; there is no separate sizing property. Setting the property back to `false` disables coalescing and clears the per-shard cache immediately. +ifdef::env-cloud[] +For details about updating cluster properties in Redpanda Cloud, see xref:manage:cluster-maintenance/config-cluster.adoc[]. +endif::[] + +ifndef::env-cloud[] == Evaluate coalescing effectiveness The coalescer exports four counters on the internal `/metrics` endpoint, one series per shard. The metrics are registered only when internal metrics are enabled (that is, when `disable_metrics` is not set). @@ -69,6 +82,7 @@ coalescing hit ratio = ---- A ratio near 0 means little fan-out is being collapsed: the reads aren't aligning on the same partition, offset, and fetch settings, and the workload is unlikely to benefit from coalescing. A high ratio means the coalescer is absorbing most of the fan-out. +endif::[] == Limitations @@ -83,4 +97,11 @@ Before enabling fetch read coalescing, familiarize yourself with the following l == Suggested reading * xref:develop:consume-data/follower-fetching.adoc[] +ifndef::env-cloud[] * xref:manage:monitoring.adoc[] +endif::[] +ifdef::env-cloud[] +* xref:manage:monitor-cloud.adoc[] +endif::[] + +// end::single-source[] From 8324dec480ff3fef7014b715bb3cfc0b7662d36c Mon Sep 17 00:00:00 2001 From: Joyce Fee Date: Thu, 23 Jul 2026 14:07:57 -0400 Subject: [PATCH 4/8] Replace doc-coined "aligned fan-out" shorthand with plain language The term was invented by this page and never defined; spell out the condition instead (same partition, same offset, same fetch settings). Co-Authored-By: Claude Fable 5 --- .../develop/pages/consume-data/fetch-read-coalescing.adoc | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/modules/develop/pages/consume-data/fetch-read-coalescing.adoc b/modules/develop/pages/consume-data/fetch-read-coalescing.adoc index 9fd98af879..f4d8e065a9 100644 --- a/modules/develop/pages/consume-data/fetch-read-coalescing.adoc +++ b/modules/develop/pages/consume-data/fetch-read-coalescing.adoc @@ -5,15 +5,15 @@ When many consumers fetch the same partition at the same offset concurrently (high read fan-out), the broker does redundant work: each fetch performs its own log read, its own serialization, and allocates its own copy of the response bytes. With a fan-out of N consumers, that is roughly N times the read CPU and N times the fetch-response memory for byte-identical output. -Fetch read coalescing removes that redundancy. The broker reads and serializes each unique read once, and shares the single result with every concurrent (and eligible back-to-back) consumer of the same data: one read, one serialization, one buffer, fanned out to all requesters. Under aligned fan-out, read CPU and fetch-response memory drop from roughly N times to one. +Fetch read coalescing removes that redundancy. The broker reads and serializes each unique read once, and shares the single result with every concurrent (and eligible back-to-back) consumer of the same data: one read, one serialization, one buffer, fanned out to all requesters. When all N consumers fetch the same partition at the same offset with the same fetch settings, read CPU and fetch-response memory drop from roughly N times to one. ifdef::env-cloud[] Fetch read coalescing is available on BYOC and Dedicated clusters. -Fetch read coalescing is disabled by default. Enable it when many consumers tail the same partitions with the same fetch settings, for example fan-out delivery of one stream to many downstream applications. Workloads with misaligned or low fan-out see little-to-no benefit and should leave it disabled. +Fetch read coalescing is disabled by default. Enable it when many consumers tail the same partitions with the same fetch settings, for example fan-out delivery of one stream to many downstream applications. Workloads where consumers read at different offsets, or where few consumers share a partition, see little-to-no benefit and should leave it disabled. endif::[] ifndef::env-cloud[] -Fetch read coalescing is disabled by default. Enable it when many consumers tail the same partitions with the same fetch settings, for example fan-out delivery of one stream to many downstream applications. Workloads with misaligned or low fan-out see little-to-no benefit and should leave it disabled. See <> for how to measure whether it helps your workload. +Fetch read coalescing is disabled by default. Enable it when many consumers tail the same partitions with the same fetch settings, for example fan-out delivery of one stream to many downstream applications. Workloads where consumers read at different offsets, or where few consumers share a partition, see little-to-no benefit and should leave it disabled. See <> for how to measure whether it helps your workload. endif::[] == How fetch read coalescing works @@ -88,7 +88,7 @@ endif::[] Before enabling fetch read coalescing, familiarize yourself with the following limitations: -* *Only identical reads coalesce.* Consumers reading the same data but with a different `max_bytes`, a different offset, or a different isolation level do not share a read. The benefit scales with how aligned the fan-out is: the ideal case is many consumers tailing the same partition at the same offset with the same fetch sizing. +* *Only identical reads coalesce.* Consumers reading the same data but with a different `max_bytes`, a different offset, or a different isolation level do not share a read. The benefit scales with how closely the concurrent fetches match: the ideal case is many consumers tailing the same partition at the same offset with the same fetch sizing. * *Obligatory and strict reads are grouped apart.* A single partition and offset fetched both ways at the same time incurs one duplicate read by design. * *Retention is best-effort.* Completed results are only reusable by later readers while a fetch still references them; the coalescer never keeps a result alive on its own. The reliable savings are on genuinely concurrent readers of the same data, and back-to-back reuse is opportunistic. * *Per-shard scope.* Coalescing collapses only the fan-out that lands on the same shard. Where consumers are spread across replicas and shards, for example with follower fetching, each shard coalesces the fan-out local to it. From 7145bcec9c2a91b6e9cfcb9c01031f775e592d92 Mon Sep 17 00:00:00 2001 From: Joyce Fee <102751339+Feediver1@users.noreply.github.com> Date: Thu, 23 Jul 2026 13:16:57 -0500 Subject: [PATCH 5/8] Apply suggestion from @Feediver1 --- modules/develop/pages/consume-data/fetch-read-coalescing.adoc | 1 - 1 file changed, 1 deletion(-) diff --git a/modules/develop/pages/consume-data/fetch-read-coalescing.adoc b/modules/develop/pages/consume-data/fetch-read-coalescing.adoc index f4d8e065a9..597243b597 100644 --- a/modules/develop/pages/consume-data/fetch-read-coalescing.adoc +++ b/modules/develop/pages/consume-data/fetch-read-coalescing.adoc @@ -92,7 +92,6 @@ Before enabling fetch read coalescing, familiarize yourself with the following l * *Obligatory and strict reads are grouped apart.* A single partition and offset fetched both ways at the same time incurs one duplicate read by design. * *Retention is best-effort.* Completed results are only reusable by later readers while a fetch still references them; the coalescer never keeps a result alive on its own. The reliable savings are on genuinely concurrent readers of the same data, and back-to-back reuse is opportunistic. * *Per-shard scope.* Coalescing collapses only the fan-out that lands on the same shard. Where consumers are spread across replicas and shards, for example with follower fetching, each shard coalesces the fan-out local to it. -* *It is a de-duplication mechanism, not a throughput mechanism.* Coalescing reduces read CPU and fetch-response memory; it does not change where reads land or reduce the total volume of unique reads. == Suggested reading From 8ebd811e55c11a78f80c37e3c9b455725b818799 Mon Sep 17 00:00:00 2001 From: Joyce Fee Date: Thu, 23 Jul 2026 14:24:16 -0400 Subject: [PATCH 6/8] Make clear cloud users cannot self-serve enable coalescing Per ballard26's review: kafka_fetch_read_coalescing_enabled is not whitelisted for user control in Redpanda Cloud. Gate the rpk procedure to Self-Managed and route cloud readers to Redpanda Support. Revert to self-serve wording if the property is whitelisted before release. Co-Authored-By: Claude Fable 5 --- .../pages/consume-data/fetch-read-coalescing.adoc | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/modules/develop/pages/consume-data/fetch-read-coalescing.adoc b/modules/develop/pages/consume-data/fetch-read-coalescing.adoc index 597243b597..1b7f3f38fe 100644 --- a/modules/develop/pages/consume-data/fetch-read-coalescing.adoc +++ b/modules/develop/pages/consume-data/fetch-read-coalescing.adoc @@ -10,7 +10,7 @@ Fetch read coalescing removes that redundancy. The broker reads and serializes e ifdef::env-cloud[] Fetch read coalescing is available on BYOC and Dedicated clusters. -Fetch read coalescing is disabled by default. Enable it when many consumers tail the same partitions with the same fetch settings, for example fan-out delivery of one stream to many downstream applications. Workloads where consumers read at different offsets, or where few consumers share a partition, see little-to-no benefit and should leave it disabled. +Fetch read coalescing is disabled by default, and the cluster property that controls it is managed by Redpanda; see <>. It benefits workloads where many consumers tail the same partitions with the same fetch settings, for example fan-out delivery of one stream to many downstream applications. Workloads where consumers read at different offsets, or where few consumers share a partition, see little-to-no benefit. endif::[] ifndef::env-cloud[] Fetch read coalescing is disabled by default. Enable it when many consumers tail the same partitions with the same fetch settings, for example fan-out delivery of one stream to many downstream applications. Workloads where consumers read at different offsets, or where few consumers share a partition, see little-to-no benefit and should leave it disabled. See <> for how to measure whether it helps your workload. @@ -38,6 +38,7 @@ Coalescing is scoped per shard: it collapses only the fan-out that lands on the == Enable fetch read coalescing +ifndef::env-cloud[] Enable coalescing with the `kafka_fetch_read_coalescing_enabled` cluster property: [,bash] @@ -46,9 +47,11 @@ rpk cluster config set kafka_fetch_read_coalescing_enabled true ---- This is a runtime change; no restart is required. The coalescing cache is fixed-size (about 0.5 MB per shard) and is allocated only while the feature is enabled; there is no separate sizing property. Setting the property back to `false` disables coalescing and clears the per-shard cache immediately. - +endif::[] ifdef::env-cloud[] -For details about updating cluster properties in Redpanda Cloud, see xref:manage:cluster-maintenance/config-cluster.adoc[]. +The `kafka_fetch_read_coalescing_enabled` cluster property that controls coalescing is not user-configurable in Redpanda Cloud. To enable coalescing on a BYOC or Dedicated cluster, contact https://support.redpanda.com/hc/en-us/requests/new[Redpanda Support^]. + +Enabling or disabling coalescing is a runtime change: no cluster restart is required, and the coalescing cache (about 0.5 MB per shard) is allocated only while the feature is enabled. endif::[] ifndef::env-cloud[] From 66eed08d9557879bcc6e38aed05685264aff342f Mon Sep 17 00:00:00 2001 From: Joyce Fee Date: Thu, 23 Jul 2026 14:31:32 -0400 Subject: [PATCH 7/8] Warn that coalescing costs CPU at low fan-out, not just "no benefit" Per ballard26: the de-duplication logic runs on every fetch read, so 1:1 or 1:2 fan-out likely increases reactor utilization. Strengthen the enable guidance in both env variants and add a limitations bullet. Co-Authored-By: Claude Fable 5 --- .../develop/pages/consume-data/fetch-read-coalescing.adoc | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/modules/develop/pages/consume-data/fetch-read-coalescing.adoc b/modules/develop/pages/consume-data/fetch-read-coalescing.adoc index 1b7f3f38fe..bf80bd83b5 100644 --- a/modules/develop/pages/consume-data/fetch-read-coalescing.adoc +++ b/modules/develop/pages/consume-data/fetch-read-coalescing.adoc @@ -10,10 +10,10 @@ Fetch read coalescing removes that redundancy. The broker reads and serializes e ifdef::env-cloud[] Fetch read coalescing is available on BYOC and Dedicated clusters. -Fetch read coalescing is disabled by default, and the cluster property that controls it is managed by Redpanda; see <>. It benefits workloads where many consumers tail the same partitions with the same fetch settings, for example fan-out delivery of one stream to many downstream applications. Workloads where consumers read at different offsets, or where few consumers share a partition, see little-to-no benefit. +Fetch read coalescing is disabled by default, and the cluster property that controls it is managed by Redpanda; see <>. It benefits workloads where many consumers tail the same partitions with the same fetch settings, for example fan-out delivery of one stream to many downstream applications. Request it only for high fan-out workloads: at low fan-out (for example, one or two consumers per partition), the de-duplication logic saves little to nothing and its overhead can increase reactor utilization. endif::[] ifndef::env-cloud[] -Fetch read coalescing is disabled by default. Enable it when many consumers tail the same partitions with the same fetch settings, for example fan-out delivery of one stream to many downstream applications. Workloads where consumers read at different offsets, or where few consumers share a partition, see little-to-no benefit and should leave it disabled. See <> for how to measure whether it helps your workload. +Fetch read coalescing is disabled by default. Enable it when many consumers tail the same partitions with the same fetch settings, for example fan-out delivery of one stream to many downstream applications. Enable it only for high fan-out workloads: at low fan-out (for example, one or two consumers per partition), the de-duplication logic saves little to nothing and its overhead can increase reactor utilization. See <> for how to measure whether it helps your workload. endif::[] == How fetch read coalescing works @@ -95,6 +95,7 @@ Before enabling fetch read coalescing, familiarize yourself with the following l * *Obligatory and strict reads are grouped apart.* A single partition and offset fetched both ways at the same time incurs one duplicate read by design. * *Retention is best-effort.* Completed results are only reusable by later readers while a fetch still references them; the coalescer never keeps a result alive on its own. The reliable savings are on genuinely concurrent readers of the same data, and back-to-back reuse is opportunistic. * *Per-shard scope.* Coalescing collapses only the fan-out that lands on the same shard. Where consumers are spread across replicas and shards, for example with follower fetching, each shard coalesces the fan-out local to it. +* *De-duplication adds per-read overhead.* The de-duplication logic runs on every fetch read, whether or not reads coalesce. At low fan-out (for example, one or two consumers per partition), this overhead can increase reactor utilization while saving little to nothing, so only enable coalescing for high fan-out workloads. == Suggested reading From 4b8a22a2ff1952a81ad7e61288a1e957683052ff Mon Sep 17 00:00:00 2001 From: Joyce Fee Date: Thu, 23 Jul 2026 16:22:25 -0400 Subject: [PATCH 8/8] Mark fetch read coalescing as an Enterprise feature Per leadership decision: add the enterprise-license NOTE and a license prerequisite to the page (Self-Managed only; Cloud portions unchanged), and list the feature in the Enterprise features table. Co-Authored-By: Claude Fable 5 --- .../pages/consume-data/fetch-read-coalescing.adoc | 13 +++++++++++++ modules/get-started/pages/licensing/overview.adoc | 4 ++++ 2 files changed, 17 insertions(+) diff --git a/modules/develop/pages/consume-data/fetch-read-coalescing.adoc b/modules/develop/pages/consume-data/fetch-read-coalescing.adoc index bf80bd83b5..9db0201f4f 100644 --- a/modules/develop/pages/consume-data/fetch-read-coalescing.adoc +++ b/modules/develop/pages/consume-data/fetch-read-coalescing.adoc @@ -3,6 +3,13 @@ :page-categories: Clients, Development // tag::single-source[] +ifndef::env-cloud[] +[NOTE] +==== +include::shared:partial$enterprise-license.adoc[] +==== +endif::[] + When many consumers fetch the same partition at the same offset concurrently (high read fan-out), the broker does redundant work: each fetch performs its own log read, its own serialization, and allocates its own copy of the response bytes. With a fan-out of N consumers, that is roughly N times the read CPU and N times the fetch-response memory for byte-identical output. Fetch read coalescing removes that redundancy. The broker reads and serializes each unique read once, and shares the single result with every concurrent (and eligible back-to-back) consumer of the same data: one read, one serialization, one buffer, fanned out to all requesters. When all N consumers fetch the same partition at the same offset with the same fetch settings, read CPU and fetch-response memory drop from roughly N times to one. @@ -16,6 +23,12 @@ ifndef::env-cloud[] Fetch read coalescing is disabled by default. Enable it when many consumers tail the same partitions with the same fetch settings, for example fan-out delivery of one stream to many downstream applications. Enable it only for high fan-out workloads: at low fan-out (for example, one or two consumers per partition), the de-duplication logic saves little to nothing and its overhead can increase reactor utilization. See <> for how to measure whether it helps your workload. endif::[] +ifndef::env-cloud[] +== Prerequisites + +* A valid xref:get-started:licensing/index.adoc[Redpanda Enterprise license]. +endif::[] + == How fetch read coalescing works The coalescer sits on the fetch read path of each shard and groups reads by the properties that make two reads produce byte-identical output: diff --git a/modules/get-started/pages/licensing/overview.adoc b/modules/get-started/pages/licensing/overview.adoc index fe8bfb81ab..8fa938ae19 100644 --- a/modules/get-started/pages/licensing/overview.adoc +++ b/modules/get-started/pages/licensing/overview.adoc @@ -128,6 +128,10 @@ Continuous Intra-Broker Partition Balancing is enabled by default for all new cl | Continuous Intra-Broker Partition Balancing is disabled. +| xref:develop:consume-data/fetch-read-coalescing.adoc[Fetch Read Coalescing] +| Shares one read result across concurrent fetches of the same data, reducing read CPU and fetch-response memory under high consumer fan-out. +| You can no longer enable `kafka_fetch_read_coalescing_enabled`. + | xref:manage:security/fips-compliance.adoc[FIPS Compliance] | Enables compliance with FIPS security standards for cryptography. | No change.