From 8848a7e22b6cdfffde0583e048b989a6275ce736 Mon Sep 17 00:00:00 2001 From: Joyce Fee Date: Tue, 21 Jul 2026 10:40:05 -0400 Subject: [PATCH 1/5] DOC-1381: Add Load Factor dashboard monitoring page (26.2) New how-to for the 26.2 "Redpanda - Load Factor" Grafana dashboard (ENG-540): what load factor is (live utilization vs estimated instance capacity), the internal-metrics-endpoint prerequisite (the dashboard queries vectorized_* series only; the public-metrics variant was an unshipped stretch goal), the instance-capacity and host-OS metric families behind it, import + panel-reading guidance, and limitations (nominal vendor capacity figures; unrecognized instance types emit no capacity gauges). Facts verified against v26.2.1-rc3 (enable_host_metrics default/restart/ visibility, instance_metrics.cc presence) and the dashboard JSON on dev (112 vectorized_ refs, 0 redpanda_; panel inventory). DRAFT-VERIFY markers cover the two facts pending engineering: dashboard distribution (Travis Downs pinged on DOC-1381) and the exact load-factor computation semantics. Metrics reference entries for instance_*/host_metrics_info/ io_queue_config_* arrive via the 26.2 metrics regen, not hand-edits. Co-Authored-By: Claude Fable 5 --- modules/ROOT/nav.adoc | 1 + modules/manage/pages/monitor-load-factor.adoc | 78 +++++++++++++++++++ 2 files changed, 79 insertions(+) create mode 100644 modules/manage/pages/monitor-load-factor.adoc diff --git a/modules/ROOT/nav.adoc b/modules/ROOT/nav.adoc index d371109319..4a12353a0d 100644 --- a/modules/ROOT/nav.adoc +++ b/modules/ROOT/nav.adoc @@ -248,6 +248,7 @@ *** xref:console:config/kafka-connect.adoc[Kafka Connect] ** xref:manage:use-admin-api.adoc[Use the Admin API] ** xref:manage:monitoring.adoc[] +*** xref:manage:monitor-load-factor.adoc[] * xref:upgrade:index.adoc[Upgrade] ** xref:upgrade:rolling-upgrade.adoc[Upgrade Redpanda in Linux] ** xref:upgrade:k-rolling-upgrade.adoc[Upgrade Redpanda in Kubernetes] diff --git a/modules/manage/pages/monitor-load-factor.adoc b/modules/manage/pages/monitor-load-factor.adoc new file mode 100644 index 0000000000..9ca617d76b --- /dev/null +++ b/modules/manage/pages/monitor-load-factor.adoc @@ -0,0 +1,78 @@ += Monitor Cluster Load with the Load Factor Dashboard +:description: Use the Redpanda Load Factor Grafana dashboard to see how close each broker is running to the capacity of its host, across CPU, disk, memory, and network. +:page-categories: Management, Monitoring +:env-linux: true +:page-topic-type: how-to +:personas: platform_admin +:learning-objective-1: Configure Prometheus to collect the metrics the Load Factor dashboard requires +:learning-objective-2: Import the Redpanda Load Factor dashboard into Grafana +:learning-objective-3: Interpret the load factor to decide when a cluster needs more capacity + +// DRAFT-VERIFY (DOC-1381): Dashboard distribution is unconfirmed with engineering +// (Travis Downs pinged 2026-07-21): repo import vs published bundle vs Console. +// The import procedure below assumes users take the JSON from the redpanda repo. +// Also confirm the exact load-factor computation semantics before publishing; +// the description below is concept-level, derived from the dashboard structure +// and ENG-540. + +Starting in Redpanda 26.2, the *Redpanda - Load Factor* Grafana dashboard shows how close each broker is running to the capacity of its host. The dashboard compares live utilization against the estimated capacity of the broker's cloud instance type and reports the result as a percentage, the load factor, for each resource: CPU, disk read and write IOPS, memory, and network. Use it to see how much headroom a cluster has, to decide when to scale, and to identify which resource saturates first as traffic grows. + +After reading this page, you will be able to: + +* [ ] {learning-objective-1} +* [ ] {learning-objective-2} +* [ ] {learning-objective-3} + +== Prerequisites + +* All brokers must be running Redpanda 26.2 or later. +* *Prometheus must scrape the internal metrics endpoint* (`/metrics`). The dashboard queries internal (`vectorized_`) metrics only; scraping the xref:manage:monitoring.adoc[public metrics endpoint] (`/public_metrics`) alone is not sufficient. +* The xref:reference:properties/cluster-properties.adoc#enable_host_metrics[`enable_host_metrics`] cluster property must be enabled (the default). If it has been disabled, re-enabling it requires a broker restart. +* A Grafana instance connected to your Prometheus data source. + +== How load factor works + +Traditional utilization metrics tell you how busy a resource is, but not how much capacity remains. Load factor closes that gap by comparing utilization against what the host can deliver: + +* *Instance capacity metrics* estimate what the broker's host can sustain. Redpanda recognizes the cloud instance type it runs on and emits the vendor's nominal figures for it: vCPUs, memory, disk size, read and write IOPS, and network throughput (the `vectorized_instance_*` metric family, labeled with `cloud_provider` and `instance_type`). Capacity figures are emitted only when Redpanda recognizes the instance type. +* *Host OS metrics* report actual host-level activity from `/proc`: disk statistics for the block devices that back the Redpanda data and cache directories, and network counters (the `vectorized_host_diskstats_*`, `vectorized_host_netstat_*`, and `vectorized_host_snmp_*` metric families, gated by the `enable_host_metrics` cluster property). The `vectorized_io_queue_config_*` metrics report the I/O rates measured for the host during startup. + +The dashboard computes a load factor for each resource and highlights the highest one, which shows the resource that saturates first. A load factor near 100% means the cluster is running close to its redline on that resource: sustained growth requires scaling up or out. + +NOTE: Redpanda does not emit a single `load_factor` metric. The load factor is computed by the dashboard from the metric families above. + +== Import the dashboard + +The dashboard is defined in https://github.com/redpanda-data/redpanda/blob/dev/tools/dashboards/single-load-factor.json[`single-load-factor.json`^] in the Redpanda repository. + +. Download `single-load-factor.json`. +. In Grafana, go to *Dashboards*, click *New*, and select *Import*. +. Upload the JSON file, select your Prometheus data source, and click *Import*. + +The dashboard appears as *Redpanda - Load Factor*. + +== Read the dashboard + +The top row answers the headline question: + +* *Load Factor*: The current load factor per resource, as a percentage of estimated capacity. +* *Highest Load Factor*: The single highest load factor across resources, which shows how close the broker is to its first bottleneck. + +The information panels describe the environment the estimates are based on: + +* *Host Info* and *Host Metrics Info*: The host-level statistics feeding the disk and network factors. If this panel warns that `host_metrics_info` is missing, check the <>: the internal metrics endpoint may not be scraped, or `enable_host_metrics` may be disabled. +* *Instance Info*: The detected cloud provider and instance type, and whether capacity estimates are available for it. +* *IO Properties*: The I/O rates measured for the host during startup. + +The remaining panels break each factor down over time so you can attribute load to a resource: reactor and CPU utilization (per instance and per shard), disk read and write IOPS, available memory, client connections, and network receive and transmit rates. + +== Limitations + +* Capacity estimates are nominal vendor figures for the detected instance type, not measured limits. Real capacity can differ, for example when disk throughput is provisioned separately from the instance type. +* When Redpanda does not recognize the instance type, capacity gauges are not emitted and the dashboard cannot compute a load factor for the affected resources. + +== Next steps + +* xref:manage:monitoring.adoc[Monitor Redpanda] +* xref:reference:internal-metrics-reference.adoc[Internal Metrics Reference] +* xref:manage:cluster-maintenance/manage-throughput.adoc[Manage Throughput] From f5f06898a189f040813d9edeae98dbceb9d0f495 Mon Sep 17 00:00:00 2001 From: Joyce Fee Date: Tue, 21 Jul 2026 11:00:28 -0400 Subject: [PATCH 2/5] Link the Grafana prerequisite to the Grafana + Prometheus getting-started guide Co-Authored-By: Claude Fable 5 --- modules/manage/pages/monitor-load-factor.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/manage/pages/monitor-load-factor.adoc b/modules/manage/pages/monitor-load-factor.adoc index 9ca617d76b..4e518ee51d 100644 --- a/modules/manage/pages/monitor-load-factor.adoc +++ b/modules/manage/pages/monitor-load-factor.adoc @@ -28,7 +28,7 @@ After reading this page, you will be able to: * All brokers must be running Redpanda 26.2 or later. * *Prometheus must scrape the internal metrics endpoint* (`/metrics`). The dashboard queries internal (`vectorized_`) metrics only; scraping the xref:manage:monitoring.adoc[public metrics endpoint] (`/public_metrics`) alone is not sufficient. * The xref:reference:properties/cluster-properties.adoc#enable_host_metrics[`enable_host_metrics`] cluster property must be enabled (the default). If it has been disabled, re-enabling it requires a broker restart. -* A Grafana instance connected to your Prometheus data source. +* A https://grafana.com/docs/grafana/latest/fundamentals/getting-started/first-dashboards/get-started-grafana-prometheus/[Grafana instance connected to your Prometheus data source^]. == How load factor works From 835c224bc72b1b769f2d28ba4eab2a565ff72bef Mon Sep 17 00:00:00 2001 From: Joyce Fee <102751339+Feediver1@users.noreply.github.com> Date: Fri, 24 Jul 2026 08:00:59 -0500 Subject: [PATCH 3/5] Apply suggestions from code review Co-authored-by: Travis Downs --- modules/manage/pages/monitor-load-factor.adoc | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/modules/manage/pages/monitor-load-factor.adoc b/modules/manage/pages/monitor-load-factor.adoc index 4e518ee51d..1f4572228c 100644 --- a/modules/manage/pages/monitor-load-factor.adoc +++ b/modules/manage/pages/monitor-load-factor.adoc @@ -15,7 +15,7 @@ // the description below is concept-level, derived from the dashboard structure // and ENG-540. -Starting in Redpanda 26.2, the *Redpanda - Load Factor* Grafana dashboard shows how close each broker is running to the capacity of its host. The dashboard compares live utilization against the estimated capacity of the broker's cloud instance type and reports the result as a percentage, the load factor, for each resource: CPU, disk read and write IOPS, memory, and network. Use it to see how much headroom a cluster has, to decide when to scale, and to identify which resource saturates first as traffic grows. +Starting in Redpanda 26.2, the *Redpanda - Load Factor* Grafana dashboard shows how close each broker is running to the capacity of its host. The dashboard compares live utilization against each resource's capacity and reports the result as a percentage, the load factor, for each resource: CPU, disk read and write IOPS, memory, and network. For disk and network, that capacity is estimated from the broker's cloud instance type; the other resources are measured against fixed limits (CPU utilization is already a percentage, memory against total RAM, connections against the Kafka connection limit). Use it to see how much headroom a cluster has, to decide when to scale, and to identify which resource saturates first as traffic grows. After reading this page, you will be able to: @@ -35,7 +35,8 @@ After reading this page, you will be able to: Traditional utilization metrics tell you how busy a resource is, but not how much capacity remains. Load factor closes that gap by comparing utilization against what the host can deliver: * *Instance capacity metrics* estimate what the broker's host can sustain. Redpanda recognizes the cloud instance type it runs on and emits the vendor's nominal figures for it: vCPUs, memory, disk size, read and write IOPS, and network throughput (the `vectorized_instance_*` metric family, labeled with `cloud_provider` and `instance_type`). Capacity figures are emitted only when Redpanda recognizes the instance type. -* *Host OS metrics* report actual host-level activity from `/proc`: disk statistics for the block devices that back the Redpanda data and cache directories, and network counters (the `vectorized_host_diskstats_*`, `vectorized_host_netstat_*`, and `vectorized_host_snmp_*` metric families, gated by the `enable_host_metrics` cluster property). The `vectorized_io_queue_config_*` metrics report the I/O rates measured for the host during startup. +* *Host OS metrics* report actual disk activity from the host's `/proc`: read and write counts and bytes for the block devices that back the Redpanda data and cache directories (the `vectorized_host_diskstats_*` family, gated by the `enable_host_metrics` cluster property). The `vectorized_io_queue_config_*` metrics report the I/O rates measured for the host during startup. +* *Network throughput* comes from Redpanda's own counters, `vectorized_network_bytes_received` and `vectorized_network_bytes_sent`, not from the host, so it is unaffected by `enable_host_metrics`. The dashboard computes a load factor for each resource and highlights the highest one, which shows the resource that saturates first. A load factor near 100% means the cluster is running close to its redline on that resource: sustained growth requires scaling up or out. @@ -64,11 +65,11 @@ The information panels describe the environment the estimates are based on: * *Instance Info*: The detected cloud provider and instance type, and whether capacity estimates are available for it. * *IO Properties*: The I/O rates measured for the host during startup. -The remaining panels break each factor down over time so you can attribute load to a resource: reactor and CPU utilization (per instance and per shard), disk read and write IOPS, available memory, client connections, and network receive and transmit rates. +The remaining panels break each factor down over time so you can attribute load to a resource: reactor and CPU utilization (per instance and per shard), I/O scheduler utilization, disk read and write IOPS, available memory, client connections, and network receive and transmit rates. == Limitations -* Capacity estimates are nominal vendor figures for the detected instance type, not measured limits. Real capacity can differ, for example when disk throughput is provisioned separately from the instance type. +* For disk and network, capacity estimates are nominal vendor figures for the detected instance type. These figures are often achievable, but actual capacity available to Redpanda may differ, for example, when achieved device performance is different than vendor figures, or when another process or activity on the same host consumes some of the resource. * When Redpanda does not recognize the instance type, capacity gauges are not emitted and the dashboard cannot compute a load factor for the affected resources. == Next steps From 1c83f2f54622b9386bbe3babbf225c5f4a1b312b Mon Sep 17 00:00:00 2001 From: Joyce Fee <102751339+Feediver1@users.noreply.github.com> Date: Fri, 24 Jul 2026 08:02:17 -0500 Subject: [PATCH 4/5] Apply suggestion from @Feediver1 --- modules/manage/pages/monitor-load-factor.adoc | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/modules/manage/pages/monitor-load-factor.adoc b/modules/manage/pages/monitor-load-factor.adoc index 1f4572228c..2f42e3167d 100644 --- a/modules/manage/pages/monitor-load-factor.adoc +++ b/modules/manage/pages/monitor-load-factor.adoc @@ -70,7 +70,8 @@ The remaining panels break each factor down over time so you can attribute load == Limitations * For disk and network, capacity estimates are nominal vendor figures for the detected instance type. These figures are often achievable, but actual capacity available to Redpanda may differ, for example, when achieved device performance is different than vendor figures, or when another process or activity on the same host consumes some of the resource. -* When Redpanda does not recognize the instance type, capacity gauges are not emitted and the dashboard cannot compute a load factor for the affected resources. +* When Redpanda does not recognize the instance type, capacity gauges are not emitted and the dashboard cannot compute a load factor for the affected resources. +* Automatic detection of host resources works only on AWS. Factors which use host resources will not be reported on other cloud providers or on self-hosted hardware. However, the provided dashboards can be modified to use fixed, known values in these cases as the denominator in the load factor calculation. == Next steps From 56252c7ab5b85273e957dc3167eff05cadd4e3ac Mon Sep 17 00:00:00 2001 From: Joyce Fee <102751339+Feediver1@users.noreply.github.com> Date: Fri, 24 Jul 2026 08:03:18 -0500 Subject: [PATCH 5/5] Apply suggestion from @Feediver1 --- modules/manage/pages/monitor-load-factor.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/manage/pages/monitor-load-factor.adoc b/modules/manage/pages/monitor-load-factor.adoc index 2f42e3167d..3fa73df506 100644 --- a/modules/manage/pages/monitor-load-factor.adoc +++ b/modules/manage/pages/monitor-load-factor.adoc @@ -15,7 +15,7 @@ // the description below is concept-level, derived from the dashboard structure // and ENG-540. -Starting in Redpanda 26.2, the *Redpanda - Load Factor* Grafana dashboard shows how close each broker is running to the capacity of its host. The dashboard compares live utilization against each resource's capacity and reports the result as a percentage, the load factor, for each resource: CPU, disk read and write IOPS, memory, and network. For disk and network, that capacity is estimated from the broker's cloud instance type; the other resources are measured against fixed limits (CPU utilization is already a percentage, memory against total RAM, connections against the Kafka connection limit). Use it to see how much headroom a cluster has, to decide when to scale, and to identify which resource saturates first as traffic grows. +Starting in Redpanda v26.2, the *Redpanda - Load Factor* Grafana dashboard shows how close each broker is running to the capacity of its host. The dashboard compares live utilization against each resource's capacity and reports the result as a percentage, the load factor, for each resource: CPU, disk read and write IOPS, memory, and network. For disk and network, that capacity is estimated from the broker's cloud instance type; the other resources are measured against fixed limits (CPU utilization is already a percentage, memory against total RAM, connections against the Kafka connection limit). Use it to see how much headroom a cluster has, to decide when to scale, and to identify which resource saturates first as traffic grows. After reading this page, you will be able to: