Benchmarks: Micro benchmark - add nvbench based kernel-launch, sleep-kernel & auto-throughput by WenqingLan1 · Pull Request #750 · microsoft/superbenchmark

WenqingLan1 · 2025-10-09T23:12:33Z

This pull request adds support for NVBench-based GPU micro-benchmarks to SuperBench.

Integrated the NVBench submodule
Implemented three benchmarks
- nvbench-sleep-kernel
- nvbench-kernel-launch
- nvbench-auto-throughput
updated documentation and added example scripts

Example config:

version: v0.12
superbench:
  enable:
  # nvbench benchmarks
  - nvbench-sleep-kernel:single
  - nvbench-sleep-kernel:list
  - nvbench-sleep-kernel:range
  - nvbench-sleep-kernel:range-step
  - nvbench-kernel-launch
  - nvbench-auto-throughput
  - nvbench-auto-throughput:stride-list
  - nvbench-auto-throughput:stride-range
  var:
    default_local_mode: &default_local_mode
      modes:
      - name: local
        proc_num: 4
        prefix: CUDA_VISIBLE_DEVICES={proc_rank}
        parallel: yes
  benchmarks:
    nvbench-sleep-kernel:single:
      <<: *default_local_mode
      timeout: 300
      parameters:
        duration_us: "50"                   # Single value format
        timeout: 30
    nvbench-sleep-kernel:list:
      <<: *default_local_mode
      timeout: 300
      parameters:
        duration_us: "[25,50,75]"         # List format - no spaces after commas
        timeout: 30
    nvbench-sleep-kernel:range:
      <<: *default_local_mode
      timeout: 300
      parameters:
        duration_us: "[0:5]"           # Range format
        timeout: 30
    nvbench-sleep-kernel:range-step:
      <<: *default_local_mode
      timeout: 300
      parameters:
        duration_us: "[0:50:10]"         # Range with step format
        timeout: 30
    nvbench-kernel-launch:
      <<: *default_local_mode
      timeout: 300
    nvbench-auto-throughput:
      <<: *default_local_mode
      timeout: 600
      parameters:
        stride: "[1,2,4,8]"              # List format for stride
        block_size: "[128,256,512,1024]"  # List format for block size
    nvbench-auto-throughput:stride-list:
      <<: *default_local_mode
      timeout: 600
      parameters:
        stride: "[1,2,4,8]"              # List format
        block_size: "[256,512]"
    nvbench-auto-throughput:stride-range:
      <<: *default_local_mode
      timeout: 600
      parameters:
        stride: "[1:8:2]"                # Range with step format
        block_size: "256"                 # Single value format

codecov · 2025-10-10T20:44:21Z

Codecov Report

❌ Patch coverage is 98.20628% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 86.35%. Comparing base (67298ae) to head (63e7ed2).

Files with missing lines	Patch %	Lines
...rbench/benchmarks/micro_benchmarks/nvbench_base.py	97.91%	2 Missing ⚠️
...hmarks/micro_benchmarks/nvbench_auto_throughput.py	98.07%	1 Missing ⚠️
...enchmarks/micro_benchmarks/nvbench_sleep_kernel.py	97.67%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #750      +/-   ##
==========================================
+ Coverage   86.02%   86.35%   +0.33%     
==========================================
  Files         103      107       +4     
  Lines        7950     8173     +223     
==========================================
+ Hits         6839     7058     +219     
- Misses       1111     1115       +4

Flag	Coverage Δ
cpu-python3.10-unit-test	`71.62% <98.17%> (+0.74%)`	⬆️
cpu-python3.12-unit-test	`71.62% <98.17%> (+0.74%)`	⬆️
cpu-python3.7-unit-test	`71.08% <98.20%> (+0.77%)`	⬆️
cuda-unit-test	`84.34% <98.17%> (+0.38%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Copilot

Pull request overview

Copilot reviewed 25 out of 29 changed files in this pull request and generated 5 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review. Take the survey.

Copilot

Pull request overview

Copilot reviewed 25 out of 29 changed files in this pull request and generated 4 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review. Take the survey.

Copilot

Pull request overview

Copilot reviewed 25 out of 29 changed files in this pull request and generated 7 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot

Pull request overview

Copilot reviewed 25 out of 29 changed files in this pull request and generated 2 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-04-22T19:59:10Z

+cmake_minimum_required(VERSION 3.18)
+project(nvbench_benchmarks LANGUAGES CUDA)
+
+# Check if we have a recent enough CMake for nvbench (which requires 3.30.4)
+if(CMAKE_VERSION VERSION_LESS "3.30.4")
+  message(STATUS "CMake version ${CMAKE_VERSION} is less than 3.30.4 (required by nvbench), skipping nvbench benchmarks")
+  return()
+endif()


This CMakeLists declares project(... LANGUAGES CUDA) before checking the CMake version / CUDA availability. If this directory is configured on a machine without a CUDA toolchain (or when CMake < 3.30.4), configuration can fail before reaching the intended “skip” logic. Consider moving the CMake version guard above project() and using project(... LANGUAGES CXX) + include(cuda_common.cmake)/enable_language(CUDA) only inside the CUDAToolkit_FOUND branch.

Copilot · 2026-04-22T19:59:11Z

 	cd ./nvbandwidth && git apply ../nvbandwidth.patch && cp ../nvbandwidth_testcases_patched.h ./testcases_patched.h && cmake . && make && cd ..
 	cp -v ./nvbandwidth/nvbandwidth $(SB_MICRO_PATH)/bin
+
+# Build nvbench


New cuda_nvbench target isn’t listed in the Makefile’s .PHONY targets. If a file/directory named cuda_nvbench exists, make cuda_nvbench may become a no-op. Add cuda_nvbench to the .PHONY list to ensure the recipe always runs.

Suggested change

# Build nvbench

# Build nvbench

.PHONY: cuda_nvbench

Copilot

Pull request overview

Copilot reviewed 25 out of 29 changed files in this pull request and generated 4 comments.

+def parse_time_to_us(raw: str) -> float:
+    """Parse a time string like '123.45 us' or '1.5 s' to float microseconds."""
+    raw = raw.strip()
+    m = re.match(r'^([\d.]+)\s*([mun]?s)?$', raw)
+    if not m:
+        raise ValueError(f'Invalid time string: {raw!r}')
+    val, unit = float(m.group(1)), (m.group(2) or 'us')
+    if unit == 's':
+        return val * 1e6


+    'BlasLtBaseBenchmark', 'ComputationCommunicationOverlap', 'CpuMemBwLatencyBenchmark', 'CpuHplBenchmark',
+    'CpuStreamBenchmark', 'CublasBenchmark', 'CublasLtBenchmark', 'CudaGemmFlopsBenchmark', 'CudaMemBwBenchmark',
+    'CudaNcclBwBenchmark', 'CudnnBenchmark', 'DiskBenchmark', 'DistInference', 'HipBlasLtBenchmark', 'GPCNetBenchmark',
+    'GemmFlopsBenchmark', 'GpuBurnBenchmark', 'GpuCopyBwBenchmark', 'GpuStreamBenchmark', 'IBBenchmark',
+    'IBLoopbackBenchmark', 'KernelLaunch', 'MemBwBenchmark', 'MicroBenchmark', 'MicroBenchmarkWithInvoke',
+    'ORTInferenceBenchmark', 'RocmGemmFlopsBenchmark', 'RocmMemBwBenchmark', 'ShardingMatmul',
+    'TCPConnectivityBenchmark', 'TensorRTInferenceBenchmark', 'DirectXGPUEncodingLatency', 'DirectXGPUCopyBw',
+    'DirectXGPUMemBw', 'DirectXGPUCoreFlops', 'NvBandwidthBenchmark', 'NvbenchKernelLaunch', 'NvbenchSleepKernel',
+    'NvbenchAutoThroughput'


+find_package(CUDAToolkit QUIET)
+if (CUDAToolkit_FOUND)
+  include(../cuda_common.cmake)
+
+  # Try to find nvbench, but don't require it
+  find_package(nvbench CONFIG QUIET)
+
+  if (nvbench_FOUND)
+    message(STATUS "Found nvbench, building nvbench benchmarks")


+        uses: lukka/get-cmake@latest
+        with:
+          cmakeVersion: '3.20.0'


polarG · 2026-07-22T04:40:50Z

+def parse_time_to_us(raw: str) -> float:
+    """Parse a time string like '123.45 us' or '1.5 s' to float microseconds."""
+    raw = raw.strip()
+    m = re.match(r'^([\d.]+)\s*([mun]?s)?$', raw)


[SHOULD-FIX] parse_time_to_us does not accept the μ (micro sign) unit that callers' regexes expect

m = re.match(r'^([\d.]+)\s*([mun]?s)?$', raw)

The row-matching regexes in nvbench_kernel_launch.py, nvbench_sleep_kernel.py, and nvbench_auto_throughput.py all explicitly anticipate a μ (U+03BC MICRO SIGN) unit prefix, e.g.:

r'([\d.]+\s*[μmun]?s)\s*\|\s*' # CPU Time (μs, ns, ms, us, s)

but this helper's unit character class ([mun]?s) does not include μ. If the row regex ever captures a value like "123.45 μs", passing it here raises ValueError('Invalid time string: ...'). That exception propagates out of the per-row parsing loop in each caller's _process_raw_result, is caught by the outer except BaseException, and triggers _handle_parsing_error() — marking the entire result as MICROBENCHMARK_RESULT_PARSING_FAILURE and discarding any rows already parsed in that call.

Recommendation: extend the regex to accept μ and normalize it before comparing, e.g.:

m = re.match(r'^([\d.]+)\s*([μmun]?s)?$', raw) ... unit = (m.group(2) or 'us').replace('μ', 'u')

polarG · 2026-07-22T04:40:50Z

-          DEBIAN_FRONTEND=noninteractive apt-get install -y ffmpeg libavcodec-dev libavformat-dev libavutil-dev libswresample-dev sudo
+          DEBIAN_FRONTEND=noninteractive apt-get install -y ffmpeg libavcodec-dev libavformat-dev libavutil-dev libswresample-dev sudo build-essential
+      - name: Setup CMake
+        uses: lukka/get-cmake@latest


[SHOULD-FIX] Third-party Action pinned to the floating @latest tag

uses: lukka/get-cmake@latest

@latest is a mutable ref — whoever controls lukka/get-cmake (or anyone who compromises that repo/release pipeline) can change what code runs in this CI job on every subsequent run, with no PR diff or review gate on this repo's side. This is a supply-chain compromise vector for the CodeQL analysis workflow, which typically runs with elevated permissions/secrets.

Recommendation: pin to an immutable commit SHA (preferred) or at minimum a specific release tag, e.g. lukka/get-cmake@<full-commit-sha> or lukka/get-cmake@v3.30.5.

polarG · 2026-07-22T04:40:50Z

+        *) CMAKE_ARCH="x86_64" ;; \
+    esac && \
+    echo "Detected architecture: ${ARCH}, using CMAKE_ARCH: ${CMAKE_ARCH}" && \
+    wget -q https://github.com/Kitware/CMake/releases/download/v3.30.4/cmake-3.30.4-linux-${CMAKE_ARCH}.tar.gz && \


[SHOULD-FIX] CMake release tarball downloaded and executed without integrity verification

wget -q https://github.com/Kitware/CMake/releases/download/v3.30.4/cmake-3.30.4-linux-${CMAKE_ARCH}.tar.gz && \ tar -xzf cmake-3.30.4-linux-${CMAKE_ARCH}.tar.gz && \ mv cmake-3.30.4-linux-${CMAKE_ARCH} /opt/cmake && \ ln -sf /opt/cmake/bin/* /usr/local/bin/ && \

The tarball is fetched over HTTPS but its checksum/signature is never verified before being extracted and symlinked onto PATH, where it's trusted for the rest of the image build (including compiling nvbench). A compromised release asset, mirror substitution, or interception on the download path would let arbitrary code execute during the image build with no detection (OWASP A08:2021 — Software and Data Integrity Failures).

Recommendation: verify a pinned SHA-256 checksum before extracting, e.g.:

echo "<expected-sha256> cmake-3.30.4-linux-${CMAKE_ARCH}.tar.gz" | sha256sum -c - && \ tar -xzf cmake-3.30.4-linux-${CMAKE_ARCH}.tar.gz && \

(Kitware publishes cmake-3.30.4-SHA-256.txt alongside each release.) This same block is duplicated in dockerfile/cuda12.9.dockerfile and needs the same fix there.

polarG · 2026-07-22T04:40:50Z

+        *) CMAKE_ARCH="x86_64" ;; \
+    esac && \
+    echo "Detected architecture: ${ARCH}, using CMAKE_ARCH: ${CMAKE_ARCH}" && \
+    wget -q https://github.com/Kitware/CMake/releases/download/v3.30.4/cmake-3.30.4-linux-${CMAKE_ARCH}.tar.gz && \


[SHOULD-FIX] CMake release tarball downloaded and executed without integrity verification

Same issue as dockerfile/cuda12.8.dockerfile: the CMake tarball is downloaded over HTTPS but never checksum-verified before being extracted and symlinked onto PATH. See the comment on cuda12.8.dockerfile for the recommended sha256sum -c fix.

[NON-BLOCKING] Also: this ~20-line CMake-install block is duplicated verbatim in both Dockerfiles. Any future version bump or fix (like the checksum verification above) has to be applied in two places and can silently drift. Consider factoring it into a shared script (e.g. dockerfile/scripts/install-cmake.sh) that both Dockerfiles invoke.

Copilot

Pull request overview

Copilot reviewed 25 out of 29 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (2)

superbench/benchmarks/micro_benchmarks/nvbench_base.py:19

parse_time_to_us() doesn't handle the microsecond symbols "μs" / "µs" even though the NVBench parsers' regexes accept them. If NVBench emits "μs" in table output, parsing will fail with ValueError and the benchmark will report MICROBENCHMARK_RESULT_PARSING_FAILURE. Consider normalizing "μ"/"µ" to "u" before unit handling.

    raw = raw.strip()
    m = re.match(r'^([\d.]+)\s*([mun]?s)?$', raw)
    if not m:
        raise ValueError(f'Invalid time string: {raw!r}')
    val, unit = float(m.group(1)), (m.group(2) or 'us')

.github/workflows/codeql-analysis.yml:59

The workflow uses lukka/get-cmake@latest, which is not version-pinned and can change behavior over time (supply-chain/reproducibility risk). Pin to a specific tag or commit SHA so CodeQL runs remain deterministic.

      - name: Setup CMake
        uses: lukka/get-cmake@latest
        with:
          cmakeVersion: '3.20.0'

WenqingLan1 and others added 15 commits July 22, 2025 16:03

add nvbench kernel launch

741ee98

submodule update

0ae7864

init sleep kernel

35bfb61

Merge branch 'microsoft:main' into feat/third_party/nvbench

66b4786

Merge branch 'microsoft:main' into feat/third_party/nvbench

82aed0c

Merge branch 'microsoft:main' into feat/third_party/nvbench

24ee0a5

test sleep kernel

bd87f50

add sm 103

a663db6

add arg parsing logic

32fe197

Merge branch 'microsoft:main' into feat/third_party/nvbench

76562dc

add arg parsing tests

3eb5525

refactor

4785fe6

refine logic - remove gpu_id

1fb7c05

add doc

83c442c

refine regex & update nvbench submodule

4b274c4

WenqingLan1 requested a review from a team as a code owner October 9, 2025 23:12

WenqingLan1 added benchmarks SuperBench Benchmarks micro-benchmarks Micro Benchmark Test for SuperBench Benchmarks labels Oct 9, 2025

WenqingLan1 added 8 commits October 10, 2025 16:48

update cmake

0cf48bb

fix lint

5905647

fix lint

baa57c9

fix import

ecce2d9

fix

3a58ead

fix

d0d8773

fix

fbb5969

fix

f007745

WenqingLan1 added 3 commits October 10, 2025 21:23

fix

b6b6082

fix

0f2c838

fix

5bd20f6

Copilot AI review requested due to automatic review settings March 10, 2026 20:55

Copilot AI reviewed Mar 10, 2026

View reviewed changes

WenqingLan1 added 2 commits March 10, 2026 14:18

fix pipeline & resolve comments

9643150

fix lint

f1a3b6d

Copilot AI review requested due to automatic review settings March 10, 2026 21:31

Copilot AI reviewed Mar 10, 2026

View reviewed changes

Comment thread tests/benchmarks/micro_benchmarks/test_nvbench_sleep_kernel.py Outdated

microsoft deleted a comment from Copilot AI Mar 10, 2026

fix test

fe48e35

WenqingLan1 changed the title ~~Benchmarks: Micro benchmark - add nvbench based kernel-launch & sleep-kernel~~ Benchmarks: Micro benchmark - add nvbench based kernel-launch, sleep-kernel & auto-throughput Mar 25, 2026

Merge branch 'microsoft:main' into feat/third_party/nvbench

e1e12d2

Copilot AI review requested due to automatic review settings April 8, 2026 20:27

Copilot started reviewing on behalf of WenqingLan1 April 8, 2026 20:29 View session

Copilot AI reviewed Apr 8, 2026

View reviewed changes

WenqingLan1 and others added 2 commits April 22, 2026 11:04

Merge branch 'microsoft:main' into feat/third_party/nvbench

6fc5afb

resolve comments

e253b85

Copilot AI review requested due to automatic review settings April 22, 2026 19:50

Copilot started reviewing on behalf of WenqingLan1 April 22, 2026 19:51 View session

Copilot AI reviewed Apr 22, 2026

View reviewed changes

WenqingLan1 added 2 commits May 13, 2026 11:44

Merge branch 'microsoft:main' into feat/third_party/nvbench

7dd26d3

Merge branch 'microsoft:main' into feat/third_party/nvbench

4c6113d

Copilot AI review requested due to automatic review settings May 20, 2026 18:48

Copilot started reviewing on behalf of WenqingLan1 May 20, 2026 18:48 View session

Copilot AI reviewed May 20, 2026

View reviewed changes

Merge branch 'main' into feat/third_party/nvbench

63e7ed2

polarG requested changes Jul 22, 2026

View reviewed changes

polarG requested a review from Copilot July 22, 2026 05:17

Copilot started reviewing on behalf of polarG July 22, 2026 05:18 View session

Copilot AI reviewed Jul 22, 2026

View reviewed changes

Uh oh!

Conversation

WenqingLan1 commented Oct 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov Bot commented Oct 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Apr 22, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 22, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

polarG Jul 22, 2026

Choose a reason for hiding this comment

Uh oh!

polarG Jul 22, 2026

Choose a reason for hiding this comment

Uh oh!

polarG Jul 22, 2026

Choose a reason for hiding this comment

Uh oh!

polarG Jul 22, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

WenqingLan1 commented Oct 9, 2025 •

edited

Loading

codecov Bot commented Oct 10, 2025 •

edited

Loading