Add SqlClient performance test pipeline with baseline comparison, noise reduction, and ADX ingestion

[cheenamalhotra]

Summary

Adds a performance-test pipeline for Microsoft.Data.SqlClient under eng/pipelines/perf/. It runs the BenchmarkDotNet perf suite on a dedicated performance test lab (Azure Dedicated Hosts), compares the branch under test against a released NuGet baseline, applies noise-reduction controls, and optionally ingests results into an Azure Data Explorer (Kusto) database.

What's included

• Pipeline (sqlclient-perf-pipeline.yml) — extends the reusable Perf.Test.Job template; provisions a VM, runs benchmarks over SSH, collects results, and runs post-test translation/ingestion on the agent.
• On-VM harness (scripts/run-perf-tests.sh / .ps1) — installs the SDK, creates the perf DB, injects the VM connection string, pins the client to a reserved CPU set, and runs the benchmarks.
• Two-pass build model — current build via ProjectReference (branch under test) vs. baseline via PackageReference (released MDS from NuGet.org), pinned with VersionOverride under Central Package Management. Also supports building against the 7.1.0-preview1 MDS package, pinning Microsoft.Data.SqlClient.Extensions.Abstractions/Microsoft.Data.SqlClient.Internal.Logging to 1.0.0 only for that preview baseline.
• Noise reduction — interleaved per-unit baseline↔candidate runs + best-of-N confirmation (interleave_perf.py), client CPU pinning (with a safe fallback for CPU indices ≥ 64), warm-up, allocator/network tuning on Linux, fail-loud guards, and diagnostics capture.
• Comparison (compare_perf.py) — matches benchmarks by (Type, Method, Parameters), emits comparison.md/comparison.json, and can gate the run on confirmed regressions.
• Kusto ingestion (perf_to_kusto.py, ingest_kusto.py) — translates BenchmarkDotNet JSON into PerfRun + PerfBenchmarkResult NDJSON and performs a queued ingestion, with post-ingestion row-count verification that tolerates slow async flush without failing the step when the data has actually landed. ADX cluster/database/service-connection are sourced from a pipeline library variable group (ADX Cluster Variables), not hard-coded.
• Runner config capture — PerfRun.Config now records a JSON snapshot of the runner's boolean AppContext switches at the time of the run (UseManagedSniOnWindows, UseOptimizedAsyncBehaviour, UseConnectionPoolV2, WaitForProfiler/UseNativeMemoryAndETWProfiler, excluding the connection string and benchmark selection). UseManagedSniOnWindows, UseOptimizedAsyncBehaviour, and UseConnectionPoolV2 are also exposed as pipeline parameters so a run can override them and have the override reflected in Config.
• Runner harness changes — PerformanceTests/Program.cs gains PERF_LIST_BENCHMARKS / PERF_BENCHMARK modes to support single-unit interleaving; BenchmarkConfig.cs enables the JSON "full" exporter.

Notes

• The pipeline runs on internal perf-lab infrastructure and consumes an internal extends template; it is manual/scheduled only (pr: none, trigger: none) and does not run on public PRs or commits.
• Ingestion is optional and gated at runtime — the pipeline still runs and compares when the ADX variables are empty, publishing the translated NDJSON as an artifact for manual/backfill ingestion.

TODOs for later:

• Add support for running benchmarks against .NET Franework 4.8 on Windows.

Checklist

• Tests added or updated (perf harness/runner)
• Public API changes documented (none — no public API changes)
• Ensure no breaking changes introduced
• Perf Pipeline setup and verified.

[Copilot]

Pull request overview

Adds an internal Azure DevOps performance-testing pipeline for Microsoft.Data.SqlClient, including on-VM harness scripts to run BenchmarkDotNet in a noise-reduced baseline-vs-candidate model and (optionally) translate/ingest results into ADX (Kusto). This fits the repo’s existing eng/pipelines/ infrastructure and extends the existing perf test project to support single-unit execution needed for interleaving.

Changes:

• Adds a new perf pipeline (eng/pipelines/perf/sqlclient-perf-pipeline.yml) that provisions a perf-lab VM, runs benchmarks, publishes results, and optionally ingests translated NDJSON into Kusto.
• Adds Linux/Windows VM harness scripts plus Python utilities for interleaving (best-of-N confirmation), baseline comparison, and Kusto translation/ingestion.
• Updates the perf test runner to support enumerating/running individual benchmark “units”, and ensures BenchmarkDotNet “full” JSON reports are emitted.

Reviewed changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
src/Microsoft.Data.SqlClient/tests/PerformanceTests/Program.cs	Adds PERF_LIST_BENCHMARKS / PERF_BENCHMARK modes to enable interleaved per-unit runs.
src/Microsoft.Data.SqlClient/tests/PerformanceTests/Microsoft.Data.SqlClient.PerformanceTests.csproj	Enables baseline pinning in Package mode via MdsPackageVersion + VersionOverride under CPM.
src/Microsoft.Data.SqlClient/tests/PerformanceTests/Config/BenchmarkConfig.cs	Enables BenchmarkDotNet JSON “full” exporter needed for comparison/translation.
eng/pipelines/perf/sqlclient-perf-pipeline.yml	New ADO perf pipeline extending internal perf-lab template; publishes artifacts and optionally ingests to Kusto.
eng/pipelines/perf/scripts/run-perf-tests.sh	Linux on-VM harness: install SDK, create DB, run interleaved/sequential passes, compare results.
eng/pipelines/perf/scripts/run-perf-tests.ps1	Windows on-VM harness counterpart (affinity pinning via ProcessorAffinity).
eng/pipelines/perf/scripts/interleave_perf.py	Interleaved + best-of-N orchestrator invoking single-unit runner mode and producing comparison outputs.
eng/pipelines/perf/scripts/compare_perf.py	Baseline vs current comparison over *-report-full.json with markdown/JSON outputs.
eng/pipelines/perf/scripts/perf_to_kusto.py	Translates BenchmarkDotNet “full” JSON into Kusto PerfRun/PerfBenchmarkResult NDJSON.
eng/pipelines/perf/scripts/ingest_kusto.py	Queued ingestion into ADX using az-cli auth + post-ingest verification polling.
eng/pipelines/perf/README.md	Documents pipeline architecture, parameters, noise-reduction controls, and ingestion setup.

[Copilot]

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (2)

eng/pipelines/perf/scripts/run-perf-tests.sh:167

• The “reuse pre-installed SDK” check doesn’t actually verify the pinned SDK version from global.json. It only checks for any 10.0.* SDK, which could pick an incompatible feature band/patch and cause builds/restores to fail (global.json pins 10.0.300). Also, DOTNET_ROOT is derived from the dotnet shim path; if dotnet is a symlink in /usr/bin this can point to the wrong root.

# Reuse a pre-installed SDK only if it already satisfies global.json; otherwise install locally.
if command -v dotnet >/dev/null 2>&1 && dotnet --list-sdks 2>/dev/null | grep -q '10\.0\.'; then
    echo "Using pre-installed dotnet: $(command -v dotnet)"
    export DOTNET_ROOT="$(dirname -- "$(command -v dotnet)")"
else

eng/pipelines/perf/scripts/run-perf-tests.ps1:127

• The pre-installed dotnet detection checks only for any 10.0.* SDK, but global.json pins a specific SDK version (10.0.300). If the VM image has a different 10.0 SDK, builds can fail despite this check passing. Prefer verifying the exact pinned version (or at least the pinned feature band) before skipping Install-DotNet.

$hasNet10Sdk = $false
if (Get-Command dotnet -ErrorAction SilentlyContinue) {
    if ((dotnet --list-sdks) -match '^10\.0\.') { $hasNet10Sdk = $true }
}
if ($hasNet10Sdk) {
    Write-Host "Using pre-installed dotnet: $((Get-Command dotnet).Source)"
} else {
    Install-DotNet
}

[Copilot]

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (2)

eng/pipelines/perf/scripts/run-perf-tests.sh:167

• The preinstalled-SDK check is too permissive and can skip installing the SDK required by global.json (e.g., a host with 10.0.1xx will satisfy grep '10\.0\.' but dotnet build will later fail because global.json requires 10.0.3xx). Also, setting DOTNET_ROOT to the directory containing the dotnet shim (often /usr/bin) is incorrect and can break host resolution.

Consider checking for a compatible feature band derived from global.json and avoiding DOTNET_ROOT overrides when using the system installation.

# Reuse a pre-installed SDK only if it already satisfies global.json; otherwise install locally.
if command -v dotnet >/dev/null 2>&1 && dotnet --list-sdks 2>/dev/null | grep -q '10\.0\.'; then
    echo "Using pre-installed dotnet: $(command -v dotnet)"
    export DOTNET_ROOT="$(dirname -- "$(command -v dotnet)")"
else

eng/pipelines/perf/scripts/run-perf-tests.ps1:127

• The script decides whether to install the SDK based on a generic ^10\.0\. check, which can accept a preinstalled SDK that doesn't satisfy global.json (e.g., 10.0.1xx vs required 10.0.3xx). This can cause the subsequent build to fail.

Consider deriving the required SDK feature band from global.json (rollForward=patch ⇒ same feature band) and checking dotnet --list-sdks against that instead.

$hasNet10Sdk = $false
if (Get-Command dotnet -ErrorAction SilentlyContinue) {
    if ((dotnet --list-sdks) -match '^10\.0\.') { $hasNet10Sdk = $true }
}
if ($hasNet10Sdk) {
    Write-Host "Using pre-installed dotnet: $((Get-Command dotnet).Source)"
} else {
    Install-DotNet
}

[Copilot]

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

eng/pipelines/perf/scripts/run-perf-tests.sh:123

• SQL_SERVER is defaulted to localhost when unset, but the script does not export it after assignment. Later, the Python runner-config rewrite reads os.environ["SQL_SERVER"], so if SQL_SERVER was not injected by the template, the script will still fail with a KeyError instead of using the intended default.

: "${SQL_SERVER:=localhost}"
if [[ -z "${SQL_PASSWORD:-}" ]]; then
    echo "ERROR: SQL_PASSWORD environment variable is not set (expected from the perf template)." >&2
    exit 1
fi

[Copilot]

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated 1 comment.