Quarantine flaky pipeline tests; add triage prompt

[paulmedynski]

Description

Quarantines intermittently-failing and hang-prone tests observed in the public CI pipelines, and adds a reusable prompt for triaging pipeline test failures. Each quarantined test is marked [Trait("Category", "flaky")] (excluded by the category!=flaky CI filter) with an accurate root-cause comment; where useful, the captured xUnit output/stack trace is included.

This branch has been merged up to main so it includes the background pool warmup/replenishment work from #4452; the newly-flaky tests from that feature are quarantined here as well (see below).

No product code changes; no public API changes. All quarantined tests are timing/harness-sensitive under CI load — they pass locally and on main's own CI.

Pre-existing flaky/hang quarantines (retry/timing family)

• UnitTests.SimulatedServerTests.ConnectionTests.TransientFault_RetryEnabled_ShouldSucceed_Async — the transient error occasionally resurfaces on the retry login under CI load, so the async open propagates a SqlException instead of succeeding.
• UnitTests.SimulatedServerTests.ConnectionTests.NetworkError_RetryEnabled_ShouldSucceed_Async — post-login Connection Timeout on a slow agent (pre-login handshake ~4.4s exhausts the connect-timeout budget); same family as the above.
• UnitTests.SimulatedServerTests.ConnectionFailoverTests.NetworkError_WithUserProvidedPartner_RetryDisabled_ShouldConnectToFailoverPartner — the tight 5s ConnectTimeout can be exhausted by the failover connection itself on a slow agent.
• UnitTests.ConnectionPool.WaitHandleDbConnectionPoolTransactionTest.TwoThreads_SharedTransaction_AccessSameTransactedEntry — Task.Run thread-pool starvation on a loaded x86 agent; task1 starts late and misses task2's 10s wait.
• UnitTests.ConnectionPool.ChannelDbConnectionPoolTest.ConcurrentCallers_ShouldTimeoutIndependently — hang (aborted by the 10-minute --blame-hang timeout); suspected virtual-time race, root cause unconfirmed pending hang-dump analysis (requires WinDbg+SOS on Windows), so quarantined rather than "fixed" to avoid masking a possible product deadlock.
• ManualTests.SQL.ConnectionPoolTest.TransactionPoolTest.TransactionCleanupTest — process-global pool count race under TransactionScope (Expected 2, Actual 3); test-isolation issue.

#4452 background-warmup pool tests (flaky under CI load)

These 12 tests come from #4452 (background warmup/replenishment for ChannelDbConnectionPool). They pass locally (3/3) and on main's own CI, but fail on PR merge builds across multiple jobs with instant Assert Expected:True/Actual:False — a background-thread timing race under CI load, not a deterministic defect. Origin is #4452, not this PR.

• UnitTests.ConnectionPool.ChannelDbConnectionPoolTest: ErrorOccurred_OnFailure_FollowsBlockingPeriod, ErrorOccurred_BlockingEnabled_SubsequentRequestFastFails, Clear_InErrorState_ResetsErrorOccurred, Failure_ThenBlockingPeriodExpiry_AllowsSuccessfulCreate
• UnitTests.ConnectionPool.ChannelDbConnectionPoolWarmupTest: Warmup_AllCreationsFail_AbsorbedAndEntersErrorState, Warmup_Fails_UserRequestFastFailsDuringBlockingPeriod, Warmup_RespectsErrorState_StandsDownWhileBlocking
• UnitTests.ConnectionPool.WaitHandleDbConnectionPoolBlockingPeriodTest: TryGetConnection_WhenFactoryThrows_EntersBlockingPeriod, TryGetConnection_WhileBlocked_FastFailsWithCachedExceptionWithoutInvokingFactory, TryGetConnection_AfterBlockingPeriodExpires_RetriesFactoryAndRecovers, TryGetConnection_SuccessfulCreate_ResetsBackoffToInitialWait, TryGetConnection_FailingAgainAfterExitTimer_StillDoublesBackoff

Note for reviewers: the #4452 tests are quarantined here to unblock CI, but the underlying flakiness belongs to that feature. The #4452 author should confirm/fix at the source and lift the flaky trait when addressed.

Tooling

• Adds .github/prompts/triage-pipeline-failures.prompt.md — an access-agnostic (ADO MCP / az CLI / REST) workflow to find and classify failing tests in the SqlClient pipelines at/after a given commit, capture full xUnit output/stack traces, distinguish regression vs pre-existing, and present findings for approval before fixing or quarantining.

[Copilot]

Pull request overview

Quarantines two intermittently failing tests in the SqlClient CI by marking them with the existing Category=flaky trait, and adds a reusable Copilot prompt to triage pipeline test failures at/after a specific commit SHA.

Changes:

• Mark TransientFault_RetryEnabled_ShouldSucceed_Async (simulated server unit test) as quarantined via [Trait("Category", "flaky")] with an embedded failure signature comment.
• Mark TransactionCleanupTest (manual pooling test) as quarantined via [Trait("Category", "flaky")] with an embedded failure signature comment.
• Add .github/prompts/triage-pipeline-failures.prompt.md to guide ADO/GitHub pipeline failure investigation and classification workflows.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

File	Description
src/Microsoft.Data.SqlClient/tests/UnitTests/SimulatedServerTests/ConnectionTests.cs	Quarantines an intermittently failing transient-retry simulated-server test.
src/Microsoft.Data.SqlClient/tests/ManualTests/SQL/ConnectionPoolTest/TransactionPoolTest.cs	Quarantines an intermittently failing transaction-pool cleanup test.
.github/prompts/triage-pipeline-failures.prompt.md	Adds a triage workflow prompt for classifying pipeline test failures and deciding fix vs quarantine.

Comments suppressed due to low confidence (1)

src/Microsoft.Data.SqlClient/tests/ManualTests/SQL/ConnectionPoolTest/TransactionPoolTest.cs:93

• Quarantining this test as [Trait("Category", "flaky")] reduces regular CI coverage, but the described failure mode (pool ConnectionCount off by 1) can be made deterministic by isolating the pool key per invocation. Since pools are process-global and keyed by connection string, appending a unique ApplicationName (via SqlConnectionStringBuilder) prevents cross-test contamination without quarantining the test.

        [Trait("Category", "flaky")]
        [ConditionalTheory(typeof(DataTestUtility), nameof(DataTestUtility.AreConnStringsSetup), nameof(DataTestUtility.IsNotAzureSynapse))]
        [ClassData(typeof(ConnectionPoolConnectionStringProvider))]
        public static void TransactionCleanupTest(string connectionString)
        {

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 62.61%. Comparing base (a684c99) to head (e073c35).
⚠️ Report is 4 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4476      +/-   ##
==========================================
- Coverage   64.64%   62.61%   -2.04%     
==========================================
  Files         288      283       -5     
  Lines       44009    66930   +22921     
==========================================
+ Hits        28451    41909   +13458     
- Misses      15558    25021    +9463     

Flag	Coverage Δ
CI-SqlClient	?
PR-SqlClient-Project	62.61% <ø> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 3 comments.

[Copilot]

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 4 comments.

Comments suppressed due to low confidence (9)

src/Microsoft.Data.SqlClient/tests/UnitTests/ConnectionPool/WaitHandleDbConnectionPoolBlockingPeriodTest.cs:138

• The // Flaky ... block sits between the /// XML docs and the method attributes, which causes CS1587 (XML comment not placed on a valid language element). With TreatWarningsAsErrors enabled, this will break the build. Fold this text into an XML doc <remarks> block instead.

    // Flaky under CI load only (passes locally 3/3 and on main CI; failed on a PR merge
    // build): the WaitHandle pool's background create/prune thread races the assertion under
    // load, and this file shares the DbConnectionInternal/SqlConnectionFactory changes from
    // #4452. Not a defect in this PR.
    [Trait("Category", "flaky")]

src/Microsoft.Data.SqlClient/tests/UnitTests/ConnectionPool/WaitHandleDbConnectionPoolBlockingPeriodTest.cs:224

• This quarantine comment is placed between the /// XML doc comment and the method declaration, which triggers CS1587 and will fail the build (warnings-as-errors). Move it into an XML doc <remarks> block so the doc comment remains contiguous with the member.

    // Flaky under CI load only (passes locally 3/3 and on main CI; failed on a PR merge
    // build): the WaitHandle pool's background create/prune thread races the ErrorOccurred/timer
    // assertions under load, and this file shares the DbConnectionInternal/SqlConnectionFactory
    // changes from #4452. Not a defect in this PR.
    [Trait("Category", "flaky")]

src/Microsoft.Data.SqlClient/tests/UnitTests/ConnectionPool/WaitHandleDbConnectionPoolBlockingPeriodTest.cs:270

• The non-XML // Flaky ... comment breaks up the /// XML documentation block from the attribute/method it documents, producing CS1587 (warnings-as-errors). Please convert this to XML doc <remarks> lines.

    // Flaky under CI load only (passes locally 3/3 and on main CI; failed on a PR merge
    // build): the WaitHandle pool's background create/prune thread races the ErrorOccurred/timer
    // assertions under load, and this file shares the DbConnectionInternal/SqlConnectionFactory
    // changes from #4452. Not a defect in this PR.
    [Trait("Category", "flaky")]

src/Microsoft.Data.SqlClient/tests/UnitTests/ConnectionPool/WaitHandleDbConnectionPoolBlockingPeriodTest.cs:327

• This // Flaky ... block is between XML docs and the method attributes, which causes CS1587 and breaks the build under TreatWarningsAsErrors. Convert it into XML doc <remarks> (or move it above the /// block).

    // Flaky under CI load only (passes locally 3/3 and on main CI; failed on a PR merge
    // build): the WaitHandle pool's background create/prune thread races the ErrorOccurred/timer
    // assertions under load, and this file shares the DbConnectionInternal/SqlConnectionFactory
    // changes from #4452. Not a defect in this PR.
    [Trait("Category", "flaky")]

src/Microsoft.Data.SqlClient/tests/UnitTests/ConnectionPool/ChannelDbConnectionPoolWarmupTest.cs:377

• The // Flaky ... block breaks up the /// XML documentation and the method declaration, producing CS1587 (and build failure due to TreatWarningsAsErrors). Convert this quarantine note into XML doc <remarks> lines.

        // Flaky under CI load only (passes locally 3/3 and on main CI; fails on PR merge
        // builds): the background warmup thread added in #4452 races the assertion so the
        // pool's error-state transition is observed late (Assert Expected:True/Actual:False).
        // Not a defect in this PR.
        [Trait("Category", "flaky")]

src/Microsoft.Data.SqlClient/tests/UnitTests/ConnectionPool/ChannelDbConnectionPoolWarmupTest.cs:407

• Non-XML // comments inserted between XML docs and the attributes will raise CS1587 and break the build (warnings-as-errors). Please move this quarantine text into an XML doc <remarks> block so the /// docs remain contiguous with the member.

        // Flaky under CI load only (passes locally 3/3 and on main CI; fails on PR merge
        // builds): the background warmup thread added in #4452 races the assertion so the
        // pool's error-state transition is observed late (Assert Expected:True/Actual:False).
        // Not a defect in this PR.
        [Trait("Category", "flaky")]

src/Microsoft.Data.SqlClient/tests/UnitTests/ConnectionPool/ChannelDbConnectionPoolTest.cs:1665

• The // Flaky ... block splits the /// XML doc comment from the attributes/method, which will raise CS1587 and fail the build (warnings-as-errors). Please move this quarantine rationale into an XML doc <remarks> block instead.

        // Flaky under CI load only (passes locally 3/3 and on main CI; fails on PR merge
        // builds): races the background warmup/replenishment work added in #4452 before the
        // pool's error-state transition is observable. Not a defect in this PR.
        [Trait("Category", "flaky")]
        [Fact]

src/Microsoft.Data.SqlClient/tests/UnitTests/ConnectionPool/ChannelDbConnectionPoolTest.cs:1709

• This quarantine comment is placed between the /// XML doc block and the method attributes, which triggers CS1587 and breaks builds with TreatWarningsAsErrors. Convert it into an XML doc <remarks> block so the documentation remains attached to the test.

        // Flaky under CI load only (passes locally 3/3 and on main CI; fails on PR merge
        // builds): races the background warmup/replenishment work added in #4452 before the
        // pool's error-state transition is observable. Not a defect in this PR.
        [Trait("Category", "flaky")]
        [Fact]

src/Microsoft.Data.SqlClient/tests/UnitTests/ConnectionPool/ChannelDbConnectionPoolTest.cs:1749

• The non-XML // Flaky ... block breaks the XML doc comment’s adjacency to the method/attributes, producing CS1587 (warnings-as-errors). Please move this quarantine note into an XML doc <remarks> block.

        // Flaky under CI load only (passes locally 3/3 and on main CI; fails on PR merge
        // builds): races the background warmup/replenishment work added in #4452 before the
        // pool's error-state transition is observable. Not a defect in this PR.
        [Trait("Category", "flaky")]
        [Fact]

[paulmedynski]

I've accumulated enough new quarantines to warrant getting this reviewed and merged. New flaky tests will be in a subsequent PR.

@mdaigle - Assigned you to review since I had to quarantine a bunch of new pool tests. See the description.

[mdaigle]

If I open a PR that breaks a test, that PR's pipeline failures will surface in this list and the test could get marked as flaky even though it's only breaking due to my change. Are there really tests that are only flaky at PR time? Can we limit to only CI runs as the input?

[paulmedynski]

Updated to exclude pipeline runs for PR commits.

[Copilot]

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (2)

.github/prompts/triage-pipeline-failures.prompt.md:26

• This prompt states it will prefer an Azure DevOps MCP server when connected, but the tools: frontmatter doesn't include any MCP/extension tool identifiers (per generate-prompt.prompt.md, extension tools must be explicitly listed, e.g. <server-name>/*). To avoid misleading readers in environments where tool scoping is enforced, qualify the MCP option as "if MCP tools are available/in-scope" and otherwise fall back to az/REST.

1. An **Azure DevOps MCP server**, if one is connected (preferred — no shell needed).
2. The **`az` CLI** (`az rest --resource 499b84ac-1321-427f-aa17-267ca6975798 ...`,
   `az pipelines ...`, `az boards ...`).
3. **Direct ADO REST** calls over HTTPS with a bearer token.

Do not assume a specific mechanism. If the first choice is unavailable or errors,
fall back to the next. Keep read operations read-only; only edits to test source
files (quarantine/fix) modify state, and those happen in the repo, not in ADO.

.github/prompts/triage-pipeline-failures.prompt.md:11

• The prompt encourages users to pass extra scoping text inside ${input:commit}, but later steps (e.g., git merge-base --is-ancestor <target> ...) require ${input:commit} to be a valid SHA. As written, a value like " main" would break the git ancestry checks. Clarify that only the first whitespace-delimited token is the commit SHA and any remaining text is optional scope (or use a separate input variable for scope).

Triage failing tests in the Microsoft.Data.SqlClient pipelines for commit
`${input:commit}` and later. If the user supplied extra scope in `${input:commit}`
(e.g. a pipeline name or branch), honor it; otherwise use the defaults below.

[Copilot]

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (2)

src/Microsoft.Data.SqlClient/tests/UnitTests/SimulatedServerTests/ConnectionTests.cs:99

• TransientFault_RetryEnabled_ShouldSucceed_Async uses an ephemeral localhost port but leaves pooling enabled. Most other simulated-server tests in this file set Pooling = false to avoid leaking pooled connections bound to short-lived ports; leaving pooling on here can retain broken connections and create cross-test interference (which may contribute to the CI-only flakiness you’re quarantining). Consider disabling pooling for this test as well.

        [Trait("Category", "flaky")]
        [Theory]
        [InlineData(40613)]
        [InlineData(42108)]
        [InlineData(42109)]
        public async Task TransientFault_RetryEnabled_ShouldSucceed_Async(uint errorCode)

src/Microsoft.Data.SqlClient/tests/ManualTests/SQL/ConnectionPoolTest/TransactionPoolTest.cs:92

• TransactionCleanupTest is quarantined due to a process-global pool sharing race, but this can likely be made deterministic by isolating the pool via a unique connection string key (e.g., set a unique Application Name for each invocation), similar to how this file already appends ;App=... in other places. If the pool is isolated per invocation, the test should no longer need the [Trait("Category", "flaky")] quarantine.

        //     at Microsoft.Data.SqlClient.ManualTesting.Tests.TransactionPoolTest.TransactionCleanupTest(String connectionString) in TransactionPoolTest.cs:line 101
        [Trait("Category", "flaky")]
        [ConditionalTheory(typeof(DataTestUtility), nameof(DataTestUtility.AreConnStringsSetup), nameof(DataTestUtility.IsNotAzureSynapse))]
        [ClassData(typeof(ConnectionPoolConnectionStringProvider))]
        public static void TransactionCleanupTest(string connectionString)

[apoorvdeshmukh]

Is this resource identifier required to be mentioned here?

[paulmedynski]

Removed all specific identifiers.

[Copilot]

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

.github/prompts/triage-pipeline-failures.prompt.md:48

• The prompt defines the projects as <CI project> and <mirror project> in the Environment section, but Step 1 refers to public and ADO.Net. This inconsistency makes the workflow ambiguous for users copying the prompt. Align Step 1 with the same project placeholders (or the same concrete names) used above.

**Operation:** list build definitions in both `public` and `ADO.Net`, with each
definition's folder path, `queueStatus`, and `repository.type` / `repository.name`.