AE tests: create ConversionTests keys once per class to slow error 3807 identifier exhaustion

[paulmedynski]

Description

Fixes the recurring Create failed because all available identifiers have been exhausted. (SQL Server error 3807) failures in the Always Encrypted (AE) manual tests, seen on the sqlclient-ci-kerberos pipeline.

Root cause

SQL Server assigns each Column Master Key (CMK) and Column Encryption Key (CEK) an identifier from a monotonic, per-database counter that DROP never reclaims. Diagnosis on the shared Kerberos server confirmed this: with 0 live keys, the CMK counter was already at 143,789, and CREATE COLUMN ENCRYPTION KEY failed with 3807 — while the same statement in a fresh database succeeded with id=1. So the exhausted resource is the counter, not accumulated/leaked keys, and no amount of key cleanup can fix it.

The dominant consumer of that counter is ConversionTests: it created a fresh CMK+CEK in its constructor, and xUnit constructs a test-class instance per test case. With ~79 data rows across 3 theories, each run issued dozens of CREATE COLUMN MASTER/ENCRYPTION KEY statements from this one class, steadily advancing the counter until it hit the ceiling on the long-lived server. (Every other AE class already creates its keys once, via an IClassFixture.)

Change

• Add ConversionTestFixture (an IClassFixture) that creates the CMK+CEK once per test class and drops them on dispose.
• Refactor ConversionTests to consume that fixture instead of creating keys in its per-test constructor. Per-test tables are still created/dropped per case (tables don't consume the constrained identifier).

This reduces this class's key-creation churn by roughly two orders of magnitude, dramatically slowing counter growth on any persistent AE database (both the Kerberos Northwind server and the enclave CI servers, which are on the same trajectory).

Notes / scope

• This slows growth; it does not reset an already-exhausted counter. The only reset is recreating the database, which was done manually on sqldrv-sql22 to unblock the pipeline. AE databases require external environment setup that can't currently be provisioned per-run, so a fresh-DB-per-run approach isn't viable.
• Earlier commits in this branch explored an orphan-sweep / leak-cleanup approach (AlwaysEncryptedCleanup, idempotent drops, resilient teardown, and a dropOrphanedAlwaysEncryptedKeys.sql script). Those targeted key leakage, which turned out not to be the cause of 3807, so they have been reverted. The net change of this PR is just the two files above.

Issues

Diagnosed from Kerberos pipeline runs 20260724.1 and 20260727.3.

Testing

• sqlclient-ci-kerberos: 20260727.4

[Copilot]

Pull request overview

Adds a standalone T-SQL maintenance script to clean up orphaned Always Encrypted test artifacts (AE-prefixed CMKs/CEKs and dependent tables) in long-lived shared CI databases, mitigating “identifier exhaustion” failures during AE test setup.

Changes:

• Introduces dropOrphanedAlwaysEncryptedKeys.sql with a default “WhatIf” preview mode that prints DROP statements.
• Implements dependency-ordered cleanup (tables → CEKs → CMKs) with per-object TRY/CATCH to continue on failures.
• Provides a summary and reports any remaining AE-prefixed keys after execution.

[Copilot]

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 2 comments.

Comments suppressed due to low confidence (1)

src/Microsoft.Data.SqlClient/tests/ManualTests/AlwaysEncrypted/TestFixtures/AlwaysEncryptedCleanup.cs:97

• In the orphan-sweep T-SQL, the cursor for tables selects a schema-qualified, QUOTENAME'd identifier (e.g. [schema].[table]). Storing that in @name sysname (nvarchar(128)) can truncate longer schema-qualified names, leading to invalid DROP statements and leaving orphans behind. Use a wider NVARCHAR for @name (or separate @tableName/@keyName variables).

SET NOCOUNT ON;
DECLARE @name sysname, @sql nvarchar(max);

[Copilot]

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 3 comments.

Comments suppressed due to low confidence (2)

tools/testsql/dropOrphanedAlwaysEncryptedKeys.sql:55

• @label is declared as nvarchar(400) but is assigned values from @dropTables.tableName (nvarchar(517)). If schema/table names are long, this can truncate the identifier and generate invalid DROP statements. Use nvarchar(517) for @label to match the max length of schema-qualified QUOTENAME output.

DECLARE @stmt   nvarchar(max);
DECLARE @label  nvarchar(400);
DECLARE @errMsg nvarchar(4000);

src/Microsoft.Data.SqlClient/tests/ManualTests/AlwaysEncrypted/TestFixtures/AlwaysEncryptedCleanup.cs:128

• The T-SQL sweep uses DECLARE @name sysname, but the table cursor stores a schema-qualified, QUOTENAME'd name which can be up to 517 characters. Using sysname (128) risks truncation and incomplete cleanup for long names.

DECLARE @name sysname, @sql nvarchar(max);

[Copilot]

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 62.90%. Comparing base (a684c99) to head (0e83eee).
⚠️ Report is 4 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4478      +/-   ##
==========================================
- Coverage   64.64%   62.90%   -1.75%     
==========================================
  Files         288      283       -5     
  Lines       44009    66928   +22919     
==========================================
+ Hits        28451    42101   +13650     
- Misses      15558    24827    +9269     

Flag	Coverage Δ
CI-SqlClient	?
PR-SqlClient-Project	62.90% <ø> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[cheenamalhotra]

Must consider backporting to release branches as well.