Treat BeforeAll module-local setup failure as catastrophic: skip tests, run AfterAll cleanup, and report root cause

[MariusStorhaug]

The module-local setup/teardown model assumes tests/BeforeAll.ps1 prepares state and tests/AfterAll.ps1 cleans up any created resources.

Request

When BeforeAll fails, Test-ModuleLocal should be treated as blocked by a catastrophic setup failure.

Current behavior allows test execution semantics that do not clearly enforce setup-as-gate, and this can produce noisy/invalid test runs after failed setup. Cleanup still needs to run to remove partially created resources.

From observed failure output (run 29487711781 / job 87586287510):

• BeforeAll script failed: tests/BeforeAll.ps1 - Cannot bind argument to parameter 'Token' because it is null.

Desired experience

• A failed BeforeAll is treated as catastrophic for module-local tests
• Test-ModuleLocal does not run when BeforeAll fails
• AfterAll-ModuleLocal still runs (best-effort cleanup)
• Final workflow summary clearly states setup failure as the root cause

Why this matters

Running tests after setup failure can produce misleading failures and can leave test resources behind if cleanup is not guaranteed.

Acceptance criteria

• Test-ModuleLocal is skipped when BeforeAll-ModuleLocal fails
• AfterAll-ModuleLocal runs with always() semantics even when BeforeAll-ModuleLocal fails
• AfterAll-ModuleLocal can access required context/secrets for cleanup in this failure path
• Workflow summary includes a clear catastrophic setup-failure message
• Overall workflow concludes as failed because setup failed (even if cleanup succeeds)

---

Technical decisions

Setup failure as hard gate: Treat BeforeAll-ModuleLocal failure as a hard prerequisite failure for test execution.

Cleanup must be independent of test execution: Keep AfterAll-ModuleLocal wired with always() and dependency conditions that permit execution after setup failure.

Explicit status propagation: Introduce/propagate a setup status flag (or rely on job result conditions) so downstream jobs and summary logic can distinguish:

• setup failed
• tests skipped due to setup failure
• cleanup executed (success/failure)

Accurate final reporting: Ensure the final summarization step reports setup failure as root cause and does not mask it behind skipped tests.

Non-breaking for happy path: Existing behavior remains unchanged when BeforeAll succeeds.

---

Implementation plan

Workflow gating

• Update Test-ModuleLocal job condition to require successful BeforeAll-ModuleLocal
• Ensure BeforeAll failure leads to explicit skip of test job (not ambiguous state)

Cleanup path

• Confirm AfterAll-ModuleLocal condition includes always() and allows run after setup failure
• Verify cleanup job has needed inputs/env/context in setup-failure scenarios
• Ensure cleanup failures are surfaced without overriding the primary setup-failure diagnosis

Summary and status

• Update summary logic to emit a dedicated "BeforeAll failed (catastrophic)" section
• Include setup, test, and cleanup job outcomes in final summary
• Ensure workflow result remains failed when setup fails

Tests

• Add/extend workflow tests for path: BeforeAll fails -> Test skipped -> AfterAll runs
• Add assertion that final status is failed with setup-failure root cause
• Add assertion that cleanup still executes in this path