Skip to content

[Bug] Python 3.14 unit tests intermittently stall during pytest shutdown #935

Description

@RyAuld

Description

Python 3.14 unit tests intermittently stall after all tests pass (reaches 99-100% completion), hanging until the 5-minute step timeout kills the process. Other Python versions (3.9–3.13) consistently complete in ~1.5 minutes.

Observed Behavior

  • All tests pass successfully
  • The last output before the stall is typically a deprecation warning or the final test result
  • The process hangs during pytest session teardown/finalization
  • The stall is intermittent — sometimes Python 3.14 passes, sometimes it hangs

Builds Exhibiting This Issue

  • 1651220 — canceled at 30-min timeout
  • 1651218 — canceled at 30-min timeout
  • 1654753 — failed at 5-min timeout (after timeout was reduced)

Mitigations Already Applied (PR #934)

  • Replaced deprecated datetime.utcnow() with datetime.now(timezone.utc) — did not resolve the stall
  • Removed 2>&1 | tee pipe (replaced with --log-file) — did not resolve the stall
  • Added timeoutInMinutes: 5 on the pytest step to fail fast

Suspected Root Causes

Without agent-level access, we cannot definitively determine the cause. Likely candidates:

  1. pytest-azurepipelines plugin — may hang during post-test result upload on 3.14
  2. Python 3.14 interpreter shutdown — thread/GC finalization deadlock in pre-release runtime
  3. Dependencies compiled from source (no pre-built 3.14 wheels) — buggy atexit/cleanup handler
  4. Hosted agent resource contention — network timeout to ADO during result publishing

Suggested Next Steps

  • Try running with -p no:azurepipelines to rule out plugin issues
  • Add continueOnError: true for Python 3.14 matrix entry to unblock PRs
  • Consider removing Python 3.14 from the CI matrix until official release and ecosystem wheel availability
  • Revisit once Python 3.14 reaches stable release

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions