Skip to content

[CI] Stabilize the Newton environment smoke tests and run them kit-less - #6804

Open
hujc7 wants to merge 1 commit into
isaac-sim:developfrom
hujc7:jichuanh/usd-core-physics-parser-race
Open

[CI] Stabilize the Newton environment smoke tests and run them kit-less#6804
hujc7 wants to merge 1 commit into
isaac-sim:developfrom
hujc7:jichuanh/usd-core-physics-parser-race

Conversation

@hujc7

@hujc7 hujc7 commented Jul 30, 2026

Copy link
Copy Markdown
Collaborator

Description

The isaaclab_tasks shards intermittently aborted with SIGSEGV or SIGABRT while creating a
Newton environment — measured at ~5% per environment build. The abort is glibc catching a
corrupted heap (malloc(): unaligned tcache chunk detected) at an unrelated later allocation, so
pytest recorded CRASHED with no Python traceback and no failing node.

Root cause is a race in OpenUSD's physics parser: collecting collider descriptors for several
colliders under one rigid body reallocates a shared std::vector across worker threads.

"Fixed a bug in UsdPhysicsParsingUtility which could cause a race condition and crash when parsing
multiple colliders under the same rigid body in a multi-threaded context"
— OpenUSD CHANGELOG
[26.05] - 2026-04-24, upstream issue 4002

This PR contains three related changes.

1. Work around the parser race

Limit OpenUSD's work-thread pool for this suite, mirroring the workaround already in
source/isaaclab_tasks/test/rendering_test_utils.py and carrying the same removal note.

Applied at module scope rather than in a fixture: kit-less, OpenUSD builds its task arena while
tests are collected, and WorkSetConcurrencyLimit does not shrink an arena that already exists
(measured — it still reported 128 workers and still crashed).

2. Run the suite kit-less

The MJWarp preset is a kit-less backend, so AppLauncher is dropped and the suite filters to tasks
that run without Kit. The split is derived from scan(...).needs_kit — the same decision
launch_simulation() makes — so a task follows its own config instead of a hand-maintained list.

Tasks carrying Kit camera sensors are excluded; test_environments.py already smoke-tests each of
them on its default backend, so no task loses coverage.

get_isaac_sim_version() in _run_environments is now called only when create_stage_in_memory is
set, since it imports isaacsim — that unconditional call was what previously forced Kit.

3. Make the physics preset selection actually apply

Selecting a preset in this suite had no effect. parse_env_cfg resolves every PresetCfg wrapper to
its default before returning, and the follow-up apply_overrides call had no choice node left to
act on, so each task ran whichever backend it defaulted to.

parse_env_cfg now takes the selection and applies it while the wrappers are resolved, which is
what register_task_to_hydra already does. Default presets=() keeps every existing caller
unchanged. The raw-config / collect_presets / apply_overrides dance in the test util collapses to
a single parse_env_cfg(...) call.

Consequence: Isaac-Velocity-Flat-G1 and Isaac-Velocity-Flat-Spot now run Newton in this
suite rather than PhysX. Both pass.

Evidence

Standalone reproducer — pxr only, no Isaac Lab, Newton, CUDA or Kit; 15 processes x 40 parses of
the Franka Reach stage:

usd-core Has the 26.05 fix Crashed processes
25.11 (current pin) no 14 / 15
26.3 no 10 / 15
26.8 yes 0 / 15

Crash rate against OpenUSD's thread count, which is what this PR limits:

PXR_WORK_THREAD_LIMIT 1 2 4 8 16 32 128
crashed procs (of 15) 0 0 10 10 12 11 14

Captured stack, previously unavailable because the process died without a Python traceback:

#0  moveDescsToDict<UsdPhysicsRigidBodyDesc>(...)        pxr/UsdPhysics/_usdPhysics.so
#1  LoadUsdPhysicsFromRange(stage, paths, callback, ...) libusd_ms
    newton/_src/utils/import_usd.py:546  parse_usd
    isaaclab_newton/cloner/newton_clone_utils.py:188  _build_source_builder
    isaaclab/scene/interactive_scene.py:187  __init__

Local verification:

Check Result
test_parse_env_cfg_presets.py (new) 3 passed — fails on develop
kit-less collection, no isaacsim import 58 tests in 7.3s
Isaac-Reach-Franka, -OSC, Isaac-Reorient-Cube-Shadow-Direct 6 passed
Isaac-Velocity-Flat-G1, -Spot on Newton for the first time 4 passed

Why not raise the usd-core pin

usd-core>=26.5 is the real fix, but it cannot be delivered yet: newton[importers] declares
usd-core>=25.5,<26.5, whose exclusive bound excludes the fixing release. [tool.uv] override-dependencies gets uv sync past that, but isaaclab.sh --install resolves
[project.dependencies] with pip, which has no override mechanism (see the comment at
source/isaaclab/isaaclab/cli/commands/install.py:341), so the Docker image build fails with
ResolutionImpossible.

Raising usd-core<26.5 to <27 in newton[importers] is the prerequisite. Once that lands, the
pin can move and both this workaround and the one in rendering_test_utils.py can be removed.

Notes for review

  • This is a mitigation for the crash, not a fix. Users creating Newton environments outside this
    suite remain exposed until the pin can move.
  • Change 3 alters what CI exercises: any task whose default is not newton_mjwarp now genuinely runs
    Newton in this suite. G1 and Spot pass; other suites are unaffected because presets defaults to
    an empty selection.
  • apply_overrides still accepts a presets argument that it never reads. It no longer has callers
    in env_test_utils.py, but rendering_test_utils.py and test_hydra.py still pass it. Removing
    it is a separate cleanup across those call sites.

Type of change

  • Bug fix (non-breaking change which fixes an issue)

Checklist

  • Pre-commit hooks pass
  • Changelog fragment added
  • Full CI validation

@github-actions github-actions Bot added isaac-lab Related to Isaac Lab team infrastructure labels Jul 30, 2026
@hujc7
hujc7 marked this pull request as ready for review July 30, 2026 12:18
@hujc7
hujc7 requested a review from a team July 30, 2026 12:18
@hujc7
hujc7 requested review from StafaH and kellyguo11 as code owners July 30, 2026 12:18
@hujc7
hujc7 marked this pull request as draft July 30, 2026 12:19
Comment thread pyproject.toml Outdated
# they must never co-install: usd-core supplies pxr where it has wheels (x86_64),
# usd-exchange supplies it on aarch64 (where usd-core has no wheel).
"usd-core>=25.11,<26.0 ; platform_machine == 'x86_64' or platform_machine == 'AMD64'",
"usd-core>=26.5,<27.0 ; platform_machine == 'x86_64' or platform_machine == 'AMD64'",

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Stale USD version assertions

When the metadata and benchmark tests run after this dependency bump, they still require the old usd-core>=25.11,<26.0 constraint and installed version 25.11, causing deterministic assertion failures with the newly resolved 26.8 version.

Comment thread pyproject.toml Outdated
# they must never co-install: usd-core supplies pxr where it has wheels (x86_64),
# usd-exchange supplies it on aarch64 (where usd-core has no wheel).
"usd-core>=25.11,<26.0 ; platform_machine == 'x86_64' or platform_machine == 'AMD64'",
"usd-core>=26.5,<27.0 ; platform_machine == 'x86_64' or platform_machine == 'AMD64'",

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Obsolete single-thread USD workaround

The rendering test utility still forces PXR_WORK_THREAD_LIMIT=1 even though its TODO names usd-core 26.5 as the removal threshold. Leaving it in place slows rendering tests and prevents them from exercising normal concurrent USD parsing.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

@greptile-apps

greptile-apps Bot commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

Stabilizes Newton environment smoke testing and moves it to the kit-less path.

  • Limits OpenUSD worker concurrency to avoid the parser race on the currently pinned USD release.
  • Resolves requested physics presets directly in parse_env_cfg.
  • Filters smoke-test tasks using the resolved configuration’s Kit requirement.
  • Adds focused coverage for default and explicitly selected physics presets.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename Overview
source/isaaclab_tasks/isaaclab_tasks/utils/parse_cfg.py Adds preset selection to environment configuration resolution while preserving default behavior for existing callers.
source/isaaclab_tasks/test/env_test_utils.py Uses resolved presets for execution and introduces configuration-derived filtering for Kit requirements.
source/isaaclab_tasks/test/core/test_environments_newton.py Converts the Newton smoke suite to kit-less execution and applies the OpenUSD concurrency workaround before collection.
source/isaaclab_tasks/test/core/test_parse_env_cfg_presets.py Verifies default preset resolution, explicit Newton selection, and compatibility with other configuration overrides.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart LR
    A["Collect registered task"] --> B["Load raw environment config"]
    B --> C["Resolve newton_mjwarp preset"]
    C --> D["Scan resolved config for Kit requirement"]
    D -->|Kit required| E["Exclude from kit-less suite"]
    D -->|Kit-less| F["Create environment with resolved preset"]
    F --> G["Run Newton smoke test"]
Loading

Reviews (2): Last reviewed commit: "Run the Newton environment smoke tests k..." | Re-trigger Greptile

@hujc7 hujc7 changed the title Require usd-core 26.5 to fix Newton USD import crash Require usd-core 26.5 to fix Newton USD import crash (blocked on Newton bound) Jul 30, 2026
@hujc7 hujc7 changed the title Require usd-core 26.5 to fix Newton USD import crash (blocked on Newton bound) [BUG] Require usd-core 26.5 to fix Newton USD import crash Jul 30, 2026
@hujc7
hujc7 force-pushed the jichuanh/usd-core-physics-parser-race branch from 413048f to bd014cd Compare July 30, 2026 16:39
@hujc7 hujc7 changed the title [BUG] Require usd-core 26.5 to fix Newton USD import crash Run the Newton environment smoke tests kit-less Jul 30, 2026
@hujc7 hujc7 changed the title Run the Newton environment smoke tests kit-less [CI] Stabilize the Newton environment smoke tests and run them kit-less Jul 30, 2026
The suite aborted intermittently with SIGSEGV or SIGABRT while creating an
environment, at roughly 5% per environment build. The abort is glibc catching a
corrupted heap at a later allocation, so pytest recorded CRASHED with no Python
traceback and no failing node.

The corruption comes from a race in OpenUSD's physics parser: collecting
collider descriptors for several colliders under one rigid body reallocates a
shared vector across worker threads. OpenUSD fixes this in 26.05, which
usd-core cannot reach yet because newton[importers] caps it below 26.5, so
limit OpenUSD's work-thread pool here instead. This mirrors the workaround in
rendering_test_utils.py and carries the same removal note. The limit is applied
at module scope because OpenUSD builds its task arena while tests are
collected, and WorkSetConcurrencyLimit does not shrink an existing arena.

The MJWarp preset is a kit-less backend, so drop AppLauncher and filter the
suite to the tasks that run without Kit. The split comes from the launcher's
own scan, so a task follows its config instead of a hand-maintained list.
Tasks carrying Kit camera sensors are excluded; test_environments.py already
smoke-tests each of them on its default backend.

get_isaac_sim_version() is now only called when create_stage_in_memory is set,
since it imports isaacsim, which a kit-less caller does not have.

Selecting a physics preset also had no effect. parse_env_cfg resolved every
PresetCfg wrapper to its default before returning, and the follow-up
apply_overrides call had no choice node left to act on, so each task ran
whichever backend it defaulted to. Thread the selection through parse_env_cfg
so it is applied while the wrappers are resolved, which is what
register_task_to_hydra already does. Isaac-Velocity-Flat-G1 and
Isaac-Velocity-Flat-Spot now run Newton in this suite rather than PhysX.
@hujc7
hujc7 force-pushed the jichuanh/usd-core-physics-parser-race branch from bd014cd to d4c4a8f Compare July 30, 2026 17:15

@AntoineRichard AntoineRichard left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some comments could be shorter, but if it works it works.

@AntoineRichard
AntoineRichard marked this pull request as ready for review July 31, 2026 14:43
@AntoineRichard
AntoineRichard requested a review from ooctipus as a code owner July 31, 2026 14:43

@isaaclab-review-bot isaaclab-review-bot Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isaac Lab Review Bot

The preset-selection plumbing and kit-less Newton smoke-test path are coherent, but the changelog fragment incorrectly classifies the PR as test-only despite changing the public parse_env_cfg API and behavior.

  • Design and architecture: The kit-less split is derived from sim_launcher.scan(...).needs_kit, keeping task selection aligned with launcher behavior rather than a separate maintained list. Applying preset selection during PresetCfg resolution also removes the previous ineffective post-resolution override path.
  • API: parse_env_cfg gains a backward-compatible optional presets parameter, but this public API and behavior change is hidden by a .skip changelog fragment. Replace it with an appropriate .rst fragment describing the new preset-selection support; the current fragment also claims a file split that is not present in the patch.
  • Implementation: Preset resolution occurs before the existing device and environment-count overrides, and the Isaac Sim version check now short-circuits correctly for kit-less execution. The module-scope OpenUSD worker limit is explicitly documented as a temporary mitigation with a removal condition.

Minor fixes needed. Posted 1 actionable finding inline.

Automated review; human maintainers own approval decisions.

@@ -0,0 +1,3 @@
Test-only: ran the Newton environment smoke tests kit-less, split the Kit-requiring tasks into

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Warning · Api — Public API change filed as skip fragment

The fragment uses the .skip tier, which the repository rules reserve for CI/docs/test-only changes and which yields no changelog entry and no version bump. This PR also changes public source: parse_env_cfg gains a presets parameter and new preset-resolution behavior. File an .rst fragment for isaaclab_tasks with an Added/Fixed entry describing parse_env_cfg(..., presets=...). The text also mentions a file split not present in this diff.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

infrastructure isaac-lab Related to Isaac Lab team

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants