[CI] Stabilize the Newton environment smoke tests and run them kit-less - #6804
[CI] Stabilize the Newton environment smoke tests and run them kit-less#6804hujc7 wants to merge 1 commit into
Conversation
| # they must never co-install: usd-core supplies pxr where it has wheels (x86_64), | ||
| # usd-exchange supplies it on aarch64 (where usd-core has no wheel). | ||
| "usd-core>=25.11,<26.0 ; platform_machine == 'x86_64' or platform_machine == 'AMD64'", | ||
| "usd-core>=26.5,<27.0 ; platform_machine == 'x86_64' or platform_machine == 'AMD64'", |
There was a problem hiding this comment.
| # they must never co-install: usd-core supplies pxr where it has wheels (x86_64), | ||
| # usd-exchange supplies it on aarch64 (where usd-core has no wheel). | ||
| "usd-core>=25.11,<26.0 ; platform_machine == 'x86_64' or platform_machine == 'AMD64'", | ||
| "usd-core>=26.5,<27.0 ; platform_machine == 'x86_64' or platform_machine == 'AMD64'", |
There was a problem hiding this comment.
Obsolete single-thread USD workaround
The rendering test utility still forces PXR_WORK_THREAD_LIMIT=1 even though its TODO names usd-core 26.5 as the removal threshold. Leaving it in place slows rendering tests and prevents them from exercising normal concurrent USD parsing.
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
Greptile SummaryStabilizes Newton environment smoke testing and moves it to the kit-less path.
Confidence Score: 5/5The PR appears safe to merge. No blocking failure remains. Important Files Changed
Flowchart%%{init: {'theme': 'neutral'}}%%
flowchart LR
A["Collect registered task"] --> B["Load raw environment config"]
B --> C["Resolve newton_mjwarp preset"]
C --> D["Scan resolved config for Kit requirement"]
D -->|Kit required| E["Exclude from kit-less suite"]
D -->|Kit-less| F["Create environment with resolved preset"]
F --> G["Run Newton smoke test"]
Reviews (2): Last reviewed commit: "Run the Newton environment smoke tests k..." | Re-trigger Greptile |
413048f to
bd014cd
Compare
The suite aborted intermittently with SIGSEGV or SIGABRT while creating an environment, at roughly 5% per environment build. The abort is glibc catching a corrupted heap at a later allocation, so pytest recorded CRASHED with no Python traceback and no failing node. The corruption comes from a race in OpenUSD's physics parser: collecting collider descriptors for several colliders under one rigid body reallocates a shared vector across worker threads. OpenUSD fixes this in 26.05, which usd-core cannot reach yet because newton[importers] caps it below 26.5, so limit OpenUSD's work-thread pool here instead. This mirrors the workaround in rendering_test_utils.py and carries the same removal note. The limit is applied at module scope because OpenUSD builds its task arena while tests are collected, and WorkSetConcurrencyLimit does not shrink an existing arena. The MJWarp preset is a kit-less backend, so drop AppLauncher and filter the suite to the tasks that run without Kit. The split comes from the launcher's own scan, so a task follows its config instead of a hand-maintained list. Tasks carrying Kit camera sensors are excluded; test_environments.py already smoke-tests each of them on its default backend. get_isaac_sim_version() is now only called when create_stage_in_memory is set, since it imports isaacsim, which a kit-less caller does not have. Selecting a physics preset also had no effect. parse_env_cfg resolved every PresetCfg wrapper to its default before returning, and the follow-up apply_overrides call had no choice node left to act on, so each task ran whichever backend it defaulted to. Thread the selection through parse_env_cfg so it is applied while the wrappers are resolved, which is what register_task_to_hydra already does. Isaac-Velocity-Flat-G1 and Isaac-Velocity-Flat-Spot now run Newton in this suite rather than PhysX.
bd014cd to
d4c4a8f
Compare
AntoineRichard
left a comment
There was a problem hiding this comment.
Some comments could be shorter, but if it works it works.
There was a problem hiding this comment.
Isaac Lab Review Bot
The preset-selection plumbing and kit-less Newton smoke-test path are coherent, but the changelog fragment incorrectly classifies the PR as test-only despite changing the public parse_env_cfg API and behavior.
- Design and architecture: The kit-less split is derived from
sim_launcher.scan(...).needs_kit, keeping task selection aligned with launcher behavior rather than a separate maintained list. Applying preset selection duringPresetCfgresolution also removes the previous ineffective post-resolution override path. - API:
parse_env_cfggains a backward-compatible optionalpresetsparameter, but this public API and behavior change is hidden by a.skipchangelog fragment. Replace it with an appropriate.rstfragment describing the new preset-selection support; the current fragment also claims a file split that is not present in the patch. - Implementation: Preset resolution occurs before the existing device and environment-count overrides, and the Isaac Sim version check now short-circuits correctly for kit-less execution. The module-scope OpenUSD worker limit is explicitly documented as a temporary mitigation with a removal condition.
Minor fixes needed. Posted 1 actionable finding inline.
Automated review; human maintainers own approval decisions.
| @@ -0,0 +1,3 @@ | |||
| Test-only: ran the Newton environment smoke tests kit-less, split the Kit-requiring tasks into | |||
There was a problem hiding this comment.
🟡 Warning · Api — Public API change filed as skip fragment
The fragment uses the .skip tier, which the repository rules reserve for CI/docs/test-only changes and which yields no changelog entry and no version bump. This PR also changes public source: parse_env_cfg gains a presets parameter and new preset-resolution behavior. File an .rst fragment for isaaclab_tasks with an Added/Fixed entry describing parse_env_cfg(..., presets=...). The text also mentions a file split not present in this diff.
Description
The
isaaclab_tasksshards intermittently aborted withSIGSEGVorSIGABRTwhile creating aNewton environment — measured at ~5% per environment build. The abort is glibc catching a
corrupted heap (
malloc(): unaligned tcache chunk detected) at an unrelated later allocation, sopytest recorded
CRASHEDwith no Python traceback and no failing node.Root cause is a race in OpenUSD's physics parser: collecting collider descriptors for several
colliders under one rigid body reallocates a shared
std::vectoracross worker threads.This PR contains three related changes.
1. Work around the parser race
Limit OpenUSD's work-thread pool for this suite, mirroring the workaround already in
source/isaaclab_tasks/test/rendering_test_utils.pyand carrying the same removal note.Applied at module scope rather than in a fixture: kit-less, OpenUSD builds its task arena while
tests are collected, and
WorkSetConcurrencyLimitdoes not shrink an arena that already exists(measured — it still reported 128 workers and still crashed).
2. Run the suite kit-less
The MJWarp preset is a kit-less backend, so
AppLauncheris dropped and the suite filters to tasksthat run without Kit. The split is derived from
scan(...).needs_kit— the same decisionlaunch_simulation()makes — so a task follows its own config instead of a hand-maintained list.Tasks carrying Kit camera sensors are excluded;
test_environments.pyalready smoke-tests each ofthem on its default backend, so no task loses coverage.
get_isaac_sim_version()in_run_environmentsis now called only whencreate_stage_in_memoryisset, since it imports
isaacsim— that unconditional call was what previously forced Kit.3. Make the physics preset selection actually apply
Selecting a preset in this suite had no effect.
parse_env_cfgresolves everyPresetCfgwrapper toits
defaultbefore returning, and the follow-upapply_overridescall had no choice node left toact on, so each task ran whichever backend it defaulted to.
parse_env_cfgnow takes the selection and applies it while the wrappers are resolved, which iswhat
register_task_to_hydraalready does. Defaultpresets=()keeps every existing callerunchanged. The raw-config /
collect_presets/apply_overridesdance in the test util collapses toa single
parse_env_cfg(...)call.Consequence:
Isaac-Velocity-Flat-G1andIsaac-Velocity-Flat-Spotnow run Newton in thissuite rather than PhysX. Both pass.
Evidence
Standalone reproducer —
pxronly, no Isaac Lab, Newton, CUDA or Kit; 15 processes x 40 parses ofthe Franka Reach stage:
Crash rate against OpenUSD's thread count, which is what this PR limits:
PXR_WORK_THREAD_LIMITCaptured stack, previously unavailable because the process died without a Python traceback:
Local verification:
test_parse_env_cfg_presets.py(new)isaacsimimportIsaac-Reach-Franka,-OSC,Isaac-Reorient-Cube-Shadow-DirectIsaac-Velocity-Flat-G1,-Spoton Newton for the first timeWhy not raise the usd-core pin
usd-core>=26.5is the real fix, but it cannot be delivered yet:newton[importers]declaresusd-core>=25.5,<26.5, whose exclusive bound excludes the fixing release.[tool.uv] override-dependenciesgetsuv syncpast that, butisaaclab.sh --installresolves[project.dependencies]with pip, which has no override mechanism (see the comment atsource/isaaclab/isaaclab/cli/commands/install.py:341), so the Docker image build fails withResolutionImpossible.Raising
usd-core<26.5to<27innewton[importers]is the prerequisite. Once that lands, thepin can move and both this workaround and the one in
rendering_test_utils.pycan be removed.Notes for review
suite remain exposed until the pin can move.
newton_mjwarpnow genuinely runsNewton in this suite. G1 and Spot pass; other suites are unaffected because
presetsdefaults toan empty selection.
apply_overridesstill accepts apresetsargument that it never reads. It no longer has callersin
env_test_utils.py, butrendering_test_utils.pyandtest_hydra.pystill pass it. Removingit is a separate cleanup across those call sites.
Type of change
Checklist