Fix ManiSkill2 frame extraction during evaluation

[Arison591]

Summary

Fix ManiSkill2 video-frame extraction by making _extract_frame an instance method. The method selects frames from self.enabled_cameras, but it was declared as a static method, so both reset() and step() raised NameError: name 'self' is not defined when recording observations.

User-Visible Behavior

ManiSkill2 benchmark episodes can now record the first configured camera's RGB frame during reset and step instead of crashing before evaluation proceeds.

Affected Pipeline / Benchmark

• Pipeline(s): N/A
• Benchmark(s): maniskill2
• Runtime profile(s): N/A
• Public entrypoint(s): ManiSkill2 benchmark execution through the evaluation runner

Change Type

• Bug fix
• Model integration
• Benchmark integration
• Pipeline/runtime change
• Documentation only
• Test/QA tooling

Asset / API / GPU Requirements

• Downloads or large assets: None for the focused regression; a real end-to-end run requires the normal ManiSkill2/SAPIEN installation and assets.
• API keys or quota: None
• GPU / simulator requirements: No GPU is required for the fixed frame-selection logic. Full simulator validation was not run because ManiSkill2/SAPIEN is not installed in the validation environment.
• Official repository or checkpoint assumptions: None

Commands Run

Command	Result	Notes
Focused stubbed reset() + step() harness (below)	Pass	Recorded the configured RGB frame at both call sites without importing ManiSkill2/SAPIEN.
python -m ruff check worldfoundry/evaluation/tasks/embodied/simulators/maniskill2/benchmark.py	Pass	No lint errors in the touched file.
python -m ruff format --check worldfoundry/evaluation/tasks/embodied/simulators/maniskill2/benchmark.py	Pass	File is formatted.
make lint	Pass	Compile, shell, catalog, and runtime-registry checks passed.
make compile-eval && make docs-check	Pass	Evaluation modules compile and documented CLI entrypoints load.
make cli-check	Pass	Existing-results evaluation completed with one successful sample/artifact.
bash scripts/docs/build.sh --skip-bootstrap	Skipped	Local blocker: npm is not installed. No docs were changed.

Focused regression command

python - <<'PY'
import sys
import types
import numpy as np

from worldfoundry.evaluation.tasks.embodied.simulators.maniskill2.benchmark import ManiSkill2Benchmark

rgb = np.arange(18, dtype=np.uint8).reshape(2, 3, 3)
obs = {"image": {"base_camera": {"rgb": rgb}}}

class FakeEnv:
    def __init__(self):
        self.unwrapped = self
    def reset(self):
        return obs, {}
    def step(self, action):
        assert action.shape == (7,)
        return obs, 0.25, False, False, {"success": False}
    def close(self):
        pass

fake_env = FakeEnv()
gymnasium = types.ModuleType("gymnasium")
gymnasium.make = lambda *args, **kwargs: fake_env
mani_skill2 = types.ModuleType("mani_skill2")
mani_skill2.__path__ = []
mani_skill2_envs = types.ModuleType("mani_skill2.envs")
mani_skill2.envs = mani_skill2_envs
sys.modules.update({
    "gymnasium": gymnasium,
    "mani_skill2": mani_skill2,
    "mani_skill2.envs": mani_skill2_envs,
})

class Recorder:
    def __init__(self):
        self.frames = []
    def record_video(self, frame):
        self.frames.append(frame)
    def record_step(self, **kwargs):
        pass

benchmark = ManiSkill2Benchmark()
benchmark._recorder = Recorder()
assert benchmark.reset({"env_name": "PickCube-v0"}) is obs
assert benchmark.step({"action": [0, 0, 0, 0, 0, 0, 1]}).obs is obs
assert len(benchmark._recorder.frames) == 2
for frame in benchmark._recorder.frames:
    np.testing.assert_array_equal(frame, rgb)
print("stubbed reset/step frame recording: PASS")
PY

Checkpoint / API Key Needs

• Checkpoints / weights: None
• Required environment variables: None
• API providers: None
• Local cache assumptions: None

Sample Artifact Evidence

• Artifact path or link: Console output stubbed reset/step frame recording: PASS
• What it demonstrates: Both frame-recording call sites select the configured camera's RGB observation without raising NameError.
• Known limitations: This uses an in-memory simulator stub; a full ManiSkill2/SAPIEN rollout was not run locally.

Validation Matrix Status

Area	Status	Evidence / Command
Unit or focused regression	Pass	Stubbed reset() + step() harness above
Real inference validation	N/A	No model inference behavior changed
Streaming or multi-turn path	N/A	Not affected
Benchmark runner / metric path	Pass	make cli-check; focused adapter harness
Docs build or link check	Pass	make docs-check; full Node build skipped because npm is unavailable
GPU/API-key dependent path	N/A	Frame selection is CPU-only and requires no credentials

Leaderboard Validity Impact

• Readiness status changed: No
• leaderboard_valid impact: None
• Scorecard/preflight evidence: make cli-check passed; no scorecard schema changed
• Demo or contract-only limitations: Full ManiSkill2 simulator rollout was not claimed

Compatibility And Risk

• Backward compatibility impact: None for public APIs. _extract_frame is private and all in-tree callers invoke it on an instance.
• Expected resource cost: None
• Failure modes or rollout concerns: Camera lookup behavior and configured-camera ordering are unchanged; only access to the instance configuration is restored.

Checklist

• I kept the change narrowly scoped.
• I did not commit secrets, API keys, large checkpoints, or generated cache files.
• I reused fixtures from data/test_cases/ or assets from data/benchmarks/ where practical.
• I updated docs when public behavior, install steps, or entry commands changed.
• I documented API/GPU/checkpoint/official-repo requirements or confirmed none are needed.
• I did not promote demo, contract-only, normalizer-only, API-blocked, or partial evidence to leaderboard validity.

[iChubai]

Thank you so much for catching and fixing this! The _extract_frame issue was easy to miss but would directly break ManiSkill2 video recording during evaluation. The fix is clean and well-scoped. Really appreciate your first contribution to WorldFoundry — looking forward to more!

[Copilot]

Pull request overview

Fixes ManiSkill2 evaluation video-frame extraction by correcting _extract_frame to be an instance method so it can access self.enabled_cameras during reset() and step() without raising NameError.

Changes:

• Converted _extract_frame from @staticmethod to an instance method (self-aware) to support configured camera selection.
• Kept frame selection behavior aligned with enabled_cameras ordering (first available configured camera wins).
• Reformatted the long specs import into a multi-line import block (no functional change).

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.