Fix the 404 when uploading a recorded dataset to the Hub

[nicolas-rabault]

Users reported a 404 when pushing a recorded dataset to the Hub. Two separate bugs turned up: the one behind the report, and one that surfaced while verifying the fix on hardware.

The 404

LeRobotDataset.finalize() writes meta/episodes/ and the parquet footers. record_with_web_events never called it, so a finished recording left a directory with no episode index. LeRobot reads that as "not downloaded yet", asks the Hub for the dataset's refs, and 404s because it was never pushed.

It looked intermittent because DatasetWriter.__del__ finalizes as a safety net. But recording_active flips to False before the worker frame dies, and finalize() itself blocks on video and image flushing, so the frontend reaches /dataset-info and /upload-dataset while the dataset is still being written. Small recordings healed before anyone noticed; camera recordings did not.

Recording with push_to_hub=True was worse: it pushed before finalize, so it uploaded a dataset with a truncated data parquet rather than 404ing.

Reproduced deterministically:

### without finalize
on-disk meta/: ['info.json', 'stats.json', 'tasks.parquet']
RESULT: RepositoryNotFoundError 404 Client Error.

### with finalize
on-disk meta/: ['file-000.parquet', 'info.json', 'stats.json', 'tasks.parquet']
RESULT: loaded OK, episodes = 1

Repairing datasets already on disk

The fix is forward-only, so lelab/dataset_repair.py rebuilds the episode index of a recording that was interrupted: episode boundaries and frame ranges from the data parquet, tasks from tasks.parquet, video windows from cumulative durations probed with PyAV. It runs before every LeRobotDataset(repo_id) on a recorded dataset and is a no-op when the index already loads.

Recovery is bounded by the parquet footer. Only the newest data file is ever left open, so recovery stops there; trailing unreadable files are renamed out of the reader's glob and info.json counters are rewritten to match what survived. A killed short session recovers nothing, because LeRobot rotates data files at 100 MB and below that there is a single open file whose footer was never written. Those now get an explicit "re-record it" message instead of a Hub 404. Long sessions recover everything up to the last rotation.

Verified against a SIGKILLed recording: 4 of 6 episodes recovered, frames decode, video stays aligned to the right episode (worst pixel deviation 2/255, which is codec noise), action data exact, second call a no-op.

The second bug

With the upload path fixed, recording on real arms died a couple of seconds into the loop:

Failed to sync read 'Present_Position' on ids=[1, 2, 3, 4, 5, 6] after 1 tries.
[TxRxResult] There is no status packet!

robot.connect() ends with configure(), which runs inside bus.torque_disabled() and so re-enables torque on exit. Recording then wrote the calibration registers (Homing_Offset, Min_Position_Limit, Max_Position_Limit) with torque on and no guard of its own, which left the bus unable to answer once record_loop started reading it.

Upstream writes the calibration before configure(), and teleoperate.py already did the same. That is the discriminator: teleoperation worked, recording did not. Recording now follows the teleoperation sequence, connect the buses, write the calibration, connect the cameras, configure the motors. Going through bus.connect() also sidesteps connect()'s stdin prompt, which raised EOFError under the server whenever the motors disagreed with the calibration file.

Why this was hard to see

logger.exception("Recording session failed") printed no traceback. lerobot's init_logging(), called from lelab/calibrate.py, installs a root formatter that returns only record.getMessage(), dropping exc_info process-wide. The recording error handler now folds the traceback into the message, /recording-status carries the failure reason, and the frontend shows it instead of sending the user to the upload page for a dataset that was never recorded.

Testing

197 tests pass, 9 new ones covering the repair (rebuild with and without video, idempotency, truncated file quarantine, unrecoverable case, no-op on healthy and absent datasets). ruff check and ruff format clean.

Verified by hand on SO-101 hardware for the connect ordering and the recording loop. The end to end record then upload run is worth repeating by a reviewer with arms attached.

Note for reviewers

One thing this does not address: the log showed Record loop is running slower (14.8 Hz) than the target FPS (30 Hz) with two 640x480@30 cameras. Unrelated to these bugs, but it will drop frames.

[Copilot]

Pull request overview

Fixes dataset upload 404s by finalizing recordings, repairing interrupted datasets, and surfacing recording failures.

Changes:

• Finalizes datasets before upload and corrects hardware connection ordering.
• Adds recovery for interrupted datasets with tests.
• Displays backend recording errors in the frontend.

Reviewed changes

Copilot reviewed 5 out of 8 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
lelab/record.py	Finalizes recordings, repairs uploads, and improves device setup/errors.
lelab/dataset_repair.py	Reconstructs interrupted dataset metadata.
tests/test_dataset_repair.py	Tests repair and truncation scenarios.
frontend/src/pages/Recording.tsx	Displays recording failures.
frontend/dist/index.html	References rebuilt assets.
frontend/dist/assets/index-C5KQGL1C.css	Rebuilt stylesheet.
CLAUDE.md	Documents dataset repair.

Comments suppressed due to low confidence (1)

lelab/record.py:891

• If finalize() fails while flushing video/images or closing parquet writers, execution leaves this finally immediately and both device disconnects are skipped. Nest finalization in its own try/finally so hardware resources are released even when dataset finalization raises.

        dataset.finalize()

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.