feat: live control-loop rate readout during inference

[ZouzouWP]

The problem

There is currently no way to answer a basic question about a running policy: is it actually stepping at 30 Hz?

lerobot does notice when the control loop misses its target, but it reports it as a warning per missed iteration, buried in the inference log:

WARNING Record loop is running slower (0.4 Hz) than the target FPS (30.0 Hz).

That number is 1/dt for a single slow iteration, so it swings wildly and says nothing about the run as a whole. Meanwhile the UI shows the same thing whether the policy is holding 30 Hz or crawling at 8 — a stopwatch and a progress bar. Since a policy behaves differently when it is stepped at a fraction of the rate it was trained at, that gap matters when you are trying to work out whether a disappointing rollout is a bad checkpoint or a starved control loop.

What this adds

A live readout of the real control-loop rate, measured rather than assumed.

On the Inference page

Three figures under the timer, plus the last minute of history against the target:

 CONTROL LOOP                                    target 30 Hz
┌──────────────┬──────────────┬──────────────┐
│ 28.6         │ 27.9         │ 11.4         │
│ FPS now      │ average      │ worst second │
└──────────────┴──────────────┴──────────────┘
   ╭╮      ╭─╮                            ╌╌╌╌╌╌╌ 30 Hz
 ──╯╰──────╯ ╰────╮   ╭──────────────────
                  ╰───╯
 longest stall 452 ms · 1 742 loop ticks

figure	what it answers
FPS now	is it keeping up right now — colour-coded against the target
average	did the run as a whole hold its rate
worst second	did it ever drop, and how far
longest stall	was it one long freeze or a steady sag — the number that points at the cause

The average is also carried into the completion toast, so a finished run states the rate it actually ran at.

In demo mode

The same measurement, presented for a different audience. Demo mode (#75) is meant to be watched from across a room, so it gets a single colour-coded pill next to the running indicator rather than a panel:

  ● Robot running   ( 28 fps )

Green while the loop holds its target, amber below 85%, red below 55% — enough to catch a degrading demo at a glance, small enough to stay out of the way.

Ordering note: the pill lives in Demo.tsx, which does not exist on main yet — it arrives with #75. This PR is deliberately standalone and touches only the inference path, so the two can be reviewed and merged in either order; whichever lands second picks up the other half. Happy to fold them together if you would rather review it as one change.

How it works

The measurement point is ThreadSafeRobot.get_observation: the rollout control loop calls it exactly once per iteration, so counting those calls over a window is the loop rate. No lerobot patch, no instrumentation inside the loop body.

  rollout subprocess          lelab server              browser
 ┌──────────────────┐      ┌────────────────┐      ┌──────────────┐
 │ rollout_metrics  │      │ _pump_stdout   │      │ Inference /  │
 │  counts ticks    │─────▶│  parses the    │─────▶│ Demo         │
 │  prints 1 line/s │stdout│  line          │ HTTP │  render      │
 └──────────────────┘      └────────────────┘      └──────────────┘
      [LELAB_FPS]            /inference-status         1 Hz poll

• lelab/rollout_metrics.py — a thin wrapper around lerobot.scripts.lerobot_rollout (the module the inference subprocess now spawns). It installs the counter, then hands over. Every failure path is swallowed: a run whose meter fails to install simply reports no rate.
• lelab/rollout.py — _pump_stdout already read every output line to find the rollout-start marker, so it now also picks up the rate samples and lerobot's own Robot: … | FPS: 30 | … banner. The target is read, not assumed from RolloutConfig's default, so the display stays honest if the config changes.
• Frontend — both pages already polled /inference-status once a second. Nothing new is plumbed; they just render fields that are now in the response.

New fields on /inference-status: target_fps, fps_now, fps_avg, fps_min, fps_worst_gap_ms, fps_ticks, fps_stale.

Two details worth a reviewer's eye

Windows are timed by their own ticks, not by the reporter's clock. The obvious implementation — count ticks, divide by one second — reports a phantom slowdown for the half-filled window left behind when a run ends. A healthy 20 Hz run finishing mid-window read as 10 Hz, which would have put a scary "worst second" on every clean run. Each window is now timed from its own first to its own last tick, and a window with fewer than two ticks reports nothing at all. That silence is also what lets the server mark the sample stale (3 s) instead of showing a frozen number as live.

This is not lerobot's warning number. lerobot logs 1/dt for one slow iteration; a single 230 ms hiccup among 29 healthy iterations makes it shout 0.4 Hz while the second genuinely ran at 24 Hz. Both are true, they answer different questions. That is exactly why the longest-stall figure is reported separately — it is where a single freeze shows up, instead of being averaged away.

Testing

• 27 unit tests pass, including new coverage for the line format (the contract between the subprocess and the pump), the target-FPS parsing, and the staleness/finished-run branches of the status payload.
• The meter was driven against a simulated control loop with an injected 400 ms stall, verifying it reports the true rate, captures the stall, and goes quiet — rather than reporting a phantom slowdown — once the loop stops.
• Exercised on a real SO-101 rollout: the meter tracks 30 Hz with intermittent stalls, consistent with what the profiler reports for the same run.

frontend/dist/ is intentionally untouched — build_frontend.yml rebuilds it on main.