(WIP) Obs geometry is the policy interface: dropout-invariant layout, eval inheritance, fail-fast

[eugenevinitsky]

What

Three related changes with one theme — observation geometry is the policy's interface and must be identical between training and eval envs:

1. Dropout-invariant layout: road dropout (env noise) no longer changes the obs vector size. The layout is always the full obs_slots_* counts; dropout only limits how many slots carry data, zero-padded with the existing valid-count features reporting what was kept. Content selection and its RNG path unchanged; at p=0 bit-identical.
2. Eval sections inherit obs geometry: [eval.validation_defaults] pinned obs_slots_lane_n/boundary_n = 80 and [eval.behaviors_defaults] pinned obs_slots_partners_n = 32. Any training config with different geometry produced eval observations the policy could not unpack. Those pins are removed; eval envs take geometry from the training config.
3. Fail fast: the policy records the obs length it was built for (expected_obs_dim), and EvalManager._run_inline raises with an actionable message on mismatch — instead of an asynchronous CUDA gather assert ~2h into training that poisons the context and kills the job after the manager already printed "continuing".

Why

Both the 2026-07-04 and 2026-07-05 nightly multi runs (trained at 60/40 slots with dropout 0.2/0.3) died at their first inline eval with ScatterGatherKernel device asserts. Night 1 was the dropout-shrunken layout meeting the clean macro; night 2 — after pinning eval dropout — was the section's hard 80/80 slot pins. Same disease: eval env obs shape ≠ policy obs shape. This PR removes both causes and makes any future instance a one-line error at eval start.

Notes

• Compat: checkpoints trained with dropout > 0 under the old layout won't load against the new one (only the two crashed nightly batches exist in that regime). Dropout-free checkpoints unaffected; smoke goldens should not move.
• Verified: full C test suite; 74 unit tests incl. 3 new (obs space invariant to dropout; valid counts/padding; the exact regression — dropout-trained policy on clean-env obs); CPU training run with dropout whose shutdown clean-eval passes. Eval-vs-train num_obs equality for the nightly config verified against the merged eval args (691 == 691).
• The cluster working tree currently carries the equivalent config edits as a stopgap; they come out when this merges.
• The explicit env.obs_dropout_* = 0.0 in validation_defaults stays — zeroing noise at eval is semantics, not geometry, and is safe after (1).

🤖 Generated with Claude Code