(WIP) Add nightly wandb report builder#508
Open
eugenevinitsky wants to merge 12 commits into
Open
Conversation
scripts/make_nightly_report.py creates or rewrites the 'PufferDrive Nightlies' report: per-project bar charts of each night's final metric values (grouped by run group = launch date, mean over seeds, stderr), per-seed finals on a time axis, and per-night mean training curves. Panels are live queries, so new nights appear without edits. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The scatter x axis was run end time, not the launch date, and scatter panels have no group/mean aggregation. The date-indexed view is the bar charts grouped by run group. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
update_nightly_trends.py maintains one trend run per seed, logging each metric's final value at step = night index (days since 2026-07-04, with night_ts mapping back to the date). A line panel grouped over the trend runs plots the mean across seeds with stderr bands natively. Both nightly launchers refresh the trend runs before submitting, so the report stays current with no extra scheduler; the bar/curve runsets exclude the trend group. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Stamp each trend row's _timestamp with its night's midnight and plot the trend panels on the wall-time axis, so the x axis shows calendar dates with true spacing instead of a night index. Step stays days since NIGHT_ZERO for monotonicity. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A single logged night autoranges the wall-time axis to [0, 2x], which renders as the year 1969; pin the minimum to the day before the first night instead. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…_per_infraction Keeps derived bookkeeping runs out of the real projects' run lists: one trend run per (source project, seed), grouped by source project. The report reads trends from nightly-trends and no longer needs to filter them out of the real runsets. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
scripts/nightly_report.py replaces update_nightly_trends.py and make_nightly_report.py: 'update' (default) rebuilds the per-seed trend runs in nightly-trends, 'report [--create]' rewrites the report. The launcher hooks call the update subcommand; wandb-workspaces is only imported on the report path. ruff-formatted to satisfy pre-commit. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
eugenevinitsky
changed the title
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
scripts/make_nightly_report.py— creates or rewrites the "PufferDrive Nightlies" wandb report. For each nightly project (nightly-multi,nightly-single):Why
The nightly launchers date-stamp every run's wandb group, but nothing turned that into a regression view. All panels are native wandb live queries — new nights appear automatically with nothing to re-run or schedule.
Notes
pip install wandb-workspaces(report-authoring API; not a training dependency, so not added to install_requires).WANDB_BASE_URL=https://api.wandb.aiby default since the Greene wandb settings point at the TORC instance; an explicit env var still wins.--createmints a fresh one.🤖 Generated with Claude Code