Run orchestrator: model cards, supervised runs, onboarding gauntlet, fold-board#105
Merged
Conversation
Every special-case tuple that accumulated during the open-weight roster work (JSON_CONTRACT_MODELS, CHUNKED_OPEN_WEIGHT_MODELS, the reasoning- default tuples, the qwen timeout constant) becomes one declarative ModelCard registry; family-prefix heuristics remain the fallback for uncarded models. tests/test_model_cards.py locks every roster model's (contract, chunk, timeout, budget) so treatment changes are always visible in a diff. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Infrastructure so model additions are routine instead of the three-day open-weight saga: - policybench run: a per-scenario work queue (no more slow scenarios holding 25-scenario ranges hostage) with process-per-scenario workers reusing the sync path's cache/repairs/SIGALRM, adaptive concurrency that backs off on timeout spikes, a budget governor that projects spend from live per-scenario costs and stops cleanly at 90% instead of dying in a 402 storm, a run_state.json heartbeat, and lossless resume. - policybench onboard: probes a new model's serving stack (forced tool vs JSON contracts, whole-scenario convergence, latency, cost) and prints a suggested ModelCard; funding/auth failures abort the gauntlet instead of shaping the card. Live-validated against minimax-m3. - policybench fold-board: generalizes the release fold scripts with the row-count/duplicate gates that keep partial runs off the board. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
generate_scenarios() loads the certified populace person frame, which OOM-killed the CI runner twice (jobs canceled mid-suite with every test passing). A hand-built Scenario exercises the same gauntlet paths in a second with no data dependency. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Turns the lessons of the 2026-07-05/06 open-weight expansion (three days, three credit exhaustions, two zsh launch failures, one concurrency collapse, ~19 hours of range-workers hostage to slow scenarios) into infrastructure:
policybench/model_cards.py— every per-model serving treatment (answer contract, chunking, timeout, thinking-class budget) in one declarative registry; family-prefix heuristics remain the fallback.tests/test_model_cards.pylocks all 21 roster treatments so changes always appear in a diff.policybench run— supervised benchmark runs:run_state.jsonheartbeat for external visibility; rerunning the same command resumes losslesslypolicybench onboard— the probe gauntlet: forced-tool vs JSON at 3 variables, the surviving contract at 16 (the whole-scenario shape that broke kimi/GLM/qwen), latency and cost measurement → suggested ModelCard. Funding/auth errors abort with "fix the environment" rather than deriving a card from noise. Live-validated against minimax-m3 (correctly derived its tool contract; caught the credit-exhaustion case in the same run).policybench fold-board— the release fold generalized, with the row-count/duplicate/already-on-board gates.446 tests pass (40 new).
🤖 Generated with Claude Code