Add five open-weight models with per-model answer contracts#104
Open
MaxGhenis wants to merge 1 commit into
Open
Add five open-weight models with per-model answer contracts#104MaxGhenis wants to merge 1 commit into
MaxGhenis wants to merge 1 commit into
Conversation
Roster: deepseek-v4-pro, kimi-k2.6, glm-5.2, minimax-m3, qwen-3.7-max, with OpenRouter price overrides from the live API (2026-07-05). Reasoning-by-default open-weight models get the thinking-class treatment (16,384-token completion budget, 300s timeout) on BOTH request phases: the answers-only phase previously fell to the 256-token base, so the dynamic per-variable estimate became the budget and default-effort reasoning exhausted it before emitting any output. Kimi, GLM, and Qwen use the JSON answer contract: Moonshot and Alibaba reject forced tool_choice in thinking mode outright, and GLM accepts it but reasons to the full token ceiling without ever emitting the call. MiniMax handles forced tool calls cleanly and keeps the tool contract. Full forensics in results/local/openweight_run_20260705.md (untracked). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds five open-weight models to the benchmark roster with per-model answer-contract treatment, all probed live on 2026-07-05:
tool_choice=requiredin thinking mode (400)Harness changes
REASONING_DEFAULT_OPEN_WEIGHT_MODELS: open-weight reasoning-by-default models join gpt-5.5 in the thinking-class treatment (16,384-token completion budget, 300s timeout). The budget now applies to both explanation arms — the answers-only arm previously fell to the 256-token base, so the dynamic per-variable estimate became the cap and default-effort reasoning exhausted it before emitting output.JSON_CONTRACT_MODELS: per-model JSON contract for serving stacks that reject or degrade forced tool calls.CHUNKED_OPEN_WEIGHT_MODELS: kimi/glm chunk at 3 variables per call (the gpt-5.5 chunking precedent).Smoke evidence: glm chunked 36/36 (9s mean, $0.006); kimi chunked 33/36 (the 3 misses are repeated request deaths on one hard chunk — model behavior under the uniform contract, not harness truncation); qwen whole-scenario json 16/16 (93s); minimax tool 16/16-equivalent probe. Full forensics in
results/local/openweight_run_20260705.md(untracked, local).Full runs for all five are in flight; board fold + publish are a separate step gated on review of the results.
🤖 Generated with Claude Code