feat(speculation): generator contract and bestfirst impl

[behinddwalls]

Summary

Why?

The standard Speculator (#445 defines the contract it implements) has to answer one question on every run: of all the paths the queue could build right now, which are worth the budget? That question splits cleanly in two — enumerate and rank the candidates versus decide how many to fund and what to preempt — and this branch lands the first half.

The split is deliberately not a controller-facing extension. The speculate controller depends only on the Speculator contract; an alternate Speculator need not use or expose a Generator at all. So there is no Config or Factory here — a Generator is chosen when the standard Speculator is constructed, not routed per queue by the wiring layer.

The reason it is a pull-based stream rather than a returned slice is cost. A head's outcome space is exponential in its unresolved dependency count, and the budget usually funds a handful of paths. Returning a ranked slice would mean enumerating everything to hand back a few; an iterator lets the consumer stop as soon as the budget is spent, and lets the producer compute only what is pulled.

What?

Adds submitqueue/extension/speculation/generator — the contract — and generator/bestfirst, the first implementation.

The contract. Generator.Open starts a candidate stream over the queue's live batches and their path sets, returning a PathIterator. Next yields one candidate at a time in the generator's chosen order, reporting ok == false once the coherent, depth-capped space is exhausted. Candidates descend in score, never repeat, and never contradict a resolved fact. Each carries a transient ranking score for the current run only — scores go stale across runs and are never stored. This is the repository's first pull-based iterator; Next follows the (value, ok, error) shape used elsewhere for found/not-found returns.

Scoring. A path scores as the product of its dependencies' landing chances: an included bet contributes the chance, an excluded bet its complement, and a dependency already resolved is pinned (landed → included, failed or cancelled → excluded), contributing a certainty of 1 and dropping out of the search entirely. Chances for unresolved dependencies come from an injected scorer.Scorer, memoized so a dependency shared by many heads is scored once; a dependency absent from the live batches falls back to a coin flip. All scoring happens in Open, so Next never calls the scorer and never fails.

Laziness, at two levels. This is the part worth reviewing closely.

A cross-head heap holds one real, non-terminal candidate per head — its current front, never an optimistic upper bound. Because a head's later candidates never outrank the front it currently holds, the best entry in that heap is simply the best candidate in the queue, and Next reduces to a pop plus a refill of the head it came from.

Each head is in turn its own lazy stream. It keeps a small frontier of materialized combinations; asked for its next candidate, it pops the best and expands it into at most two children. Pulling k candidates therefore materializes O(k) paths in total, however large the space behind them — a head with twelve unresolved dependencies costs the same three paths at the front of the queue as one with two.

The per-head walk. A head's best path takes the preferred bet on every unresolved dependency — the likelier outcome, so included when the dependency more likely lands and excluded when it more likely does not. Every other path is that best path with some subset of its choices flipped to the alternative, and flipping choice i multiplies the score by its penalty: the alternative chance over the preferred chance, a factor in [0, 1].

Choices are ordered by descending penalty, and a combination whose last flipped choice is i expands to the same set plus i+1, and to the same set with i replaced by i+1. Every non-empty subset has exactly one parent under that rule — if its last two flipped choices are adjacent it can only have come from the first rule, otherwise only from the second — so each combination is generated exactly once with no visited set. And because both children flip a choice no cheaper than the one they extend or replace, a child never outranks its parent, which is what makes the heap pop in true descending order. The descending-penalty sort is load-bearing for that second property, not a cosmetic ordering.

Nothing is pruned: given enough pulls, bestfirst still yields every combination within the depth bound. Heads over the depth bound are skipped until their dependencies resolve; candidates already terminal in a head's path set are passed over (but still expanded, so skipping one never prunes its descendants), so across successive runs a failed path falls away and the next-best surfaces. No conflict relaxation (dropped bets) — that is a Speculator policy, not a generator concern.

One doc correction. The README previously said the all-included path leads. That is only true when every dependency is better than even; the leading path is the preferred-bet path, which excludes any dependency likelier to fail than to land. Corrected here.

Also adds the package README.md and the generated mock/ package.

Test Plan

✅ bazel test //submitqueue/extension/speculation/... — passes (20 tests, 174 including subtests).
✅ make lint — passes, including license headers.
✅ make check-gazelle — BUILD files up to date.

The strongest check is a brute-force cross-check: for randomized chance vectors over 1–6 dependencies (150 trials, fixed seed), the lazy walk must emit exactly the 2ⁿ combinations, with scores equal to the eagerly-enumerated scores in descending order. That pins both completeness and ordering of the canonical subset expansion.

Laziness is asserted directly, via an in-package counter of materialized combinations:

• With 12 unresolved dependencies (a 4096-path space), Open materializes at most 3 paths and three pulls at most 9.
• Consuming one head's candidate advances that head only; a sibling head's stream is untouched.
• Next makes no scorer calls — the count after Open is unchanged by a full drain.

Ordering and semantics:

• A full drain yields every combination exactly once, with bets in queue order.
• Scores are non-increasing across a multi-head drain (17 candidates over four heads, one dependency below even).
• Equal scores are deterministic: tied dependency-free heads come out in ascending path-ID order, and repeated runs over an all-even-chance head agree exactly.
• A shared dependency is scored once across three heads.

The twelve pre-existing tests — resolved-dependency pinning, depth bound (including zero), missing dependencies, empty dependencies, head-state eligibility, terminal filtering, and scorer-error propagation — were kept unchanged and pass against the new implementation.

Issues