diff --git a/docs/adr/ADR-0008-storage-economics-and-payment-protocol.md b/docs/adr/ADR-0008-storage-economics-and-payment-protocol.md new file mode 100644 index 00000000..a4acb5f4 --- /dev/null +++ b/docs/adr/ADR-0008-storage-economics-and-payment-protocol.md @@ -0,0 +1,516 @@ +# ADR-0008: Storage economics and the quoting + payment protocol + +- **Status:** Proposed +- **Date:** 2026-07-28 +- **Decision owners:** Anselme (@grumbach) +- **Reviewers:** +- **Supersedes:** none +- **Superseded by:** none +- **Related:** ADR-0002 (storage audit, merged), ADR-0004 (commitment-bound + quote pricing, merged), ADR-0006 (receiver-side revenue floor, merged). + **Not merged at this head, referenced only as pending work:** ADR-0005 + (earned-reward eligibility, on `adr-0005-earned-reward-eligibility`), + ADR-0007 (audit proof shape, on the verified-slice-audit branch), + ant-client ADR-0003 (client-side eligibility gate). + +## Context + +Every ADR so far amends one corner of the economics — ADR-0004 puts a ceiling +on what a node may charge, ADR-0006 puts a floor under what it will accept, the +pending eligibility work decides who may be paid at all — but the model those +amendments modify was never written down. Anyone reading the ADR set today can +see the patches and not the machine. + +This ADR is **mostly descriptive**: it records the pricing and payment design +as built, so that later ADRs can amend a stated baseline instead of an implied +one. Two things follow from that and are called out wherever they appear: + +- **Shipped vs pending.** Sections below mark what is live at this head, what + is behind a rollout flag, and what lives on an unmerged branch. Nothing here + should be read as deployed unless it says so. +- **One normative decision.** Writing the model down surfaced a defect: the + merkle batch path settled a third of what the single-node path settles for an + identical chunk, because the 3× multiplier was never applied when pool + commitments are built. This ADR **decides** to raise merkle to 3×, and the + change ships **client-first**: a released ant-client pays 3× on every batch + from the moment it is used, and nodes begin *requiring* 3× at a fixed instant + set after that release, **2026-08-04 15:00 UTC** (Unix `1785855600`). There is + no shadow mode and no flag to flip afterwards. Because it is a pricing + decision and not just an arithmetic repair, it still needs the economic + owner's sign-off before the client is published; the alternatives that were + weighed are recorded below. + +Terms: *record* (one stored chunk, the priced unit), *close group* (the 7 nodes +closest to an address, which hold the replicas), *quote* (a node's signed +offer), *commitment* (ADR-0002's signed Merkle root plus key count over what a +node holds), *ANT* (the ERC-20 payment token on Arbitrum), *atto* (10⁻¹⁸ ANT). + +## Decision Drivers + +- One payment, stored indefinitely — the client transacts once and never again + for that data. +- The price must be a public function anyone can recompute. No negotiation, no + oracle, no per-operator configuration. +- The price must rise as the network fills, so scarcity attracts new supply, + and must be non-zero when empty, so storage is never free to spam. +- Every storer must be able to verify payment independently, from the chain, + without trusting the client or the paying node. +- Gas must not dominate: a large upload cannot mean one transaction per chunk. +- Client and node must compute prices from the same code, so they can never + disagree about what was owed. + +## Considered Options + +1. **Per-byte pricing.** Rejected: the network's unit of work is a record — + replication, audit, and routing all cost the same per chunk regardless of + how full it is. Per-byte pricing prices the wrong thing and invites padding + games. +2. **Rent / recurring payment.** Rejected: it makes permanence conditional on + a live payer, requires an eviction-for-non-payment path, and puts a + recurring on-chain cost on every stored object. +3. **Pay every member of the close group.** Rejected on gas: seven settlements + per chunk. The chosen design pays one node a multiple instead, so the + network receives the same money for a seventh of the transactions. +4. **Fixed network price.** Rejected: it carries no supply signal, so a + filling network cannot pay more to attract capacity. +5. **Quadratic per-record price, median-of-group selection, one on-chain + settlement, re-verified by every storer (chosen).** + +## Decision + +We price and settle storage as follows. + +### 1. What is sold + +A **record** — one chunk of up to 4 MiB — is the unit of sale. The price is +per record and does not depend on how many bytes the chunk actually carries. +Payment is once; the data is then stored indefinitely. There is no rent, no +renewal, no retrieval fee, and no emission or inflation subsidy: **client +payments are the network's only revenue**. + +### 2. The price curve + + price_per_record(n) = BASELINE + K × (n / D)² + +with `BASELINE = 0.00390625 ANT`, `K = 0.03515625 ANT`, `D = 6000 records`, +computed in `u256` wei. `n` is the node's **committed** key count — the count +in its signed storage commitment (ADR-0004) — not its raw on-disk count. A +node with no live commitment can only quote the baseline. + +The formula lives once, in `ant-protocol`, and both sides import it, so a +quote's price is recomputable by the client before paying and by every storer +after (see `ant-protocol/src/payment/pricing.rs`). + +At $0.10/ANT and full 4 MiB chunks (256 chunks/GiB), the quoted price per GiB +is roughly: + +| Committed records | ANT / chunk | Quoted $/GiB | Client pays (3×) | +|---|---|---|---| +| 0 (empty) | 0.0039 | $0.10 | $0.30 | +| 6 000 | 0.0391 | $1.00 | $3.00 | +| 12 000 | 0.1445 | $3.70 | $11.10 | + +Chunks smaller than 4 MiB cost the same each, so the effective $/GiB rises for +small-chunk data. + +### 3. Quoting + +The client asks the **witnessed close group**: the 7 nodes closest to the +address, each recognised by a quorum of its neighbours' reported views, so the +client cannot be handed a fabricated group. The request carries the address, +data size, type, and a fresh 32-byte nonce. + +The quorum is **not a fixed 5-of-7**. It starts at `ceil(7 × 2/3) = 5` and is +reduced by one for every close-group peer that returned no responder view, +floored at 1 (`witnessed_close_group_quorum_for_missing_views`). Under churn or +partial responses the bar therefore degrades, in the worst case to a single +witness. That is a deliberate liveness choice, but it means "witnessed" is a +best-effort check, not a 5-of-7 guarantee. + +Each responder returns, signed with ML-DSA-65: the **quote** (content address, +timestamp, price, rewards address, committed key count, commitment pin) and the +**signed commitment** the price was derived from. A responder that already +holds the chunk says so, and the client skips payment for that address +entirely. + +Before paying, the client drops any quote it cannot fully resolve: wrong +public-key-to-peer binding, bad signature, wrong content address, a price that +is not exactly `calculate_price(committed_key_count)`, or a commitment that +does not parse, does not belong to the quoter, or does not hash to the quote's +pin. + +*Pending, not at this head:* the quote response also carries a nonce-bound +**audit report**, which feeds a payee-eligibility gate excluding quoters their +neighbours have not attested clean at the size they are monetizing. That is +ADR-0005 / ant-client ADR-0003 work on unmerged branches, observe-only even +there. It is described here because it shapes the model, not because it is +running. + +### 4. Paying — two shapes + +#### 4a. Shipped baseline + +**Single-node (default below 64 chunks).** The quotes the client collected are +sorted by price; it pays the **upper-median** issuer **3× its quoted price** and +the others nothing, in one `payForQuotes` call. Cost per chunk is 3 × that +median. The multiplier keeps the network's revenue equal to paying three of the +group while costing one transaction's gas. The median is taken over *the set +the client chose to supply*, which a storer cannot audit for completeness — +ADR-0006's receiver-side floor exists for exactly that reason. + +**Merkle batch (64 chunks and above).** The client builds a Merkle tree over up +to 256 addresses, derives `2^ceil(depth/2)` candidate pools from its midpoints, +and collects 16 quotes per pool — one `payForMerkleTree` call for the whole +batch. The contract selects a winner pool and pays `depth` of its 16 +candidates: `total = median16(amount) × 2^depth`, split evenly. Today the +client submits the **bare quoted price** as the payable `amount`, so a batch +settles `1 × median16 × 2^depth`. The receipt is valid for one week; larger +uploads split into successive batches. + +So the shipped baseline is asymmetric: **3× the close-group median per chunk on +the single-node path, 1× the winner pool's own 16-candidate median per padded +leaf on the merkle path.** + +The client needs ANT and Arbitrum gas, and approves the vault once. + +#### 4b. Decision: raise merkle to 3×, client-first, enforced from a fixed date + +Apply the same 3× multiplier on the merkle path, to the on-chain payable +`amount` rather than to the signed quote, so a batch settles +`3 × median16 × 2^depth`. Signed candidate prices and the pool hash stay +untouched, so every existing proof still verifies against the quotes the nodes +actually signed. + +Two PRs: ant-client#161 pays the multiplier, this one makes ant-node require it. +**The requirement is on by default** — no shadow mode, no follow-up flag — but +the two halves do not arrive at the same time, and the gap between them is what +makes the change safe. + +##### Five events, not one + +Most of the reasoning below is about ordering, so the events have to be kept +apart. Saying "the release" conflates all five: + +| Event | What it is | Timing | +|---|---|---| +| **PR landing** | ant-client#161 and this PR merge | either order, no user-visible effect | +| **Client publication** | a released ant-client that pays 3× | **must precede the boundary** | +| **Client adoption** | users actually running that client | rolling, never complete | +| **Node deployment** | operators running a node that knows the boundary | rolling, before or after the boundary | +| **Enforcement** | upgraded nodes start requiring 3× | **2026-08-04 15:00 UTC** (`1785855600`) | + +Only the last is a date in the code. The others are release and operations +events, and the boundary is chosen to sit after client publication with room +for adoption. + +##### Why client-first works + +The client's 3× is unconditional and immediate: from the first upload made with +the published client, batches settle at `3 × median16 × 2^depth`. It carries no +date and consults no flag. + +Nodes that have not upgraded accept those payments anyway, because they require +1× as a *minimum* and 3× clears it. So the client can go out — and be adopted — +well ahead of any node change, paying the network the intended amount the whole +time. That is the whole reason this ordering was chosen: the money is corrected +from the moment the client ships, and the node-side requirement only closes the +door behind it. + +##### The cutover, and the receipts it must not destroy + +A merkle receipt stays spendable for one week. Refusing every 1× settlement the +moment a node upgrades would therefore reject uploads whose payment had already +been made, correctly, under the previous rule, with no way to refund it. So the +requirement keys off **the receipt's own timestamp**, against a compiled-in +boundary (`MERKLE_PARITY_ENFORCED_FROM_UNIX` = `1785855600`, 2026-08-04 +15:00 UTC — after client publication, with room for adoption): + +| Receipt stamped | Must settle | +|---|---| +| before the boundary | 1× (the rule it was bought under) | +| at or after the boundary | 3× | + +Note what this keys off: not when the chunk was stored, not when the node +upgraded, but when the *receipt* was bought. A node deployed after the boundary +enforces immediately on fresh receipts and still honours the unexpired legacy +ones. + +Two properties make that a bounded compatibility window rather than an open +loophole: + +- **It self-retires.** Receipts older than one week are expired anyway, so once + `boundary + 1 week` has passed, every still-valid receipt is necessarily + stamped at or after the boundary and the 1× branch is unreachable. Nobody has + to remember to remove it; the branch becomes dead code that can be deleted. +- **Evasion is possible, but bounded and self-closing.** The timestamp is + client-chosen: it is passed to `payForMerkleTree`, and the contract only + checks that it is not in the future and not more than a week old. Quoting + nodes sign whatever stamp the request carries. So during the week after the + boundary, a *modified* client can deliberately stamp a receipt just before the + boundary, pay 1×, and be admitted — the same route the honest grace uses. What + closes it is expiry, not detection: a stamp before the boundary is unusable + once it is more than a week old, so the route disappears at + `boundary + 1 week` and cannot be extended. Nothing prevents backdating + *during* the window; it is accepted as the cost of not destroying legitimately + bought receipts, and it is bounded at one week of underpayment by clients who + went out of their way to underpay. + +What this does **not** protect: a client still running the *old* code after the +boundary builds a genuinely fresh 1× receipt, and every upgraded node refuses +it. That is the deliberate cost of enforcing rather than shadowing. The boundary +is placed after client publication precisely so adoption can run ahead of it, +and the failure is loud — the store is refused with the required multiplier +named — rather than a silent underpayment. Nodes that never upgrade keep +accepting 1×; enforcement is per-node, so the old client degrades gradually as +the fleet upgrades rather than failing everywhere at once. + +What that does and does not equalise, stated precisely, because "parity" is +easy to over-read: + +- It equalises **aggregate client spend per padded tree leaf**, at 3× a median. +- The two medians are **different quantities**: the single-node median is over + the close-group quotes the client supplied for that one chunk; the merkle + median is over the winner pool's 16 candidates for a midpoint address. Equal + multipliers do not make them equal prices. +- It does **not** equalise what any individual node earns. Merkle pays `depth` + candidates drawn from the batch's winner pool, not the storer of each chunk; + a node can store a merkle chunk and be paid nothing for it. Single-node pays + one close-group issuer per chunk. Both are lotteries, with different draws. +- Per *chunk* rather than per leaf, it is exact only at power-of-two batch + sizes. The tree pads to `2^ceil(log2(N))` and the contract charges for every + leaf, so 65 chunks pay for 128. Cost per chunk is + `3 × median16 × 2^depth / N`, from 3× up to just under 6× the median. This + padding premium predates the proposal and is unchanged by it. + +**Alternatives considered.** (1) Retain the differential and document it as a +volume discount. (2) Lower single-node to 1×, which cuts per-chunk revenue +network-wide by two thirds. (3) Raise merkle to 3× — **chosen**, and the +direction the multiplier was designed for. (4) Ship it in shadow mode first and +enforce later, which was the earlier plan and was rejected: it leaves the +network underpaid for as long as it takes someone to flip a flag, and the +timestamp boundary already provides the compatibility a shadow period was +meant to buy. Tripling the common large-upload path is a pricing decision +rather than an arithmetic repair, so (3) needs the economic owner's approval +before the client is published — the client, not the node, is what starts +charging 3×. + +### 5. Storing and verifying + +The PUT carries the payment proof. Every storer independently re-verifies, +before writing anything: the bundle's structure (**1 to 7 quotes** — a +single-quote bundle is valid, so "the median of seven" is the intended shape, +not an enforced one), the median selection, and then, **for the paid median +candidate only**: its content address, that its public key hashes to the peer +it claims, that the issuer is among the storer's own K closest peers for that +address, its ML-DSA-65 signature, and the on-chain settlement — which must be +at least 3× the median **and recorded for the quote's own rewards address**, so +a client cannot redirect the money to itself and still be admitted. + +Two limits of that check, both current behaviour: + +- **Non-paid quotes are not fully re-verified.** They position the median but + their signatures and content addresses are not checked, so a bad signature on + an unpaid quote is accepted (`test_legacy_unpaid_quote_bad_signature_accepted` + pins this). They still influence which quote gets paid. +- **The on-curve price check is off.** ADR-0004's + `price == calculate_price(committed_key_count)` recheck over every quote is + rollout-gated by `QUOTE_ARITHMETIC_RECHECK_ENABLED`, which is `false`; today + off-curve quotes are only logged. + +Merkle proofs are checked against the pool's on-chain record: every candidate's +signature, the pool's closeness to the midpoint, the tree proofs, and each paid +node's index, address and amount. The required per-node amount depends on the +receipt's own timestamp: **3× parity for receipts stamped from +`MERKLE_PARITY_ENFORCED_FROM_UNIX` onward, the historic 1× for receipts bought +before it** (see §4b). Since a receipt expires after a week, the 1× branch +becomes unreachable one lifetime past the boundary. + +The chunk then replicates to the close group; peers admit the fan-out under +the same proof. Later repair between neighbours carries no proof and is +authorized from network evidence instead. Each node keeps an LRU of verified +addresses and a persistent paid list, so a chunk already paid for at a node is +free to store there again. + +Payment verification cannot be turned off, and a node without a rewards +address does not start. + +### 6. What payment does not buy + +Payment is not proof of storage. Reads are free and unmetered. Whether a node +keeps what it was paid for is settled entirely by the audit and eviction path +(ADR-0002, ADR-0003, ADR-0007). The two layers meet in one place only: the +committed count that pricing binds to is the same artifact the audits check, +and a node that was actually paid is nominated for a first audit. + +### 7. Enforcement state at the time of writing + +| Gate | State at this head | +|---|---| +| Client resolve-before-pay quote binding | **enforced** | +| Node re-verification of the **paid median** quote (signature, content, K-closeness, settlement, payee address) | **enforced** | +| Node re-verification of **non-paid** quotes in the bundle | not done (bad signature on an unpaid quote is accepted) | +| Node rejection of off-curve quotes (`QUOTE_ARITHMETIC_RECHECK_ENABLED`) | off (telemetry only) | +| Quote/commitment mismatch reported to trust | off (telemetry only) | +| Receiver-side price floor (ADR-0006) | shadow (`ANT_PRICE_FLOOR_ENFORCE`, 50% tolerance) | +| Payee eligibility gate (ADR-0005) | **not merged**; observe-only on its branch (`ADR5_ENFORCE`) | +| Client pays the 3× multiplier on the merkle path | **enforced** by the published client, immediately and unconditionally (no flag, no date) | +| Storer requires it (`MERKLE_PARITY_ENFORCED_FROM_UNIX`) | **enforced** by upgraded nodes, for receipts stamped from 2026-08-04 15:00 UTC onward | + +So the guarantees once both halves are out are: the client pays only prices it +can recompute and resolve to a signed commitment; every **single-node** stored +chunk is settled on-chain at ≥3× the supplied-set median, to the quoting node's +own address; and every **merkle** chunk from an upgraded client is settled at +≥3× the winner pool's median per padded leaf. Upgraded nodes require that from +the boundary, except on receipts stamped before it, which keep the 1× rule until +they expire. Everything else in the table is still instrumented rather than +enforced. + +The parity telemetry no longer gates a decision — the decision has been made. +It now answers one operational question: how much traffic is still arriving on +pre-boundary receipts, which is how we know the compatibility window has drained +and the 1× branch is dead code that can be deleted. Traffic on pre-boundary +receipts *after* the boundary is also the only visibility we have into +deliberate backdating, since honest and evasive use of that branch are +indistinguishable from a single receipt. + +## Consequences + +### Positive + +- Anyone — client, storer, or observer — can recompute what a chunk should + cost from public data. There is no price oracle to attack and no per-node + price configuration to misconfigure. +- The curve carries a supply signal: as nodes fill, quotes rise and new + capacity is worth adding; an empty node still charges a spam barrier rather + than zero. +- Gas cost is one settlement per chunk, or one per 256-chunk batch, rather + than one per replica. +- Payment is verified by every storer against the chain, so a forged, + underpaid, or redirected payment is rejected everywhere it lands rather than + at one gatekeeper. + +### Negative / Trade-offs + +- **Batch uploads get three times more expensive.** Raising the merkle path + from 1× to 3× the median per padded leaf is a real price increase for every + upload of 64 chunks or more — the common path for files. The alternative, + lowering the single-node path to 1×, would instead cut per-chunk revenue + across the whole network by two thirds; parity had to be restored in one + direction or the other, and this is the one the 3× multiplier was designed + for. +- **An un-upgraded client's batch uploads start failing at the boundary.** It + pays 1× on a fresh receipt and upgraded storers refuse it. There is no way to + both require 3× and accept a new 1× payment, so this is the unavoidable cost + of enforcing by default rather than shadowing. The boundary sits after client + publication to give adoption time, and the rejection names the required + multiplier so the cause is obvious rather than looking like a random store + failure. Anyone pinned to an old client past the boundary is affected and + needs telling before it, not after. +- **The 1× branch is a one-week underpayment window for a determined client.** + Because the receipt timestamp is client-chosen and quoting nodes sign what + they are asked for, a modified client can keep paying 1× until + `boundary + 1 week` by stamping receipts just before the boundary. The window + is bounded and closes on its own, and the same branch is what protects honest + in-flight receipts, so it is accepted rather than fixed. It cannot be + narrowed without also refusing legitimately bought receipts. +- **The boundary is a compiled-in date.** It has to sit after the client is + published and after operators have had time to deploy nodes carrying it. If + the client release slips past it, upgraded nodes begin requiring 3× while + clients are still paying 1×, and batch uploads fail for everyone. If node + rollout slips, enforcement is simply thinner than intended for a while, which + is far less harmful. So the constant must be moved forward if the **client** + release slips, must never be set earlier than the client that pays it, and + changing it is a fleet-wide recompile — a node built with the old value keeps + the old behaviour. +- **Parity is exact only up to the contract's integer division.** The vault + computes `amountPerNode = total / depth`, which discards a remainder when + `depth` does not divide `median × 2^depth`. The loss is under one wei per + paid node per batch — economically nil, but it means the invariant is + "within rounding", not exact. +- **Batches still pay for padding leaves.** The tree rounds up to a power of + two and the contract charges for every leaf, so a batch of 65 chunks pays + for 128. Per chunk that is up to 2× the parity price, worst just above a + power-of-two boundary. Pre-existing and untouched here; closing it needs a + contract change, so it is recorded rather than fixed. +- **Node revenue is a lottery.** One of 7 quoters is paid per chunk, or + `depth` of a 16-node pool per batch. Fair in expectation over many chunks, + high variance for a small or new node — and a node earns nothing for merely + being online and healthy. +- **Price tracks the committed count, not cost or demand.** It does not + respond to bandwidth, hardware price, or how much clients actually want to + store, and USD cost per GiB moves one-for-one with the ANT price. +- **One payment funds unbounded future cost.** A node's storage, replication, + and audit costs continue indefinitely against a fixed past payment, with no + mechanism to reprice. +- **Median-of-K is only as honest as the quote set the client presents.** A + modified client can shop the neighbourhood and settle the cheapest valid + quote; the receiver-side floor is the counterweight and is not yet enforcing. + +### Neutral / Operational + +- The economic constants — `BASELINE`, `K`, `D`, the 3× multiplier, close + group 7, 16 candidates per pool, the 64-chunk merkle threshold, the one-week + receipt life — are compile-time. The price constants live in `ant-protocol`, + so client and node must move together; changing any of them is a coordinated + fleet change. +- ANT is an ERC-20 on Arbitrum; vault and token addresses are per-network + configuration. Clients need both ANT and gas. +- Cost per GiB is a function of chunk fill. Data that self-encrypts into small + chunks pays more per byte than data that fills 4 MiB chunks. + +## Validation + +- **Price parity.** Client and node import one `calculate_price`; round-trip + and monotonicity tests hold in `ant-protocol`. A second implementation of the + formula anywhere is a defect. +- **End to end against a real chain.** Anvil-backed tests pay and verify both + shapes, including the redirect-rejection and underpayment cases. **Still owed + before the client is published:** a client → vault → node test proving 3× + construction, settlement and acceptance against a real vault, plus the + just-below-3× and redirected-payee cases. Unit tests cover the cutover matrix + (1× refused after the boundary, 1× honoured before it, 3× accepted on both + sides) but not against a live chain. +- **The cutover is exercised in both directions.** Tests pin the boundary + explicitly rather than reading the production constant, because every merkle + receipt is stamped within a week of now — a test left on the real boundary + would silently change which regime it exercised once the wall clock crossed + that date. +- **The parity arithmetic is pinned by unit tests.** They assert that a merkle + **leaf** settles for the single-node 3× median at every tree depth; that the + per-node expectation is the floor of the multiplied total, which is *not* the + 1× expectation scaled (they differ by up to `multiplier - 1` wei, so a + refactor that divided before multiplying fails); and, client-side, that + applying the multiplier leaves the signed candidate prices and the pool hash + untouched. Any future change to either path must keep these passing; a + per-path price difference is a defect unless an ADR says otherwise. +- **Rollout telemetry has to be trustworthy before it is trusted.** The parity + line is emitted only after a proof's paid indices, addresses and amounts have + all validated, and only for store admissions, so rejected proofs and + paid-list replays cannot enter the signal. Cardinality is one line per + settlement per storer, not one per chunk: the line carries the pool hash, and + a bounded, pool-keyed first-emission cache — written only after that full + validation, and deliberately separate from the on-chain pool cache, which is + populated before it — drops the rest of the batch. Its check-and-insert is a + single locked operation, so the concurrent admissions a batch upload actually + produces still emit once. Regression tests pin all of it: zero events from an + invalid paid index, a wrong reward address, an underpayment or a paid-list + verification; one event from two chunks of one pool; one more from a second + pool; one from eight concurrent admissions of one pool. +- **Re-open triggers.** The ANT price moving enough to put $/GiB outside its + intended band; the **client** release slipping past the parity boundary date, + which requires moving the constant forward; node rollout not reaching enough + of the fleet before the boundary, which weakens enforcement without breaking + anything; a rise in refused batch uploads after the boundary, indicating + clients that never upgraded; parity telemetry still reporting pre-boundary + receipts more than a week after the boundary, which would mean the expiry + assumption is wrong; a volume of pre-boundary receipts during the window that + looks like deliberate backdating rather than honest in-flight traffic; + enabling any of the gates still listed as instrumented; any move to pay nodes + for uptime rather than for stores, which would make client payments no longer + the only revenue and invalidate this ADR's central assumption. + +## Notes for AI-assisted work + +AI tools may help draft this ADR, but **must not mark it Accepted without human +review**. Accepted ADRs are immutable: create a new superseding ADR rather than +editing an Accepted ADR. diff --git a/src/payment/verifier.rs b/src/payment/verifier.rs index 143366bc..a67e8bb8 100644 --- a/src/payment/verifier.rs +++ b/src/payment/verifier.rs @@ -176,6 +176,87 @@ fn median_quote_index(quote_count: usize) -> usize { quote_count / 2 } +/// The settlement multiplier a merkle receipt must satisfy, from the instant it +/// was stamped (ADR-0008). +/// +/// `PAID_QUOTE_PAYMENT_MULTIPLIER` (3x, matching the single-node path) for a +/// receipt stamped at or after `enforced_from`; the historic `1` for one +/// stamped before it, which was bought under the previous rule and cannot be +/// refunded. +/// +/// The boundary needs no sunset logic because receipt expiry supplies it: a +/// receipt older than one week is refused regardless, so once +/// `enforced_from + one week` has passed no valid receipt can still reach the +/// `1` branch. +/// +/// Expiry bounds backdating rather than preventing it. The stamp is +/// client-chosen and quoting nodes sign what they are asked for, so during the +/// week after `enforced_from` a modified client can stamp just before the +/// boundary and still settle at 1x — the same route honest in-flight receipts +/// take. It closes when such stamps expire, at `enforced_from + one week`, and +/// it cannot be narrowed without also refusing receipts bought in good faith +/// under the previous rule. +fn merkle_required_multiplier(receipt_timestamp: u64, enforced_from: u64) -> u64 { + if receipt_timestamp >= enforced_from { + PAID_QUOTE_PAYMENT_MULTIPLIER + } else { + 1 + } +} + +/// Per-node payment implied by the merkle contract formula, for a given +/// payment multiplier. +/// +/// The contract settles `total = median16(amount) x 2^depth` and splits it +/// evenly across the `depth` nodes it paid, so +/// `per_node = multiplier x median16(price) x 2^depth / depth`. +/// +/// `candidate_prices` are the candidates' **signed quoted** prices as they +/// appear in the proof — always 1x, because the multiplier is applied by the +/// client to the on-chain payable `amount` and never to the signed quote +/// (ADR-0008). Passing `multiplier = 1` therefore reproduces the historic +/// merkle expectation, and `PAID_QUOTE_PAYMENT_MULTIPLIER` the single-node +/// parity expectation. +/// +/// `median16` is the upper median (index `len / 2`), matching Solidity's +/// `median16` with `k = 8`. Depth `0` yields zero: there is nothing to pay. +/// +/// **Not linear in `multiplier`.** The trailing `/ depth` is integer division, +/// so `expected(m)` is `floor(m x median x 2^depth / depth)`, which is NOT +/// generally `m x expected(1)` — the two differ by up to `m - 1` wei whenever +/// `depth` does not divide `median x 2^depth`. For example median 901 at depth +/// 7 gives `expected(3) = 49426` but `3 x expected(1) = 49425`. Callers must +/// therefore ask for the multiplier they mean and compare against that value; +/// scaling a 1x result is wrong. The order here — multiply the total, then +/// divide once — is deliberate, and matches the contract, which computes +/// `totalAmount` before splitting it. +fn merkle_expected_per_node( + candidate_prices: &[Amount], + depth: u8, + multiplier: u64, +) -> Result { + if depth == 0 { + return Ok(Amount::ZERO); + } + + let mut sorted = candidate_prices.to_vec(); + sorted.sort_unstable(); // ascending + let median_price = *sorted + .get(sorted.len() / 2) + .ok_or_else(|| Error::Payment("empty candidate pool in merkle proof".into()))?; + + let leaves = 1u64 + .checked_shl(u32::from(depth)) + .ok_or_else(|| Error::Payment("merkle proof depth too large".into()))?; + + let total_amount = median_price + .checked_mul(Amount::from(leaves)) + .and_then(|total| total.checked_mul(Amount::from(multiplier))) + .ok_or_else(|| Error::Payment("merkle total payment overflow".into()))?; + + Ok(total_amount / Amount::from(u64::from(depth))) +} + fn payment_proof_type_label(payment_proof: Option<&[u8]>) -> &'static str { match payment_proof.and_then(detect_proof_type) { Some(ProofType::Merkle) => "merkle", @@ -328,6 +409,23 @@ impl PaymentStatus { /// Default capacity for the merkle pool cache (number of pool hashes to cache). const DEFAULT_POOL_CACHE_CAPACITY: usize = 1_000; +/// Capacity of the ADR-0008 parity-telemetry first-emission cache: how many +/// recently measured pool hashes this node remembers in order to emit one +/// sample per settlement instead of one per admitted chunk. +/// +/// Bounded, and deliberately its own cache rather than a flag on the pool +/// cache: the pool cache is populated as soon as the on-chain record is read, +/// which happens *before* the paid indices, addresses and amounts are checked, +/// so keying first-emission off it would let a proof that is rejected moments +/// later suppress (or produce) a sample. This one is written only after full +/// admission validation. +/// +/// LRU eviction means a pool that goes quiet for more than +/// `MERKLE_PARITY_TELEMETRY_CACHE_CAPACITY` distinct pools can be sampled a +/// second time. That is the intended trade: the cache bounds memory, and the +/// stream stays deduplicable by `pool` downstream. +const MERKLE_PARITY_TELEMETRY_CACHE_CAPACITY: usize = 1_000; + /// ADR-0004: max commitment sidecars processed per bundle. A legitimate bundle /// carries at most one commitment per quote/candidate — `CANDIDATES_PER_POOL` /// (16) is the larger of the single-node (`CLOSE_GROUP_SIZE` = 7) and merkle @@ -445,6 +543,30 @@ pub struct PaymentVerifier { /// future ops surface) can flip enforcement without rebuilding the /// verifier. price_floor: RwLock, + /// ADR-0008: the instant from which a merkle receipt must settle at the + /// single-node 3x multiplier, defaulting to + /// [`crate::replication::config::MERKLE_PARITY_ENFORCED_FROM_UNIX`]. + /// + /// Behind a lock purely so tests can pin it. Every merkle receipt is + /// necessarily stamped within one week of now (older ones are expired, + /// future ones refused), so a test that used the production boundary would + /// silently change which side of it the test data fell on as the wall clock + /// crossed that date. Pinning the boundary keeps those tests deterministic + /// forever. + merkle_parity_from: RwLock, + /// ADR-0008: pool hashes this node has already emitted a parity telemetry + /// line for. Bounded by + /// [`MERKLE_PARITY_TELEMETRY_CACHE_CAPACITY`]; see that constant for why + /// this is separate from [`Self::pool_cache`]. + /// + /// Check-and-insert happens under this one lock, so concurrent admissions + /// of different chunks from the same batch emit exactly one line between + /// them rather than one each. + merkle_parity_logged: Mutex>, + /// Count of ADR-0008 parity lines this verifier has actually emitted. + /// Incremented only on first emission for a pool, so it measures + /// settlements sampled rather than chunks admitted. + merkle_parity_emissions: std::sync::atomic::AtomicUsize, /// Configuration. config: PaymentVerifierConfig, } @@ -574,6 +696,14 @@ impl PaymentVerifier { monetized_pin_tx: RwLock::new(None), local_commitment_source: RwLock::new(None), price_floor: RwLock::new(config.price_floor), + merkle_parity_from: RwLock::new( + crate::replication::config::MERKLE_PARITY_ENFORCED_FROM_UNIX, + ), + merkle_parity_logged: Mutex::new(LruCache::new( + NonZeroUsize::new(MERKLE_PARITY_TELEMETRY_CACHE_CAPACITY) + .unwrap_or(NonZeroUsize::MIN), + )), + merkle_parity_emissions: std::sync::atomic::AtomicUsize::new(0), config, } } @@ -664,6 +794,28 @@ impl PaymentVerifier { *self.price_floor.write() = config; } + /// Test-only setter for the ADR-0008 merkle parity boundary, so both the + /// pre-boundary (historic 1x) and enforcing (3x) regimes can be exercised + /// against fixed receipt timestamps instead of the wall clock. `0` enforces + /// on every receipt; `u64::MAX` puts every receipt in the legacy window. + #[cfg(any(test, feature = "test-utils"))] + pub fn set_merkle_parity_from_for_tests(&self, enforced_from: u64) { + *self.merkle_parity_from.write() = enforced_from; + } + + /// Number of ADR-0008 parity telemetry lines this verifier has emitted. + /// + /// One per settlement pool this node admitted a store for — not one per + /// chunk, and none at all for proofs that failed validation or for + /// non-store contexts. Test-only surface for the cardinality regression + /// tests; the production signal is the log line itself. + #[cfg(any(test, feature = "test-utils"))] + #[must_use] + pub fn merkle_parity_emission_count(&self) -> usize { + self.merkle_parity_emissions + .load(std::sync::atomic::Ordering::Relaxed) + } + /// Test-only setter for local closest peers used by the paid-quote /// issuer K-closest check. #[cfg(any(test, feature = "test-utils"))] @@ -2064,6 +2216,81 @@ impl PaymentVerifier { } } + /// ADR-0008 merkle payment-parity telemetry. + /// + /// Records what a batch actually settled per paid node against what + /// single-node parity requires, and which regime the receipt's timestamp put + /// it in. Parity is enforced, so a post-boundary shortfall has already been + /// rejected before this runs and cannot appear here; what this measures is + /// how much traffic is still arriving on pre-boundary (1x) receipts, which + /// is what tells us the compatibility window can be considered closed. + /// Observational only: it never rejects and never feeds trust scoring — a + /// client on a legacy receipt is running old code, not misbehaving. + /// + /// Two call-site requirements, both load-bearing for the signal: + /// - **After full admission validation.** Every paid index, reward address + /// and amount must already have been checked. Logging earlier lets a + /// proof that is rejected moments later enter the rollout signal, which + /// would understate parity with samples that never resulted in a store. + /// - **Store admissions only.** A paid-list admission reprices nothing and + /// would double-count a batch already measured when it was stored. + /// + /// Keyed by `pool_hash`, not by chunk address: one on-chain settlement + /// covers up to 256 chunks, so a line per chunk would emit 256 duplicate + /// samples of one economic event and let a big batch outvote a small one. + /// + /// Carrying the pool hash in the line is not enough on its own — it makes + /// the stream *deduplicable* but leaves the emitted cardinality at one line + /// per chunk. So the first emission for a pool is recorded in + /// [`Self::merkle_parity_logged`] and later admissions from the same batch + /// return without logging. The check-and-insert is a single locked + /// operation, so concurrent admissions from one batch also produce exactly + /// one line. "Once per pool" is per node/verifier: every storer that admits + /// a chunk from the batch still contributes its own sample, which is what + /// makes the signal a measure of traffic reaching the fleet. + fn log_merkle_parity( + &self, + pool_hash: PoolHash, + payment_info: &OnChainPaymentInfo, + expected_bare: Amount, + expected_parity: Amount, + required_multiplier: u64, + ) { + // Check-and-insert under one lock: the loser of a race sees its own + // insert report a prior entry and stays quiet. + if self + .merkle_parity_logged + .lock() + .put(pool_hash, ()) + .is_some() + { + return; + } + self.merkle_parity_emissions + .fetch_add(1, std::sync::atomic::Ordering::Relaxed); + + let min_paid = payment_info + .paid_node_addresses + .iter() + .map(|(_, _, amount)| *amount) + .min() + .unwrap_or(Amount::ZERO); + let at_parity = min_paid >= expected_parity; + let legacy_receipt = required_multiplier < PAID_QUOTE_PAYMENT_MULTIPLIER; + + info!( + target: "ant_node::payment::merkle_parity", + "Merkle parity: pool={}, depth={}, paid_nodes={}, receipt_ts={}, \ + min_paid={min_paid}, expected_bare={expected_bare}, \ + expected_parity={expected_parity}, required_multiplier={required_multiplier}, \ + legacy_receipt={legacy_receipt}, at_parity={at_parity}", + hex::encode(pool_hash), + payment_info.depth, + payment_info.paid_node_addresses.len(), + payment_info.merkle_payment_timestamp, + ); + } + /// Pure curve-canonicality predicate: does `price` lie exactly on the /// pricing curve? Equivalent to "there exists some non-negative integer /// `n` such that `calculate_price(n) == price`". @@ -2749,38 +2976,41 @@ impl PaymentVerifier { ))); } - // Compute expected per-node payment using the contract formula: - // totalAmount = median16(candidate_prices) * (1 << depth) - // amountPerNode = totalAmount / depth - let expected_per_node = if payment_info.depth > 0 { - let mut candidate_prices: Vec = merkle_proof - .winner_pool - .candidate_nodes - .iter() - .map(|c| c.price) - .collect(); - candidate_prices.sort_unstable(); // ascending - // Upper median (index 8 of 16) — matches Solidity's median16 (k = 8) - let median_price = *candidate_prices - .get(candidate_prices.len() / 2) - .ok_or_else(|| Error::Payment("empty candidate pool in merkle proof".into()))?; - let shift = u32::from(payment_info.depth); - let multiplier = 1u64 - .checked_shl(shift) - .ok_or_else(|| Error::Payment("merkle proof depth too large".into()))?; - let total_amount = median_price * Amount::from(multiplier); - total_amount / Amount::from(u64::from(payment_info.depth)) - } else { - Amount::ZERO - }; + // Expected per-node payment from the contract formula. ADR-0008: which + // multiplier is REQUIRED depends on when the receipt was stamped — + // 3x from the parity boundary onward, the historic 1x before it, since + // that money was already spent under the old rule. The expiry check + // above bounds the legacy branch: no receipt older than a week is + // valid, so the branch is unreachable one week past the boundary. + let candidate_prices: Vec = merkle_proof + .winner_pool + .candidate_nodes + .iter() + .map(|c| c.price) + .collect(); + let parity_from = *self.merkle_parity_from.read(); + let required_multiplier = + merkle_required_multiplier(payment_info.merkle_payment_timestamp, parity_from); + let expected_per_node_bare = + merkle_expected_per_node(&candidate_prices, payment_info.depth, 1)?; + let expected_per_node_parity = merkle_expected_per_node( + &candidate_prices, + payment_info.depth, + PAID_QUOTE_PAYMENT_MULTIPLIER, + )?; + let expected_per_node = + merkle_expected_per_node(&candidate_prices, payment_info.depth, required_multiplier)?; // Verify paid node indices, addresses, and amounts against the candidate pool. // // Each paid node must: // 1. Have a valid index within the candidate pool // 2. Match the expected reward address at that index - // 3. Have been paid at least the expected per-node amount from the - // contract formula: median16(prices) * 2^depth / depth + // 3. Have been paid at least `expected_per_node`, which is the contract + // formula `median16(prices) * 2^depth / depth` at the multiplier the + // receipt's own timestamp selects (3x from the parity boundary, the + // historic 1x before it). Do NOT re-assume the 1x formula here — + // read `expected_per_node` rather than recomputing it. // // Note: unlike single-node payments, merkle proofs are NOT bound to a // specific storing node. The contract pays `depth` random nodes from the @@ -2810,8 +3040,9 @@ impl PaymentVerifier { return Err(Error::Payment(format!( "Underpayment for node at index {idx}: paid {paid_amount}, \ expected at least {expected_per_node} \ - (median16 formula, depth={})", - payment_info.depth + (median16 formula, depth={}, {required_multiplier}x required for a \ + receipt stamped {} vs parity boundary {parity_from})", + payment_info.depth, payment_info.merkle_payment_timestamp ))); } } @@ -2824,6 +3055,24 @@ impl PaymentVerifier { ); } + // ADR-0008 parity telemetry, emitted only now: every paid index, reward + // address and amount above has been validated, so a proof that is + // rejected later in this function can never enter the rollout signal. + // Store admissions only, matching the cross-check below — a paid-list + // receipt reprices no fresh economic decision and would otherwise + // double-count a batch already measured at its store admission. + // `log_merkle_parity` itself drops all but the first admission per + // pool, so a 256-chunk batch contributes one line here, not 256. + if context.is_store_admission() { + self.log_merkle_parity( + pool_hash, + &payment_info, + expected_per_node_bare, + expected_per_node_parity, + required_multiplier, + ); + } + // ADR-0004: route the merkle-batch candidates through the SAME // cross-check + first-audit funnel as single-node quotes, AFTER on-chain // verification has succeeded (so an unpaid pool cannot drive audits or @@ -2979,6 +3228,142 @@ mod tests { assert!(issuer_closeness_width > close_group_size); } + /// 16 candidate prices 100..=1600; the upper median (index 8) is 900. + fn varied_candidate_prices() -> Vec { + (1..=16u64).map(|i| Amount::from(i * 100)).collect() + } + + #[test] + fn merkle_expected_per_node_reproduces_the_contract_formula() { + // total = median16 x 2^depth = 900 x 16 = 14400, over depth=4 nodes. + let per_node = merkle_expected_per_node(&varied_candidate_prices(), 4, 1) + .expect("depth 4 pool is payable"); + assert_eq!(per_node, Amount::from(3_600u64)); + } + + /// 16 identical candidate prices, so the upper median is exactly `median`. + fn prices_with_median(median: u64) -> Vec { + vec![Amount::from(median); 16] + } + + /// The parity expectation is the floor of the multiplied total, which is + /// **not** the bare expectation times the multiplier. + /// + /// `expected(m) = floor(m x median x 2^depth / depth)` sits in + /// `[m x expected(1), m x expected(1) + (m - 1)]`: equal when `depth` + /// divides the total, up to `m - 1` wei above it when it does not. + /// Asserting exact equality would encode `floor(3x/d) == 3 x floor(x/d)`, + /// which is false in general and only happened to hold for the + /// evenly-dividing case the first version of this test used. + #[test] + fn merkle_expected_per_node_scales_within_the_division_remainder() { + let multiplier = PAID_QUOTE_PAYMENT_MULTIPLIER; + + for median in [1u64, 899, 900, 901, 6_000, 1_000_003] { + for depth in 1..=8u8 { + let prices = prices_with_median(median); + let bare = merkle_expected_per_node(&prices, depth, 1) + .expect("bare expectation is payable"); + let parity = merkle_expected_per_node(&prices, depth, multiplier) + .expect("parity expectation is payable"); + + let scaled_bare = bare * Amount::from(multiplier); + let slack = Amount::from(multiplier - 1); + + assert!( + parity >= scaled_bare && parity <= scaled_bare + slack, + "median {median} depth {depth}: parity {parity} must lie in \ + [{scaled_bare}, {}]", + scaled_bare + slack + ); + + // And it is exactly the floor of the multiplied total. + let leaves = 1u64 << u32::from(depth); + let expected = + Amount::from(median * leaves * multiplier) / Amount::from(u64::from(depth)); + assert_eq!(parity, expected, "median {median} depth {depth}"); + } + } + } + + /// Regression for the arithmetic order: the helper multiplies the total and + /// then divides once, as the contract does. Median 901 at depth 7 is a case + /// where that differs from scaling a 1x result by one wei, so a refactor + /// that divided first would fail here. + #[test] + fn merkle_expected_per_node_multiplies_before_dividing() { + let prices = prices_with_median(901); + let bare = merkle_expected_per_node(&prices, 7, 1).expect("bare expectation is payable"); + let parity = merkle_expected_per_node(&prices, 7, PAID_QUOTE_PAYMENT_MULTIPLIER) + .expect("parity expectation is payable"); + + assert_eq!(bare, Amount::from(16_475u64)); + assert_eq!(parity, Amount::from(49_426u64)); + assert_eq!( + parity, + bare * Amount::from(PAID_QUOTE_PAYMENT_MULTIPLIER) + Amount::from(1u64), + "dividing before multiplying would lose this wei" + ); + } + + /// The economic invariant ADR-0008 restores: a merkle **leaf** settles + /// for what the single-node path settles — 3x the median quote — up to + /// the contract's integer division. + /// + /// Leaf, not chunk: the tree pads to `2^ceil(log2(N))` and the contract + /// charges for every leaf, so a batch whose size is not a power of two + /// pays a padding premium on top. That premium is a property of the + /// contract formula, not of the multiplier under test here. + /// + /// `amountPerNode = totalAmount / depth` discards a remainder whenever + /// `depth` does not divide `median x 2^depth` (e.g. depth 7). The loss is + /// strictly under one wei per paid node, spread across `2^depth` chunks, + /// so it is economically irrelevant — but the invariant is "within + /// rounding", not exact equality, and the test says so rather than + /// picking depths that happen to divide evenly. + #[test] + fn merkle_parity_per_chunk_equals_single_node_settlement() { + let prices = varied_candidate_prices(); + let median = Amount::from(900u64); + + for depth in 1..=8u8 { + let per_node = merkle_expected_per_node(&prices, depth, PAID_QUOTE_PAYMENT_MULTIPLIER) + .expect("every supported depth is payable"); + let required_total = per_node * Amount::from(u64::from(depth)); + + let leaves = Amount::from(1u64 << u32::from(depth)); + let ideal_total = median * Amount::from(PAID_QUOTE_PAYMENT_MULTIPLIER) * leaves; + + assert!( + required_total <= ideal_total, + "at depth {depth} the required total {required_total} must not \ + exceed the 3x-median ideal {ideal_total}" + ); + assert!( + ideal_total - required_total < Amount::from(u64::from(depth)), + "at depth {depth} the shortfall {} must be pure division \ + remainder (< depth wei), not a missing multiplier", + ideal_total - required_total + ); + } + } + + #[test] + fn merkle_expected_per_node_is_zero_at_depth_zero() { + let per_node = merkle_expected_per_node(&varied_candidate_prices(), 0, 3) + .expect("depth zero is a valid no-op, not an error"); + assert_eq!(per_node, Amount::ZERO); + } + + #[test] + fn merkle_expected_per_node_rejects_an_empty_pool() { + let Err(err) = merkle_expected_per_node(&[], 4, 3) else { + panic!("an empty candidate pool has no median and must not be payable"); + }; + let err = err.to_string(); + assert!(err.contains("empty candidate pool"), "got: {err}"); + } + fn make_signed_quote( xorname: XorName, price: Amount, @@ -4487,7 +4872,7 @@ mod tests { async fn test_merkle_tagged_proof_invalid_data_rejected() { use crate::ant_protocol::PROOF_TAG_MERKLE; - let verifier = create_test_verifier(); + let verifier = merkle_test_verifier(); let xorname = [0xA1u8; 32]; // Build a merkle-tagged proof with garbage body. @@ -5001,6 +5386,28 @@ mod tests { /// Helper: build a minimal valid `MerklePaymentProof` with real ML-DSA-65 /// signatures. Returns `(xorname, serialized_tagged_proof, pool_hash, timestamp)`. + /// Verifier for merkle admission tests, pinned to the **enforcing** side of + /// the ADR-0008 parity boundary. + /// + /// Merkle receipts must be stamped within one week of now or they are + /// expired, so these tests necessarily build near-`now` timestamps. Left on + /// the production boundary they would sit in the legacy 1x window today and + /// silently switch to the 3x rule when the wall clock crossed that date, + /// changing what each assertion means. Pinning to `0` means "every receipt + /// is post-boundary", which is the rule the release ships with. + fn merkle_test_verifier() -> PaymentVerifier { + let verifier = create_test_verifier(); + verifier.set_merkle_parity_from_for_tests(0); + verifier + } + + /// Per-node amount a depth-2 pool of 1024-priced candidates must settle + /// under parity: `median(1024) * 2^2 * 3 / 2`. + const MERKLE_PARITY_PER_NODE_DEPTH2: u64 = 6144; + + /// The same pool's historic pre-parity amount: `median(1024) * 2^2 / 2`. + const MERKLE_LEGACY_PER_NODE_DEPTH2: u64 = 2048; + fn make_valid_merkle_proof_bytes() -> ( [u8; 32], Vec, @@ -5015,7 +5422,7 @@ mod tests { #[tokio::test] async fn test_merkle_address_mismatch_rejected() { - let verifier = create_test_verifier(); + let verifier = merkle_test_verifier(); let (_correct_xorname, tagged_proof, _pool_hash, _ts) = make_valid_merkle_proof_bytes(); // Use a DIFFERENT xorname than what the proof was built for @@ -5042,7 +5449,7 @@ mod tests { #[tokio::test] async fn test_merkle_malformed_body_rejected() { - let verifier = create_test_verifier(); + let verifier = merkle_test_verifier(); let xorname = [0xA3u8; 32]; // Valid merkle tag but truncated/corrupted msgpack body @@ -5195,7 +5602,7 @@ mod tests { #[tokio::test] async fn test_merkle_tampered_candidate_signature_rejected() { - let verifier = create_test_verifier(); + let verifier = merkle_test_verifier(); let (mut merkle_proof, _pool_hash, xorname, timestamp) = make_valid_merkle_proof(); @@ -5242,7 +5649,7 @@ mod tests { #[tokio::test] async fn test_merkle_timestamp_mismatch_rejected() { - let verifier = create_test_verifier(); + let verifier = merkle_test_verifier(); let (xorname, tagged, pool_hash, timestamp) = make_valid_merkle_proof_bytes(); @@ -5274,7 +5681,7 @@ mod tests { #[tokio::test] async fn test_merkle_paid_node_index_out_of_bounds_rejected() { - let verifier = create_test_verifier(); + let verifier = merkle_test_verifier(); let (xorname, tagged_proof, pool_hash, ts) = make_valid_merkle_proof_bytes(); // The test tree has 4 addresses → depth 2. We must match the tree depth @@ -5285,11 +5692,19 @@ mod tests { depth: 2, merkle_payment_timestamp: ts, paid_node_addresses: vec![ - // First paid node: valid (matches candidate 0, amount matches formula) - // Expected per-node: median(1024) * 2^2 / 2 = 2048 - (RewardsAddress::new([0u8; 20]), 0, Amount::from(2048u64)), + // First paid node: valid (matches candidate 0, amount clears + // the parity formula so the test reaches the index check) + ( + RewardsAddress::new([0u8; 20]), + 0, + Amount::from(MERKLE_PARITY_PER_NODE_DEPTH2), + ), // Second paid node: index 999 is way beyond CANDIDATES_PER_POOL (16) - (RewardsAddress::new([1u8; 20]), 999, Amount::from(2048u64)), + ( + RewardsAddress::new([1u8; 20]), + 999, + Amount::from(MERKLE_PARITY_PER_NODE_DEPTH2), + ), ], }; verifier.pool_cache.lock().put(pool_hash, info); @@ -5316,7 +5731,7 @@ mod tests { #[tokio::test] async fn test_merkle_paid_node_address_mismatch_rejected() { - let verifier = create_test_verifier(); + let verifier = merkle_test_verifier(); let (xorname, tagged_proof, pool_hash, ts) = make_valid_merkle_proof_bytes(); // Tree has depth 2, so provide 2 paid node entries. @@ -5326,11 +5741,19 @@ mod tests { depth: 2, merkle_payment_timestamp: ts, paid_node_addresses: vec![ - // Index 0 with matching address [0x00; 20] - // Expected per-node: median(1024) * 2^2 / 2 = 2048 - (RewardsAddress::new([0u8; 20]), 0, Amount::from(2048u64)), + // Index 0 with matching address [0x00; 20], paid enough to + // clear the parity formula so the test reaches index 1 + ( + RewardsAddress::new([0u8; 20]), + 0, + Amount::from(MERKLE_PARITY_PER_NODE_DEPTH2), + ), // Index 1 with WRONG address — candidate 1's address is [0x01; 20] - (RewardsAddress::new([0xFF; 20]), 1, Amount::from(2048u64)), + ( + RewardsAddress::new([0xFF; 20]), + 1, + Amount::from(MERKLE_PARITY_PER_NODE_DEPTH2), + ), ], }; verifier.pool_cache.lock().put(pool_hash, info); @@ -5354,7 +5777,7 @@ mod tests { #[tokio::test] async fn test_merkle_wrong_depth_rejected() { - let verifier = create_test_verifier(); + let verifier = merkle_test_verifier(); let (xorname, tagged_proof, pool_hash, ts) = make_valid_merkle_proof_bytes(); // Pre-populate pool cache with depth=3 but only 1 paid node address @@ -5394,7 +5817,7 @@ mod tests { #[tokio::test] async fn test_merkle_underpayment_rejected() { - let verifier = create_test_verifier(); + let verifier = merkle_test_verifier(); let (xorname, tagged_proof, pool_hash, ts) = make_valid_merkle_proof_bytes(); // Tree depth=2, so 2 paid nodes required. Candidates all quote price=1024. @@ -5431,8 +5854,547 @@ mod tests { ); } + /// A batch settled at the historic 1x is refused once its receipt is stamped + /// at or after the parity boundary. This is the fix being active by default: + /// no flag, no shadow mode. + #[tokio::test] + async fn merkle_legacy_1x_settlement_rejected_after_the_parity_boundary() { + let verifier = merkle_test_verifier(); // boundary pinned at 0 = always enforcing + let (xorname, tagged_proof, pool_hash, ts) = make_valid_merkle_proof_bytes(); + + { + let info = evmlib::merkle_payments::OnChainPaymentInfo { + depth: 2, + merkle_payment_timestamp: ts, + paid_node_addresses: vec![ + ( + RewardsAddress::new([0u8; 20]), + 0, + Amount::from(MERKLE_LEGACY_PER_NODE_DEPTH2), + ), + ( + RewardsAddress::new([1u8; 20]), + 1, + Amount::from(MERKLE_LEGACY_PER_NODE_DEPTH2), + ), + ], + }; + verifier.pool_cache.lock().put(pool_hash, info); + } + + let err_msg = format!( + "{}", + verifier + .verify_payment( + &xorname, + Some(&tagged_proof), + VerificationContext::ClientPut, + ) + .await + .expect_err("a 1x settlement must be refused past the boundary") + ); + assert!( + err_msg.contains("Underpayment") && err_msg.contains("3x required"), + "Error should name the required multiplier: {err_msg}" + ); + } + + /// A receipt stamped BEFORE the boundary keeps its 1x price. The money was + /// already spent under the old rule and cannot be refunded, so refusing it + /// would destroy value. This is the compatibility half of the cutover. + #[tokio::test] + async fn merkle_legacy_1x_settlement_accepted_before_the_parity_boundary() { + let verifier = create_test_verifier(); + let (xorname, tagged_proof, pool_hash, ts) = make_valid_merkle_proof_bytes(); + // Boundary just after this receipt's stamp: the receipt is legacy. + verifier.set_merkle_parity_from_for_tests(ts + 1); + + { + let info = evmlib::merkle_payments::OnChainPaymentInfo { + depth: 2, + merkle_payment_timestamp: ts, + paid_node_addresses: vec![ + ( + RewardsAddress::new([0u8; 20]), + 0, + Amount::from(MERKLE_LEGACY_PER_NODE_DEPTH2), + ), + ( + RewardsAddress::new([1u8; 20]), + 1, + Amount::from(MERKLE_LEGACY_PER_NODE_DEPTH2), + ), + ], + }; + verifier.pool_cache.lock().put(pool_hash, info); + } + + let result = verifier + .verify_payment( + &xorname, + Some(&tagged_proof), + VerificationContext::ClientPut, + ) + .await; + assert!( + result.is_ok(), + "a receipt stamped before the boundary keeps the 1x rule: {result:?}" + ); + } + + /// An upgraded client's 3x settlement is accepted on both sides of the + /// boundary. This is what makes the client-first rollout work: the client + /// pays 3x from the day it ships, and neither a node that predates the + /// boundary nor one enforcing past it has any reason to refuse. + #[tokio::test] + async fn merkle_parity_settlement_accepted_in_both_regimes() { + for parity_from in [0u64, u64::MAX] { + let verifier = create_test_verifier(); + verifier.set_merkle_parity_from_for_tests(parity_from); + let (xorname, tagged_proof, pool_hash, ts) = make_valid_merkle_proof_bytes(); + + { + let info = evmlib::merkle_payments::OnChainPaymentInfo { + depth: 2, + merkle_payment_timestamp: ts, + paid_node_addresses: vec![ + ( + RewardsAddress::new([0u8; 20]), + 0, + Amount::from(MERKLE_PARITY_PER_NODE_DEPTH2), + ), + ( + RewardsAddress::new([1u8; 20]), + 1, + Amount::from(MERKLE_PARITY_PER_NODE_DEPTH2), + ), + ], + }; + verifier.pool_cache.lock().put(pool_hash, info); + } + + let result = verifier + .verify_payment( + &xorname, + Some(&tagged_proof), + VerificationContext::ClientPut, + ) + .await; + assert!( + result.is_ok(), + "3x must be accepted with boundary {parity_from}: {result:?}" + ); + } + } + + /// One chunk's payment proof as the verifier receives it: the stored + /// address and the tagged, serialized proof bytes. + type TaggedProof = ([u8; 32], Vec); + + /// Build valid proofs for several addresses of ONE tree, all resolving to + /// the same winner pool hash — the shape a real batch upload has, and the + /// case the ADR-0008 parity telemetry must collapse to a single sample. + /// Returns `(Vec<(xorname, tagged_proof)>, pool_hash, timestamp)`. + fn make_valid_merkle_proofs_for_one_pool( + indices: &[usize], + ) -> ( + Vec, + evmlib::merkle_batch_payment::PoolHash, + u64, + ) { + use evmlib::merkle_payments::{MerklePaymentCandidatePool, MerklePaymentProof, MerkleTree}; + + let timestamp = std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .expect("system time") + .as_secs(); + + let addresses: Vec = (0..4u8) + .map(|i| xor_name::XorName::from_content(&[i])) + .collect(); + let tree = MerkleTree::from_xornames(addresses.clone()).expect("tree"); + + let midpoint_proof = tree + .reward_candidates(timestamp) + .expect("reward candidates") + .first() + .expect("at least one candidate") + .clone(); + let pool = MerklePaymentCandidatePool { + midpoint_proof, + candidate_nodes: make_candidate_nodes(timestamp), + }; + + let mut proofs = Vec::with_capacity(indices.len()); + let mut pool_hash = None; + for &index in indices { + let address = *addresses.get(index).expect("address index within the tree"); + let address_proof = tree + .generate_address_proof(index, address) + .expect("address proof"); + let proof = MerklePaymentProof::new(address, address_proof, pool.clone()); + let hash = proof.winner_pool_hash(); + match pool_hash { + None => pool_hash = Some(hash), + Some(first) => assert_eq!( + hash, first, + "every address in one tree must share one winner pool" + ), + } + let tagged = + crate::payment::proof::serialize_merkle_proof(&proof).expect("serialize proof"); + proofs.push((address.0, tagged)); + } + + (proofs, pool_hash.expect("at least one proof"), timestamp) + } + + /// The depth-2 test pool's on-chain record, settled at 3x parity so the + /// proof is admitted and the telemetry call site is actually reached. + fn parity_settled_pool_info(ts: u64) -> evmlib::merkle_payments::OnChainPaymentInfo { + evmlib::merkle_payments::OnChainPaymentInfo { + depth: 2, + merkle_payment_timestamp: ts, + paid_node_addresses: vec![ + ( + RewardsAddress::new([0u8; 20]), + 0, + Amount::from(MERKLE_PARITY_PER_NODE_DEPTH2), + ), + ( + RewardsAddress::new([1u8; 20]), + 1, + Amount::from(MERKLE_PARITY_PER_NODE_DEPTH2), + ), + ], + } + } + + // ========================================================================= + // ADR-0008 parity telemetry: what may enter the rollout signal, and at what + // cardinality. The signal is read to decide when the pre-boundary (1x) + // compatibility window has drained, so a sample from a proof that was + // rejected — or 256 samples from one settlement — would mislead that call. // ========================================================================= - // Closeness-window constants regression tests + + /// A proof naming a paid index outside the pool is rejected in the paid-node + /// loop, before the telemetry call site. Nothing may be measured. + #[tokio::test] + async fn parity_telemetry_silent_when_a_paid_index_is_invalid() { + let verifier = merkle_test_verifier(); + let (xorname, tagged_proof, pool_hash, ts) = make_valid_merkle_proof_bytes(); + + { + let mut info = parity_settled_pool_info(ts); + // Second entry points past CANDIDATES_PER_POOL (16). + if let Some(entry) = info.paid_node_addresses.get_mut(1) { + entry.1 = 999; + } + verifier.pool_cache.lock().put(pool_hash, info); + } + + let result = verifier + .verify_payment( + &xorname, + Some(&tagged_proof), + VerificationContext::ClientPut, + ) + .await; + + assert!( + result.is_err(), + "an out-of-bounds paid index must be refused" + ); + assert_eq!( + verifier.merkle_parity_emission_count(), + 0, + "a refused proof must not enter the parity signal" + ); + } + + /// A paid entry whose reward address does not match the candidate at that + /// index is rejected before the telemetry call site. + #[tokio::test] + async fn parity_telemetry_silent_when_a_reward_address_is_wrong() { + let verifier = merkle_test_verifier(); + let (xorname, tagged_proof, pool_hash, ts) = make_valid_merkle_proof_bytes(); + + { + let mut info = parity_settled_pool_info(ts); + // Candidate 1's address is [0x01; 20], not [0xFF; 20]. + if let Some(entry) = info.paid_node_addresses.get_mut(1) { + entry.0 = RewardsAddress::new([0xFF; 20]); + } + verifier.pool_cache.lock().put(pool_hash, info); + } + + let result = verifier + .verify_payment( + &xorname, + Some(&tagged_proof), + VerificationContext::ClientPut, + ) + .await; + + assert!(result.is_err(), "a redirected payee must be refused"); + assert_eq!( + verifier.merkle_parity_emission_count(), + 0, + "a refused proof must not enter the parity signal" + ); + } + + /// An underpaid batch is the case the signal most obviously must not + /// contain: it is rejected, so it never became a store. + #[tokio::test] + async fn parity_telemetry_silent_on_underpayment() { + let verifier = merkle_test_verifier(); + let (xorname, tagged_proof, pool_hash, ts) = make_valid_merkle_proof_bytes(); + + { + let info = evmlib::merkle_payments::OnChainPaymentInfo { + depth: 2, + merkle_payment_timestamp: ts, + paid_node_addresses: vec![ + (RewardsAddress::new([0u8; 20]), 0, Amount::from(1u64)), + (RewardsAddress::new([1u8; 20]), 1, Amount::from(1u64)), + ], + }; + verifier.pool_cache.lock().put(pool_hash, info); + } + + let result = verifier + .verify_payment( + &xorname, + Some(&tagged_proof), + VerificationContext::ClientPut, + ) + .await; + + assert!(result.is_err(), "an underpaid batch must be refused"); + assert_eq!( + verifier.merkle_parity_emission_count(), + 0, + "a refused proof must not enter the parity signal" + ); + } + + /// A paid-list admission verifies the same proof but reprices nothing. It + /// must not be measured, or a batch already sampled at its store admission + /// would be counted twice. + #[tokio::test] + async fn parity_telemetry_silent_for_paid_list_admission() { + let verifier = merkle_test_verifier(); + let (xorname, tagged_proof, pool_hash, ts) = make_valid_merkle_proof_bytes(); + verifier + .pool_cache + .lock() + .put(pool_hash, parity_settled_pool_info(ts)); + + let result = verifier + .verify_payment( + &xorname, + Some(&tagged_proof), + VerificationContext::PaidListAdmission, + ) + .await; + + assert!(result.is_ok(), "the proof itself is valid: {result:?}"); + assert_eq!( + verifier.merkle_parity_emission_count(), + 0, + "only store admissions may be measured" + ); + } + + /// Two chunks of one batch are one economic event. The pool hash in the + /// line makes the stream deduplicable; the first-emission cache is what + /// keeps the emitted cardinality at one. + #[tokio::test] + async fn parity_telemetry_emits_once_for_two_chunks_of_one_pool() { + let verifier = merkle_test_verifier(); + let (proofs, pool_hash, ts) = make_valid_merkle_proofs_for_one_pool(&[0, 1]); + verifier + .pool_cache + .lock() + .put(pool_hash, parity_settled_pool_info(ts)); + + for (xorname, tagged) in &proofs { + let result = verifier + .verify_payment(xorname, Some(tagged), VerificationContext::ClientPut) + .await; + assert!(result.is_ok(), "both chunks must be admitted: {result:?}"); + } + + assert_eq!( + verifier.merkle_parity_emission_count(), + 1, + "one settlement is one sample, however many chunks it covers" + ); + } + + /// Deduplication is per pool, not a global mute: a genuinely separate + /// settlement is a separate sample. + #[tokio::test] + async fn parity_telemetry_emits_once_more_for_a_second_pool() { + let verifier = merkle_test_verifier(); + + let (first, first_pool, first_ts) = make_valid_merkle_proofs_for_one_pool(&[0, 1]); + verifier + .pool_cache + .lock() + .put(first_pool, parity_settled_pool_info(first_ts)); + for (xorname, tagged) in &first { + verifier + .verify_payment(xorname, Some(tagged), VerificationContext::ClientPut) + .await + .expect("first pool admitted"); + } + assert_eq!(verifier.merkle_parity_emission_count(), 1); + + // A fresh batch: new candidate keypairs and leaf salts give it a + // different pool hash. Its chunk is one the first batch did not carry, + // so this exercises verification rather than the verified-address cache. + let (second, second_pool, second_ts) = make_valid_merkle_proofs_for_one_pool(&[2]); + let (second_xorname, second_tagged) = second.first().expect("one proof").clone(); + assert_ne!(first_pool, second_pool, "the second batch is a new pool"); + verifier + .pool_cache + .lock() + .put(second_pool, parity_settled_pool_info(second_ts)); + verifier + .verify_payment( + &second_xorname, + Some(&second_tagged), + VerificationContext::ClientPut, + ) + .await + .expect("second pool admitted"); + + assert_eq!( + verifier.merkle_parity_emission_count(), + 2, + "a second settlement contributes a second sample" + ); + } + + /// Concurrent admissions of one batch — the normal case, since a client + /// PUTs a batch's chunks in parallel — must still emit once. The + /// check-and-insert is a single locked operation for exactly this reason. + #[tokio::test(flavor = "multi_thread", worker_threads = 4)] + async fn parity_telemetry_emits_once_under_concurrent_admissions() { + let verifier = Arc::new(merkle_test_verifier()); + let (proofs, pool_hash, ts) = make_valid_merkle_proofs_for_one_pool(&[0, 1, 2, 3]); + verifier + .pool_cache + .lock() + .put(pool_hash, parity_settled_pool_info(ts)); + + let mut handles = Vec::new(); + for (xorname, tagged) in proofs { + // Two racing admissions per address, as replication and the direct + // PUT can both arrive for the same chunk. + for _ in 0..2 { + let verifier = Arc::clone(&verifier); + let tagged = tagged.clone(); + handles.push(tokio::spawn(async move { + verifier + .verify_payment(&xorname, Some(&tagged), VerificationContext::ClientPut) + .await + })); + } + } + + for handle in handles { + let result = handle.await.expect("task panicked"); + assert!( + result.is_ok(), + "every concurrent admission must succeed: {result:?}" + ); + } + + assert_eq!( + verifier.merkle_parity_emission_count(), + 1, + "concurrent admissions of one pool must not race into extra samples" + ); + } + + /// The first-emission cache is bounded, so a node admitting an unbounded + /// stream of distinct pools cannot grow it without limit. + #[test] + fn parity_telemetry_first_emission_cache_is_bounded() { + let verifier = create_test_verifier(); + for i in 0..(MERKLE_PARITY_TELEMETRY_CACHE_CAPACITY + 50) { + let mut hash: PoolHash = [0u8; 32]; + for (j, b) in i.to_le_bytes().iter().enumerate() { + if let Some(slot) = hash.get_mut(j) { + *slot = *b; + } + } + verifier.merkle_parity_logged.lock().put(hash, ()); + } + + assert_eq!( + verifier.merkle_parity_logged.lock().len(), + MERKLE_PARITY_TELEMETRY_CACHE_CAPACITY, + "the telemetry cache must stay at its capacity" + ); + } + + #[test] + fn merkle_required_multiplier_switches_exactly_at_the_boundary() { + let boundary = 1_785_855_600u64; + + assert_eq!(merkle_required_multiplier(boundary - 1, boundary), 1); + assert_eq!( + merkle_required_multiplier(boundary, boundary), + PAID_QUOTE_PAYMENT_MULTIPLIER, + "the boundary instant itself is enforcing" + ); + assert_eq!( + merkle_required_multiplier(boundary + 1, boundary), + PAID_QUOTE_PAYMENT_MULTIPLIER + ); + } + + /// The legacy branch cannot outlive the receipt lifetime, which is what + /// makes the cutover self-completing. + /// + /// It bounds backdating rather than preventing it: mid-window, a stamp just + /// before the boundary is both unexpired and eligible for 1x, so a modified + /// client can take that route until it expires. Both halves are asserted + /// here so neither claim drifts. + #[test] + fn merkle_legacy_window_closes_one_receipt_lifetime_after_the_boundary() { + let boundary = 1_785_855_600u64; + let expiry = evmlib::merkle_payments::MERKLE_PAYMENT_EXPIRATION; + + // Mid-window: a deliberately backdated stamp is still unexpired, and + // still gets the 1x rule. This route is open for exactly one week. + let mid_window = boundary + expiry / 2; + let backdated = boundary - 1; + assert!( + mid_window - backdated < expiry, + "the backdated stamp is still within its lifetime" + ); + assert_eq!( + merkle_required_multiplier(backdated, boundary), + 1, + "backdating is bounded by expiry, not blocked outright" + ); + + // Once now is a full receipt lifetime past the boundary, the oldest + // still-valid stamp is itself at or after the boundary. + let now = boundary + expiry; + let oldest_valid_stamp = now - expiry; + assert_eq!( + merkle_required_multiplier(oldest_valid_stamp, boundary), + PAID_QUOTE_PAYMENT_MULTIPLIER, + "no unexpired receipt can still claim the 1x rule" + ); + } + // // These constants are load-bearing for both correctness (the storer // must look at the same window the client picks from, otherwise honest diff --git a/src/replication/config.rs b/src/replication/config.rs index 2ce8f8c3..bd5225e9 100644 --- a/src/replication/config.rs +++ b/src/replication/config.rs @@ -334,6 +334,53 @@ pub const QUOTE_ARITHMETIC_RECHECK_ENABLED: bool = false; /// off-curve price, so it deserves its own dial. pub const QUOTE_COMMITMENT_MISMATCH_TRUST_ENABLED: bool = false; +/// Instant from which a merkle batch payment must carry the same 3x settlement +/// multiplier the single-node path has always applied (ADR-0008). Unix seconds. +/// +/// The single-node path settles a chunk at 3x the median quoted price. The +/// merkle path submitted the bare quoted price as the on-chain payable amount, +/// so the contract's `median16(amount) x 2^depth` came to 1x the median per +/// padded leaf: the same chunk, stored and replicated identically, earned a +/// third as much when it arrived in a batch. +/// +/// **The rollout is client-first, and this constant is only its second half.** +/// ant-client (ant-client#161) pays 3x immediately and unconditionally once +/// released — no date, no flag — and un-upgraded nodes accept that, because 3x +/// clears the 1x minimum they require. So the network is paid correctly from +/// the moment the client ships. This instant is when upgraded nodes stop +/// accepting anything less. +/// +/// **Enforcement is on by default.** There is no shadow mode and no flag to +/// flip later: a receipt stamped at or after this instant must settle at 3x or +/// the store is refused. What the boundary buys is not a soft launch, it is +/// compatibility with money that was already spent: +/// +/// - A receipt stamped **before** the boundary is held to the historic 1x. It +/// was paid in good faith under the old rule and the payer cannot get it +/// back, so refusing it would destroy value rather than protect it. +/// - Merkle receipts expire after `MERKLE_PAYMENT_EXPIRATION` (one week), and +/// a receipt stamped in the future is refused outright. So once +/// `boundary + one week` has passed, **every** still-valid receipt is +/// necessarily stamped at or after the boundary and the 1x branch becomes +/// unreachable. It retires itself; nobody has to remember to turn it off. +/// - Backdating is **not** prevented during that week. The timestamp is +/// client-chosen — it is a `payForMerkleTree` argument, the contract only +/// rejects future and week-old stamps, and quoting nodes sign whatever stamp +/// the request carries — so a modified client can stamp just before the +/// boundary and keep paying 1x until such a stamp expires. Expiry is what +/// closes the route, not detection, and it closes it at +/// `boundary + one week`. The same branch protects honest in-flight receipts, +/// so the window cannot be narrowed without also destroying those. +/// +/// Set to 2026-08-04 15:00 UTC, after the client release with room for +/// adoption. Two invariants when moving it: **it must never be earlier than the +/// client release that pays 3x** (an earlier value refuses receipts bought +/// under the previous rule, the one outcome this constant exists to avoid), and +/// it must move forward if that **client** release slips — a node enforcing +/// before clients pay fails every batch upload. A slip in *node* rollout needs +/// no change: it only means fewer nodes enforce for a while. +pub const MERKLE_PARITY_ENFORCED_FROM_UNIX: u64 = 1_785_855_600; + /// ADR-0004: max unresolved quote pins to fetch per payment bundle. /// /// A bundle has at most `CLOSE_GROUP_SIZE` quotes; capping fetches per bundle