docs(runbooks): flatKV↔memIAVL parity via sharded historical replay

[bdchatham]

Adds an agent-first runbook to .agent/runbooks/ for driving a flatKV-vs-memIAVL storage-engine correctness validation on harbor at scale (50+ shards), plus its README index row.

What it captures

The method a cold Claude session (or operator) needs to run the validation end-to-end, with the load-bearing correctness traps front-loaded:

• Compare the two replay nodes to each other, never to the archive — comparing a re-executing shadow to the archive's stored pre-v6.5 results conflates the storage engine with version drift and manufactures false divergences. The archive is the block source only.
• Verify the flatKV node's migration is complete before trusting any result (sei_chain_seidb_migration_version == target) — migrate_evm is a boundary-split router that serves un-migrated EVM reads from memIAVL, so a premature comparison is silently vacuous (memIAVL-vs-memIAVL). Free at EVM genesis; high-height shards need evm_migrated/flatkv_only/forced completion.
• historical_replay build tag for pre-v6.5 non-canonical tx bodies (else the strict decoder skips their execution).

Then: the replay-pair topology (same binary + snapshot, blocks from a shared full-history archive), standing up a pair, the seictl result-export shadow comparator (L1 + L2), result aggregation, the Notion report, the 50+ shard fan-out, and a failure-modes table.

Provenance

Distilled from an end-to-end run of this validation on harbor eng-fromtherain. Every technical claim was accuracy-reviewed against the controller + sei-db + seictl source and the live cluster (a systems-engineer on storage/comparator/metrics, a kubernetes-specialist on the CRD/GitOps surface), and legibility-reviewed for agent-first execution (a prose-steward) before this PR.

🤖 Generated with Claude Code

[cursor]

PR Summary

Low Risk
Documentation-only change under .agent/runbooks/; no controller, CRD, or runtime behavior is modified.

Overview
Adds .agent/runbooks/validating-flatkv-memiavl-parity-via-sharded-replay.md, an agent-first operator runbook for flatKV vs memIAVL correctness validation on harbor, and a matching row in .agent/runbooks/README.md.

The new doc encodes the differential replay method (flatKV + memIAVL SeiNode pairs, shared archive, same image/snapshot) and front-loads traps that make naïve runs look green while measuring nothing: compare replay nodes to each other, not the archive; gate on sei_chain_seidb_migration_version before trusting flatKV reads; use mock_chain_validation + historical_replay builds (with the documented sei-chain PR dependency). It also covers SeiNode overrides, Flux/harbor-dev rollout, seictl result-export comparator params, S3 aggregation, Notion reporting, 50+ shard fan-out, and a failure-modes table—with L1 as the flatKV verdict and L2 explicitly as same-history sanity only.

^{Reviewed by Cursor Bugbot for commit b34fdbf. Bugbot is set up for automated code reviews on this repo. Configure here.}