feat(cutover): rollback handlers, reverse payment path, and admin surface (ENG-401)#431
Conversation
…n surface (ENG-401)
Makes the ENG-364 rollback contingency plan executable. The state machine
had rollback_started/rolled_back statuses but nothing could enter or
process them — the runbook's "roll back from snapshot on 1-cent drift"
promise was unexecutable.
State machine: every forward/failed/review/complete status may now enter
rollback_started (skipped_already_migrated and rolled_back excluded by
design); rollback_started self-loops to persist sub-step artifacts and
fails closed into requires_operator_review.
Rollback worker (rollback-worker.ts): field-driven resumable executor
mirroring the forward pipeline's field-presence guards —
1. restore default-wallet pointer (previousDefaultWalletId, captured at
forward flip; legacy USD wallet as constructive fallback)
2. reverse the balance move: no-amount invoice on the legacy USD wallet
paid from the USDT wallet with the exact forward amount
(destinationAmountUsdtMicros); fails closed if the USDT balance no
longer covers it (user transacted post-cutover)
3. treasury tops up any legacy USD shortfall vs sourceBalanceUsdCents
(1-cent dust tolerance; hard no-double-pay guard)
4. finalize to rolled_back with rolledBackAt
Pre-money migrations short-circuit straight to rolled_back. Reverse
payments traced by account, migration id, run id (cwco-rb memos), and
operator (rollbackRequestedBy/reason audit fields).
Orchestration (rollback.ts): requestPrimaryCashWalletRollback (single
account or whole run, dry-run, idempotent — already-rolled-back are
skips), runPrimaryCashWalletRollbackBatch (reuses forward batch
locking/stale-lock recovery; all failures marked
requires_operator_review), completePrimaryCashWalletRollback (flips the
cutover config to the new rolled_back state only when nothing is
mid-flight or in review).
Admin GraphQL: cashWalletCutoverRollback mutation (admin-only),
cashWalletMigrations query exposing full migration records incl.
previousDefaultWalletId and the rollback audit trail, ROLLED_BACK added
to CashWalletCutoverState (additive; public SDL change is enum-value
only). CLI: rollback-request / rollback-batch / rollback-complete.
Tests: 11 new unit tests (rollback-worker.spec.ts) covering request
semantics, every executor path (short-circuit, pointer restore, reverse
pay, fail-closed insufficiency, resume, shortfall top-up, dust tolerance,
no-double-pay), and decimal-cent shortfall math. tsc + eslint clean;
all 7 cutover suites pass (34 tests).
Linear: ENG-401 (implements ENG-364 modes 3A/3C/3D)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ing, pointer-restore gate
Review fixes on the ENG-401 rollback implementation:
1. (High) completePrimaryCashWalletRollback could flip the run-level
config to rolled_back after a single-account rollback while every
other migration remained `complete`. Completion now requires the
ENTIRE run to be accounted for: zero migrations outside
{rolled_back, skipped_already_migrated} — checked via
listMigrationsByStatuses over every other status, with the offending
migration named in the error. Single-account rollbacks never complete
the run.
2. (High) evaluateCashWalletCutoverGuard now handles the rolled_back
config state explicitly instead of falling through to the in_progress
branch: rolled_back/absent/not_started -> legacy_usd,
skipped_already_migrated -> usdt (they were USDT before the run),
rollback_started -> in-progress error, and anything else (e.g. a
leftover `complete` under a supposedly rolled-back run) fails closed
with CashWalletMigrationFailedError rather than guessing a route.
3. (Bonus, same class as 2) the `complete` config state blanket-routed
every account to usdt — including accounts rolled back AFTER global
complete (ENG-364 mode 3D), whose funds are back on legacy USD. The
complete branch now routes rolled_back -> legacy_usd and blocks
rollback_started, and only then blanket-routes usdt.
4. (Medium) pointer restore is now gated on previousDefaultWalletId
being present — the forward pointer flip is that field's only writer,
so its absence proves no flip happened. A pre-money migration whose
account defaults to USDT (e.g. native USDT-default signup) is no
longer touched; the fallback flip to legacyUsdWalletId is removed.
9 new unit tests (rollback.spec.ts + rollback-worker.spec.ts): all 8
cutover suites pass, 44 total. tsc + eslint clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Review fixes pushed — all three findings addressed, plus a fourth of the same class the review surfaced: High 1 — completion gating. High 2 — guard Bonus (same class as High 2) — mode 3D under Medium — pointer-restore gate. Restore now requires On "is partial rollback feasible for a permanent cutover?" Yes, as a T+72h incident tool: (a) routing is per-account (migration-record based) and the phased runbook already operates a mixed fleet for days (Phase 1 internal-only), so one rolled-back account is infrastructurally identical to a not-yet-migrated one; (b) the ENG-364 decision matrix requires it ("$1 discrepancy in any migrated account → rollback that account only") — the alternative would be full-run rollback for one bad account; (c) records are unique per (accountId, runId), so the account re-migrates cleanly in a later run. The limits are now enforced in code: single-account rollback never touches the run config (High-1), and post-complete rollbacks of accounts that have since transacted fail closed to operator review — the mechanical form of the runbook's point-of-no-return. "Permanent" holds at the program level: the exception path's end state is always re-migrate-before-GA, never a standing USD account. Validation: 9 new tests, all 8 cutover suites green (44 total), tsc + eslint clean. |
Implements ENG-401 — makes the ENG-364 rollback contingency plan executable. Until now
rollback_started/rolled_backexisted in the state machine but nothing could enter or process them: the cutover runbook's "roll back from snapshot if reconciliation drifts by 1 cent" was unexecutable. This is a go/no-go gate for cutover Phase 1 (ENG-461).The 6 prerequisites → what shipped
failed/requires_operator_review/completestatus →rollback_started; self-loop persists sub-step artifacts;executeCashWalletMigrationRollbackStep(newrollback-worker.ts)previousDefaultWalletIdadmin read accesscashWalletMigrationsadmin query returns full migration records incl. pointer + rollback audit fieldsCashWalletCutoverRepository.listMigrationsByStatuses(status-set driven; powers request + batch execution)CashWalletCutoverStateenumROLLED_BACKadded (config-level terminal state; public SDL change is an additive enum value only — non-breaking)cashWalletCutoverRollback: single-account (accountId) or full-run,dryRun, idempotent (already-rolled-back = skips), resumable; admin-onlyPlus CLI:
rollback-request/rollback-batch/rollback-completeonscripts/cash-wallet-cutover.ts.Rollback sequence (field-driven, resumable — mirrors forward's field-presence guards)
previousDefaultWalletId(legacy USD wallet as constructive fallback)destinationAmountUsdtMicros; fails closed (requires_operator_review) if the USDT balance no longer covers it (user transacted post-cutover)sourceBalanceUsdCentswithin a 1-cent LN-dust tolerance; hard no-double-pay guard (still short after one top-up → operator review, never a second automatic payment)rolled_back(+rolledBackAt)Pre-money migrations (no forward payment ever sent) short-circuit straight to
rolled_back.Acceptance criteria mapping
cwco-rb:<runId>:<id>:move|shortfallmemos), and operator (rollbackRequestedBy/rollbackReason)requires_operator_review(batch runner override: rollback failures are never plainfailed)skipped_already_migratedaccounts left in USDT (transition rejected by the state machine — unit-tested)Validation
tsc --noEmitclean (both configs) · eslint clean ·yarn test:unitcutover suites 34/34 · SDL regenerated (write-sdl), admin schema gains the new surfaceLinear: ENG-401 · builds on ENG-345 (forward orchestration) · implements ENG-364 modes 3A (single account), 3C (full-run pre-complete), 3D (post-complete)
🤖 Generated with Claude Code