Skip to content

Backfill MongoDB data into PostgreSQL and verify ongoing parity #175

Description

@coder13

Goal

Turn PostgreSQL dual writes into a trustworthy migration target by copying historical MongoDB data and continuously checking parity.

Acceptance criteria

  • Implement an idempotent, resumable backfill for users/preferences, rooms/participant state, attempts, durable solve results, and sanitized metrics.
  • Support dry-run, bounded batches, progress checkpoints, and safe retries.
  • Do not copy OAuth access tokens or prohibited metric fields.
  • Produce reconciliation output for record counts, missing identifiers, duplicate identifiers, penalty fields, and representative aggregates.
  • Verify writes arriving during the backfill are not lost or duplicated.
  • Document operational invocation, expected runtime/load, alerts, failure recovery, and how to rerun only failed ranges.
  • Add focused tests for deterministic IDs, cursor boundaries, retry behavior, and reconciliation failures.

Dependencies

Unlocks

Metadata

Metadata

Assignees

No one assigned

    Labels

    area: dataMongoDB, PostgreSQL, migrations, backfills, and data integrityenhancementNew feature or requestpriority: P1High-priority work for the next delivery cycle

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions