Zero Data Loss: Migrating Millions of Credit Records

Migrating consumer credit records is the kind of work where a 99.9% success rate is a failure. Every one of those records is someone's ability to get a loan, rent an apartment, or buy a car. "We lost a few thousand" is not a sentence you ever want to say. So when we moved millions of records off a legacy credit system, the goal wasn't "fast" — it was provably zero loss.

Treat the migration as a system, not a script

The instinct is to write a big script, run it overnight, and check the row counts in the morning. That instinct loses data. We built the migration as a small system with three independent parts: a mover, a reconciler, and a verifier — each able to run repeatedly and idempotently.

A migration you can only run once is a migration you can't trust. Make it re-runnable and the fear goes away.

Dual-write, then backfill, then cut over

Rather than a single big-bang switch, we ran the old and new systems in parallel:

1. Dual-write. New writes went to both systems, so the gap between them never grew.
2. Backfill. A throttled job copied historical records in batches, idempotently, so a crash mid-run just resumed.
3. Reconcile continuously. A separate process compared the two systems and reported drift in real time.

def reconcile(batch):
    mismatches = []
    for record in batch:
        old = legacy.get(record.id)
        new = target.get(record.id)
        if checksum(old) != checksum(new):
            mismatches.append(record.id)
    return mismatches  # must be empty before cutover

Only when reconciliation reported zero drift across the entire dataset — sustained, not a single lucky snapshot — did we flip reads to the new system.

Checksums over counts

Row counts lie. Two systems can have identical counts and different data. We checksummed the meaningful fields of every record and compared those. It is more expensive and it is the only thing that actually proves equivalence.

The unglamorous part that saved us

The verifier ran after cutover too, for weeks, quietly confirming the systems still agreed as new writes flowed in. It caught one subtle encoding difference in a date field that counts would never have surfaced. We fixed it before a single lender noticed.

Zero data loss isn't a heroic overnight push. It's dual-writes, idempotent backfills, continuous reconciliation, and the patience to let the numbers prove themselves before you trust them.