5 Billion Records, 10 Terabytes, Four Weeks: A Real Data Migration Playbook

8 hours ago
8 min read

A Real Data Migration Playbook by Wix Engineering

By the time you finish reading this, someone at Wix is probably in the middle of a data migration. We've run migrations that lasted anywhere from a few months to two years.

More recently, we've completed large and complex migrations in a few weeks. The difference wasn't the tooling, which barely changed. It was that we had finally learned which problems show up in every migration, and started dealing with them before the migration began instead of in the middle of it.

Recap: Why We Migrate, And How

Nobody rewrites a live system for fun. You do it because legal requires stronger guarantees, because the old architecture won't survive future growth, because tech debt has made every change expensive, or because new business requirements simply don't fit the old API. Usually, it's several of these at once.

The "how" is well documented, so here's the compressed version: design the V2 API, put an API proxy in front of V1, roll out to new users behind a population-based toggle, migrate the data with CDC, keep syncing ongoing changes until V1 is idle, then route everyone to V2 and decommission the old system.

If any of that was new to you, read our Wix Engineering's The Great Rewrite, Part 2, by Roni Enzel Elman and Oded Apel, which walks through the playbook end to end, and Dalia Simons' How to get a service migration to the finish line successfully, which covers compare mode in depth.

The playbook is a strong foundation. But in the real world, things get messy. The rest of this post is about the problems that still show up when you follow it correctly - and what to do about them.

Yes, Your Database Has Secrets

Before you write a single line of migration code, you must be familiar with your source database.

We cannot stress this enough. Legacy services carry years of changes, and they are almost always more corrupted than you expect. In migrations we've run, we found UUIDs that didn't conform to the UUID spec, first names longer than any validation should have allowed, lists that contained more entries than the documented limit, and at least one birthdate set to the year 19,653.

These records exist because validations change over time, because old code had bugs, because data was imported from systems with different rules.

The reason this matters: your V2 API should have stricter validations. Records that your V1 system silently accepts will break your migration service when you try to convert them.

The way to overcome this obstacle is straightforward and we promise, it takes almost zero effort and no time while the visibility gains are enormous:

Implement the read phase of your migration service before anything else, bind it to the source database, and add logging for every field and structure you suspect might be inconsistent. Run a snapshot dry run and let it produce a dashboard. You will find things. Find them now, not after you've already migrated 5 billion records across 10 terabytes and discovered that a percentage of them can't be converted.

In our logistics data migration, we used this phase to verify that every destination in the V1 API was handled by our V2 destination resolver.

Parity Gaps Will Find You. Find Them First.

A parity gap is when V1 and V2 return different results for the same input. They happen because of missing functionality, different validations between systems, or plain bugs in your V2 implementation. You cannot roll out to users with parity gaps. Finding them late costs weeks.

Two approaches work well together.

First: invest in tests early. Migrations are one of the places where a strong test suite pays for itself many times over.

At Wix, we rely heavily on end-to-end tests because they validate the system the way users & servers actually experience it. Before touching production traffic, run your V1 E2E test suite against your V2 implementation through the API proxy. This immediately exposes obvious parity gaps while giving you confidence that the critical user journeys still behave correctly.

The important point isn't just reusing existing tests - it's having meaningful E2E coverage in the first place. Unit tests verify individual pieces of logic, but migrations fail at the boundaries between components, where assumptions about data, validations, and side effects meet. Well-designed E2E tests catch those integration issues before your users do.

Second: use an LLM to analyze actual production usage of your V1 API. Give it your V1 and V2 codebases, your test suites, and a sample of real production traffic. Ask it to identify use cases that might behave differently between versions. This surfaces gaps that your existing tests don't cover, because those tests were written against V1 assumptions.

The API proxy also provides a compare mode: every read request to V1 triggers an async shadow call to V2, the responses are compared, and you get a match and diff score dashboard. Run this on a subset of production traffic before committing to a full migration. The key word is before - running compare mode after you've migrated all the data means you've already paid the cost of the migration before discovering the bugs.

A Single Bad Record Can Stop Everything

At Wix we use Kafka and CDC (Change Data Capture) for data transfers. CDC preserves message order per partition. That's what makes it reliable. It's also what makes it fragile: if one message fails, everything behind it on that partition stalls.

Think about what this means for your plans. You start the connector, let the snapshot run, go home, and come back in the morning expecting to analyze your compare dashboard. But one record failed at a certain position, and your migration service has been stuck since midnight.

There are three approaches, and the right one depends on how much you trust your data:

Blocking retries. The pipeline stops on failure and stays stopped until a human fixes the handler. You lose speed and you'll need someone on call, but nothing is ever silently skipped. Choose this when data consistency outranks everything else.

Try, catch, skip. Wrap the conversion, log the failure with enough context to investigate, emit a metric, and move on. The pipeline finishes on schedule and you triage the failures afterward: exclude those tenants, fix the bug, re-migrate the affected slice. Choose this when you need the run to complete without a human in the loop and can tolerate a known set of stragglers.

Dead letter queue. Failed messages divert to a separate topic. The main pipeline never stalls, nothing is lost, and you repair and replay the failures on your own schedule. The most robust option and the most work to build, which is exactly the tradeoff you'd expect.

We've used all three across different migrations. The point is not which one is best; it's that "what happens when a record fails" should be a line in your design doc, not a discovery.

The Math of Migration Speed

If you have 5 billion records in your database - in our contacts migration that meant roughly 10 terabytes with a service handling around 200,000 RPM - and the DBAs tell you not to exceed 5,000 RPM to protect live traffic, your migration will take 694 days.

But that calculation is incomplete. While your eager migration is processing historical records, your database is accumulating changes at roughly 15,000 RPM on average - about 22 million changes per day. Your lazy migration has to handle those changes too. If you're running the eager migration at 5,000 RPM and changes are arriving at 15,000 RPM, you will never finish.

This means there are really only two practical operating speeds for a migration: fast enough to outrun the incoming change rate, or slow enough that you're willing to let it run for years.

To run at the speeds you actually need, you have two levers.

Partitions. CDC produces messages into partitions, which allows parallelization. But you have to set the partition count before you start, not after. Come back to a running migration wanting to add partitions and you'll hear "too late, you have to start again." Calculate your required throughput before you begin, divide by your expected per-partition processing rate, and set your partitions accordingly. For our contacts migration, 350 partitions were needed to sustain 100,000 RPM.

Database capacity. In our case, the old contacts database was running out of space - 14 terabytes out of a 16 terabyte maximum, with write load between 15,000 and 50,000 RPM. All running in a mySQL DB with a single master. The DBAs, our platform engineering and database infrastructure teams implemented sharding, splitting our database into eight smaller databases, multiplying the write throughput by the number of shards. With sharding, we could run at the throughput we needed and actually finish. If your migration timeline is blocked by database capacity, this conversation needs to happen early.

One more thing: if your migration service touches and produces side effects on other systems, those systems have limits too. Some of those limits are documented. Many aren't. You'll find the undocumented ones when people come to you and tell you to stop your migration because you're taking down something upstream.

The solution is to decouple. Produce BI events or Kafka messages instead of calling other systems directly during migration, then run a separate follow-up migration for anything that requires those calls. Reduce your blast radius.

The Cleanup Is Part of the Migration

Migrations don't end when the data is in V2. They end when V1 is gone.

Verify there's no traffic hitting your V1 API. Remove the pods and kill the service. Delete the code, and be especially careful with proto packages because stale proto references cause incidents. Delete the migrated population data, because population storage has real costs. And finally, drop the old database.

That last step feels scary. It shouldn't. The data is in V2. Keeping the V1 database around doesn't make you safer, it just costs money and creates ambiguity about which source of truth to trust.

Discount V2 finished in four weeks. Contacts V5 also finished recently. Compared to the seven-month and two-year migrations earlier in our careers, that difference came almost entirely from treating the problems in this post as first-class planning concerns rather than surprises to be handled mid-migration.

What's Next: Supervised, agentic migrations

Look back at everything above as a checklist: define a dry-run population, run it, study the logs, fix conversion bugs and open PRs, size the partitions, negotiate database resources, run compare mode, chase the diffs, kick off the real migration, watch the dashboards, re-migrate the broken subsets, clean up.

It is long, it is repetitive, and almost every step follows a pattern we can now describe precisely, because we've done it enough times to write this post.

That is more or less the definition of work an agent should be doing. Picture three pieces: something that continuously perceives the migration's state from the dashboards and logs we already have, something that reasons about that state and decides the next move, and something that executes it, opening the PR that fixes a failing conversion, adjusting resources, triggering a subset re-migration when compare scores dip.

The word doing the heavy lifting in this section's title is supervised. An agent is able to follow the playbook, that's the whole point of having one. The hard part is teaching it to recognize the moments that fall outside the playbook, the ones this post is about, and to stop and ask a human. We suspect the first version will ask often and the tenth version rarely, and that the migrations our teams run two years from now will look less like operating a pipeline and more like reviewing an agent's pull requests.

Until then - plan for the mess, build for safety, clean up after yourself, and celebrate when V1 is finally, truly dead.

This post is based on Avraham Rosenzweig and Roy Noyman's Wix Engineering Conference 2026's session - "The Migration Strikes Back: What 3 Massive Rewrites Taught Us About Doing It Right":

https://youtu.be/syil90x2c5g?si=eWUYA5dXr5whAaXC

This post was written by: