Scaling Etsy Payments with Vitess: Part 1 – The Data Model

Etsy's payment databases faced scalability issues in 2020, necessitating a migration to a sharded environment managed by Vitess. The project involved moving 40 billion rows from four databases into a single sharded environment, using Vitess vindexes for sharding. The first phase involved migrating the seller ledger infrastructure, which had an ideal data model for sharding. The second phase focused on reducing load on the primary payments database, which had a complex data model that required a more tailored approach. Two options were considered for the second phase: remodeling the core data model or finding shardifiers within the existing model. The latter option was chosen due to time constraints and the importance of maintaining reliability. The team used existing primary and foreign keys as shardifiers, reducing the workload compared to the first phase. Vitess' re-sharding features provide flexibility for future adjustments to the shard design. Secondary indexes in Vitess mitigate limitations of the non-ideal data model, allowing for shard targeting without using the shardifier in queries. This migration improved scalability by 60% and provided room for future growth.