Migrating Etsy’s database sharding to Vitess
Etsy's database architecture relied on sharded MySQL clusters managed by a proprietary ORM since 2010. This system, with thousands of shards, faced challenges in manual scaling and a single point of failure in the index database. The index database stored shard mappings, hindering the ORM’s ability to route queries efficiently, and causing potential site outages. To address these issues, Etsy adopted Vitess, an open-source database scaling tool, in 2018. They implemented custom vindexes within Vitess to replicate their existing sharding logic, avoiding a full data migration. This involved creating SQLite lookup vindexes and hybrid vindexes for a gradual transition. The migration to Vitess was done incrementally, table by table, to minimize risks and test for query incompatibilities. Challenges with database transactions required simultaneous migration of related tables, highlighting interdependencies. Vitess's cross-shard querying capabilities significantly improved performance, though scatter queries were carefully managed. Ultimately, the Vitess migration eliminated the single point of failure and simplified data access for developers, while allowing a nearly invisible improvement in performance for the users. The move also dramatically improved the manual scaling and enabled new Vitess features.