Netflix TechBlog | Medium
Follow
The Evolution of Cassandra Data Movement at Netflix
Netflix uses Data Bridge as a unified management plane for batch data movement. Historically, custom connectors were developed for specific needs, but efforts are underway to centralize these offerings. The Cassandra to Iceberg connector, Casspactor, was an in-house solution for critical applications like Member and Billing. Casspactor handled significant data volumes but faced challenges with fragile metadata dependencies and inherited limitations across data abstractions. These issues included skewed partition failures, lack of data model awareness, intermediate table bloat, and an inability to time travel. A new layered architecture was developed to address these shortcomings.This new engine, built on Apache Cassandra Analytics and the Move Data framework, features a core S3 reading capability and a Connector Factory model. It processes data directly into Spark DataFrames, eliminating intermediate tables. The new stack efficiently handles skewed partitions and offers autosizing capabilities for jobs. It also significantly reduces dependencies by relying solely on S3 for metadata, enhancing reliability and enabling time travel. Rigorous validation, visibility, and safety measures were crucial for migrating from Casspactor. A like-for-like migration strategy ensured minimal disruption and no changes for downstream users. Shadow testing was employed to guarantee data consistency between the old and new systems. This involved verifying that the output of the new system was an exact replica of the legacy system's output.