Netflix TechBlog | Medium
Follow
Dynamically Splitting Wide Partitions in Cassandra for Time Series Workloads
Netflix's TimeSeries Abstraction ingests and queries petabytes of temporal event data with millisecond latency, using Apache Cassandra as its storage. Wide partitions, where a single partition accumulates a large volume of events over time, pose a significant challenge for TimeSeries workloads. This leads to high read latencies, timeouts, increased CPU utilization, and garbage collection pauses in Cassandra clusters. To address this, TimeSeries data is partitioned into discrete time chunks, creating manageable segments.The initial provisioning strategy relied on user-specified workload characteristics and Monte Carlo simulations to determine optimal infrastructure and partition configurations. However, this approach proved insufficient when workloads were unknown, inaccurately estimated, evolved over time, or contained data outliers. To automate adjustments, a background worker was introduced to monitor partition histograms and dynamically re-partition future time slices based on observed data density. This Time Slice Re-Partitioning strategy effectively reduces read latencies and timeouts when most data exhibits similar wide partition behavior.However, this strategy doesn't address scenarios where only a small percentage of IDs within a table are wide. For such cases, and when callers require all data even with elevated latencies, Dynamic Partitioning per ID was developed. This asynchronous pipeline detects wide partitions during read operations and transparently splits them into optimal sizes. The process involves detection, planning and splitting, and serving reads by re-routing queries to the split partitions.Detection occurs when a read operation exceeds a configured byte threshold, emitting an event to Kafka. The system initially focuses on immutable partitions for simplicity. The planning stage reads the entire partition to create a split plan, using checkpointing to handle failures. Splitting involves delegating the data division to specific strategies, like assigning more event buckets to a time bucket. Validating splits is crucial, with checksums ensuring data integrity before marking a split as complete. Finally, the TimeSeries servers use in-memory Bloom filters to efficiently divert read queries to the split partitions, making the diversion practically invisible to callers.