TimeSeries Tiered Storage Jour... Note

TimeSeries Tiered Storage Journey: Kafka/Flink Streams to Native Cassandra Cold Reads

Netflix's TimeSeries Abstraction platform, crucial for petabyte-scale temporal data, faces significant storage cost challenges. To address this, two solutions have been implemented to move older, less accessed data to cheaper object storage while maintaining low-latency access. The first solution uses Apache Kafka and Flink to stream data to S3, followed by a daily compaction job that merges and optimizes Parquet files. This compaction process involves creating small row groups, sorting events by time, and using Bloom filters for efficient reads. The compacted files are then registered in the control plane, allowing hot storage truncation. Reads are fanned out across hot and cold tiers, with the cold tier leveraging compaction metadata to minimize S3 reads.Despite success, the streaming infrastructure introduced complexity and cost. This led to the development of a second, more integrated solution leveraging Cassandra backups. This approach aims to read data directly from S3 backups without separate data transfer. It introduced a new Cassandra compaction strategy called ReadOnlyCompactionStrategy (ROCS) for immutable time slices. ROCS aligns SSTables with token ranges, reducing read fan-out and improving latency across all tiers.Cassandra backs up its data to S3, creating manifests that list SSTable paths. The second solution customizes Cassandra to read directly from these S3 backups for immutable time slices. It uses pluggable Memtables, specifically a "backupmemtable," to redirect reads. This backup Memtable downloads necessary index and metadata files from S3 to the node. Robust resilience mechanisms ensure reliable component downloads and initialization. Once local components are ready, Cassandra serves reads from these backups, parallelizing S3 reads and utilizing caching for further optimization. This strategy significantly reduces operational overhead and infrastructure costs by eliminating separate streaming infrastructure.
CdXz5zHNQW_qmkDWPsHdU.png