Introducing Netflix TimeSeries... Note

Introducing Netflix TimeSeries Data Abstraction Layer

Netflix has developed the TimeSeries Abstraction to efficiently store and query large volumes of temporal event data with low millisecond latencies. This system is designed to handle high-throughput writes, efficient querying in large datasets, global reads and writes, tunable configuration, and cost efficiency. The TimeSeries Abstraction is built around core design principles including partitioned data, flexible storage, configurability, scalability, and sharded infrastructure. The data model consists of event items, events, time series IDs, and namespaces. Event items are key-value pairs that store data for a given event, while events are structured collections of one or more event items. Time series IDs are collections of events over a dataset's retention period, and namespaces are collections of time series IDs and event data. The TimeSeries Abstraction provides APIs for interacting with event data, including WriteEventRecordsSync, WriteEventRecords, ReadEventRecords, SearchEventRecords, and AggregateEventRecords. The storage layer consists of a primary data store and an optional index data store, with Apache Cassandra and Elasticsearch being the preferred choices for storing durable data and indexing, respectively. The primary data store uses a temporal partitioning scheme to divide data into manageable chunks based on time intervals, allowing for efficient querying of specific time ranges and optimizing storage and query performance. The data is further partitioned into time buckets and event buckets to facilitate effective range scans and manage high-throughput write operations. The TimeSeries Abstraction is designed to handle the challenges of storing and querying temporal event data at scale, including high throughput, efficient querying, global reads and writes, tunable configuration, and cost efficiency. By using a unique event data model and a scalable storage layer, the TimeSeries Abstraction provides a versatile and cost-effective solution for managing temporal data at Netflix.