Netflix Stratum Media Processi... Note

Netflix Stratum Media Processing: Automated Container Right-Sizing

Stratum is Netflix's internal serverless media processing platform, crucial for handling diverse media workloads like encoding and packaging. Engineers define Stratum functions as OCI images, which are then scheduled as batch jobs on the Titus compute platform. Managing hundreds of thousands of containers, Stratum needs dynamic scaling for new content formats such as podcasts and live events. Resource requests, specified in YAML, are often over-provisioned due to evolving infrastructure and application changes, leading to inefficiencies and increased operational costs.To address this, Netflix developed the Stratum Resource Tuner (SRT), an automated system designed to right-size container resources in production. SRT aims to eliminate manual resource configuration, focusing initially on high-throughput workloads with relaxed latency requirements, which constitute 70-80% of resource usage. Function owners can opt-in to automatic right-sizing per resource type (CPU, memory, disk, network).The SRT workflow consists of five steps: collecting resource usage metrics, generating recommendations, verifying recommendations via experiments, applying or rejecting recommendations, and monitoring post-application metrics. Recommendations progress through states: CREATED, EVALUATING, APPLIED, REJECTED, and MONITORED. For example, the VideoEncoder.encodeV1 function's over-provisioned memory and CPU were successfully tuned.Metrics collection is robust, with execution-level summaries stored in Elasticsearch for short-term lookups and high-resolution raw usage metrics in Iceberg tables for long-term trend analysis. A Conductor workflow runs daily, using a month of Iceberg data to generate recommendations for all functions and resource types. This process involves filtering high-throughput workloads, aggressively tuning CPU and network (p99 usage), and conservatively tuning memory and disk (p99.9 usage with a safety buffer).Safeguards are in place, such as discarding recommendations if the reduction is too small, and clamping maximum reductions for memory and disk to 15% per cycle. Recommendations are created in the CREATED state and become immutable once they advance past this stage. The example shows CPU recommendations reducing from 8 to 4, and memory from 18000MB to 15300MB in a single step, illustrating SRT's practical application in cost optimization.
CdXz5zHNQW_2pPMwVXJcD.png