Netflix Conductor: The Next Ch... Note

Netflix Conductor: The Next Chapter

Netflix Conductor has evolved significantly beyond its initial open-source release. While maintenance of the open-source version ceased, internal development accelerated, leading to a complete rebuild. Conductor's scale has exploded, processing billions of workflow executions annually, a stark contrast to its earlier milestone achievements. This growth necessitated modernization across nearly every system layer, from the data plane to the evaluation engine. Conductor now supports a vast number of workflow definitions and is integral to various Netflix businesses, including Content and Studio Engineering, Ads, and Games. The Media Production Suite, for instance, relies on Conductor for critical media ingestion and processing pipelines. Architecturally, Conductor transitioned from a Dynomite-backed stack to using Cassandra as its primary datastore and Timestone for queueing, significantly improving scalability and reliability. The introduction of Conductor 3.0 addressed foundational issues, with Conductor 4.0 rebuilding the core evaluation engine. This rewrite partitioned data for improved memory efficiency and performance, and introduced an asynchronous evaluation model to eliminate synchronous bottlenecks. The overall result is a more robust, scalable, and performant workflow orchestration engine capable of handling Netflix's massive operational demands.
CdXz5zHNQW_fpGlfFLsWS.png