Netflix TechBlog | Medium
Follow
Maestro: Netflix’s Workflow Orchestrator
Maestro, Netflix's open-source workflow orchestrator, is designed for large-scale data pipelines and machine learning workflows. It supports both acyclic and cyclic workflows, including reusable patterns like foreach loops, subworkflows, and conditional branching. Maestro provides horizontal scalability to manage a large number of workflows and jobs within a single workflow. It offers a flexible workflow definition format in JSON, combining user-supplied and Maestro-managed fields. Maestro's workflow run strategies include sequential, strict sequential, first-only, last-only, and parallel with concurrency limit options. Maestro's parameters and expression language support allow for dynamic parameters, code injection, and state sharing between workflows and steps. It also provides output parameters for callable step execution and parameterized workflows for flexibility and ease of management. Maestro includes common building blocks like foreach support, subworkflows, and conditional branching, enabling easy definition of dataflow and other workflow patterns.