Netflix TechBlog | Medium
Follow
Introducing Configurable Metaflow
Metaflow empowers teams at Netflix to efficiently develop and manage ML/AI projects. Configs, a new Metaflow feature, enables configurable flows by allowing users to define flow parameters, resource requirements, and application-specific settings in human-readable configuration files. Configs extend existing Metaflow artifacts and Parameters, providing greater flexibility and customization options. They can be dynamically loaded and modified, even when using remote execution or production deployments. Configs unlock advanced use cases such as mixing fixed deployments with runtime configurability, validating configurations, managing hierarchies of configuration files, and generating configurations on the fly. In combination with configuration managers like Hydra, Configs enable orchestrating experiments over multiple configurations and sweeping over parameter spaces. Metaboost, an internal Netflix tool, exemplifies the use of Configs in practice, providing a single interface to manage ML projects across different platforms and enhance project coherence and reduce risk.