Democratizing Machine Learning... Note

Democratizing Machine Learning at Netflix: Building the Model Lifecycle Graph

Netflix utilizes machine learning across various business domains like personalization, studio production, payments, and advertising. As ML adoption grew, a challenge emerged: a fragmented landscape where models and data were siloed, hindering collaboration and discovery. ML practitioners struggled to understand model lineage, feature sources, and impact across different systems. This fragmentation prevented easy answers to questions about existing features, data sources, pipeline dependencies, and the effects of changes.The core difficulty lay in connecting disparate ML infrastructure components that generated metadata. Dozens of systems, from pipeline orchestrators to experimentation platforms and feature stores, produced data in various formats. Solving this required collecting heterogeneous metadata, transforming it into a unified model, and building a connected graph for exploration.The solution is the Metadata Service (MDS), which constructs a Model Lifecycle Graph to interconnect ML entities at Netflix. MDS ingests ML metadata in real-time, enabling cross-domain queries like identifying experiments using a specific model or models sharing certain features. The vision is to make all ML assets discoverable, understandable, and reusable across the company.MDS operates on core abstractions: Components, each with a unique AIP URI; Entities, which are ML-specific assets with properties; Entity Types, defining data shapes; Domains, which group related entity types; and Providers, concrete implementations of domains from source systems. This URI-based addressing allows any service to reference any ML asset universally. The process of building the graph involves several stages.First, MDS integrates with source systems via Kafka and AWS SNS/SQS to consume thin events indicating changes. Dedicated event handlers process events from systems like the Pipeline Orchestration, Model Registry, Feature Store, Experimentation Platform, and Identity Platform. Second, MDS implements a hydration contract, validating events and calling source system APIs to fetch complete state, which is then transformed into a normalized entity. This "notification of change" pattern ensures robustness against event order issues but places read load on source systems.Third, raw events are transformed into a unified entity model with standardized fields, creating a consistent interface for downstream consumers. Normalized entities standardize field names, formats, and convert platform-specific IDs into global AIP URIs. Finally, normalized entities are persisted to Datomic for caching and relationship storage, and simultaneously indexed in Elasticsearch. Datomic, with its immutable fact model, supports complex graph traversals and entity relationships, enabling queries across multiple domains without inefficient N+1 query patterns.
CdXz5zHNQW_EXeyjNVmx8.png