Netflix TechBlog | Medium - TheNote.app

Netflix TechBlog | Medium
Follow

Netflix Tech Blog offers insights into how Netflix handles technology. They provide research on data science, engineering, design, and technology innovations. They showcase their innovations, like their proprietary content delivery network and provide insights into their service reliability efforts.

Netflix TechBlog - Medium netflixtechblog.com

RSS netflixtechblog.com

RSS Hunter • Aug 19, 2024

Thread Of Notes

MAPS: Netflix’s Multimodal Asset Personalization at Scale

Netflix aims to connect members with content they'll love through personalized visual assets like artwork and video previews. A significant challenge arises with new titles, where insufficient interaction data prevents effective personalization, a problem known as the cold-start issue. Traditionally, models treated assets as opaque IDs, relying on popularity heuristics until enough data accumulated. This meant personalization for new content was delayed, impacting the discovery experience.Netflix's solution involves using multimodal embeddings to allow models to "see" and "hear" the assets. By encoding artwork with CLIP, a pre-trained image-text model, assets gain visual understanding. This CLIP embedding is concatenated with the asset's ID embedding, creating a richer representation that captures visual themes and talent. This enables personalization based on member preferences for visual styles, even before the asset has any interaction history, facilitating knowledge transfer across titles.This approach also enables model consolidation, merging separate models for different artwork canvases into a single unified model. Because CLIP embeddings are largely invariant to cropping and resizing, near-identical renderings of artwork map to similar vectors. This pooling of interaction signals across canvases, especially benefiting low-data canvases, allows a single model to personalize all artwork placements. To address data imbalance across canvases, reward-based weighting is used, prioritizing interactions based on their long-term value rather than impression volume.The effectiveness of this approach was validated through offline evaluations using inverse propensity scoring and large-scale A/B tests. Combining image embeddings with a unified model (V3) significantly outperformed models with only one of these improvements. This combined approach proved crucial during a major Netflix home-screen redesign, where a dominant canvas with minimal historical data was introduced. The unified model with CLIP embeddings successfully personalized content for this new layout, showing statistically significant gains in discovery metrics and streaming hours, highlighting the power of multimodal embeddings for overcoming the cold-start problem.

https://netflixtechblog.com/maps-netflixs-multimodal-asset-personalization-at-scale-32f96320785e?source=rss----2615bd06b42e---4 netflixtechblog.com

RSS Hunter • Aug 28

A Tale of Two Flink Autoscalers

Netflix currently runs two Flink autoscalers, aiming to consolidate to one open-source solution. They initially built an in-house autoscaler to manage thousands of Flink jobs, which proved effective but limited in handling complex, multi-operator jobs. This homegrown system relied on external metrics, which could be inaccurate or miss internal job issues. The Apache Flink community later developed a more sophisticated autoscaler that reasons from within the job itself. This second autoscaler estimates an operator's true processing rate to determine optimal parallelism per vertex. Netflix is converging on this open-source solution due to its ability to scale stateful jobs and allow granular configuration. Implementing the OSS autoscaler at Netflix required significant adaptation, including integrating it into their existing control plane. They also made modifications to Flink's runtime for better metric collection and to handle specific job structures like forward-connected subgraphs and sink limitations. By adopting the OSS autoscaler, Netflix has achieved substantial cost savings and improved resource utilization. Lessons learned emphasize that metric quality is crucial, tunable defaults are beneficial, and adopting existing solutions before extending them is a wise strategy. They plan to fully migrate to the OSS-based autoscaler to streamline operations.

https://netflixtechblog.com/a-tale-of-two-flink-autoscalers-e9f6a1b1492b?source=rss----2615bd06b42e---4 netflixtechblog.com

RSS Hunter • Aug 21

How and Why Netflix Built a Real-Time Distributed Graph: Part 3 — Querying the graph with gRPC…

Netflix built a Real-Time Distributed Graph to power real-time insights for their internal partners, and the third part of this blog series focuses on querying the graph efficiently. The graph is a complex network of billions of nodes and edges, and querying it requires a fast and flexible serving layer. The authors discuss the challenges of querying the graph, including handling a wide range of access patterns and supporting both shallow-wide and deep-narrow queries. They explain how they designed a serving layer to efficiently query the graph, using a breadth-first approach and asynchronous composition to minimize latency. The authors also discuss the importance of caching, opt-in enrichments, and eventual consistency in the design of the query layer. The query layer is composed of three layers: the Graph Query Service, the Storage Abstraction Layer, and the Enrichment Layer, which work together to execute queries efficiently. The authors walk through an example query to demonstrate how the query layer works in practice, highlighting the importance of reading and interpreting the request, reading from storage efficiently, executing traversal with breadth-first levels, running many operations in parallel, filtering smartly, and making repeat queries faster with caching. The goal of the query layer is to complete queries in under 100ms, and the authors demonstrate how their design achieves this goal. The query layer is designed to handle tens of thousands of queries per second, each potentially different, and the authors discuss the trade-offs they made in designing the query layer to achieve this level of performance. Overall, the authors provide a detailed overview of the design and implementation of the query layer, highlighting the challenges and trade-offs involved in building a high-performance query layer for a large-scale distributed graph. The query layer is a critical component of the Real-Time Distributed Graph, and the authors' design and implementation have enabled Netflix to power real-time insights for their internal partners.

https://netflixtechblog.com/how-and-why-netflix-built-a-real-time-distributed-graph-part-3-querying-the-graph-with-grpc-0f3468349607?source=rss----2615bd06b42e---4 netflixtechblog.com

RSS Hunter • Aug 7

Modeling Device Capabilities for Analytics

Netflix offers a wide range of features and content types across various devices, but hardware limitations can restrict certain features on specific devices. To address this, the company has developed a comprehensive device capability data model to understand device capabilities and ensure the best user experience. The data model integrates feature flags from internal systems, enabling smarter feature management across the global device landscape. A cumulative table is used to store information about device capabilities, such as screen resolutions, video profiles, and RAM size, making it ideal for analytics and reporting. The table is structured to capture the latest state of each device and its associated capabilities, allowing for efficient processing of information. For aggregate analytics, a histogram table is used to capture active device counts over the past 28 days, broken down by device model and software version. This table also records the number of devices supporting specific capabilities, enabling detailed distribution analysis. The histogram data can be used to analyze the distribution of external display capabilities, such as the percentage of devices supporting HD or UHD profiles. By leveraging these datasets, Netflix has built analytical products to provide a comprehensive view of feature reach, including 4K Ultra HD, Netflix Spatial Audio, and Cloud Gaming. The company uses data-driven insights to make informed decisions about which features to enable on specific devices, ensuring both performance and reliability.

https://netflixtechblog.com/modeling-device-capabilities-for-analytics-e7607acebde8?source=rss----2615bd06b42e---4 netflixtechblog.com

RSS Hunter • Jul 31

GenRec: Towards LLM-Native Recommendation at Netflix

Recommendations are a crucial part of the Netflix experience, and the company has been using complex production models that rely on thousands of hand-crafted features and specialized architectures. However, these models are costly to maintain and update, and the company is looking for more efficient and effective solutions. Large language models have shown promise in this area, but they are not yet production-ready and often have limitations such as over-recommending popular content and ignoring business constraints. To address this, Netflix built GenRec, an LLM-backed recommendation ranker that post-trains an internal foundation LLM on Netflix-specific data and objectives. GenRec verbalizes user histories, item metadata, and context as text and uses a catalog-aware scoring head to rank items. The model is trained with a multi-objective loss that combines ranking, language, and reward signals to align with business goals and long-term member satisfaction. GenRec has shown statistically significant improvements in both short-term and long-term online metrics compared to a well-tuned production ranker, while using only a small fraction of the labeled data and input signals. The model is served on Netflix's internal LLM stack using vLLM, and the company has implemented strategies to control serving cost, such as using smaller models, aggressive context compaction, and prefill-only inference. Overall, GenRec represents a significant step towards a more LLM-centric future for recommendation at Netflix, and its success has the potential to transform the way the company approaches recommendation and personalization. The model's ability to learn from natural language inputs and generate personalized rankings has the potential to improve the user experience and increase engagement. By leveraging the power of large language models, Netflix can create more effective and efficient recommendation systems that drive business success.

https://netflixtechblog.com/genrec-towards-llm-native-recommendation-at-netflix-f20be6f643e3?source=rss----2615bd06b42e---4 netflixtechblog.com

RSS Hunter • Jul 30

In-House LLM Serving at Netflix

Netflix runs its entire large language model stack internally, managing deployment and inference directly. They integrated this into their existing production environment instead of a separate machine learning silo. This approach involved carefully selecting an inference engine, deciding how models would be packaged, designing the API surface, defining deployment strategies, and enforcing output constraints.The chosen inference engine is vLLM, selected for its operational fit, ability to load custom architectures, extensibility, debuggability, and familiarity among practitioners. Models are packaged using the vLLM backend in Triton to allow dynamic I/O tensor specification, promoting independent evolution of models and frontends. An OpenAI-compatible HTTP frontend was added alongside their existing gRPC interface for broader ecosystem compatibility.For deployment, Netflix uses a Versioned strategy to handle potential schema changes between model versions, allowing consumers to update independently. When model interfaces are stable, the less costly Red-Black deployment strategy is employed. One key operational challenge was slow model startup times, which they addressed by materializing large models on Amazon FSx for faster access.Observability was improved by creating a unified /metrics endpoint that aggregates data from both vLLM and Triton. A significant feature is constrained decoding at scale, implemented via vLLM's custom logits processor. This allows models to generate compliant outputs directly, avoiding post-inference correction.An initial pure-Python implementation of constrained decoding struggled with scaling due to the Global Interpreter Lock (GIL) and sequential processing on the CPU. This only became apparent under high concurrency, leading to significant tail latencies. The system relies on a Java control plane for managing deployment, versioning, and autoscaling. Both real-time and cached batch inference paths are supported by their serving system. This unified system handles the complete downstream consumer flow, including routing, candidate generation, and logging.

https://netflixtechblog.com/in-house-llm-serving-at-netflix-a5a8e799ea2c?source=rss----2615bd06b42e---4 netflixtechblog.com

RSS Hunter • Jul 17

Building Service Topology at Scale: Architecture, Challenges, and Lessons Learned

Netflix developed a real-time service dependency map to aid engineers in troubleshooting and understanding their distributed architecture. The system combines eBPF network flows, IPC metrics, and distributed tracing into independent graph layers. While the initial version worked locally, production revealed significant scaling challenges, including Kafka consumer lag and memory issues.The core architectural decision was a streaming-first approach, providing near real-time topology updates instead of hourly batch processing. This was crucial for incident response and live event monitoring. To handle millions of flow records per second without data loss, the system employs reactive streams with backpressure, signaling upstream components to slow down when downstream systems are overwhelmed. This ensures graceful degradation rather than crashes or data loss.The architecture uses a multi-layer design with physical storage isolation for network, IPC, and tracing layers, allowing independent optimization. The network layer's ingestion relies on a three-stage distributed aggregation pipeline. Stage 1 performs initial aggregation from Kafka, Stage 2 resolves network intermediaries (like load balancers) into direct application-level dependencies, and Stage 3 handles final aggregation, enrichment with external data, and persistence to the graph database.The three-stage pipeline was a critical evolution from an initial two-stage design, which suffered from "hot nodes" due to data concentration during intermediary resolution and enrichment. Splitting these responsibilities across three stages distributes the workload, preventing bottlenecks. Server-Sent Events (SSE) replaced gRPC for inter-stage communication due to gRPC's performance overhead, which consumed excessive CPU and memory at scale.

https://netflixtechblog.com/building-service-topology-at-scale-architecture-challenges-and-lessons-learned-f4b792f3f0d8?source=rss----2615bd06b42e---4 netflixtechblog.com

RSS Hunter • Jul 13

GenPage: Towards End-to-End Generative Homepage Construction at Netflix

The Netflix homepage is a highly personalized and structured interface for content discovery. Traditionally, its generation involved a complex multi-stage pipeline. GenPage, a novel approach, utilizes a single generative transformer model to autoregressively construct the entire homepage. This end-to-end model takes user context as a prompt and generates rows, entities, and layout simultaneously. Key goals include simplifying the recommendation stack, enabling whole-page optimization via reinforcement learning, and improving scalability and flexibility. Production challenges involved real-time serving latency, entity cold start, and maintaining model freshness. GenPage has demonstrated significant improvements in user engagement and reduced serving latency in A/B tests. The data is tokenized into context tokens representing user information and page tokens for the homepage structure. Domain-specific tokenization enhances computational efficiency and product control compared to generic text tokenization. User history, profile, and request context form the prompt, while entities and rows are represented as tokens in the generated page. The system uses a reward system based on user feedback to quantify recommendation value and guide training. GenPage employs a standard decoder-only transformer architecture and follows an LLM-like training recipe of pretraining and post-training. Pretraining uses next-token prediction to teach the model the homepage language, followed by post-training with weighted binary classification or reinforcement learning. This generative approach offers a more integrated and direct path to optimizing the entire Netflix homepage experience.

https://netflixtechblog.com/genpage-towards-end-to-end-generative-homepage-construction-at-netflix-77146fba8a08?source=rss----2615bd06b42e---4 netflixtechblog.com

RSS Hunter • Jun 29

Toward More Controllable AI Video Editing: An Early Research Exploration at Netflix

Netflix researchers have developed two AI tools to assist video editors in creating promotional assets. The first tool, Vera, is a layered video diffusion model designed for content-preserving edits. Vera generates edits as separate layers, leaving untouched portions of the original footage unaltered, thus preserving identities and details. To train Vera, they created a specialized dataset and employed a Mixture-of-Transformers architecture for efficient output generation. Evaluations showed Vera significantly outperformed existing methods in content preservation with comparable video quality. The second tool, VOID, is a video object and interaction deletion framework. VOID addresses the challenge of physically implausible results when removing objects with significant interactions. It uses a two-pass inference pipeline, with the first pass generating a physically plausible counterfactual video. A second pass refines the output to prevent artifacts like object morphing. VOID was trained using synthetic data generated from simulations and real-world motion capture data. Experiments indicate that VOID better preserves consistent scene dynamics compared to previous approaches. These research explorations aim to advance AI video editing responsibly while empowering artists. Both Vera and VOID are detailed in publicly released research papers.

https://netflixtechblog.com/toward-more-controllable-ai-video-editing-an-early-research-exploration-at-netflix-eb8160ed60a2?source=rss----2615bd06b42e---4 netflixtechblog.com

RSS Hunter • Jun 23

How Netflix Simplified Batch Compute with Kueue

Netflix is transitioning its compute infrastructure to a Kubernetes-native model, integrating components like Kueue into its Titus platform. Kueue, a cloud-native job queueing system, has largely replaced the custom logic of their homegrown batch solution, Compute Managed Batch (CMB). Motivations for this migration included CMB's age, the evolution of Kubernetes, and the growing difficulty of adding features like preemption to the existing system. Kueue was chosen over alternatives due to its compatibility with existing Titus scheduling profiles, its adoption momentum, and its native support for features like preemption.The migration, known as Netflix Batch, aimed for zero user impact and no reduction in throughput. The process involved rerouting job submissions to Kueue-enabled Titus cells, with Titus federation handling job distribution. Operator adjustments were minimal, primarily involving a simple UI toggle to enroll tenants into Kueue. This enrollment translates CMB's existing tenant hierarchy and capacity configurations into Kueue's concepts of Cohorts, ClusterQueues, and LocalQueues.Key lessons learned include maintaining API parity with the old system to ensure a smooth user experience and migrating complex use cases early to build confidence. It was also crucial to load test Kueue to ensure it could handle Netflix's high throughput requirements, necessitating adjustments to default configurations. Kueue is now fully operational, managing millions of batch workloads, and has enabled enhanced fair sharing and preemption features. These improvements allow for better utilization of reserved capacity, reduced job starvation, and faster turnaround for critical workloads, significantly increasing average resource utilization.

https://netflixtechblog.com/how-netflix-simplified-batch-compute-with-kueue-87860682629c?source=rss----2615bd06b42e---4 netflixtechblog.com

RSS Hunter • Jun 22

The Data Canary: How Netflix Validates Catalog Metadata

Netflix built an automated data canary system to ensure catalog data reliability. This system validates data transformations using production traffic. It detects issues in under 10 minutes, preventing corrupted data from reaching millions of viewers. A past incident revealed gaps in their resilience strategy when manual actions corrupted a data feed. This corrupted data caused manifest generation failures and playback issues. Existing code canaries were ineffective because no code had changed, only the data. The challenge was validating high-velocity data pipelines with short time constraints. They needed to validate the actual output consumed by clients, not just inputs. Production traffic was deemed essential for detecting real customer impact. Their solution involved a dedicated orchestrator pattern with permanent baseline and canary clusters. They extended their chaos platform, customizing thresholds and using sticky canaries. Behavioral metrics like Starts Per Second became their primary signal for detection. The system also incorporated robust edge case handling for production environments. Controlled failure injection experiments validated the system's effectiveness. Issues were identified in 2.5–4 minutes, with automatic blocking of corrupted data. This effort established that data deployments require the same rigor as code deployments.

https://netflixtechblog.com/the-data-canary-how-netflix-validates-catalog-metadata-18b699d58e36?source=rss----2615bd06b42e---4 netflixtechblog.com

RSS Hunter • Jun 19

Data Projects: Managing Data Assets at Netflix Scale

Netflix faced challenges managing millions of data assets with an individual asset-based access control system. Organizational changes caused permission updates to be burdensome and inconsistent, leading to either flooded support teams or overly broad access grants. Workloads were tied to human identities, causing failures when engineers changed roles or left the company, resulting in a "permissions whack-a-mole" issue. To address these problems, Netflix introduced Data Projects, which act as containers for related assets and provide durable, project-specific identities for workloads. This shifts management granularity from individual assets to logical project groupings. Each Data Project has grants and roles, allowing users, groups, or applications to be assigned specific access levels. The project’s Netflix application identity executes workloads, independent of human lifecycles, and can be cryptographically exchanged for AWS IAM roles. A key feature called "gravity" automatically adds newly created assets to their project, ensuring organizational consistency. Data Projects have proven successful in stabilizing critical pipelines and simplifying access management for complex analytics domains. Onboarding involves creating a project, granting roles, and configuring workflows to use the project's identity, with new assets automatically associated via gravity. Future plans include expanding projects beyond data to software and studio assets, integrating rightsizing for permissions, and adding features like cost attribution and auditing. Ultimately, Data Projects provide a scalable and stable solution for identity and access management at Netflix, aligning with how teams organize their work.

https://netflixtechblog.com/data-projects-managing-data-assets-at-netflix-scale-7ca25888591e?source=rss----2615bd06b42e---4 netflixtechblog.com

RSS Hunter • Jun 19

Predicting Risk in Content Launches: How Data-Driven Insights can Transform Launch Planning

Netflix's Analytics Summit showcases how analytics informs business decisions, including content launch risk. Content goes through development, production, post-production, and launch preparation, with launch preparation relying on finalized media assets. Teams face a trade-off between waiting for the final IMF, risking delays, or starting early with a less final Locked Cut, risking rework. Manually provided production schedules for these assets often lack accuracy and coverage, especially far from delivery. This inaccuracy is strongly correlated with content launch misses. To address this, Netflix developed predictive models to estimate media asset delivery dates. These boosted tree regression models use production-level signals, metadata, and seasonal data, updated daily. The models aim to fill schedule gaps and improve existing date accuracy. Evaluating these models involves metrics like mean absolute error, bias, and error distribution. Backtests show significant improvements in predictive accuracy compared to manual schedules, reducing forecast errors. These improved predictions lead to lower Accumulated Error Days, a metric tied to launch misses. The predictive dates integrate into existing workflows, offering earlier accuracy than scheduled dates. Serving logic defaults to scheduled dates where the model underperforms, while other teams can view both and use their judgment. This initiative streamlines launch workflows and mitigates launch risks by providing more reliable delivery estimates.

https://netflixtechblog.com/predicting-risk-in-content-launches-how-data-driven-insights-can-transform-launch-planning-587b1f2de928?source=rss----2615bd06b42e---4 netflixtechblog.com

RSS Hunter • Jun 19

The Evolution of Cassandra Data Movement at Netflix

Netflix uses Data Bridge as a unified management plane for batch data movement. Historically, custom connectors were developed for specific needs, but efforts are underway to centralize these offerings. The Cassandra to Iceberg connector, Casspactor, was an in-house solution for critical applications like Member and Billing. Casspactor handled significant data volumes but faced challenges with fragile metadata dependencies and inherited limitations across data abstractions. These issues included skewed partition failures, lack of data model awareness, intermediate table bloat, and an inability to time travel. A new layered architecture was developed to address these shortcomings.This new engine, built on Apache Cassandra Analytics and the Move Data framework, features a core S3 reading capability and a Connector Factory model. It processes data directly into Spark DataFrames, eliminating intermediate tables. The new stack efficiently handles skewed partitions and offers autosizing capabilities for jobs. It also significantly reduces dependencies by relying solely on S3 for metadata, enhancing reliability and enabling time travel. Rigorous validation, visibility, and safety measures were crucial for migrating from Casspactor. A like-for-like migration strategy ensured minimal disruption and no changes for downstream users. Shadow testing was employed to guarantee data consistency between the old and new systems. This involved verifying that the output of the new system was an exact replica of the legacy system's output.

https://netflixtechblog.com/the-evolution-of-cassandra-data-movement-at-netflix-6e13329c80a1?source=rss----2615bd06b42e---4 netflixtechblog.com

RSS Hunter • Jun 19

Thinking Fast & Slow for a Personalized Notification System

Daniel Kahneman's dual-process theory describes two cognitive systems: System 1 for automatic, quick thinking and System 2 for deliberate, focused effort. This concept applies to designing intelligent systems that balance immediate responsiveness with foresight. Netflix's personalized notification platform faces a similar challenge, optimizing hundreds of millions of daily messages. A core tension exists between maximizing short-term engagement and ensuring a positive long-term member experience. Over-messaging can lead to fatigue and opt-outs, while under-messaging risks missing valuable content discoveries.To address this, Netflix implemented a hierarchical framework with a "slow" policy for strategic, weekly messaging plans and a "fast" policy for tactical, real-time message selection. Previously, single-message outcome models optimized for short-term gains but overlooked cumulative effects and coupled ranking with pacing decisions. The new "slow" policy defines a personalized message pacing over a defined horizon, considering long-term engagement. This is achieved by maximizing a utility function that balances positive engagement signals against the long-term cost of messaging.A universal message cost term is introduced to prevent models from always opting for maximum frequency. The "slow" policy's decisions are stored in a feature store, enabling asynchronous communication with the "fast" policy. The "fast" policy then executes tactical send decisions within the strategic guardrails set by the "slow" policy. This decoupling allows for independent evolution of both strategies and ensures a consistent member experience. The hierarchical architecture led to significant metric lifts, particularly benefiting casual viewers by improving content awareness. Ultimately, separating frequency planning from message selection proved transformative, allowing for independent iteration on both aspects.

https://netflixtechblog.com/thinking-fast-slow-for-a-personalized-notification-system-4d89b26525cd?source=rss----2615bd06b42e---4 netflixtechblog.com

RSS Hunter • Jun 19

A Human-Augmenting Agentic Workflow for Causal Inference

Data analysis is increasingly delegated to software agents, necessitating oversight for valid results, especially in Observational Causal Inference (OCI). This work introduces an agentic workflow for OCI that adheres to rigorous templates while augmenting human inspection. The workflow aims to reduce repetitive tasks for OCI practitioners, allowing them to focus on nuanced aspects like question framing and assumption evaluation. An open-sourced version of their oci-agent is available for improvement, and evaluations show it outperforms one-shot iterations and rivals hand-tuned benchmarks. The philosophy relies on target trial emulation, outlining ideal A/B tests to inform necessary assumptions for causal inference. This involves design diagnostics like covariate balance, overlap, placebo outcomes, and sensitivity analysis. The agent orchestrates an actor-critic loop with three personas: Principal (human user), Actor (analysis executor), and Critic (results synthesizer). Principals define the plan and context, Actors refine and execute the analysis with diagnostics, and Critics identify gaps and assess credibility. To empower human evaluation, the workflow makes each analytic step transparent through published artifacts like plans, specifications, and notebooks. A case study at Netflix estimated the impact of a new entertainment type, revealing significant early adopter bias. Standard regression gave a large effect, but the agentic workflow, using diagnostics, identified poor overlap and placebo test failures. Addressing these, the agent implemented trimming, significantly reducing the estimated effect to a more credible range for the overlapping population. Follow-up analyses, such as sensitivity tests and time-series generation, are facilitated by the agent's ability to manage complex, multi-version executions.

https://netflixtechblog.com/a-human-augmenting-agentic-workflow-for-causal-inference-4623f0a9c5af?source=rss----2615bd06b42e---4 netflixtechblog.com

RSS Hunter • Jun 19

VMAF v1: Good Is Not Good Enough

Netflix has open-sourced a new version of its video quality metric, VMAF v1. VMAF is crucial for optimizing video encoding and ensuring a good viewing experience for members. VMAF v1 addresses limitations of the previous version, VMAF v0, to more accurately assess visual quality. Key improvements include enhanced sensitivity to compression artifacts by adding the AIM component. VMAF v1 also allows for a single model to be used across various viewing conditions by adjusting feature values based on viewing distance. This approach improves accuracy and generalizability over older mapping functions. Banding artifacts are now addressed by integrating the CAMBI feature, and chroma artifacts are handled by modifying SpEED-QA for chroma channels. The no-enhancement gain (NEG) mode is now enabled by default to preserve creative intent and is used in codec development. VMAF v1 features an improved motion feature with a hard threshold and an option for a larger temporal window to better handle high frame rates. The new version includes multiple models for different resolutions and viewing distances, such as standard 1080p, phone, and 4K. VMAF v1 has been calibrated to align with v0 scores, preserving interpretability while offering enhanced accuracy. The new version also boasts reduced computational complexity, making it faster than VMAF v0. Netflix encourages the community to test VMAF v1, report issues, and contribute to its ongoing development.

https://netflixtechblog.com/vmaf-v1-good-is-not-good-enough-60d7e4244ea8?source=rss----2615bd06b42e---4 netflixtechblog.com

RSS Hunter • Jun 19

Dynamically Splitting Wide Partitions in Cassandra for Time Series Workloads

Netflix's TimeSeries Abstraction ingests and queries petabytes of temporal event data with millisecond latency, using Apache Cassandra as its storage. Wide partitions, where a single partition accumulates a large volume of events over time, pose a significant challenge for TimeSeries workloads. This leads to high read latencies, timeouts, increased CPU utilization, and garbage collection pauses in Cassandra clusters. To address this, TimeSeries data is partitioned into discrete time chunks, creating manageable segments.The initial provisioning strategy relied on user-specified workload characteristics and Monte Carlo simulations to determine optimal infrastructure and partition configurations. However, this approach proved insufficient when workloads were unknown, inaccurately estimated, evolved over time, or contained data outliers. To automate adjustments, a background worker was introduced to monitor partition histograms and dynamically re-partition future time slices based on observed data density. This Time Slice Re-Partitioning strategy effectively reduces read latencies and timeouts when most data exhibits similar wide partition behavior.However, this strategy doesn't address scenarios where only a small percentage of IDs within a table are wide. For such cases, and when callers require all data even with elevated latencies, Dynamic Partitioning per ID was developed. This asynchronous pipeline detects wide partitions during read operations and transparently splits them into optimal sizes. The process involves detection, planning and splitting, and serving reads by re-routing queries to the split partitions.Detection occurs when a read operation exceeds a configured byte threshold, emitting an event to Kafka. The system initially focuses on immutable partitions for simplicity. The planning stage reads the entire partition to create a split plan, using checkpointing to handle failures. Splitting involves delegating the data division to specific strategies, like assigning more event buckets to a time bucket. Validating splits is crucial, with checksums ensuring data integrity before marking a split as complete. Finally, the TimeSeries servers use in-memory Bloom filters to efficiently divert read queries to the split partitions, making the diversion practically invisible to callers.

https://netflixtechblog.com/dynamically-splitting-wide-partitions-in-cassandra-for-time-series-workloads-0eded064f456?source=rss----2615bd06b42e---4 netflixtechblog.com

RSS Hunter • Jun 3

High-Throughput Graph Abstraction at Netflix: Part I

Netflix built a Graph Abstraction to handle high-throughput, low-latency graph operations, particularly for use cases like Real-Time Distributed Graphs and Social Graphs. It operates within two categories: OLAP for in-depth analysis and OLTP for streaming user experiences. The abstraction uses a Property Graph model with strongly typed nodes and edges, organized into isolated namespaces. Each namespace has a predefined graph schema managed through a Data Gateway Control Plane. This schema enables optimizations like data quality enforcement and efficient query planning. The real-time index uses key-value storage for nodes and edges, employing separate indexes for links and properties. Edge links are indexed by source-destination relationships. To ensure access regardless of direction, this organizes identifiers lexicographically. Caching is used to minimize write and read amplification. The Abstraction architecture prioritizes high performance and incorporates strategies like write-aside caching.

https://netflixtechblog.com/high-throughput-graph-abstraction-at-netflix-part-i-e88063e6f6d5?source=rss----2615bd06b42e---4 netflixtechblog.com

RSS Hunter • May 29

From Silos to Service Topology: Why Netflix Built a Real-Time Service Map

Netflix developed a "living map" of its distributed infrastructure, called Service Topology, to help engineers understand service dependencies and troubleshoot issues. The map addresses the need for quicker identification of service relationships and potential impact during outages. It answers crucial questions like which services depend on each other and identifies the source of problems. They gathered data from three sources: eBPF network flows, IPC metrics, and end-to-end tracing. Each data source provides a unique perspective – network connectivity, application-level details, and actual request flows. This multi-layered architecture combines these views into a unified, real-time map. The system uses separate graph databases for each data source to maintain independence and enable parallel querying. Engineers can view each graph individually or combine them for a comprehensive understanding of service interactions. The system ingests flow logs from Kafka in multiple regions, processes them, and stores the data in graph databases. This living map helps Netflix engineers quickly diagnose and resolve problems, ensuring a smooth streaming experience.

https://netflixtechblog.com/from-silos-to-service-topology-why-netflix-built-a-real-time-service-map-0165ba13a7bc?source=rss----2615bd06b42e---4 netflixtechblog.com

RSS Hunter • May 29

Scaling ArchUnit with Nebula ArchRules

Netflix uses a polyrepo strategy with thousands of Java repositories, requiring efficient build logic sharing. They built the Nebula suite of Gradle plugins, including ArchRules, to manage dependencies and enforce code standards. This initiative arose from a need to improve Java library lifecycle management following a backwards-incompatible change incident. Netflix utilizes API lifecycle annotations (@Deprecated, @Public, @Experimental) to identify potential issues with deprecated code. ArchUnit, a popular library for enforcing architectural rules, was chosen to detect the misuse of these annotations and other technical debt issues. Nebula ArchRules plugins enable sharing and applying ArchUnit rules across many repositories, providing enhanced capabilities. ArchRules uses bytecode analysis (ASM) for cross-language support and an easy to use builder pattern simplifying rule creation. Rules can be bundled with libraries or defined in standalone rule libraries, automatically detected and run by the plugin. The ArchRules Runner plugin evaluates rules against source sets and generates JSON and console reports, enabling better reporting capabilities. By using ArchRules, Netflix provides a platform for library authors to track API usage and detect deprecated API usage.

https://netflixtechblog.com/scaling-archunit-with-nebula-archrules-b4642c464c5a?source=rss----2615bd06b42e---4 netflixtechblog.com

RSS Hunter • May 8

Democratizing Machine Learning at Netflix: Building the Model Lifecycle Graph

Netflix utilizes machine learning across various business domains like personalization, studio production, payments, and advertising. As ML adoption grew, a challenge emerged: a fragmented landscape where models and data were siloed, hindering collaboration and discovery. ML practitioners struggled to understand model lineage, feature sources, and impact across different systems. This fragmentation prevented easy answers to questions about existing features, data sources, pipeline dependencies, and the effects of changes.The core difficulty lay in connecting disparate ML infrastructure components that generated metadata. Dozens of systems, from pipeline orchestrators to experimentation platforms and feature stores, produced data in various formats. Solving this required collecting heterogeneous metadata, transforming it into a unified model, and building a connected graph for exploration.The solution is the Metadata Service (MDS), which constructs a Model Lifecycle Graph to interconnect ML entities at Netflix. MDS ingests ML metadata in real-time, enabling cross-domain queries like identifying experiments using a specific model or models sharing certain features. The vision is to make all ML assets discoverable, understandable, and reusable across the company.MDS operates on core abstractions: Components, each with a unique AIP URI; Entities, which are ML-specific assets with properties; Entity Types, defining data shapes; Domains, which group related entity types; and Providers, concrete implementations of domains from source systems. This URI-based addressing allows any service to reference any ML asset universally. The process of building the graph involves several stages.First, MDS integrates with source systems via Kafka and AWS SNS/SQS to consume thin events indicating changes. Dedicated event handlers process events from systems like the Pipeline Orchestration, Model Registry, Feature Store, Experimentation Platform, and Identity Platform. Second, MDS implements a hydration contract, validating events and calling source system APIs to fetch complete state, which is then transformed into a normalized entity. This "notification of change" pattern ensures robustness against event order issues but places read load on source systems.Third, raw events are transformed into a unified entity model with standardized fields, creating a consistent interface for downstream consumers. Normalized entities standardize field names, formats, and convert platform-specific IDs into global AIP URIs. Finally, normalized entities are persisted to Datomic for caching and relationship storage, and simultaneously indexed in Elasticsearch. Datomic, with its immutable fact model, supports complex graph traversals and entity relationships, enabling queries across multiple domains without inefficient N+1 query patterns.

https://netflixtechblog.com/democratizing-machine-learning-at-netflix-building-the-model-lifecycle-graph-5cc6d5828bb1?source=rss----2615bd06b42e---4 netflixtechblog.com

RSS Hunter • May 4

State of Routing in Model Serving

The blog post discusses the technical insights into how Netflix's ML model serving infrastructure powers personalized experiences at scale across various domains. The central ML model serving platform exposes a domain-independent API abstraction and traffic routing capabilities to several domain-specific microservices for model inference. This singular API has increased the speed of innovation for iterating on newer versions of existing ML experiences and enabling new product experiences with ML. The success of the ML model serving infrastructure depends on enabling researchers to rapidly experiment with new hypotheses and safely release their models into production. The platform serves hundreds of model types and versions, netting 1 million requests per second, and operates at the level of workflows, not just individual scoring functions.The model definition contains a list of facts that it needs to compute features, and it relies on the model serving platform to supply these facts at serving time by calling several other microservices. The calling services only need to provide standard request context, and the relevant domain context, and the model can itself compute features and perform inference as part of the execution flow. The platform acts as an enabler of rapid ML innovation and limits the exposure of ML model iterations to the client apps. The key principles of the platform include model innovation independent of client apps, decoupling clients from model sharding, and flexible traffic routing rules.The platform uses a custom service called Switchboard, which serves as a flexible proxy layer for all traffic, handling over 1 million requests per second while maintaining high availability and reliability. Switchboard provides a single point of contact for all clients' model needs and can route a request based on a rich set of contextual features. The platform also introduces the concept of an "Objective", which is an enumeration defined by the serving platform that every request into the system must provide. The Objective decouples clients from concrete models and guides the platform's routing and model selection decisions.Switchboard Rules is a JavaScript configuration that allows researchers to attach model variants, experiments, and traffic splits to Objectives without changing client code. The rules dictate the default model to use for a given Objective, A/B experiments to configure for a set of Objectives, and customizations to gradually shift traffic to a new model. The rules are consumed by both Switchboard and the Model Serving clusters, and the serving platform components can take various actions based on these rules. Overall, the platform provides a scalable and flexible solution for serving ML models at Netflix, enabling rapid innovation and experimentation while minimizing the impact on client apps.

https://netflixtechblog.com/state-of-routing-in-model-serving-16e22fe18741?source=rss----2615bd06b42e---4 netflixtechblog.com

RSS Hunter • May 1

Scaling Camera File Processing at Netflix

Netflix built its Media Production Suite (MPS) to streamline media workflows for global productions, aiming for efficiency and consistency. The core of MPS leverages FilmLight's API (FLAPI) for image processing, a collaboration that avoids building everything from scratch. MPS addresses challenges like file wrangling and inconsistent media handling, automating tasks and minimizing errors. The FLAPI is used to inspect camera file metadata, normalize it, and make it searchable, ensuring consistency. FLAPI also generates VFX plates and deliverables with accurate color management and debayering. MPS is cloud-integrated using Cosmos, allowing for CPU-only instances in a Dockerized environment, maximizing efficiency. Production workloads are handled elastically to scale on demand, ensuring quick turnaround times. MPS is designed for both experienced teams and those needing guidance, with FLAPI handling the complexities. This partnership allows for open communication, joint validation, and benefits the standardization ecosystem. The overall impact is fewer delays, faster turnaround, and more efficient production processes.

https://netflixtechblog.com/scaling-camera-file-processing-at-netflix-6dab2b1e80be?source=rss----2615bd06b42e---4 netflixtechblog.com

RSS Hunter • Apr 24

The Human Infrastructure: How Netflix Built the Operations Layer Behind Live at Scale

Netflix's live streaming operations have rapidly expanded since their first live show, now streaming multiple events daily. Initial live shows relied on engineers and improvised setups, lacking dedicated operational teams. The evolution required building dedicated Broadcast Operations Centers (BOCs) to manage the live broadcast process. To ensure signal reliability, Netflix enforces strict specifications on video and audio feed contributions from venues. Netflix's operational model evolved through several phases, from engineering-led operations to specialized engineering teams. The Transmission Operations Center (TOC) model was developed to improve efficiency and manage a high volume of events concurrently. Within the TOC model distinct roles like Transmission, Streaming, and Broadcast Control Operators are crucial. For high-profile events, a specialized "Big Bet" model provides dedicated resources. The Live Command Center (LCC) monitors the entire live streaming pipeline, offering real-time data to operators. The LCC uses a custom observability stack to manage massive data volumes and react to issues promptly. LCC Operations Leads and Technical Launch Managers are responsible for incident response.

https://netflixtechblog.com/the-human-infrastructure-how-netflix-built-the-operations-layer-behind-live-at-scale-33e2a311c597?source=rss----2615bd06b42e---4 netflixtechblog.com

RSS Hunter • Apr 17

Evaluating Netflix Show Synopses with LLM-as-a-Judge

Netflix aims to improve the synopsis selection process for its vast content library. Thousands of titles complicate the user's choice, making synopses crucial for decision-making. High-quality synopses enhance user engagement, while poor ones lead to frustration and abandonment. The challenge lies in scaling quality validation across hundreds of thousands of synopses. Netflix developed an LLM-based approach to evaluate synopsis quality across four key dimensions. This system achieves over 85% agreement with creative writers, ensuring expert-driven quality standards. Synopsis quality is assessed through both creative quality defined by writers and member implicit feedback via streaming metrics. The LLM-as-a-Judge system employs techniques like tiered rationales, consensus scoring, and factuality agents to optimize accuracy. These LLM-derived quality scores are correlated with key streaming metrics like take fraction and abandonment rate. This allows Netflix to proactively identify and fix synopsis issues, improving the member experience and content discovery.

https://netflixtechblog.com/evaluating-netflix-show-synopses-with-llm-as-a-judge-6269251e6f28?source=rss----2615bd06b42e---4 netflixtechblog.com

RSS Hunter • Apr 10

Stop Answering the Same Question Twice: Interval-Aware Caching for Druid at Netflix Scale

Netflix implemented an experimental caching layer to improve Druid query performance at its massive scale. This caching system addresses the problem of redundant queries from dashboards, especially during high-profile events. The core idea is to cache portions of results and only query Druid for the freshest data, introducing a small staleness trade-off. Exponential time-to-live (TTL) values are used, with older data cached longer to handle late-arriving events. The caching system uses a map-of-maps structure for efficient range scans of cached data. The cache intercepts requests at the Druid Router, serving results or querying Druid as needed. The solution combines cached and fresh data, returning a complete result and caching the fresh data asynchronously. Negative caching is used for empty buckets, with precautions against caching trailing empty buckets. The system uses Netflix's Key-Value Data Abstraction Layer (KVDAL), backed by Cassandra, which offers independent TTLs on each data point. The caching layer has significantly reduced Druid query load and improved query times, particularly during high-volume events. The caching system is still experimental, with future goals including integration into Druid itself.

https://netflixtechblog.com/stop-answering-the-same-question-twice-interval-aware-caching-for-druid-at-netflix-scale-22fadc9b840e?source=rss----2615bd06b42e---4 netflixtechblog.com

RSS Hunter • Apr 6

Powering Multimodal Intelligence for Video Search

The article discusses the challenges and solutions for building an advanced video search engine. The core problem is the overwhelming amount of video footage and the difficulty in extracting relevant moments. The solution involves a multimodal approach, integrating various AI models to analyze different aspects of video content. This approach requires unifying diverse data from specialized models into a cohesive, real-time intelligence system. The system's architecture focuses on processing at scale, handling billions of data points efficiently. A three-stage process involving transactional persistence, offline data fusion, and indexing for real-time search ensures data integrity and responsiveness. The process uses a decoupled pipeline to avoid bottlenecking during the ingestion process. The search service offers features like query preprocessing, fine-tuning semantic search, and advanced textual analysis for precise results. It also includes phrase matching, N-gram analysis, and fuzzy matching to enhance search accuracy. The system provides aggregations and flexible grouping to allow for more nuanced searching. Overall, the aim is to empower filmmakers by providing a powerful tool for discovering relevant video content.

https://netflixtechblog.com/powering-multimodal-intelligence-for-video-search-3e0020cf1202?source=rss----2615bd06b42e---4 netflixtechblog.com

RSS Hunter • Apr 4

Smarter Live Streaming at Scale: Rolling Out VBR for All Netflix Live Events

Netflix transitioned its Live events from Constant Bitrate (CBR) to Variable Bitrate (VBR) encoding for improved efficiency. VBR dynamically adjusts bitrate based on scene complexity, leading to reduced average traffic and improved quality of experience. A major challenge with VBR is the unpredictability of traffic, which can lead to server overload and instability due to large bitrate swings. To address this, Netflix implemented a new capacity reservation system based on nominal bitrates, mitigating the risks associated with VBR's variable nature. They also adjusted nominal bitrates across their streaming ladder to match the quality of CBR streams, using VMAF as a guiding metric. The shift to VBR allowed for efficiency gains like lower rebuffering and initial start delays. Netflix is now working on improving its adaptive bitrate algorithms and capacity reservation to further optimize VBR. The implementation of VBR was a company-wide project involving several teams and contributors.

https://netflixtechblog.com/smarter-live-streaming-at-scale-rolling-out-vbr-for-all-netflix-live-events-c8f833b238cc?source=rss----2615bd06b42e---4 netflixtechblog.com

RSS Hunter • Apr 2

Scaling Global Storytelling: Modernizing Localization Analytics at Netflix

Netflix's Analytics Engineering community gathered for an Analytics Summit to share insights across the business. The core focus is on enhancing the localization process for a global audience of over 300 million members. This involves scaling dubbing and subtitling to meet the needs of diverse language preferences. Previously, localization metrics were fragmented, leading to inconsistencies and maintenance issues. Netflix is modernizing its analytics through consolidation, standardization, and trust. The first pillar involves auditing and consolidating dashboards and backend pipelines. The second pillar tackles "Not-So-Tech Debt" by improving user experience and reporting on unified metrics. The third pillar is investing in a write-once, read-many architecture by centralizing business logic. This approach creates unified tables, solving problems once and propagating updates instantly. The future involves event-level analytics, analyzing granular timed-text data like subtitle lines. The goal is to understand how subtitle characteristics impact member engagement and refine guidelines. Ultimately, this modernization effort aims to improve the Netflix experience for all members worldwide.

https://netflixtechblog.com/scaling-global-storytelling-modernizing-localization-analytics-at-netflix-816f47290641?source=rss----2615bd06b42e---4 netflixtechblog.com

RSS Hunter • Mar 6

Optimizing Recommendation Systems with JDK’s Vector API

Netflix's Ranker service, responsible for personalized recommendations, had a CPU hotspot in its video serendipity scoring feature. Initially, the scoring utilized a costly, nested loop structure for calculating similarities. The first optimization involved batching calculations to leverage matrix multiplication, improving the algorithm. However, an initial implementation using matrix multiplication introduced overhead, like GC pressure and poor cache behavior. Further optimization involved switching to flat buffers, improving data layout and cache locality. Then, a thread-local buffer reuse strategy eliminated per-request allocations. BLAS libraries were explored but didn't provide significant gains due to JNI overhead. Finally, the JDK Vector API was integrated, enabling SIMD, with a fallback to optimized scalar code for compatibility. This streamlined approach resulted in a 7% drop in CPU utilization and a 12% drop in average latency. Ultimately, the CPU consumption of the scoring feature was reduced from 7.5% down to roughly 1%, and the CPU per request significantly decreased by 10%.

https://netflixtechblog.com/optimizing-recommendation-systems-with-jdks-vector-api-30d2830401ec?source=rss----2615bd06b42e---4 netflixtechblog.com

RSS Hunter • Mar 3

Mount Mayhem at Netflix: Scaling Containers on Modern CPUs

Netflix faced container scaling bottlenecks during their container runtime modernization. The issue arose because the new runtime used user namespaces for enhanced container security, employing kernel idmap features for efficient resource allocation, triggering frequent mount operations. These mount operations heavily contended for global kernel locks, particularly affecting nodes with many container layers. This contention was amplified on r5.metal instances, which have dual NUMA architecture. Benchmarks revealed instance type differences, with older Intel architectures showing higher container launch times. The bottleneck was rooted in Linux VFS path lookup code and was linked to the kernel's mount table's global locking. NUMA architecture amplified the problem due to memory access latency. Disabling hyperthreading improved performance by reducing resource competition. Centralized cache architectures in some CPUs exacerbated lock contention due to a single queueing structure. Distributed cache architectures, like those in AMD chips, showed better performance as contention was spread across multiple domains. Microbenchmarks validated the impact of NUMA, hyperthreading, and CPU architecture.

https://netflixtechblog.com/mount-mayhem-at-netflix-scaling-containers-on-modern-cpus-f3b09b68beac?source=rss----2615bd06b42e---4 netflixtechblog.com

RSS Hunter • Feb 28

MediaFM: The Multimodal AI Foundation for Media Understanding at Netflix

Netflix developed MediaFM, an in-house multimodal content embedding model to understand its vast catalog. This model aims to create a machine-readable understanding of content by integrating audio, video, and text. MediaFM uses a Transformer-based encoder to generate shot-level embeddings, capturing temporal relationships. The model's inputs are shot-level embeddings from video, audio, and timed text, concatenated and processed. A Masked Shot Modeling objective trains the model to predict original embeddings. Evaluation involves task-specific linear layers, proving MediaFM's superior performance compared to baselines. The model significantly improves in tasks demanding nuanced understanding, such as ad relevancy. Contextualization plays a critical role with an added modality. MediaFM benefits applications across Netflix, including ad relevancy and clip popularity prediction. The team is now exploring how to use pre-trained multimodal LLMs for future model development. The blog post concludes with gratitude for contributors and a glimpse into future work.

https://netflixtechblog.com/mediafm-the-multimodal-ai-foundation-for-media-understanding-at-netflix-e8c28df82e2d?source=rss----2615bd06b42e---4 netflixtechblog.com

RSS Hunter • Feb 23

Scaling LLM Post-Training at Netflix

Netflix developed a post-training framework to adapt large language models (LLMs) for their specific needs in recommendation, personalization, and search. This framework addresses the complexities of post-training at scale, including data pipelines, distributed computing, and workflow orchestration. The framework aims to help model developers focus on model innovation rather than infrastructure. Key challenges in post-training include data preparation, model sharding, and setting up training loops for production. The Netflix framework provides modular components for data, model, compute, and workflow management to simplify training. The framework supports SFT, DPO, RL, and knowledge distillation training recipes with reusable utilities. The framework evolved to support RL by integrating open-source tools and adopting a hybrid execution model. It prioritizes compatibility with the Hugging Face ecosystem to facilitate easy integration of models and tools. The post-training framework allows teams to experiment with advanced techniques and iterate more quickly.

https://netflixtechblog.com/scaling-llm-post-training-at-netflix-0046f8790194?source=rss----2615bd06b42e---4 netflixtechblog.com

RSS Hunter • Feb 13

Automating RDS Postgres to Aurora Postgres Migration

In 2024, Netflix standardized on Amazon Aurora PostgreSQL for its relational database needs after evaluating various technologies. This decision was driven by PostgreSQL's existing prevalence, industry momentum, and Aurora's scalability and feature advantages. A key initiative was migrating existing users to Aurora PostgreSQL, starting with RDS migrations. Migrations involve more than data transfer; they require orchestrated data and functionality transitions. Netflix faced substantial operational challenges with nearly 400 PostgreSQL clusters, necessitating a self-service migration workflow. This workflow addressed issues like zero data loss, minimal downtime, and the lack of direct access to credentials, promoting a seamless process. The chosen strategy utilized Aurora Read Replicas, trading complexity for reduced client downtime. Netflix's Data Access Layer managed secure connections between applications and databases, facilitating transparent migrations. A fully automated, self-service workflow was developed, orchestrating the entire migration process without manual intervention. The process included enabling backups, porting parameters, and creating an Aurora read replica for continuous synchronization. The quiescence phase focuses on a smooth transition to the Aurora cluster.

https://netflixtechblog.com/automating-rds-postgres-to-aurora-postgres-migration-261ca045447f?source=rss----2615bd06b42e---4 netflixtechblog.com

RSS Hunter • Feb 12

The AI Evolution of Graph Search at Netflix

Netflix's Graph Search platform is evolving to incorporate natural language search using AI, addressing limitations of its structured query language. Natural language search simplifies data retrieval, improving user experience across various applications. The core challenge involves translating user queries into valid Graph Search Filter statements. This process uses Large Language Models (LLMs) to generate queries, incorporating context engineering for accuracy. Context engineering leverages GraphQL schema information and controlled vocabularies to inform the LLM. Retrieval-Augmented Generation (RAG) is employed to manage context, focusing on relevant fields and controlled vocabulary values. Field RAG and Controlled Vocabularies RAG enhance precision by matching user intent to index elements. The process involves vector search and deduplication for efficient context management. The system provides instructions to the LLM to generate syntactically, semantically, and pragmatically correct filters. The effectiveness of the solution requires careful parameter tuning for RAG components. The described work paves the potential for building a RAG system on top of Graph Search. Future articles will detail the implementation, performance evaluation, and platform evolution.

https://netflixtechblog.com/the-ai-evolution-of-graph-search-at-netflix-d416ec5b1151?source=rss----2615bd06b42e---4 netflixtechblog.com

RSS Hunter • Jan 26

How Temporal Powers Reliable Cloud Operations at Netflix

Netflix adopted Temporal, a durable execution platform, to address deployment failures in its Spinnaker continuous delivery platform. Spinnaker, used for most Netflix deployments, faced issues in its Orca and Clouddriver services, particularly with Cloud Operations. Cloud Operations, responsible for cloud infrastructure changes, experienced transient failures and operational complexity within Clouddriver. This led to a 4% failure rate in deployments, impacting engineering productivity. Temporal offers Workflows and Activities, enabling resilient execution despite failures. Netflix migrated Cloud Operations to Temporal, simplifying the process and reducing the failure rate. The migration involved transitioning Cloud Operations into Temporal workflows, leveraging the platform's retry mechanisms and state management. The integration required abstracting and dynamically configuring the system. The result of this migration was a significant reduction in deployment failures, down to 0.0001%. This improved reliability and engineering efficiency.

https://netflixtechblog.com/how-temporal-powers-reliable-cloud-operations-at-netflix-73c69ccb5953?source=rss----2615bd06b42e---4 netflixtechblog.com

RSS Hunter • Dec 15, 2025

Netflix Live Origin

The Netflix Live Origin is a custom-built origin server that plays a crucial role in the company's live streaming pipeline, acting as a broker between the cloud live streaming pipelines and the distribution system, Open Connect. The Live Origin is a multi-tenant microservice operating on EC2 instances within the AWS cloud, using standard HTTP protocol features to communicate with the Live Origin. The architecture of the Live Origin is influenced by key technical decisions, including resilience achieved through redundant regional live streaming pipelines and the implementation of epoch locking at the cloud encoder. The Live Origin features multi-pipeline and multi-region awareness, allowing it to select the first valid segment from each pipeline in a deterministic order. The system also detects segment defects via lightweight media inspection at the packager and provides this information as metadata when the segment is published to the Live Origin. To optimize interactions with the Origin Server, the proxy-caching functionality of nginx has been extended to address Live-specific needs, including millisecond grain caching and the ability to hold open requests for segments that are not yet available. The Live Origin also provides streaming metadata enhancement through the use of HTTP headers, which can be used to convey notifications of events within the stream to client devices. The system includes an invalidation system that can be used to flush all content associated with an event, as well as an enhanced cache invalidation system that takes into account the encoding pipeline and region used to generate each segment. The Origin storage architecture was initially based on AWS S3 but was later optimized to meet the unique latency and workload requirements of live streaming, using a KeyValue Storage Abstraction that leverages Apache Cassandra to provide chunked storage of large values.

https://netflixtechblog.com/netflix-live-origin-41f1b0ad5371?source=rss----2615bd06b42e---4 netflixtechblog.com

RSS Hunter • Dec 15, 2025

AV1 — Now Powering 30% of Netflix Streaming

Netflix has been working to deliver the best possible entertainment experience to its members, and one key technology enabling this is AV1, a modern open video codec. AV1 powers approximately 30% of all Netflix viewing, marking a major milestone in efforts to bring more efficient and higher-quality streaming to members. Netflix co-founded the Alliance for Open Media to develop and promote next-generation open source media technologies, with AV1 being the first major project. The AV1 codec was officially released in 2018, with goals to deliver significant improvements in compression efficiency and introduce rich features that enable new use cases. Netflix first brought AV1 streaming to Android devices in 2020, which proved valuable for mobile users who are mindful of their data usage and network conditions. The success of AV1 on Android motivated Netflix to expand support to smart TVs and other large-screen devices, where most members watch their favorite shows. Today, AV1 accounts for approximately 30% of all Netflix streaming, making it the second most-used codec, and it's on track to become number one soon. AV1's superior compression efficiency has allowed Netflix to provide high-quality streaming experiences using less data, making it more accessible and reliable. The company is also exploring AV1's unique features to unlock advanced and immersive experiences for members, including high-dynamic-range and cinematic film grain. Netflix sees significant opportunities for AV1 beyond traditional video-on-demand streaming, including live streaming and cloud gaming, and is working to productize AV1 for these use cases.

https://netflixtechblog.com/av1-now-powering-30-of-netflix-streaming-02f592242d80?source=rss----2615bd06b42e---4 netflixtechblog.com

RSS Hunter • Dec 4, 2025

Behind the Streams: Live at Netflix. Part 1

Three years ago, Netflix asked how they could entertain the world through live streaming, a format almost as old as television itself. This question led to the development of hundreds of live events, including comedy shows, sports, and WWE events. In a series called "Behind the Streams," Netflix will share the technical journey of building live streaming capabilities. Live streaming introduced new considerations for architecture and technology choices, requiring significant building to make it work well on Netflix. The key pillars of Netflix's live architecture include dedicated broadcast facilities, cloud-based redundant transcoding and packaging pipelines, scaling live content delivery, optimizing live playback, and running discovery and playback control services in the cloud. Netflix also centralized real-time metrics in the cloud with specialized tools and facilities. Building live functionality brought fresh challenges and opportunities to learn, and Netflix is still learning every day how to deliver live events more effectively. Key learnings so far include the importance of extensive testing, regular practice, viewership predictions, graceful degradation, and retry storms. Netflix has made significant progress in building a robust live streaming system, but there is still more to learn and improve.

https://netflixtechblog.com/behind-the-streams-live-at-netflix-part-1-d23f917c2f40?source=rss----2615bd06b42e---4 netflixtechblog.com

RSS Hunter • Jul 15, 2025

Netflix Tudum Architecture: from CQRS with Kafka to CQRS with RAW Hollow

Tudum.com is Netflix's official fan destination, offering exclusive content, behind-the-scenes insights, and interactive experiences to over 20 million members each month. The platform's architecture is designed to be maintainable, extensible, and flexible, using a server-driven UI approach similar to Command Query Responsibility Segregation (CQRS). Tudum's editorial team creates content, which is stored in a write database, and then converted into a read-optimized format for consumption by users. The initial architecture used Kafka to separate the write and read databases, allowing for independent scaling and eventual consistency. However, this approach introduced a delay between content edits and their reflection on the website. The team identified the source of the delay as the Page Data Service, which used a near cache to accelerate page building. To address this issue, Netflix developed RAW Hollow, an in-memory, co-located, compressed object database that provides strong read-after-write consistency and low latency. Tudum was a perfect fit to test RAW Hollow, which significantly reduced I/O and enabled synchronous data access in O(1) time. The updated architecture eliminated the need for Kafka infrastructure and reduced data propagation times, allowing writers and editors to preview changes in seconds. The migration also led to faster request times, with homepage construction time decreasing from 1.4 seconds to 0.4 seconds.

https://netflixtechblog.com/netflix-tudum-architecture-from-cqrs-with-kafka-to-cqrs-with-raw-hollow-86d141b72e52?source=rss----2615bd06b42e---4 netflixtechblog.com

RSS Hunter • Jul 10, 2025

Driving Content Delivery Efficiency Through Classifying Cache Misses

Netflix's Open Connect program is a content delivery network (CDN) that aims to provide the best quality of experience (QoE) to its members by localizing content delivery through partnerships with internet service providers (ISPs) worldwide. The program uses custom-built servers called Open Connect Appliances (OCAs) that are designed for efficiency and cost-effectiveness. To evaluate the efficiency of Open Connect, Netflix uses a framework to identify sources of inefficiencies, specifically cache misses, which occur when bytes are not served from the best available OCA for a given client. Cache misses are classified into three categories: content misses, health misses, and other misses. Content misses occur when files are not found on OCAs in the local site, while health misses occur when the local site's OCA hardware resources are saturated. Netflix logs two critical data components to compute cache misses: steering playback manifest logs and OCA server logs. These logs are joined to compute detailed cache miss metrics at different aggregation levels. The system architecture for computing cache misses involves log emission, log consolidation, log enrichment, and streaming window-based joins. The data model used to evaluate cache misses allows for replaying logic offline and in simulations with variable parameters to test new conditions and features without impacting production traffic.

https://netflixtechblog.com/driving-content-delivery-efficiency-through-classifying-cache-misses-ffcf08026b6c?source=rss----2615bd06b42e---4 netflixtechblog.com

RSS Hunter • Jul 2, 2025

AV1 @ Scale: Film Grain Synthesis, The Awakening

Netflix has introduced AV1 Film Grain Synthesis (FGS) streams, which preserve the artistic integrity of film grain while optimizing data efficiency. Film grain is a key element in storytelling, adding depth and realism to films, but it's difficult to compress using traditional algorithms. The AV1 FGS tool models film grain through two components: film grain pattern and film grain intensity. The film grain pattern is replicated using an auto-regressive model, while the film grain intensity is controlled by a scaling function. The encoding process removes film grain from the video, compresses it, and transmits the grain's pattern and intensity alongside the compressed video data. During playback, the film grain is recreated and reintegrated into the video using a block-based method. Enabling AV1 FGS has led to significant bitrate reduction, allowing for high-quality video streaming with less data. The technology has also improved visual quality, with synthesized noise effectively masking compression artifacts. Netflix has rolled out FGS across its platform, and users can now enjoy FGS-enabled streams on supported devices. The rollout has resulted in a smoother and more reliable Quality of Experience for Netflix members, with lower bitrate, decreased playback errors, and improved playback stability.

https://netflixtechblog.com/av1-scale-film-grain-synthesis-the-awakening-ee09cfdff40b?source=rss----2615bd06b42e---4 netflixtechblog.com

RSS Hunter • Jul 2, 2025

Model Once, Represent Everywhere: UDA (Unified Data Architecture) at Netflix

As Netflix's offerings grow, so does the complexity of the systems that support it, leading to duplicated and inconsistent models, inconsistent terminology, data quality issues, and inconsistent relationships between data. To address these challenges, a new foundation is needed to define a model once and reuse those definitions everywhere, connecting concepts to real systems and projecting those definitions outward, generating schemas and enforcing consistency across systems. This led to the development of UDA (Unified Data Architecture), which enables modeling domains once and representing them consistently across systems, powering automation, discoverability, and semantic interoperability. UDA allows users and systems to register and connect domain models, catalog and map domain models to data containers, transpile domain models into schema definition languages, move data faithfully between data containers, discover and explore domain concepts, and programmatically introspect the knowledge graph. UDA is a knowledge graph that connects business concepts to schemas and data containers, grounded in an in-house metamodel called Upper, which defines the language for domain modeling in UDA. Upper is a language for formally describing domains and their concepts, and it is used to model data container representations and mappings. UDA adopts a named-graph-first information model, ensuring resolution, modularity, and governance across the entire graph.

https://netflixtechblog.com/uda-unified-data-architecture-6a6aee261d8d?source=rss----2615bd06b42e---4 netflixtechblog.com

RSS Hunter • Jun 12, 2025

FM-Intent: Predicting User Session Intent with Hierarchical Multi-Task Learning

Recommender systems are essential components of e-commerce, streaming media, and social networks, driving significant product and business impact. At Netflix, these systems connect members with relevant content at the right time. The recommendation foundation model has made substantial progress in understanding user preferences, but there is an opportunity to further enhance its capabilities. By extending the foundation model to incorporate the prediction of underlying user intents, the model can enrich its understanding of user sessions beyond next-item prediction. Recent research has highlighted the importance of understanding user intent in online platforms, leading to more accurate and personalized recommendations. FM-Intent, a novel recommendation model, captures a user's latent session intent using short-term and long-term implicit signals as proxies, then leverages this intent prediction to improve next-item recommendations. The model establishes a hierarchical relationship between intent predictions and next-item recommendations, creating a more coherent and effective recommendation pipeline. FM-Intent makes three key contributions: a novel recommendation model, a hierarchical multi-task learning approach, and comprehensive experimental validation showing significant improvements over state-of-the-art models. FM-Intent has been successfully integrated into Netflix's recommendation ecosystem and can be leveraged for several downstream applications, including personalized UI optimization, analytics, and enhanced recommendation signals.

https://netflixtechblog.com/fm-intent-predicting-user-session-intent-with-hierarchical-multi-task-learning-94c75e18f4b8?source=rss----2615bd06b42e---4 netflixtechblog.com

RSS Hunter • May 21, 2025

Behind the Scenes: Building a Robust Ads Event Processing Pipeline

At Netflix, a robust event processing platform was built to monitor, measure, and optimize ad campaigns. The ad serving system relies on a steady stream of ad events to adjust decisions, frequency capping, pacing, and personalization. The initial ad event handling system consisted of three main components: the Microsoft Ad Server, Netflix Ads Manager, and Ad Event Handler. The system was designed to ensure the feedback loop functioned effectively, providing insights on impressions, frequency capping, and monetization processes. As the business expanded, a new persistence layer using Key-Value abstraction was introduced to address challenges such as growth in data volume and third-party tracking URLs. The event processing pipeline was further evolved to support in-house advertising technology, incorporating features such as frequency capping, pricing information, and robust reporting system. A centralized ad event collection system was planned, providing a single unified data contract to consumers and separating concerns between upstream systems and consumers. The new pipeline supported various functions such as measurement, finance/billing, reporting, frequency capping, and maintaining an essential feedback loop back to the ad server. The development of the ads event processing systems has been a carefully orchestrated journey, showcasing teamwork, planning, and coordination across various teams. The new system has significantly accelerated the ability to launch new capabilities for the business, supporting programmatic buying capabilities, sharing opt-out signals, and ensuring accurate reporting and measurement.

https://netflixtechblog.com/behind-the-scenes-building-a-robust-ads-event-processing-pipeline-e4e86caf9249?source=rss----2615bd06b42e---4 netflixtechblog.com

RSS Hunter • May 9, 2025

Measuring Dialogue Intelligibility for Netflix Content

Netflix prioritizes enhancing member experience by collaborating with technology partners to improve tools and workflows. This collaboration focuses on improving dialogue intelligibility throughout the production process, addressing issues from set to screen. A key initiative is the Dialogue Integrity Pipeline, which identifies and mitigates factors impacting clarity, such as noisy environments and audio mixing. Netflix uses industry-standard loudness meters and developed an eSTOI-based measurement system to analyze dialogue intelligibility. To improve pre-delivery optimization, Netflix partnered with Fraunhofer IDMT and Nugen Audio to create the DialogCheck plugin. This plugin, integrated into Digital Audio Workstations (DAWs), provides real-time feedback to sound engineers. The collaboration leverages Fraunhofer's machine learning-based speech intelligibility solution and Nugen Audio's expertise in audio plugins. The DialogCheck plugin allows for early detection and resolution of dialogue clarity issues, ensuring artistic intent isn't compromised. The ultimate goal is delivering immersive and accessible storytelling to all viewers, regardless of listening environment. This collaborative approach underscores Netflix's commitment to audio excellence and innovation in storytelling.

https://netflixtechblog.com/measuring-dialogue-intelligibility-for-netflix-content-58c13d2a6f6e?source=rss----2615bd06b42e---4 netflixtechblog.com

RSS Hunter • May 8, 2025

How Netflix Accurately Attributes eBPF Flow Logs

Netflix uses eBPF to capture TCP flow logs at scale for enhanced network insights, but accurately attributing flow IP addresses to workload identities was a significant challenge. The initial attribution approach relied on Sonar, an internal IP address tracking service, but it led to misattribution due to delays and failures in distributed systems. Misattribution rendered the flow data unreliable for decision-making, and a workaround of holding received flows for 15 minutes before attribution did not eliminate the issue. To solve this problem, Netflix developed a new attribution method that attributes local IP addresses by determining the local workload identity from its environment. For container workloads, Netflix leveraged IPMan, a container IP address assignment service, to attribute local IP addresses. Once local IP addresses are attributed, remote IP addresses can be attributed by learning the time ranges during which each workload owns a given IP address. FlowCollector maintains an in-memory hashmap to represent this knowledge and shares learned time ranges with other nodes using Kafka. The new method achieves accurate attribution and handles transient issues gracefully, and it is also cost-effective due to its simplicity and in-memory lookups. The method is extended to attribute cross-regional IP addresses by forwarding flows to nodes in the corresponding region. Finally, the method is further extended to attribute non-workload IP addresses, such as those belonging to Netflix's content delivery network.

https://netflixtechblog.com/how-netflix-accurately-attributes-ebpf-flow-logs-afe6d644a3bc?source=rss----2615bd06b42e---4 netflixtechblog.com

RSS Hunter • Apr 8, 2025

Globalizing Productions with Netflix’s Media Production Suite

The film industry's shift to cloud-based workflows faces challenges in global implementation. Netflix aims to solve these issues with its Media Production Suite (MPS), designed for filmmakers. The MPS streamlines media management, eliminating tedious tasks and boosting creative focus. Traditional workflows using physical tapes are slow and cumbersome, hindering collaboration. Digital workflows also face challenges in distribution and standardization. The cloud offers solutions but requires overcoming operational and technological hurdles. Netflix addresses these issues by bringing people and applications to the media instead of the reverse. MPS tackles global disparities in technology and standardization, considering diverse market needs. The suite automates processes like color management and framing using industry standards. The infrastructure combines cloud and physical capabilities optimized for user performance. MPS includes tools for ingest, media library, dailies, remote workstations, and more. Over 350 titles have utilized MPS tools, with feedback from diverse global regions. The Brazilian series "Senna" adopted MPS, showcasing its ability to overcome geographical barriers.

https://netflixtechblog.com/globalizing-productions-with-netflixs-media-production-suite-fc3c108c0a22?source=rss----2615bd06b42e---4 netflixtechblog.com

RSS Hunter • Mar 31, 2025

Foundation Model for Personalized Recommendation

Netflix is developing a foundation model for recommendations to centralize member preference learning and improve efficiency across various recommendation models. The current system has specialized models that are costly to maintain and struggle to share innovations. The foundation model aims to learn from extensive user interaction histories and content data, distributing these learnings to other models. Inspired by large language models, the approach shifts to a data-centric strategy using semi-supervised learning. Netflix tokenizes user interactions, balancing data granularity and sequence compression to handle long interaction histories. Sparse attention mechanisms and sliding window sampling are used to manage computational efficiency during training. Each token contains rich, heterogeneous information about the action and content, utilizing request-time and post-action features. The model employs an autoregressive next-token prediction objective, similar to GPT, but with modifications to account for varying interaction importance. The model predicts multiple tokens and uses auxiliary prediction objectives to capture long-term dependencies and improve accuracy. Addressing entity cold-starting, the model is built with incremental training and the ability to infer with unseen entities by using metadata information of entities and inputs.

https://netflixtechblog.com/foundation-model-for-personalized-recommendation-1a0bd8e02d39?source=rss----2615bd06b42e---4 netflixtechblog.com

RSS Hunter • Mar 29, 2025