Netflix TechBlog | Medium - TheNote.app

Netflix TechBlog | Medium
Follow

Netflix Tech Blog offers insights into how Netflix handles technology. They provide research on data science, engineering, design, and technology innovations. They showcase their innovations, like their proprietary content delivery network and provide insights into their service reliability efforts.

Netflix TechBlog - Medium netflixtechblog.com

RSS netflixtechblog.com

RSS Hunter • Aug 19, 2024

Thread Of Notes

Ontology-Driven Observability: Building an End-to-End Knowledge Graph at Netflix Scale

The company faces challenges in understanding its massive, complex infrastructure, especially during live events like the Jake Paul and Mike Tyson fight which drew 65 million streams. Debugging issues, such as a slow-rendering home screen, can take hours because information is siloed across different systems. These systems collect data through metrics, events, logs, and traces (MELT), but correlating this data effectively is difficult. The core problem is the lack of a shared, machine-readable understanding of operational entities.To address this, they developed an "Observability Ontology," a formal specification of types, properties, and relationships within their operational world. This ontology uses standards like RDF, OWL, SHACL, and SPARQL, storing data as a knowledge graph. A key design decision was to avoid a generic "Service" type, instead using specific, validated types like ops:Application. They also chose to model the reasoning and context from incident discussions, transforming unstructured chat into structured triples.The ontology creates determinism from inherently stochastic processes like LLMs and fuzzy matching through a pipelined approach. This pipeline uses "enrichers" that reduce entropy by converting noisy input into typed triples, following strict contracts. These enrichers ensure every entity has a type, URIs follow a naming scheme, and applications are validated against a catalog. The pipeline runs sequentially to avoid race conditions, with external calls being best-effort to prevent blocking.A "Correlation Engine" then assimilates signals from various sources, identifying causal or correlatory chains of events. It analyzes thirteen signal types, proposing hypotheses with evidence and confidence scores, storing supporting graph records. Each cycle of observation, enrichment, correlation, and storage forms a "Knowledge Flywheel," continuously enriching the knowledge graph and improving system context. This flywheel aims to reduce debugging time by providing a unified and interconnected view of system operations.

https://netflixtechblog.com/ontology-driven-observability-building-an-end-to-end-knowledge-graph-at-netflix-scale-421392cf5adf?source=rss----2615bd06b42e---4 netflixtechblog.com

RSS Hunter • Yesterday

Integrating Netflix’s Foundation Model into Personalization applications

Netflix developed a Foundation Model for personalization to centralize member preference learning. This powerful model, trained on extensive user interaction and content data, distributes its learnings to other models. Integrating this large model into production systems presented a challenge, leading Netflix to experiment with three distinct approaches: embeddings, subgraph, and fine-tuning. Each method caters to different application needs regarding latency, tech stack, and model utilization.The embeddings approach leverages pre-computed user profile and item representations from the Foundation Model, stored in an Embedding Store. While straightforward and cost-effective, it can lead to stale recommendations due to the time gap between computation and inference. Netflix enhances this with daily fine-tuning and an embedding stabilization technique. A near-real-time framework is being developed to address staleness and improve adaptiveness.The subgraph integration treats the Foundation Model decoder as a component within an application's larger model graph. This allows for deeper integration, uses up-to-date learnings, and unlocks application-specific value by accessing specific Foundation Model layers. However, it introduces complexity in feature generation and increases model size and inference time. This approach is reserved for high-impact use cases where metric improvements justify the costs.The fine-tuning approach adapts the Foundation Model for specific applications by further training it on product-specific data and objectives. This offers optimized performance and serves as a baseline for new applications. While requiring careful optimization for latency and SLAs, it can lead to a proliferation of models and pipelines. Netflix provides a fine-tuning framework to simplify this process.Ongoing innovations include near-real-time embedding inference and smaller, distilled Foundation Model versions to reduce latency. Standardized APIs and a robust fine-tuning framework aim to make leveraging the Foundation Model easier for application teams. These three integration patterns provide a comprehensive strategy for Netflix applications to benefit from the Foundation Model.

https://netflixtechblog.com/integrating-netflixs-foundation-model-into-personalization-applications-cf176b5860eb?source=rss----2615bd06b42e---4 netflixtechblog.com

RSS Hunter • Oct 14

DataJunction as Netflix’s answer to the missing piece of the modern data stack

Netflix faced challenges with scattered, inconsistently defined, and untrustworthy metrics, particularly within its experimentation platform. This complexity hindered data scientists and analytics engineers, making metric onboarding take weeks through the previous system. To address this, Netflix explored semantic layer solutions and chose DataJunction due to its robust SQL parsing and generation engine, along with its graph-based metadata storage. DataJunction originated at Facebook to address metric portability issues and was later open-sourced and expanded upon by Netflix engineers. A key architectural decision was to prioritize an API-first approach, focusing on a solid semantic layer engine and API with SQL input and output capabilities. The platform adopted multi-dimensional cubes as a first-class feature for familiar user interfaces and efficient data delivery. DataJunction's ability to link metric definitions back to the source simplifies troubleshooting and ad-hoc analysis. The platform's integration into Netflix's experimentation domain has streamlined metric and dimension definition and management. Future plans include unifying general analytics at Netflix by building stronger integrations with other tools. This will enable easier discovery, usage, and experimentation with metrics across the Netflix data stack.

https://netflixtechblog.com/datajunction-as-netflixs-answer-to-the-missing-piece-of-the-modern-data-stack-92af926b40a5?source=rss----2615bd06b42e---4 netflixtechblog.com

RSS Hunter • Oct 14

Measuring the Impact of Personalized Recommendations

Personalization is a key feature of online platforms like Netflix, influencing user discovery and engagement. However, measuring the true value of personalization presents a significant challenge, as recommendations are targeted based on predicted user preferences, making it difficult to disentangle recommendation effects from user appeal. A naive approach of counting recommended views is insufficient because it doesn't establish causality. Running experiments to turn off personalization entirely would negatively impact user experience and is therefore not a viable measurement strategy.Instead, researchers developed a hybrid approach combining a behavioral model with small-scale A/B tests. This model estimates how recommendations influence both viewing decisions and title selection by separating underlying user preferences from the incremental impact of a recommendation. The model represents member preferences dynamically, capturing how past viewing history shapes current tastes. It explicitly models the "bonus" effect of recommendations on title selection, reflecting increased awareness and information.To validate the model's accuracy in predicting substitution patterns, it was compared against results from randomized experiments. These experiments introduced exogenous variation in title visibility, mimicking how recommendation algorithms change. The model demonstrated strong predictive power, with its simulated diversion ratios showing high correlation with experimental estimates.Using the validated model, researchers conducted counterfactual analyses to compare the current Netflix recommender system against baselines like random, popularity-based, and basic matrix factorization recommendations. The current system significantly outperformed these baselines in terms of aggregate engagement, reducing it by 16% for random recommendations and 12% for popularity-based ones. Even compared to matrix factorization, the current system showed a 4% gain in engagement.Beyond just increasing viewing, the model also assessed consumption diversity. The current system maintained a broader distribution of viewership across titles, unlike popularity-based and matrix factorization recommendations which led to increased viewership concentration. Random recommendations offered more diversity but at a considerable cost to engagement. Therefore, personalization not only drives more viewing but also matches users to a more diverse set of titles, reflecting varied tastes.

https://netflixtechblog.com/measuring-the-impact-of-personalized-recommendations-4c26be3a4d96?source=rss----2615bd06b42e---4 netflixtechblog.com

RSS Hunter • Oct 14

TimeSeries Tiered Storage Journey: Kafka/Flink Streams to Native Cassandra Cold Reads

Netflix's TimeSeries Abstraction platform, crucial for petabyte-scale temporal data, faces significant storage cost challenges. To address this, two solutions have been implemented to move older, less accessed data to cheaper object storage while maintaining low-latency access. The first solution uses Apache Kafka and Flink to stream data to S3, followed by a daily compaction job that merges and optimizes Parquet files. This compaction process involves creating small row groups, sorting events by time, and using Bloom filters for efficient reads. The compacted files are then registered in the control plane, allowing hot storage truncation. Reads are fanned out across hot and cold tiers, with the cold tier leveraging compaction metadata to minimize S3 reads.Despite success, the streaming infrastructure introduced complexity and cost. This led to the development of a second, more integrated solution leveraging Cassandra backups. This approach aims to read data directly from S3 backups without separate data transfer. It introduced a new Cassandra compaction strategy called ReadOnlyCompactionStrategy (ROCS) for immutable time slices. ROCS aligns SSTables with token ranges, reducing read fan-out and improving latency across all tiers.Cassandra backs up its data to S3, creating manifests that list SSTable paths. The second solution customizes Cassandra to read directly from these S3 backups for immutable time slices. It uses pluggable Memtables, specifically a "backupmemtable," to redirect reads. This backup Memtable downloads necessary index and metadata files from S3 to the node. Robust resilience mechanisms ensure reliable component downloads and initialization. Once local components are ready, Cassandra serves reads from these backups, parallelizing S3 reads and utilizing caching for further optimization. This strategy significantly reduces operational overhead and infrastructure costs by eliminating separate streaming infrastructure.

https://netflixtechblog.com/timeseries-tiered-storage-journey-kafka-flink-streams-to-native-cassandra-cold-reads-e59d597c9d60?source=rss----2615bd06b42e---4 netflixtechblog.com

RSS Hunter • Oct 14

Behind the Scenes: Evolving Netflix’s Ads Event Pipeline for Live — Part II

Netflix rebuilt its Ads Event Publisher to handle increased scale and complexity, particularly for live streaming advertisements. The previous system relied on a Key-Value store for ad metadata enrichment. This approach, while functional, presented scaling bottlenecks and critical path dependencies that were unsuitable for live events. Live streams generate massive, spiky traffic requiring a more robust and elastic solution.The new architecture utilizes Apache Flink, a stateful stream processing framework, for near real-time enrichment. Ad server response logs are sent to Kafka instead of a database. Flink then performs a real-time streaming join between these responses and client-generated ad events. This eliminates direct database writes on the critical serving path, preventing outages and improving scalability.The Flink job normalizes events, routes them to the correct processing region, and importantly, performs a stateful join. It holds recent ad serving responses in memory and matches incoming client events against them within configurable windows, typically 60 minutes. This approach allows for on-the-spot enrichment when both events are available. If an event arrives first, it's held until its corresponding response appears.To address cross-region enrichment, the system routes client events to the region that served the ad, rather than replicating all serving responses globally, which is more cost-effective. The ad token now includes the serving region information. Although the 60-minute window covers most events, a separate path exists to recover late-arriving events, ensuring data completeness for billing and reporting. This stateful stream processing architecture provides the necessary scalability and resilience for Netflix's evolving advertising needs.

https://netflixtechblog.com/behind-the-scenes-evolving-netflixs-ads-event-pipeline-for-live-part-ii-826ebf9ad9fb?source=rss----2615bd06b42e---4 netflixtechblog.com

RSS Hunter • Oct 14

Netflix Conductor: The Next Chapter

Netflix Conductor has evolved significantly beyond its initial open-source release. While maintenance of the open-source version ceased, internal development accelerated, leading to a complete rebuild. Conductor's scale has exploded, processing billions of workflow executions annually, a stark contrast to its earlier milestone achievements. This growth necessitated modernization across nearly every system layer, from the data plane to the evaluation engine. Conductor now supports a vast number of workflow definitions and is integral to various Netflix businesses, including Content and Studio Engineering, Ads, and Games. The Media Production Suite, for instance, relies on Conductor for critical media ingestion and processing pipelines. Architecturally, Conductor transitioned from a Dynomite-backed stack to using Cassandra as its primary datastore and Timestone for queueing, significantly improving scalability and reliability. The introduction of Conductor 3.0 addressed foundational issues, with Conductor 4.0 rebuilding the core evaluation engine. This rewrite partitioned data for improved memory efficiency and performance, and introduced an asynchronous evaluation model to eliminate synchronous bottlenecks. The overall result is a more robust, scalable, and performant workflow orchestration engine capable of handling Netflix's massive operational demands.

https://netflixtechblog.com/netflix-conductor-the-next-chapter-41ad21067649?source=rss----2615bd06b42e---4 netflixtechblog.com

RSS Hunter • Oct 14

Running Apache Spark experiments in my sleep (and on a plane)

Netflix's member-sessionizer, a critical data pipeline, processes billions of events to create user viewing sessions. Standard debugging approaches failed at Netflix's production scale, as issues like Out-Of-Memory errors only appeared with large, unsampled datasets due to data skew. The author's breakthrough was shifting focus from speeding up single runs to enabling multiple concurrent experiments. This was achieved by leveraging Dataflow's branch-driven development to isolate each test variant.By parameterizing the scheduler and assigning each experiment its own branch and output table, multiple tests could run simultaneously. This drastically reduced total execution time from hours per test to the duration of the longest single job. An AI agent was employed to manage the operational complexity of running many parallel experiments, automating configuration updates, job triggering, and metric polling. This freed the author from tedious manual checks, allowing for immediate comparison of results.Persistent tmux sessions on Netflix's cloud development environment proved crucial for preventing multi-hour Spark jobs from being interrupted by network drops or laptop closures. The core performance bottlenecks were identified as the collect_list operator buffering entire event bags in memory and inefficient data structures. Specifically, changing session IDs to UUID strings and using Scala Options for fields significantly increased memory usage. Additionally, bloated string attributes added unnecessary weight to each event.The fixes involved unboxing primitive fields from Options and slimming down event bags by removing unused attributes. Session-invariant metadata was hoisted to the session header. Two counterintuitive findings highlighted the complexities of Spark memory management: narrowing rows could trigger faster OOMs by forcing broadcast joins, and Adaptive Query Execution silently accumulated large amounts of data in memory through broadcast hash joins. Tuning AQE's broadcast threshold resolved this.A disciplined logging system tracked each experiment, its configuration, and key metrics, creating a single source of truth and forcing isolated variable changes. A mistake was made by testing a session capping fix before the Spark Window stage, which buffered memory before truncation. The final corrected fix moved the truncation upstream to prevent memory bloat. While these tuning efforts solved immediate problems, the fundamental data shape of aggregating entire sessions into single rows poses a future scaling challenge.

https://netflixtechblog.com/running-apache-spark-experiments-in-my-sleep-and-on-a-plane-588816b6b5f8?source=rss----2615bd06b42e---4 netflixtechblog.com

RSS Hunter • Oct 14

The Lifecycle of LLM-as-a-Judge: Building, Aligning, and Monitoring at scale

Large-scale text generation with LLMs necessitates automated evaluation due to manual review limitations. The LLM-as-a-Judge approach uses a second LLM to score a primary LLM's output, but raises concerns about the evaluator's trustworthiness. This problem is addressed by building, aligning, and continuously monitoring LLM judges, ensuring they remain anchored in human judgment at every stage. Humans define quality standards, create labeled datasets, monitor judge behavior, and approve updates.A reliable LLM judge is not static; it has a lifecycle involving human ownership in its creation, alignment, operation, and maintenance. This process is illustrated through the task of generating recommendation explanations, which are short messages explaining item recommendations. Quality is crucial as misleading explanations erode user trust.The LLM judge lifecycle comprises four phases: Birth, Training, Deployment, and HITL Alignment & Drift Monitoring. Birth involves human experts crafting benchmark data, including adversarial and boundary examples, which are then labeled by human raters. Training refines the LLM judge to align with human labels and rationales, using self-reflection and rubric updates. Deployment integrates the judge as a gate and critic within the explanation generation system.HITL Alignment & Drift Monitoring ensures continued trustworthiness by periodically benchmarking the judge against human raters and requiring human approval for any system changes. This phase uses a Meta-Judge to compare the LLM judge's rationale with human rationales, catching cases where the judge arrives at the correct answer for the wrong reason. This continuous feedback loop, incorporating new human-labeled data and signals, feeds back into the benchmark and development phases. The system components include a Generator, Judge, Reflector, Meta-Judge, and human experts and raters, all working together across the lifecycle phases to maintain judge integrity.

https://netflixtechblog.com/the-lifecycle-of-llm-as-a-judge-building-aligning-and-monitoring-at-scale-c95bd8283508?source=rss----2615bd06b42e---4 netflixtechblog.com

RSS Hunter • Oct 14

Netflix Stratum Media Processing: Automated Container Right-Sizing

Stratum is Netflix's internal serverless media processing platform, crucial for handling diverse media workloads like encoding and packaging. Engineers define Stratum functions as OCI images, which are then scheduled as batch jobs on the Titus compute platform. Managing hundreds of thousands of containers, Stratum needs dynamic scaling for new content formats such as podcasts and live events. Resource requests, specified in YAML, are often over-provisioned due to evolving infrastructure and application changes, leading to inefficiencies and increased operational costs.To address this, Netflix developed the Stratum Resource Tuner (SRT), an automated system designed to right-size container resources in production. SRT aims to eliminate manual resource configuration, focusing initially on high-throughput workloads with relaxed latency requirements, which constitute 70-80% of resource usage. Function owners can opt-in to automatic right-sizing per resource type (CPU, memory, disk, network).The SRT workflow consists of five steps: collecting resource usage metrics, generating recommendations, verifying recommendations via experiments, applying or rejecting recommendations, and monitoring post-application metrics. Recommendations progress through states: CREATED, EVALUATING, APPLIED, REJECTED, and MONITORED. For example, the VideoEncoder.encodeV1 function's over-provisioned memory and CPU were successfully tuned.Metrics collection is robust, with execution-level summaries stored in Elasticsearch for short-term lookups and high-resolution raw usage metrics in Iceberg tables for long-term trend analysis. A Conductor workflow runs daily, using a month of Iceberg data to generate recommendations for all functions and resource types. This process involves filtering high-throughput workloads, aggressively tuning CPU and network (p99 usage), and conservatively tuning memory and disk (p99.9 usage with a safety buffer).Safeguards are in place, such as discarding recommendations if the reduction is too small, and clamping maximum reductions for memory and disk to 15% per cycle. Recommendations are created in the CREATED state and become immutable once they advance past this stage. The example shows CPU recommendations reducing from 8 to 4, and memory from 18000MB to 15300MB in a single step, illustrating SRT's practical application in cost optimization.

https://netflixtechblog.com/netflix-stratum-media-processing-automated-container-right-sizing-ddf9aa68a801?source=rss----2615bd06b42e---4 netflixtechblog.com

RSS Hunter • Oct 14

Reliable GPU compute for large training & inference jobs

Netflix relies heavily on GPUs for its personalized content recommendations and other machine learning applications. These GPUs are crucial for training and serving content, making their reliability a business imperative for both availability and cost efficiency. Basic cloud provider health checks are insufficient because they do not detect subtle GPU or network fabric failures that can cripple long-running training jobs. Such failures can manifest as slow performance or complete job stoppage, impacting user experience and wasting resources.Netflix employs a layered approach to ensure GPU infrastructure health. The first layer focuses on detecting and remediating failing hardware like GPUs and network adapters. The second layer involves the scheduler and platform, which retry tasks and validate machines before they are used. The third layer enables long-running jobs to survive machine loss through checkpointing and elastic workers. The fourth layer is observability, which monitors the effectiveness of the other three.Health checks operate in two tiers: device-level checks for individual GPUs and node-level checks performed by an agent. These checks report results as Kubernetes node conditions. When a problem is detected, the machine's condition is updated, and controllers take action. Initial issues are tolerated, but persistent failures escalate to machine evacuation, draining, and replacement.Remediation is graded, with actions matched to the fault severity, ranging from restarting a single GPU to terminating and replacing the entire machine. For persistently failing hardware, immutable identifiers are recorded to prevent them from rejoining the pool. Machines are validated before being released for work, ensuring they pass all diagnostics. Finally, long-running jobs are made resilient through checkpointing, allowing them to resume from the last stable state if a machine is lost.

https://netflixtechblog.com/reliable-gpu-compute-for-large-training-inference-jobs-d62b03a96d7a?source=rss----2615bd06b42e---4 netflixtechblog.com

RSS Hunter • Oct 14

Trading a Cloud Identity for Your Own: Workload Attestation on Managed Compute

Organizations often maintain two parallel identity systems: one from cloud providers like AWS and another for internal services. This post details how Apache Spark workloads on Amazon EMR bridge this gap, enabling them to gain a first-class internal identity from an initial cloud identity. Netflix utilizes a private PKI called Metatron for service-to-service authentication via mutual TLS, requiring an attestation process to verify workload identities. They also employ a "Data Project" concept, which owns data and has its own identity, ensuring consistency in access control and auditing.The core challenge is translating a cloud-based AWS execution role into the necessary internal identity for Spark jobs on managed compute. This solution establishes a direct, one-to-one mapping between each Data Project identity and a dedicated IAM role, with the Data Project service serving as the authority for this mapping. This mapping is crucial for translating statements from the cloud provider's vocabulary to the organization's internal one. To address the scalability issue of managing numerous IAM roles, these roles are deterministically sharded across a small pool of dedicated AWS accounts.Five components are involved: the control plane (launches jobs and signs metadata), the Data Project service (manages the identity-to-role mapping), the Identity service (verifies attestations and issues certificates), a Spark plugin (hooks into process bootstrapping), and AWS STS (acts as a notary). The process begins with the control plane signing a metadata payload containing workload details. Subsequently, the Spark driver plugin uses AWS credentials to create a pre-signed URL from AWS STS, proving possession of the cloud identity.The Identity service then corroborates these two independent claims: the role identified by AWS from the pre-signed URL and the signed metadata from the control plane. If they align, certificates are issued for the workload. This corroboration is vital as neither statement alone is sufficient; the provider's statement is unforgeable but underspecified, while the control plane's statement is well-specified but potentially replayable. The system addresses the fan-out problem for executors by having the driver attest once and distribute credentials securely. Certificates are short-lived and renewed through a repeatable attestation process within the driver JVM. The trust model ensures no single participant can issue an identity independently, and the workload itself is never required to vouch for its own legitimacy.

https://netflixtechblog.com/trading-a-cloud-identity-for-your-own-workload-attestation-on-managed-compute-516d5a29b252?source=rss----2615bd06b42e---4 netflixtechblog.com

RSS Hunter • Sep 25

Leave the Class Path in the Rearview Mirror

A new preview of ja and its composable command-line tools aims to modernize Java development. These tools leverage the Java Module System to simplify module creation and consumption. The module descriptor is enhanced to fully describe a project, including dependency versions and metadata. Tools like jig handle module resolution, compilation, and assembly, acting as a bridge to Maven repositories. jfmt enforces code formatting conventions for cleaner Java code. jist offers source-aware symbol searching for easier code navigation. jdocserver provides local API documentation browsing. These tools integrate with the JDK's tool discovery and execution capabilities. They provide a seamless transition to standard JDK tools like jdeps and jlink. The system addresses challenges with existing Maven artifacts by inferring module coordinates and establishing canonical coordinates. Integrity by default is enforced by allowing explicit declaration and authorization of runtime access requirements, preventing unauthorized mutations. This ensures greater security and control over module interactions.

https://netflixtechblog.com/leave-the-class-path-in-the-rearview-mirror-67a85b15b6be?source=rss----2615bd06b42e---4 netflixtechblog.com

RSS Hunter • Sep 18

MAPS: Netflix’s Multimodal Asset Personalization at Scale

Netflix aims to connect members with content they'll love through personalized visual assets like artwork and video previews. A significant challenge arises with new titles, where insufficient interaction data prevents effective personalization, a problem known as the cold-start issue. Traditionally, models treated assets as opaque IDs, relying on popularity heuristics until enough data accumulated. This meant personalization for new content was delayed, impacting the discovery experience.Netflix's solution involves using multimodal embeddings to allow models to "see" and "hear" the assets. By encoding artwork with CLIP, a pre-trained image-text model, assets gain visual understanding. This CLIP embedding is concatenated with the asset's ID embedding, creating a richer representation that captures visual themes and talent. This enables personalization based on member preferences for visual styles, even before the asset has any interaction history, facilitating knowledge transfer across titles.This approach also enables model consolidation, merging separate models for different artwork canvases into a single unified model. Because CLIP embeddings are largely invariant to cropping and resizing, near-identical renderings of artwork map to similar vectors. This pooling of interaction signals across canvases, especially benefiting low-data canvases, allows a single model to personalize all artwork placements. To address data imbalance across canvases, reward-based weighting is used, prioritizing interactions based on their long-term value rather than impression volume.The effectiveness of this approach was validated through offline evaluations using inverse propensity scoring and large-scale A/B tests. Combining image embeddings with a unified model (V3) significantly outperformed models with only one of these improvements. This combined approach proved crucial during a major Netflix home-screen redesign, where a dominant canvas with minimal historical data was introduced. The unified model with CLIP embeddings successfully personalized content for this new layout, showing statistically significant gains in discovery metrics and streaming hours, highlighting the power of multimodal embeddings for overcoming the cold-start problem.

https://netflixtechblog.com/maps-netflixs-multimodal-asset-personalization-at-scale-32f96320785e?source=rss----2615bd06b42e---4 netflixtechblog.com

RSS Hunter • Aug 28

A Tale of Two Flink Autoscalers

Netflix currently runs two Flink autoscalers, aiming to consolidate to one open-source solution. They initially built an in-house autoscaler to manage thousands of Flink jobs, which proved effective but limited in handling complex, multi-operator jobs. This homegrown system relied on external metrics, which could be inaccurate or miss internal job issues. The Apache Flink community later developed a more sophisticated autoscaler that reasons from within the job itself. This second autoscaler estimates an operator's true processing rate to determine optimal parallelism per vertex. Netflix is converging on this open-source solution due to its ability to scale stateful jobs and allow granular configuration. Implementing the OSS autoscaler at Netflix required significant adaptation, including integrating it into their existing control plane. They also made modifications to Flink's runtime for better metric collection and to handle specific job structures like forward-connected subgraphs and sink limitations. By adopting the OSS autoscaler, Netflix has achieved substantial cost savings and improved resource utilization. Lessons learned emphasize that metric quality is crucial, tunable defaults are beneficial, and adopting existing solutions before extending them is a wise strategy. They plan to fully migrate to the OSS-based autoscaler to streamline operations.

https://netflixtechblog.com/a-tale-of-two-flink-autoscalers-e9f6a1b1492b?source=rss----2615bd06b42e---4 netflixtechblog.com

RSS Hunter • Aug 21

How and Why Netflix Built a Real-Time Distributed Graph: Part 3 — Querying the graph with gRPC…

Netflix built a Real-Time Distributed Graph to power real-time insights for their internal partners, and the third part of this blog series focuses on querying the graph efficiently. The graph is a complex network of billions of nodes and edges, and querying it requires a fast and flexible serving layer. The authors discuss the challenges of querying the graph, including handling a wide range of access patterns and supporting both shallow-wide and deep-narrow queries. They explain how they designed a serving layer to efficiently query the graph, using a breadth-first approach and asynchronous composition to minimize latency. The authors also discuss the importance of caching, opt-in enrichments, and eventual consistency in the design of the query layer. The query layer is composed of three layers: the Graph Query Service, the Storage Abstraction Layer, and the Enrichment Layer, which work together to execute queries efficiently. The authors walk through an example query to demonstrate how the query layer works in practice, highlighting the importance of reading and interpreting the request, reading from storage efficiently, executing traversal with breadth-first levels, running many operations in parallel, filtering smartly, and making repeat queries faster with caching. The goal of the query layer is to complete queries in under 100ms, and the authors demonstrate how their design achieves this goal. The query layer is designed to handle tens of thousands of queries per second, each potentially different, and the authors discuss the trade-offs they made in designing the query layer to achieve this level of performance. Overall, the authors provide a detailed overview of the design and implementation of the query layer, highlighting the challenges and trade-offs involved in building a high-performance query layer for a large-scale distributed graph. The query layer is a critical component of the Real-Time Distributed Graph, and the authors' design and implementation have enabled Netflix to power real-time insights for their internal partners.

https://netflixtechblog.com/how-and-why-netflix-built-a-real-time-distributed-graph-part-3-querying-the-graph-with-grpc-0f3468349607?source=rss----2615bd06b42e---4 netflixtechblog.com

RSS Hunter • Aug 7

Modeling Device Capabilities for Analytics

Netflix offers a wide range of features and content types across various devices, but hardware limitations can restrict certain features on specific devices. To address this, the company has developed a comprehensive device capability data model to understand device capabilities and ensure the best user experience. The data model integrates feature flags from internal systems, enabling smarter feature management across the global device landscape. A cumulative table is used to store information about device capabilities, such as screen resolutions, video profiles, and RAM size, making it ideal for analytics and reporting. The table is structured to capture the latest state of each device and its associated capabilities, allowing for efficient processing of information. For aggregate analytics, a histogram table is used to capture active device counts over the past 28 days, broken down by device model and software version. This table also records the number of devices supporting specific capabilities, enabling detailed distribution analysis. The histogram data can be used to analyze the distribution of external display capabilities, such as the percentage of devices supporting HD or UHD profiles. By leveraging these datasets, Netflix has built analytical products to provide a comprehensive view of feature reach, including 4K Ultra HD, Netflix Spatial Audio, and Cloud Gaming. The company uses data-driven insights to make informed decisions about which features to enable on specific devices, ensuring both performance and reliability.

https://netflixtechblog.com/modeling-device-capabilities-for-analytics-e7607acebde8?source=rss----2615bd06b42e---4 netflixtechblog.com

RSS Hunter • Jul 31

GenRec: Towards LLM-Native Recommendation at Netflix

Recommendations are a crucial part of the Netflix experience, and the company has been using complex production models that rely on thousands of hand-crafted features and specialized architectures. However, these models are costly to maintain and update, and the company is looking for more efficient and effective solutions. Large language models have shown promise in this area, but they are not yet production-ready and often have limitations such as over-recommending popular content and ignoring business constraints. To address this, Netflix built GenRec, an LLM-backed recommendation ranker that post-trains an internal foundation LLM on Netflix-specific data and objectives. GenRec verbalizes user histories, item metadata, and context as text and uses a catalog-aware scoring head to rank items. The model is trained with a multi-objective loss that combines ranking, language, and reward signals to align with business goals and long-term member satisfaction. GenRec has shown statistically significant improvements in both short-term and long-term online metrics compared to a well-tuned production ranker, while using only a small fraction of the labeled data and input signals. The model is served on Netflix's internal LLM stack using vLLM, and the company has implemented strategies to control serving cost, such as using smaller models, aggressive context compaction, and prefill-only inference. Overall, GenRec represents a significant step towards a more LLM-centric future for recommendation at Netflix, and its success has the potential to transform the way the company approaches recommendation and personalization. The model's ability to learn from natural language inputs and generate personalized rankings has the potential to improve the user experience and increase engagement. By leveraging the power of large language models, Netflix can create more effective and efficient recommendation systems that drive business success.

https://netflixtechblog.com/genrec-towards-llm-native-recommendation-at-netflix-f20be6f643e3?source=rss----2615bd06b42e---4 netflixtechblog.com

RSS Hunter • Jul 30

In-House LLM Serving at Netflix

Netflix runs its entire large language model stack internally, managing deployment and inference directly. They integrated this into their existing production environment instead of a separate machine learning silo. This approach involved carefully selecting an inference engine, deciding how models would be packaged, designing the API surface, defining deployment strategies, and enforcing output constraints.The chosen inference engine is vLLM, selected for its operational fit, ability to load custom architectures, extensibility, debuggability, and familiarity among practitioners. Models are packaged using the vLLM backend in Triton to allow dynamic I/O tensor specification, promoting independent evolution of models and frontends. An OpenAI-compatible HTTP frontend was added alongside their existing gRPC interface for broader ecosystem compatibility.For deployment, Netflix uses a Versioned strategy to handle potential schema changes between model versions, allowing consumers to update independently. When model interfaces are stable, the less costly Red-Black deployment strategy is employed. One key operational challenge was slow model startup times, which they addressed by materializing large models on Amazon FSx for faster access.Observability was improved by creating a unified /metrics endpoint that aggregates data from both vLLM and Triton. A significant feature is constrained decoding at scale, implemented via vLLM's custom logits processor. This allows models to generate compliant outputs directly, avoiding post-inference correction.An initial pure-Python implementation of constrained decoding struggled with scaling due to the Global Interpreter Lock (GIL) and sequential processing on the CPU. This only became apparent under high concurrency, leading to significant tail latencies. The system relies on a Java control plane for managing deployment, versioning, and autoscaling. Both real-time and cached batch inference paths are supported by their serving system. This unified system handles the complete downstream consumer flow, including routing, candidate generation, and logging.

https://netflixtechblog.com/in-house-llm-serving-at-netflix-a5a8e799ea2c?source=rss----2615bd06b42e---4 netflixtechblog.com

RSS Hunter • Jul 17

Building Service Topology at Scale: Architecture, Challenges, and Lessons Learned

Netflix developed a real-time service dependency map to aid engineers in troubleshooting and understanding their distributed architecture. The system combines eBPF network flows, IPC metrics, and distributed tracing into independent graph layers. While the initial version worked locally, production revealed significant scaling challenges, including Kafka consumer lag and memory issues.The core architectural decision was a streaming-first approach, providing near real-time topology updates instead of hourly batch processing. This was crucial for incident response and live event monitoring. To handle millions of flow records per second without data loss, the system employs reactive streams with backpressure, signaling upstream components to slow down when downstream systems are overwhelmed. This ensures graceful degradation rather than crashes or data loss.The architecture uses a multi-layer design with physical storage isolation for network, IPC, and tracing layers, allowing independent optimization. The network layer's ingestion relies on a three-stage distributed aggregation pipeline. Stage 1 performs initial aggregation from Kafka, Stage 2 resolves network intermediaries (like load balancers) into direct application-level dependencies, and Stage 3 handles final aggregation, enrichment with external data, and persistence to the graph database.The three-stage pipeline was a critical evolution from an initial two-stage design, which suffered from "hot nodes" due to data concentration during intermediary resolution and enrichment. Splitting these responsibilities across three stages distributes the workload, preventing bottlenecks. Server-Sent Events (SSE) replaced gRPC for inter-stage communication due to gRPC's performance overhead, which consumed excessive CPU and memory at scale.

https://netflixtechblog.com/building-service-topology-at-scale-architecture-challenges-and-lessons-learned-f4b792f3f0d8?source=rss----2615bd06b42e---4 netflixtechblog.com

RSS Hunter • Jul 13

GenPage: Towards End-to-End Generative Homepage Construction at Netflix

The Netflix homepage is a highly personalized and structured interface for content discovery. Traditionally, its generation involved a complex multi-stage pipeline. GenPage, a novel approach, utilizes a single generative transformer model to autoregressively construct the entire homepage. This end-to-end model takes user context as a prompt and generates rows, entities, and layout simultaneously. Key goals include simplifying the recommendation stack, enabling whole-page optimization via reinforcement learning, and improving scalability and flexibility. Production challenges involved real-time serving latency, entity cold start, and maintaining model freshness. GenPage has demonstrated significant improvements in user engagement and reduced serving latency in A/B tests. The data is tokenized into context tokens representing user information and page tokens for the homepage structure. Domain-specific tokenization enhances computational efficiency and product control compared to generic text tokenization. User history, profile, and request context form the prompt, while entities and rows are represented as tokens in the generated page. The system uses a reward system based on user feedback to quantify recommendation value and guide training. GenPage employs a standard decoder-only transformer architecture and follows an LLM-like training recipe of pretraining and post-training. Pretraining uses next-token prediction to teach the model the homepage language, followed by post-training with weighted binary classification or reinforcement learning. This generative approach offers a more integrated and direct path to optimizing the entire Netflix homepage experience.

https://netflixtechblog.com/genpage-towards-end-to-end-generative-homepage-construction-at-netflix-77146fba8a08?source=rss----2615bd06b42e---4 netflixtechblog.com

RSS Hunter • Jun 29

Toward More Controllable AI Video Editing: An Early Research Exploration at Netflix

Netflix researchers have developed two AI tools to assist video editors in creating promotional assets. The first tool, Vera, is a layered video diffusion model designed for content-preserving edits. Vera generates edits as separate layers, leaving untouched portions of the original footage unaltered, thus preserving identities and details. To train Vera, they created a specialized dataset and employed a Mixture-of-Transformers architecture for efficient output generation. Evaluations showed Vera significantly outperformed existing methods in content preservation with comparable video quality. The second tool, VOID, is a video object and interaction deletion framework. VOID addresses the challenge of physically implausible results when removing objects with significant interactions. It uses a two-pass inference pipeline, with the first pass generating a physically plausible counterfactual video. A second pass refines the output to prevent artifacts like object morphing. VOID was trained using synthetic data generated from simulations and real-world motion capture data. Experiments indicate that VOID better preserves consistent scene dynamics compared to previous approaches. These research explorations aim to advance AI video editing responsibly while empowering artists. Both Vera and VOID are detailed in publicly released research papers.

https://netflixtechblog.com/toward-more-controllable-ai-video-editing-an-early-research-exploration-at-netflix-eb8160ed60a2?source=rss----2615bd06b42e---4 netflixtechblog.com

RSS Hunter • Jun 23

How Netflix Simplified Batch Compute with Kueue

Netflix is transitioning its compute infrastructure to a Kubernetes-native model, integrating components like Kueue into its Titus platform. Kueue, a cloud-native job queueing system, has largely replaced the custom logic of their homegrown batch solution, Compute Managed Batch (CMB). Motivations for this migration included CMB's age, the evolution of Kubernetes, and the growing difficulty of adding features like preemption to the existing system. Kueue was chosen over alternatives due to its compatibility with existing Titus scheduling profiles, its adoption momentum, and its native support for features like preemption.The migration, known as Netflix Batch, aimed for zero user impact and no reduction in throughput. The process involved rerouting job submissions to Kueue-enabled Titus cells, with Titus federation handling job distribution. Operator adjustments were minimal, primarily involving a simple UI toggle to enroll tenants into Kueue. This enrollment translates CMB's existing tenant hierarchy and capacity configurations into Kueue's concepts of Cohorts, ClusterQueues, and LocalQueues.Key lessons learned include maintaining API parity with the old system to ensure a smooth user experience and migrating complex use cases early to build confidence. It was also crucial to load test Kueue to ensure it could handle Netflix's high throughput requirements, necessitating adjustments to default configurations. Kueue is now fully operational, managing millions of batch workloads, and has enabled enhanced fair sharing and preemption features. These improvements allow for better utilization of reserved capacity, reduced job starvation, and faster turnaround for critical workloads, significantly increasing average resource utilization.

https://netflixtechblog.com/how-netflix-simplified-batch-compute-with-kueue-87860682629c?source=rss----2615bd06b42e---4 netflixtechblog.com

RSS Hunter • Jun 22

The Data Canary: How Netflix Validates Catalog Metadata

Netflix built an automated data canary system to ensure catalog data reliability. This system validates data transformations using production traffic. It detects issues in under 10 minutes, preventing corrupted data from reaching millions of viewers. A past incident revealed gaps in their resilience strategy when manual actions corrupted a data feed. This corrupted data caused manifest generation failures and playback issues. Existing code canaries were ineffective because no code had changed, only the data. The challenge was validating high-velocity data pipelines with short time constraints. They needed to validate the actual output consumed by clients, not just inputs. Production traffic was deemed essential for detecting real customer impact. Their solution involved a dedicated orchestrator pattern with permanent baseline and canary clusters. They extended their chaos platform, customizing thresholds and using sticky canaries. Behavioral metrics like Starts Per Second became their primary signal for detection. The system also incorporated robust edge case handling for production environments. Controlled failure injection experiments validated the system's effectiveness. Issues were identified in 2.5–4 minutes, with automatic blocking of corrupted data. This effort established that data deployments require the same rigor as code deployments.

https://netflixtechblog.com/the-data-canary-how-netflix-validates-catalog-metadata-18b699d58e36?source=rss----2615bd06b42e---4 netflixtechblog.com

RSS Hunter • Jun 19

Data Projects: Managing Data Assets at Netflix Scale

Netflix faced challenges managing millions of data assets with an individual asset-based access control system. Organizational changes caused permission updates to be burdensome and inconsistent, leading to either flooded support teams or overly broad access grants. Workloads were tied to human identities, causing failures when engineers changed roles or left the company, resulting in a "permissions whack-a-mole" issue. To address these problems, Netflix introduced Data Projects, which act as containers for related assets and provide durable, project-specific identities for workloads. This shifts management granularity from individual assets to logical project groupings. Each Data Project has grants and roles, allowing users, groups, or applications to be assigned specific access levels. The project’s Netflix application identity executes workloads, independent of human lifecycles, and can be cryptographically exchanged for AWS IAM roles. A key feature called "gravity" automatically adds newly created assets to their project, ensuring organizational consistency. Data Projects have proven successful in stabilizing critical pipelines and simplifying access management for complex analytics domains. Onboarding involves creating a project, granting roles, and configuring workflows to use the project's identity, with new assets automatically associated via gravity. Future plans include expanding projects beyond data to software and studio assets, integrating rightsizing for permissions, and adding features like cost attribution and auditing. Ultimately, Data Projects provide a scalable and stable solution for identity and access management at Netflix, aligning with how teams organize their work.

https://netflixtechblog.com/data-projects-managing-data-assets-at-netflix-scale-7ca25888591e?source=rss----2615bd06b42e---4 netflixtechblog.com

RSS Hunter • Jun 19

Predicting Risk in Content Launches: How Data-Driven Insights can Transform Launch Planning

Netflix's Analytics Summit showcases how analytics informs business decisions, including content launch risk. Content goes through development, production, post-production, and launch preparation, with launch preparation relying on finalized media assets. Teams face a trade-off between waiting for the final IMF, risking delays, or starting early with a less final Locked Cut, risking rework. Manually provided production schedules for these assets often lack accuracy and coverage, especially far from delivery. This inaccuracy is strongly correlated with content launch misses. To address this, Netflix developed predictive models to estimate media asset delivery dates. These boosted tree regression models use production-level signals, metadata, and seasonal data, updated daily. The models aim to fill schedule gaps and improve existing date accuracy. Evaluating these models involves metrics like mean absolute error, bias, and error distribution. Backtests show significant improvements in predictive accuracy compared to manual schedules, reducing forecast errors. These improved predictions lead to lower Accumulated Error Days, a metric tied to launch misses. The predictive dates integrate into existing workflows, offering earlier accuracy than scheduled dates. Serving logic defaults to scheduled dates where the model underperforms, while other teams can view both and use their judgment. This initiative streamlines launch workflows and mitigates launch risks by providing more reliable delivery estimates.

https://netflixtechblog.com/predicting-risk-in-content-launches-how-data-driven-insights-can-transform-launch-planning-587b1f2de928?source=rss----2615bd06b42e---4 netflixtechblog.com

RSS Hunter • Jun 19

The Evolution of Cassandra Data Movement at Netflix

Netflix uses Data Bridge as a unified management plane for batch data movement. Historically, custom connectors were developed for specific needs, but efforts are underway to centralize these offerings. The Cassandra to Iceberg connector, Casspactor, was an in-house solution for critical applications like Member and Billing. Casspactor handled significant data volumes but faced challenges with fragile metadata dependencies and inherited limitations across data abstractions. These issues included skewed partition failures, lack of data model awareness, intermediate table bloat, and an inability to time travel. A new layered architecture was developed to address these shortcomings.This new engine, built on Apache Cassandra Analytics and the Move Data framework, features a core S3 reading capability and a Connector Factory model. It processes data directly into Spark DataFrames, eliminating intermediate tables. The new stack efficiently handles skewed partitions and offers autosizing capabilities for jobs. It also significantly reduces dependencies by relying solely on S3 for metadata, enhancing reliability and enabling time travel. Rigorous validation, visibility, and safety measures were crucial for migrating from Casspactor. A like-for-like migration strategy ensured minimal disruption and no changes for downstream users. Shadow testing was employed to guarantee data consistency between the old and new systems. This involved verifying that the output of the new system was an exact replica of the legacy system's output.

https://netflixtechblog.com/the-evolution-of-cassandra-data-movement-at-netflix-6e13329c80a1?source=rss----2615bd06b42e---4 netflixtechblog.com

RSS Hunter • Jun 19

Thinking Fast & Slow for a Personalized Notification System

Daniel Kahneman's dual-process theory describes two cognitive systems: System 1 for automatic, quick thinking and System 2 for deliberate, focused effort. This concept applies to designing intelligent systems that balance immediate responsiveness with foresight. Netflix's personalized notification platform faces a similar challenge, optimizing hundreds of millions of daily messages. A core tension exists between maximizing short-term engagement and ensuring a positive long-term member experience. Over-messaging can lead to fatigue and opt-outs, while under-messaging risks missing valuable content discoveries.To address this, Netflix implemented a hierarchical framework with a "slow" policy for strategic, weekly messaging plans and a "fast" policy for tactical, real-time message selection. Previously, single-message outcome models optimized for short-term gains but overlooked cumulative effects and coupled ranking with pacing decisions. The new "slow" policy defines a personalized message pacing over a defined horizon, considering long-term engagement. This is achieved by maximizing a utility function that balances positive engagement signals against the long-term cost of messaging.A universal message cost term is introduced to prevent models from always opting for maximum frequency. The "slow" policy's decisions are stored in a feature store, enabling asynchronous communication with the "fast" policy. The "fast" policy then executes tactical send decisions within the strategic guardrails set by the "slow" policy. This decoupling allows for independent evolution of both strategies and ensures a consistent member experience. The hierarchical architecture led to significant metric lifts, particularly benefiting casual viewers by improving content awareness. Ultimately, separating frequency planning from message selection proved transformative, allowing for independent iteration on both aspects.

https://netflixtechblog.com/thinking-fast-slow-for-a-personalized-notification-system-4d89b26525cd?source=rss----2615bd06b42e---4 netflixtechblog.com

RSS Hunter • Jun 19

A Human-Augmenting Agentic Workflow for Causal Inference

Data analysis is increasingly delegated to software agents, necessitating oversight for valid results, especially in Observational Causal Inference (OCI). This work introduces an agentic workflow for OCI that adheres to rigorous templates while augmenting human inspection. The workflow aims to reduce repetitive tasks for OCI practitioners, allowing them to focus on nuanced aspects like question framing and assumption evaluation. An open-sourced version of their oci-agent is available for improvement, and evaluations show it outperforms one-shot iterations and rivals hand-tuned benchmarks. The philosophy relies on target trial emulation, outlining ideal A/B tests to inform necessary assumptions for causal inference. This involves design diagnostics like covariate balance, overlap, placebo outcomes, and sensitivity analysis. The agent orchestrates an actor-critic loop with three personas: Principal (human user), Actor (analysis executor), and Critic (results synthesizer). Principals define the plan and context, Actors refine and execute the analysis with diagnostics, and Critics identify gaps and assess credibility. To empower human evaluation, the workflow makes each analytic step transparent through published artifacts like plans, specifications, and notebooks. A case study at Netflix estimated the impact of a new entertainment type, revealing significant early adopter bias. Standard regression gave a large effect, but the agentic workflow, using diagnostics, identified poor overlap and placebo test failures. Addressing these, the agent implemented trimming, significantly reducing the estimated effect to a more credible range for the overlapping population. Follow-up analyses, such as sensitivity tests and time-series generation, are facilitated by the agent's ability to manage complex, multi-version executions.

https://netflixtechblog.com/a-human-augmenting-agentic-workflow-for-causal-inference-4623f0a9c5af?source=rss----2615bd06b42e---4 netflixtechblog.com

RSS Hunter • Jun 19

VMAF v1: Good Is Not Good Enough

Netflix has open-sourced a new version of its video quality metric, VMAF v1. VMAF is crucial for optimizing video encoding and ensuring a good viewing experience for members. VMAF v1 addresses limitations of the previous version, VMAF v0, to more accurately assess visual quality. Key improvements include enhanced sensitivity to compression artifacts by adding the AIM component. VMAF v1 also allows for a single model to be used across various viewing conditions by adjusting feature values based on viewing distance. This approach improves accuracy and generalizability over older mapping functions. Banding artifacts are now addressed by integrating the CAMBI feature, and chroma artifacts are handled by modifying SpEED-QA for chroma channels. The no-enhancement gain (NEG) mode is now enabled by default to preserve creative intent and is used in codec development. VMAF v1 features an improved motion feature with a hard threshold and an option for a larger temporal window to better handle high frame rates. The new version includes multiple models for different resolutions and viewing distances, such as standard 1080p, phone, and 4K. VMAF v1 has been calibrated to align with v0 scores, preserving interpretability while offering enhanced accuracy. The new version also boasts reduced computational complexity, making it faster than VMAF v0. Netflix encourages the community to test VMAF v1, report issues, and contribute to its ongoing development.

https://netflixtechblog.com/vmaf-v1-good-is-not-good-enough-60d7e4244ea8?source=rss----2615bd06b42e---4 netflixtechblog.com

RSS Hunter • Jun 19

Dynamically Splitting Wide Partitions in Cassandra for Time Series Workloads

Netflix's TimeSeries Abstraction ingests and queries petabytes of temporal event data with millisecond latency, using Apache Cassandra as its storage. Wide partitions, where a single partition accumulates a large volume of events over time, pose a significant challenge for TimeSeries workloads. This leads to high read latencies, timeouts, increased CPU utilization, and garbage collection pauses in Cassandra clusters. To address this, TimeSeries data is partitioned into discrete time chunks, creating manageable segments.The initial provisioning strategy relied on user-specified workload characteristics and Monte Carlo simulations to determine optimal infrastructure and partition configurations. However, this approach proved insufficient when workloads were unknown, inaccurately estimated, evolved over time, or contained data outliers. To automate adjustments, a background worker was introduced to monitor partition histograms and dynamically re-partition future time slices based on observed data density. This Time Slice Re-Partitioning strategy effectively reduces read latencies and timeouts when most data exhibits similar wide partition behavior.However, this strategy doesn't address scenarios where only a small percentage of IDs within a table are wide. For such cases, and when callers require all data even with elevated latencies, Dynamic Partitioning per ID was developed. This asynchronous pipeline detects wide partitions during read operations and transparently splits them into optimal sizes. The process involves detection, planning and splitting, and serving reads by re-routing queries to the split partitions.Detection occurs when a read operation exceeds a configured byte threshold, emitting an event to Kafka. The system initially focuses on immutable partitions for simplicity. The planning stage reads the entire partition to create a split plan, using checkpointing to handle failures. Splitting involves delegating the data division to specific strategies, like assigning more event buckets to a time bucket. Validating splits is crucial, with checksums ensuring data integrity before marking a split as complete. Finally, the TimeSeries servers use in-memory Bloom filters to efficiently divert read queries to the split partitions, making the diversion practically invisible to callers.

https://netflixtechblog.com/dynamically-splitting-wide-partitions-in-cassandra-for-time-series-workloads-0eded064f456?source=rss----2615bd06b42e---4 netflixtechblog.com

RSS Hunter • Jun 3

High-Throughput Graph Abstraction at Netflix: Part I

Netflix built a Graph Abstraction to handle high-throughput, low-latency graph operations, particularly for use cases like Real-Time Distributed Graphs and Social Graphs. It operates within two categories: OLAP for in-depth analysis and OLTP for streaming user experiences. The abstraction uses a Property Graph model with strongly typed nodes and edges, organized into isolated namespaces. Each namespace has a predefined graph schema managed through a Data Gateway Control Plane. This schema enables optimizations like data quality enforcement and efficient query planning. The real-time index uses key-value storage for nodes and edges, employing separate indexes for links and properties. Edge links are indexed by source-destination relationships. To ensure access regardless of direction, this organizes identifiers lexicographically. Caching is used to minimize write and read amplification. The Abstraction architecture prioritizes high performance and incorporates strategies like write-aside caching.

https://netflixtechblog.com/high-throughput-graph-abstraction-at-netflix-part-i-e88063e6f6d5?source=rss----2615bd06b42e---4 netflixtechblog.com

RSS Hunter • May 29

From Silos to Service Topology: Why Netflix Built a Real-Time Service Map

Netflix developed a "living map" of its distributed infrastructure, called Service Topology, to help engineers understand service dependencies and troubleshoot issues. The map addresses the need for quicker identification of service relationships and potential impact during outages. It answers crucial questions like which services depend on each other and identifies the source of problems. They gathered data from three sources: eBPF network flows, IPC metrics, and end-to-end tracing. Each data source provides a unique perspective – network connectivity, application-level details, and actual request flows. This multi-layered architecture combines these views into a unified, real-time map. The system uses separate graph databases for each data source to maintain independence and enable parallel querying. Engineers can view each graph individually or combine them for a comprehensive understanding of service interactions. The system ingests flow logs from Kafka in multiple regions, processes them, and stores the data in graph databases. This living map helps Netflix engineers quickly diagnose and resolve problems, ensuring a smooth streaming experience.

https://netflixtechblog.com/from-silos-to-service-topology-why-netflix-built-a-real-time-service-map-0165ba13a7bc?source=rss----2615bd06b42e---4 netflixtechblog.com

RSS Hunter • May 29

Scaling ArchUnit with Nebula ArchRules

Netflix uses a polyrepo strategy with thousands of Java repositories, requiring efficient build logic sharing. They built the Nebula suite of Gradle plugins, including ArchRules, to manage dependencies and enforce code standards. This initiative arose from a need to improve Java library lifecycle management following a backwards-incompatible change incident. Netflix utilizes API lifecycle annotations (@Deprecated, @Public, @Experimental) to identify potential issues with deprecated code. ArchUnit, a popular library for enforcing architectural rules, was chosen to detect the misuse of these annotations and other technical debt issues. Nebula ArchRules plugins enable sharing and applying ArchUnit rules across many repositories, providing enhanced capabilities. ArchRules uses bytecode analysis (ASM) for cross-language support and an easy to use builder pattern simplifying rule creation. Rules can be bundled with libraries or defined in standalone rule libraries, automatically detected and run by the plugin. The ArchRules Runner plugin evaluates rules against source sets and generates JSON and console reports, enabling better reporting capabilities. By using ArchRules, Netflix provides a platform for library authors to track API usage and detect deprecated API usage.

https://netflixtechblog.com/scaling-archunit-with-nebula-archrules-b4642c464c5a?source=rss----2615bd06b42e---4 netflixtechblog.com

RSS Hunter • May 8

Democratizing Machine Learning at Netflix: Building the Model Lifecycle Graph

Netflix utilizes machine learning across various business domains like personalization, studio production, payments, and advertising. As ML adoption grew, a challenge emerged: a fragmented landscape where models and data were siloed, hindering collaboration and discovery. ML practitioners struggled to understand model lineage, feature sources, and impact across different systems. This fragmentation prevented easy answers to questions about existing features, data sources, pipeline dependencies, and the effects of changes.The core difficulty lay in connecting disparate ML infrastructure components that generated metadata. Dozens of systems, from pipeline orchestrators to experimentation platforms and feature stores, produced data in various formats. Solving this required collecting heterogeneous metadata, transforming it into a unified model, and building a connected graph for exploration.The solution is the Metadata Service (MDS), which constructs a Model Lifecycle Graph to interconnect ML entities at Netflix. MDS ingests ML metadata in real-time, enabling cross-domain queries like identifying experiments using a specific model or models sharing certain features. The vision is to make all ML assets discoverable, understandable, and reusable across the company.MDS operates on core abstractions: Components, each with a unique AIP URI; Entities, which are ML-specific assets with properties; Entity Types, defining data shapes; Domains, which group related entity types; and Providers, concrete implementations of domains from source systems. This URI-based addressing allows any service to reference any ML asset universally. The process of building the graph involves several stages.First, MDS integrates with source systems via Kafka and AWS SNS/SQS to consume thin events indicating changes. Dedicated event handlers process events from systems like the Pipeline Orchestration, Model Registry, Feature Store, Experimentation Platform, and Identity Platform. Second, MDS implements a hydration contract, validating events and calling source system APIs to fetch complete state, which is then transformed into a normalized entity. This "notification of change" pattern ensures robustness against event order issues but places read load on source systems.Third, raw events are transformed into a unified entity model with standardized fields, creating a consistent interface for downstream consumers. Normalized entities standardize field names, formats, and convert platform-specific IDs into global AIP URIs. Finally, normalized entities are persisted to Datomic for caching and relationship storage, and simultaneously indexed in Elasticsearch. Datomic, with its immutable fact model, supports complex graph traversals and entity relationships, enabling queries across multiple domains without inefficient N+1 query patterns.

https://netflixtechblog.com/democratizing-machine-learning-at-netflix-building-the-model-lifecycle-graph-5cc6d5828bb1?source=rss----2615bd06b42e---4 netflixtechblog.com

RSS Hunter • May 4

State of Routing in Model Serving

The blog post discusses the technical insights into how Netflix's ML model serving infrastructure powers personalized experiences at scale across various domains. The central ML model serving platform exposes a domain-independent API abstraction and traffic routing capabilities to several domain-specific microservices for model inference. This singular API has increased the speed of innovation for iterating on newer versions of existing ML experiences and enabling new product experiences with ML. The success of the ML model serving infrastructure depends on enabling researchers to rapidly experiment with new hypotheses and safely release their models into production. The platform serves hundreds of model types and versions, netting 1 million requests per second, and operates at the level of workflows, not just individual scoring functions.The model definition contains a list of facts that it needs to compute features, and it relies on the model serving platform to supply these facts at serving time by calling several other microservices. The calling services only need to provide standard request context, and the relevant domain context, and the model can itself compute features and perform inference as part of the execution flow. The platform acts as an enabler of rapid ML innovation and limits the exposure of ML model iterations to the client apps. The key principles of the platform include model innovation independent of client apps, decoupling clients from model sharding, and flexible traffic routing rules.The platform uses a custom service called Switchboard, which serves as a flexible proxy layer for all traffic, handling over 1 million requests per second while maintaining high availability and reliability. Switchboard provides a single point of contact for all clients' model needs and can route a request based on a rich set of contextual features. The platform also introduces the concept of an "Objective", which is an enumeration defined by the serving platform that every request into the system must provide. The Objective decouples clients from concrete models and guides the platform's routing and model selection decisions.Switchboard Rules is a JavaScript configuration that allows researchers to attach model variants, experiments, and traffic splits to Objectives without changing client code. The rules dictate the default model to use for a given Objective, A/B experiments to configure for a set of Objectives, and customizations to gradually shift traffic to a new model. The rules are consumed by both Switchboard and the Model Serving clusters, and the serving platform components can take various actions based on these rules. Overall, the platform provides a scalable and flexible solution for serving ML models at Netflix, enabling rapid innovation and experimentation while minimizing the impact on client apps.

https://netflixtechblog.com/state-of-routing-in-model-serving-16e22fe18741?source=rss----2615bd06b42e---4 netflixtechblog.com

RSS Hunter • May 1

Scaling Camera File Processing at Netflix

Netflix built its Media Production Suite (MPS) to streamline media workflows for global productions, aiming for efficiency and consistency. The core of MPS leverages FilmLight's API (FLAPI) for image processing, a collaboration that avoids building everything from scratch. MPS addresses challenges like file wrangling and inconsistent media handling, automating tasks and minimizing errors. The FLAPI is used to inspect camera file metadata, normalize it, and make it searchable, ensuring consistency. FLAPI also generates VFX plates and deliverables with accurate color management and debayering. MPS is cloud-integrated using Cosmos, allowing for CPU-only instances in a Dockerized environment, maximizing efficiency. Production workloads are handled elastically to scale on demand, ensuring quick turnaround times. MPS is designed for both experienced teams and those needing guidance, with FLAPI handling the complexities. This partnership allows for open communication, joint validation, and benefits the standardization ecosystem. The overall impact is fewer delays, faster turnaround, and more efficient production processes.

https://netflixtechblog.com/scaling-camera-file-processing-at-netflix-6dab2b1e80be?source=rss----2615bd06b42e---4 netflixtechblog.com

RSS Hunter • Apr 24

The Human Infrastructure: How Netflix Built the Operations Layer Behind Live at Scale

Netflix's live streaming operations have rapidly expanded since their first live show, now streaming multiple events daily. Initial live shows relied on engineers and improvised setups, lacking dedicated operational teams. The evolution required building dedicated Broadcast Operations Centers (BOCs) to manage the live broadcast process. To ensure signal reliability, Netflix enforces strict specifications on video and audio feed contributions from venues. Netflix's operational model evolved through several phases, from engineering-led operations to specialized engineering teams. The Transmission Operations Center (TOC) model was developed to improve efficiency and manage a high volume of events concurrently. Within the TOC model distinct roles like Transmission, Streaming, and Broadcast Control Operators are crucial. For high-profile events, a specialized "Big Bet" model provides dedicated resources. The Live Command Center (LCC) monitors the entire live streaming pipeline, offering real-time data to operators. The LCC uses a custom observability stack to manage massive data volumes and react to issues promptly. LCC Operations Leads and Technical Launch Managers are responsible for incident response.

https://netflixtechblog.com/the-human-infrastructure-how-netflix-built-the-operations-layer-behind-live-at-scale-33e2a311c597?source=rss----2615bd06b42e---4 netflixtechblog.com

RSS Hunter • Apr 17

Evaluating Netflix Show Synopses with LLM-as-a-Judge

Netflix aims to improve the synopsis selection process for its vast content library. Thousands of titles complicate the user's choice, making synopses crucial for decision-making. High-quality synopses enhance user engagement, while poor ones lead to frustration and abandonment. The challenge lies in scaling quality validation across hundreds of thousands of synopses. Netflix developed an LLM-based approach to evaluate synopsis quality across four key dimensions. This system achieves over 85% agreement with creative writers, ensuring expert-driven quality standards. Synopsis quality is assessed through both creative quality defined by writers and member implicit feedback via streaming metrics. The LLM-as-a-Judge system employs techniques like tiered rationales, consensus scoring, and factuality agents to optimize accuracy. These LLM-derived quality scores are correlated with key streaming metrics like take fraction and abandonment rate. This allows Netflix to proactively identify and fix synopsis issues, improving the member experience and content discovery.

https://netflixtechblog.com/evaluating-netflix-show-synopses-with-llm-as-a-judge-6269251e6f28?source=rss----2615bd06b42e---4 netflixtechblog.com

RSS Hunter • Apr 10

Stop Answering the Same Question Twice: Interval-Aware Caching for Druid at Netflix Scale

Netflix implemented an experimental caching layer to improve Druid query performance at its massive scale. This caching system addresses the problem of redundant queries from dashboards, especially during high-profile events. The core idea is to cache portions of results and only query Druid for the freshest data, introducing a small staleness trade-off. Exponential time-to-live (TTL) values are used, with older data cached longer to handle late-arriving events. The caching system uses a map-of-maps structure for efficient range scans of cached data. The cache intercepts requests at the Druid Router, serving results or querying Druid as needed. The solution combines cached and fresh data, returning a complete result and caching the fresh data asynchronously. Negative caching is used for empty buckets, with precautions against caching trailing empty buckets. The system uses Netflix's Key-Value Data Abstraction Layer (KVDAL), backed by Cassandra, which offers independent TTLs on each data point. The caching layer has significantly reduced Druid query load and improved query times, particularly during high-volume events. The caching system is still experimental, with future goals including integration into Druid itself.

https://netflixtechblog.com/stop-answering-the-same-question-twice-interval-aware-caching-for-druid-at-netflix-scale-22fadc9b840e?source=rss----2615bd06b42e---4 netflixtechblog.com

RSS Hunter • Apr 6

Powering Multimodal Intelligence for Video Search

The article discusses the challenges and solutions for building an advanced video search engine. The core problem is the overwhelming amount of video footage and the difficulty in extracting relevant moments. The solution involves a multimodal approach, integrating various AI models to analyze different aspects of video content. This approach requires unifying diverse data from specialized models into a cohesive, real-time intelligence system. The system's architecture focuses on processing at scale, handling billions of data points efficiently. A three-stage process involving transactional persistence, offline data fusion, and indexing for real-time search ensures data integrity and responsiveness. The process uses a decoupled pipeline to avoid bottlenecking during the ingestion process. The search service offers features like query preprocessing, fine-tuning semantic search, and advanced textual analysis for precise results. It also includes phrase matching, N-gram analysis, and fuzzy matching to enhance search accuracy. The system provides aggregations and flexible grouping to allow for more nuanced searching. Overall, the aim is to empower filmmakers by providing a powerful tool for discovering relevant video content.

https://netflixtechblog.com/powering-multimodal-intelligence-for-video-search-3e0020cf1202?source=rss----2615bd06b42e---4 netflixtechblog.com

RSS Hunter • Apr 4

Smarter Live Streaming at Scale: Rolling Out VBR for All Netflix Live Events

Netflix transitioned its Live events from Constant Bitrate (CBR) to Variable Bitrate (VBR) encoding for improved efficiency. VBR dynamically adjusts bitrate based on scene complexity, leading to reduced average traffic and improved quality of experience. A major challenge with VBR is the unpredictability of traffic, which can lead to server overload and instability due to large bitrate swings. To address this, Netflix implemented a new capacity reservation system based on nominal bitrates, mitigating the risks associated with VBR's variable nature. They also adjusted nominal bitrates across their streaming ladder to match the quality of CBR streams, using VMAF as a guiding metric. The shift to VBR allowed for efficiency gains like lower rebuffering and initial start delays. Netflix is now working on improving its adaptive bitrate algorithms and capacity reservation to further optimize VBR. The implementation of VBR was a company-wide project involving several teams and contributors.

https://netflixtechblog.com/smarter-live-streaming-at-scale-rolling-out-vbr-for-all-netflix-live-events-c8f833b238cc?source=rss----2615bd06b42e---4 netflixtechblog.com

RSS Hunter • Apr 2

Scaling Global Storytelling: Modernizing Localization Analytics at Netflix

Netflix's Analytics Engineering community gathered for an Analytics Summit to share insights across the business. The core focus is on enhancing the localization process for a global audience of over 300 million members. This involves scaling dubbing and subtitling to meet the needs of diverse language preferences. Previously, localization metrics were fragmented, leading to inconsistencies and maintenance issues. Netflix is modernizing its analytics through consolidation, standardization, and trust. The first pillar involves auditing and consolidating dashboards and backend pipelines. The second pillar tackles "Not-So-Tech Debt" by improving user experience and reporting on unified metrics. The third pillar is investing in a write-once, read-many architecture by centralizing business logic. This approach creates unified tables, solving problems once and propagating updates instantly. The future involves event-level analytics, analyzing granular timed-text data like subtitle lines. The goal is to understand how subtitle characteristics impact member engagement and refine guidelines. Ultimately, this modernization effort aims to improve the Netflix experience for all members worldwide.

https://netflixtechblog.com/scaling-global-storytelling-modernizing-localization-analytics-at-netflix-816f47290641?source=rss----2615bd06b42e---4 netflixtechblog.com

RSS Hunter • Mar 6

Optimizing Recommendation Systems with JDK’s Vector API

Netflix's Ranker service, responsible for personalized recommendations, had a CPU hotspot in its video serendipity scoring feature. Initially, the scoring utilized a costly, nested loop structure for calculating similarities. The first optimization involved batching calculations to leverage matrix multiplication, improving the algorithm. However, an initial implementation using matrix multiplication introduced overhead, like GC pressure and poor cache behavior. Further optimization involved switching to flat buffers, improving data layout and cache locality. Then, a thread-local buffer reuse strategy eliminated per-request allocations. BLAS libraries were explored but didn't provide significant gains due to JNI overhead. Finally, the JDK Vector API was integrated, enabling SIMD, with a fallback to optimized scalar code for compatibility. This streamlined approach resulted in a 7% drop in CPU utilization and a 12% drop in average latency. Ultimately, the CPU consumption of the scoring feature was reduced from 7.5% down to roughly 1%, and the CPU per request significantly decreased by 10%.

https://netflixtechblog.com/optimizing-recommendation-systems-with-jdks-vector-api-30d2830401ec?source=rss----2615bd06b42e---4 netflixtechblog.com

RSS Hunter • Mar 3

Mount Mayhem at Netflix: Scaling Containers on Modern CPUs

Netflix faced container scaling bottlenecks during their container runtime modernization. The issue arose because the new runtime used user namespaces for enhanced container security, employing kernel idmap features for efficient resource allocation, triggering frequent mount operations. These mount operations heavily contended for global kernel locks, particularly affecting nodes with many container layers. This contention was amplified on r5.metal instances, which have dual NUMA architecture. Benchmarks revealed instance type differences, with older Intel architectures showing higher container launch times. The bottleneck was rooted in Linux VFS path lookup code and was linked to the kernel's mount table's global locking. NUMA architecture amplified the problem due to memory access latency. Disabling hyperthreading improved performance by reducing resource competition. Centralized cache architectures in some CPUs exacerbated lock contention due to a single queueing structure. Distributed cache architectures, like those in AMD chips, showed better performance as contention was spread across multiple domains. Microbenchmarks validated the impact of NUMA, hyperthreading, and CPU architecture.

https://netflixtechblog.com/mount-mayhem-at-netflix-scaling-containers-on-modern-cpus-f3b09b68beac?source=rss----2615bd06b42e---4 netflixtechblog.com

RSS Hunter • Feb 28

MediaFM: The Multimodal AI Foundation for Media Understanding at Netflix

Netflix developed MediaFM, an in-house multimodal content embedding model to understand its vast catalog. This model aims to create a machine-readable understanding of content by integrating audio, video, and text. MediaFM uses a Transformer-based encoder to generate shot-level embeddings, capturing temporal relationships. The model's inputs are shot-level embeddings from video, audio, and timed text, concatenated and processed. A Masked Shot Modeling objective trains the model to predict original embeddings. Evaluation involves task-specific linear layers, proving MediaFM's superior performance compared to baselines. The model significantly improves in tasks demanding nuanced understanding, such as ad relevancy. Contextualization plays a critical role with an added modality. MediaFM benefits applications across Netflix, including ad relevancy and clip popularity prediction. The team is now exploring how to use pre-trained multimodal LLMs for future model development. The blog post concludes with gratitude for contributors and a glimpse into future work.

https://netflixtechblog.com/mediafm-the-multimodal-ai-foundation-for-media-understanding-at-netflix-e8c28df82e2d?source=rss----2615bd06b42e---4 netflixtechblog.com

RSS Hunter • Feb 23

Scaling LLM Post-Training at Netflix

Netflix developed a post-training framework to adapt large language models (LLMs) for their specific needs in recommendation, personalization, and search. This framework addresses the complexities of post-training at scale, including data pipelines, distributed computing, and workflow orchestration. The framework aims to help model developers focus on model innovation rather than infrastructure. Key challenges in post-training include data preparation, model sharding, and setting up training loops for production. The Netflix framework provides modular components for data, model, compute, and workflow management to simplify training. The framework supports SFT, DPO, RL, and knowledge distillation training recipes with reusable utilities. The framework evolved to support RL by integrating open-source tools and adopting a hybrid execution model. It prioritizes compatibility with the Hugging Face ecosystem to facilitate easy integration of models and tools. The post-training framework allows teams to experiment with advanced techniques and iterate more quickly.

https://netflixtechblog.com/scaling-llm-post-training-at-netflix-0046f8790194?source=rss----2615bd06b42e---4 netflixtechblog.com

RSS Hunter • Feb 13

Automating RDS Postgres to Aurora Postgres Migration

In 2024, Netflix standardized on Amazon Aurora PostgreSQL for its relational database needs after evaluating various technologies. This decision was driven by PostgreSQL's existing prevalence, industry momentum, and Aurora's scalability and feature advantages. A key initiative was migrating existing users to Aurora PostgreSQL, starting with RDS migrations. Migrations involve more than data transfer; they require orchestrated data and functionality transitions. Netflix faced substantial operational challenges with nearly 400 PostgreSQL clusters, necessitating a self-service migration workflow. This workflow addressed issues like zero data loss, minimal downtime, and the lack of direct access to credentials, promoting a seamless process. The chosen strategy utilized Aurora Read Replicas, trading complexity for reduced client downtime. Netflix's Data Access Layer managed secure connections between applications and databases, facilitating transparent migrations. A fully automated, self-service workflow was developed, orchestrating the entire migration process without manual intervention. The process included enabling backups, porting parameters, and creating an Aurora read replica for continuous synchronization. The quiescence phase focuses on a smooth transition to the Aurora cluster.

https://netflixtechblog.com/automating-rds-postgres-to-aurora-postgres-migration-261ca045447f?source=rss----2615bd06b42e---4 netflixtechblog.com

RSS Hunter • Feb 12

The AI Evolution of Graph Search at Netflix

Netflix's Graph Search platform is evolving to incorporate natural language search using AI, addressing limitations of its structured query language. Natural language search simplifies data retrieval, improving user experience across various applications. The core challenge involves translating user queries into valid Graph Search Filter statements. This process uses Large Language Models (LLMs) to generate queries, incorporating context engineering for accuracy. Context engineering leverages GraphQL schema information and controlled vocabularies to inform the LLM. Retrieval-Augmented Generation (RAG) is employed to manage context, focusing on relevant fields and controlled vocabulary values. Field RAG and Controlled Vocabularies RAG enhance precision by matching user intent to index elements. The process involves vector search and deduplication for efficient context management. The system provides instructions to the LLM to generate syntactically, semantically, and pragmatically correct filters. The effectiveness of the solution requires careful parameter tuning for RAG components. The described work paves the potential for building a RAG system on top of Graph Search. Future articles will detail the implementation, performance evaluation, and platform evolution.

https://netflixtechblog.com/the-ai-evolution-of-graph-search-at-netflix-d416ec5b1151?source=rss----2615bd06b42e---4 netflixtechblog.com

RSS Hunter • Jan 26

How Temporal Powers Reliable Cloud Operations at Netflix

Netflix adopted Temporal, a durable execution platform, to address deployment failures in its Spinnaker continuous delivery platform. Spinnaker, used for most Netflix deployments, faced issues in its Orca and Clouddriver services, particularly with Cloud Operations. Cloud Operations, responsible for cloud infrastructure changes, experienced transient failures and operational complexity within Clouddriver. This led to a 4% failure rate in deployments, impacting engineering productivity. Temporal offers Workflows and Activities, enabling resilient execution despite failures. Netflix migrated Cloud Operations to Temporal, simplifying the process and reducing the failure rate. The migration involved transitioning Cloud Operations into Temporal workflows, leveraging the platform's retry mechanisms and state management. The integration required abstracting and dynamically configuring the system. The result of this migration was a significant reduction in deployment failures, down to 0.0001%. This improved reliability and engineering efficiency.

https://netflixtechblog.com/how-temporal-powers-reliable-cloud-operations-at-netflix-73c69ccb5953?source=rss----2615bd06b42e---4 netflixtechblog.com

RSS Hunter • Dec 15, 2025