Netflix TechBlog | Medium
Follow
Behind the Scenes: Evolving Netflix’s Ads Event Pipeline for Live — Part II
Netflix rebuilt its Ads Event Publisher to handle increased scale and complexity, particularly for live streaming advertisements. The previous system relied on a Key-Value store for ad metadata enrichment. This approach, while functional, presented scaling bottlenecks and critical path dependencies that were unsuitable for live events. Live streams generate massive, spiky traffic requiring a more robust and elastic solution.The new architecture utilizes Apache Flink, a stateful stream processing framework, for near real-time enrichment. Ad server response logs are sent to Kafka instead of a database. Flink then performs a real-time streaming join between these responses and client-generated ad events. This eliminates direct database writes on the critical serving path, preventing outages and improving scalability.The Flink job normalizes events, routes them to the correct processing region, and importantly, performs a stateful join. It holds recent ad serving responses in memory and matches incoming client events against them within configurable windows, typically 60 minutes. This approach allows for on-the-spot enrichment when both events are available. If an event arrives first, it's held until its corresponding response appears.To address cross-region enrichment, the system routes client events to the region that served the ad, rather than replicating all serving responses globally, which is more cost-effective. The ad token now includes the serving region information. Although the 60-minute window covers most events, a separate path exists to recover late-arriving events, ensuring data completeness for billing and reporting. This stateful stream processing architecture provides the necessary scalability and resilience for Netflix's evolving advertising needs.