Title Launch Observability at ... Note

Title Launch Observability at Netflix Scale

To achieve comprehensive title observability at Netflix, the company introduced observability endpoints across all services within its Personalization and Discovery Stack. Each microservice involved in the stack had to introduce a new "Title Health" endpoint that adheres to principles of accurately reflecting production behavior, standardization, and answering the Insight Triad. The Insight Triad requires the endpoint to answer whether a title is eligible for promotion, why it's not eligible, and what can be done to fix any problems. A stable proto request/response format was developed to standardize communication between the observability service and the personalization stack's observability endpoints. The high-level architecture of the solution involves establishing observability endpoints, proactive monitoring, tracking real-time title impressions, storing data in an optimized datastore, and offering easy-to-integrate APIs for the dashboard.The Title Health microservice runs a scheduled collector job every 30 minutes to retrieve relevant title health information from assigned row services. Real-time title impressions data is processed from a Kafka queue, which is polled every two minutes to retrieve impressions data. The data is then aggregated and presented as an additional health status indicator for stakeholders.The data is stored in a dedicated Hollow feed for each collector, which allows for high-performance read-only access and enables the monitoring of overall health and tracking of title history. The observability dashboard utilizes the title health service to present the status of titles to stakeholders, and the system also has a "Time Travel" capability that simulates traffic ahead of time to catch and fix issues before they impact members.The system's architecture and strategies have enabled Netflix to enhance title launch observability, ensuring a thrilling viewing experience and fostering trust with content creators and partners. The solution also lays the groundwork for future innovations, ensuring that every story reaches its intended audience and that every member enjoys their favorite titles on Netflix.
CdXz5zHNQW_bbTwixEEHc.png