Evaluating Netflix Show Synops... Note

Evaluating Netflix Show Synopses with LLM-as-a-Judge

Netflix aims to improve the synopsis selection process for its vast content library. Thousands of titles complicate the user's choice, making synopses crucial for decision-making. High-quality synopses enhance user engagement, while poor ones lead to frustration and abandonment. The challenge lies in scaling quality validation across hundreds of thousands of synopses. Netflix developed an LLM-based approach to evaluate synopsis quality across four key dimensions. This system achieves over 85% agreement with creative writers, ensuring expert-driven quality standards. Synopsis quality is assessed through both creative quality defined by writers and member implicit feedback via streaming metrics. The LLM-as-a-Judge system employs techniques like tiered rationales, consensus scoring, and factuality agents to optimize accuracy. These LLM-derived quality scores are correlated with key streaming metrics like take fraction and abandonment rate. This allows Netflix to proactively identify and fix synopsis issues, improving the member experience and content discovery.
CdXz5zHNQW_s8spGD1mD0.png