Netflix TechBlog | Medium
Follow
Fact-Checking at Scale: Building a More Accurate and Efficient Synopsis Factuality System
A synopsis provides a brief overview of a show, helping viewers decide what to watch. Synopses must be factual, striking a balance between informing and intriguing. Fact-checking these at Netflix's scale necessitates an automated factuality service. The existing service uses a reasoning model, but faces limitations due to high costs from long-context model calls and misaligned rationales.A new approach significantly reduces costs by 52.4% per synopsis and improves quality, with Recall-Fail rising from 0.77 to 0.86. It also achieves an 82% overlap with human rationales, up from 69%. Factuality encompasses not just literal truth, but avoiding misleading impressions, as seen in nuanced Bridgerton examples. The original pipeline involved multiple, expensive holistic LLM calls with long contexts, and often produced unreliable reasoning.Analyzing the current pipeline revealed that not all synopses need multiple checks; easy cases can be identified via confidence scores. This led to a "confidence-gated" approach, where only difficult cases receive extra scrutiny. Instead of identical holistic judges, specialized "Literal" and "Implicature" judges were introduced to target specific error types.Furthermore, the model's rationales, often containing "shaky spots" like assumptions or doubts, are now leveraged. A router model rephrases these into verifiable "atomic claims" and sends them to secondary Stage-2 judges. This cascade factuality service combines specialized judges, confidence-gated escalation, and targeted rechecking to enhance accuracy and efficiency.The new tiered system processes synopses through both judges; high-confidence verdicts are final, while low-confidence cases escalate to the router and Stage-2 judges. Stage 2 can overturn initial verdicts if secondary checks support concessions or contradict assumptions. This comprehensive evaluation focuses on quality, primarily Recall-FAIL, and cost reduction, including LLM calls and end-to-end verdict production cost.