Netflix TechBlog | Medium
Follow
A Human-Augmenting Agentic Workflow for Causal Inference
Data analysis is increasingly delegated to software agents, necessitating oversight for valid results, especially in Observational Causal Inference (OCI). This work introduces an agentic workflow for OCI that adheres to rigorous templates while augmenting human inspection. The workflow aims to reduce repetitive tasks for OCI practitioners, allowing them to focus on nuanced aspects like question framing and assumption evaluation. An open-sourced version of their oci-agent is available for improvement, and evaluations show it outperforms one-shot iterations and rivals hand-tuned benchmarks. The philosophy relies on target trial emulation, outlining ideal A/B tests to inform necessary assumptions for causal inference. This involves design diagnostics like covariate balance, overlap, placebo outcomes, and sensitivity analysis. The agent orchestrates an actor-critic loop with three personas: Principal (human user), Actor (analysis executor), and Critic (results synthesizer). Principals define the plan and context, Actors refine and execute the analysis with diagnostics, and Critics identify gaps and assess credibility. To empower human evaluation, the workflow makes each analytic step transparent through published artifacts like plans, specifications, and notebooks. A case study at Netflix estimated the impact of a new entertainment type, revealing significant early adopter bias. Standard regression gave a large effect, but the agentic workflow, using diagnostics, identified poor overlap and placebo test failures. Addressing these, the agent implemented trimming, significantly reducing the estimated effect to a more credible range for the overlapping population. Follow-up analyses, such as sensitivity tests and time-series generation, are facilitated by the agent's ability to manage complex, multi-version executions.