Improve Your Next Experiment b... Note

Improve Your Next Experiment by Learning Better Proxy Metrics From Past Experiments

Researchers at Netflix present methods to establish the relationship between short-term proxy metrics and long-term north star metrics using historical experiments. Naive approaches to understanding this relationship can be misleading due to confounders and correlated measurement error. The Total Covariance (TC) estimator calculates the OLS slope by subtracting the scaled measurement error covariance from the covariance of estimated treatment effects. Jackknife Instrumental Variables Estimation (JIVE) removes correlated measurement error by excluding each observation's data from the computation of its instrumented surrogate values. A Limited Information Maximum Likelihood (LIML) estimator is statistically efficient under the assumption of no direct effects between the treatment and the north star metric. These methods yield linear structural models of treatment effects that are easy to interpret and facilitate coordination and alignment towards the north star. They enable metric tradeoff management, metrics innovation, and independent team work. Despite the current data architecture challenges, these methods have been successfully implemented at Netflix and are actively used to develop proxy metrics. The researchers acknowledge the need for a more flexible data architecture to streamline their application and welcome potential contributors through open job postings.