An Introduction to Bayesian A/B Testing

A/B testing, or split testing, is a method used by businesses to compare different versions of a webpage or marketing asset to determine which one performs better in terms of user engagement, click-through rates, and conversion rates. The Chi2 test is a frequentist statistical test that can be used to determine if the differences observed in A/B testing are statistically significant. However, this test has limitations, such as being sensitive to sample size and not providing information on the magnitude or practical significance of the difference. Bayesian A/B testing offers more flexibility and allows for the modeling of arbitrary data-generating processes, making it more suitable for complex scenarios. It also allows for the explicit modeling of uncertainty and the use of prior beliefs. In a Bayesian setting, we can model the data-generating process directly and obtain a full probability distribution for the conversion probabilities. This approach can be applied to more complicated scenarios, such as comparing the response to an intervention over time. By using PyMC, a Python package, we can draw samples from the posterior distribution and obtain quantities of interest such as credible intervals. The Bayesian approach provides more intuitive and straightforward interpretation of results compared to the frequentist approach.