A/B Materials for In-Depth Study

A comprehensive list of articles and links to deep dive in online experiments!

Recently, I got involved in the adventure of A/B testing. I decided to go beyond simple math and online calculators and dig deeper into this field. In this journey, I found many useful articles, papers, researches and so on. If you, like me, want to become professional in A/B testing, then check the following links in this post. You’ll find many exciting things.

Overview of the A/B testing field

These links are great to refresh your knowledge, remember the basic formulas, and to learn common mistakes and solutions for them.

  1. Guidelines For Ab Testing — Hooked on Data

  2. Controlled Experiments on the Web: Survey and Practical Guide

  3. Seven Rules of Thumb for Web Site Experimenters

Advanced Topics

Sequential Analysis

  1. Simple Sequential A/B Testing — Evan Miller

  2. Rapid A/B-testing with Sequential Analysis | Audun M Øygard

  3. Estimation in Sequential Analysis | Audun M Øygard

Peeking Problem

  1. How Not To Run an A/B Test — Evan Miller with wrong conclusions and great response to it A/B Testing Rigorously (without losing your job) (and extension A/B Testing With Limited Data)

  2. The Fatal Flaw of A/B Tests: Peeking | Lucidchart Blog

  3. Peeking at A/B tests: continuous monitoring without pain | the morning paper

Bayesian

  1. Bayesian vs Frequentist A/B Testing (and Does it Even Matter?)

  2. Discussion on Reddit — Frequentist or Bayesian AB Testing Methodology? : statistics

  3. Bayesian A/B Testing at VWO

Bayesian vs. Peeking Problem

  1. Is Bayesian A/B Testing Immune to Peeking? Not Exactly — Variance Explained

  2. Bayesian AB Testing is Not Immune to Optional Stopping Issues | Analytics-Toolkit.com

A/A

  1. The A/A Test

More

In addition to authors that wrote previous papers and articles, I recommend you to check out these resources:

  1. Ronny Kohavi is a Microsoft Technical Fellow and Vice President of Analysis & Experimentation. You must explore his project ExP Platform. Also look for his recommendations on what to read.

  2. Eytan Bakshy’s blog. Eytan is a senior scientist on the Facebook Core Data Science Team, who lead the Adaptive Experimentation group.

  3. Evan Miller is a developer of statistical software. He has a series of A/B testing articles on his website.

  4. Papers from the SIGKDD conferences. It’s a community for data mining, data science, and analytics.

  5. The story “How Optimizely (Almost) Got Me Fired” and a paper about The New Stats Engine, where they fix the problems.

  6. Companies Blogs: