A/B Materials for In-Depth Study

Recently, I got involved in the adventure of A/B testing. I decided to go beyond simple math and online calculators and dig deeper into this field. In this journey, I found many useful articles, papers, researches and so on. If you, like me, want to become professional in A/B testing, then check the following links in this post. You’ll find many exciting things.

Overview of the A/B testing field

These links are great to refresh your knowledge, remember the basic formulas, and to learn common mistakes and solutions for them.

1. Guidelines For Ab Testing — Hooked on Data

2. Controlled Experiments on the Web: Survey and Practical Guide

3. Seven Rules of Thumb for Web Site Experimenters

Advanced Topics

Sequential Analysis

1. Simple Sequential A/B Testing — Evan Miller

2. Rapid A/B-testing with Sequential Analysis | Audun M Øygard

3. Estimation in Sequential Analysis | Audun M Øygard

Peeking Problem

1. How Not To Run an A/B Test — Evan Miller with wrong conclusions and great response to it A/B Testing Rigorously (without losing your job) (and extension A/B Testing With Limited Data)

2. The Fatal Flaw of A/B Tests: Peeking | Lucidchart Blog

3. Peeking at A/B tests: continuous monitoring without pain | the morning paper

Bayesian

1. Bayesian vs Frequentist A/B Testing (and Does it Even Matter?)

2. Discussion on Reddit — Frequentist or Bayesian AB Testing Methodology? : statistics

3. Bayesian A/B Testing at VWO

Bayesian vs. Peeking Problem

1. Is Bayesian A/B Testing Immune to Peeking? Not Exactly — Variance Explained

2. Bayesian AB Testing is Not Immune to Optional Stopping Issues | Analytics-Toolkit.com

A/A

1. The A/A Test

More

In addition to authors that wrote previous papers and articles, I recommend you to check out these resources:

1. Ronny Kohavi is a Microsoft Technical Fellow and Vice President of Analysis & Experimentation. You must explore his project ExP Platform. Also look for his recommendations on what to read.

2. Eytan Bakshy’s blog. Eytan is a senior scientist on the Facebook Core Data Science Team, who lead the Adaptive Experimentation group.

3. Evan Miller is a developer of statistical software. He has a series of A/B testing articles on his website.

4. Papers from the SIGKDD conferences. It’s a community for data mining, data science, and analytics.

5. The story “How Optimizely (Almost) Got Me Fired” and a paper about The New Stats Engine, where they fix the problems.

6. Companies Blogs:

   • Uber — Under the Hood of Uber’s Experimentation Platform

   • Netflix — Quasi Experimentation at Netflix — Netflix TechBlog — Medium

   • Airbnb — Scaling Airbnb’s Experimentation Platform — Airbnb Engineering & Data Science — Medium

   • Booking — How Booking.com increases the power of online experiments with CUPED

   • Facebook — Efficient tuning of online systems using Bayesian optimization — Facebook Research

   • Quora – A Robust Statistical Test for Ratio Metrics

   • Yandex – Online Evaluation for Effective Web Service Development

   • Stitch Fix Technology — Suffering from a Non-inferiority Complex?