Sign in
Experiments

A/B Materials for In-Depth Study

A comprehensive list of articles and links to deep dive in online experiments!

Антон Марцен

• 3 min read
A/B Materials for In-Depth Study
img-5e6d46d1ea160c0c8297d1b1

Recently, I got involved in the adventure of A/B testing. I decided to go beyond simple math and online calculators and dig deeper into this field. In this journey, I found many useful articles, papers, researches and so on. If you, like me, want to become professional in A/B testing, then check the following links in this post. You’ll find many exciting things.

<div class="
      image-block-outer-wrapper
      layout-caption-below
      design-layout-inline
      combination-animation-none
      individual-animation-none
      individual-text-animation-none
    " data-test="image-block-inline-outer-wrapper">

  

  
    <figure class="
          sqs-block-image-figure
          intrinsic
        " style="max-width:2500px;">
      
    
    

    
      
        
      <div class="image-block-wrapper" data-animation-role="image">
        <div class="sqs-image-shape-container-element
          
      
    
          has-aspect-ratio
        " style="
            position: relative;
            
              padding-bottom:57.84000015258789%;
            
            overflow: hidden;-webkit-mask-image: -webkit-radial-gradient(white, black);
          ">
            
              
              
              
              
              
              
              
              <img data-stretch="false" src="https://images.squarespace-cdn.com/content/v1/5e6d2f92970b8e6e6a8d1910/1584220062511-5NLE2GB6VSOS6SLZ9BUT/image-asset.jpeg" data-image="https://images.squarespace-cdn.com/content/v1/5e6d2f92970b8e6e6a8d1910/1584220062511-5NLE2GB6VSOS6SLZ9BUT/image-asset.jpeg" data-image-dimensions="2500x1446" data-image-focal-point="0.5,0.5" alt data-load="false" elementtiming="system-image-block" width="2500" height="1446" sizes="(max-width: 640px) 100vw, (max-width: 767px) 100vw, 100vw" style="display:block;object-fit: cover; width: 100%; height: 100%; object-position: 50% 50%" onload="this.classList.add(&quot;loaded&quot;)" srcset="https://images.squarespace-cdn.com/content/v1/5e6d2f92970b8e6e6a8d1910/1584220062511-5NLE2GB6VSOS6SLZ9BUT/image-asset.jpeg?format=100w 100w, https://images.squarespace-cdn.com/content/v1/5e6d2f92970b8e6e6a8d1910/1584220062511-5NLE2GB6VSOS6SLZ9BUT/image-asset.jpeg?format=300w 300w, https://images.squarespace-cdn.com/content/v1/5e6d2f92970b8e6e6a8d1910/1584220062511-5NLE2GB6VSOS6SLZ9BUT/image-asset.jpeg?format=500w 500w, https://images.squarespace-cdn.com/content/v1/5e6d2f92970b8e6e6a8d1910/1584220062511-5NLE2GB6VSOS6SLZ9BUT/image-asset.jpeg?format=750w 750w, https://images.squarespace-cdn.com/content/v1/5e6d2f92970b8e6e6a8d1910/1584220062511-5NLE2GB6VSOS6SLZ9BUT/image-asset.jpeg?format=1000w 1000w, https://images.squarespace-cdn.com/content/v1/5e6d2f92970b8e6e6a8d1910/1584220062511-5NLE2GB6VSOS6SLZ9BUT/image-asset.jpeg?format=1500w 1500w, https://images.squarespace-cdn.com/content/v1/5e6d2f92970b8e6e6a8d1910/1584220062511-5NLE2GB6VSOS6SLZ9BUT/image-asset.jpeg?format=2500w 2500w" loading="lazy" decoding="async" data-loader="sqs">

            
        </div>
      </div>
    
      
    

    
  
    </figure>
  

</div>

Overview of the A/B testing field

These links are great to refresh your knowledge, remember the basic formulas, and to learn common mistakes and solutions for them.

  1. Guidelines For Ab Testing — Hooked on Data

  2. Controlled Experiments on the Web: Survey and Practical Guide

  3. Seven Rules of Thumb for Web Site Experimenters

Advanced Topics

Sequential Analysis

  1. Simple Sequential A/B Testing — Evan Miller

  2. Rapid A/B-testing with Sequential Analysis | Audun M Øygard

  3. Estimation in Sequential Analysis | Audun M Øygard

Peeking Problem

  1. How Not To Run an A/B Test — Evan Miller with wrong conclusions and great response to it A/B Testing Rigorously (without losing your job) (and extension A/B Testing With Limited Data)

  2. The Fatal Flaw of A/B Tests: Peeking | Lucidchart Blog

  3. Peeking at A/B tests: continuous monitoring without pain | the morning paper

Bayesian

  1. Bayesian vs Frequentist A/B Testing (and Does it Even Matter?)

  2. Discussion on Reddit — Frequentist or Bayesian AB Testing Methodology? : statistics

  3. Bayesian A/B Testing at VWO

Bayesian vs. Peeking Problem

  1. Is Bayesian A/B Testing Immune to Peeking? Not Exactly — Variance Explained

  2. Bayesian AB Testing is Not Immune to Optional Stopping Issues | Analytics-Toolkit.com

A/A

  1. The A/A Test

More

In addition to authors that wrote previous papers and articles, I recommend you to check out these resources:

  1. Ronny Kohavi is a Microsoft Technical Fellow and Vice President of Analysis & Experimentation. You must explore his project ExP Platform. Also look for his recommendations on what to read.

  2. Eytan Bakshy’s blog. Eytan is a senior scientist on the Facebook Core Data Science Team, who lead the Adaptive Experimentation group.

  3. Evan Miller is a developer of statistical software. He has a series of A/B testing articles on his website.

  4. Papers from the SIGKDD conferences. It’s a community for data mining, data science, and analytics.

  5. The story “How Optimizely (Almost) Got Me Fired” and a paper about The New Stats Engine, where they fix the problems.

  6. Companies Blogs:

Sign up for more like this.

Enter your email
Subscribe