A/B Testing Course

Lesson 1 • Common Testing Pitfalls Overview

Previews the most frequent errors—peeking, novelty effects, and selection bias—so learners recognize them early. Sets expectations for rigor maintained throughout the course.

Lesson 2 • Key Metrics and KPIs

Identifies primary, secondary, and guardrail metrics for experiments. Choosing the right metric directly determines whether a test answers the intended business question.

Lesson 3 • Core Statistical Concepts

Covers probability distributions, variance, and sampling theory essential for valid tests. Provides the mathematical language used throughout the entire course.

Lesson 4 • What Is A/B Testing

Defines controlled experiments and contrasts them with observational analysis. Anchors the chapter by showing how A/B testing removes guesswork from product decisions.

Lesson 1 • Type I and Type II Errors

Defines false positives and false negatives and their business consequences. Frames the alpha-beta trade-off that governs all sample size and power decisions.

Lesson 2 • Understanding P-Values

Explains what a p-value measures and what it does not prove. Directly addresses widespread misconceptions that lead to false-positive business decisions.

Lesson 3 • Statistical Power and Sensitivity

Explains how power determines a test's ability to detect real effects. Underpowered tests waste resources; this section ensures learners size tests correctly.

Lesson 4 • Practical Significance vs. Statistical

Distinguishes statistically significant results from business-meaningful ones. Prevents teams from shipping changes that are detectable but economically irrelevant.

Lesson 1 • Using Sample Size Calculators

Demonstrates how to use online and programmatic calculators accurately. Reduces calculation errors and speeds up the pre-experiment planning phase.

Lesson 2 • Novelty and Primacy Effect Timing

Explains how user behavior changes at experiment start and stabilizes over time. Teaches when to begin measuring to avoid inflated or deflated early results.

Lesson 3 • Sample Size Fundamentals

Derives the relationship between sample size, power, alpha, and effect size. Establishes the formula learners will apply in every subsequent experiment design.

Lesson 4 • Stopping Rules and Early Termination

Covers pre-specified stopping rules that maintain error rate guarantees. Prevents the peeking problem while allowing ethical early stops for severe harm.

Lesson 1 • Instrumentation and Tracking

Defines the event logging and analytics instrumentation required before launch. Missing or incorrect tracking is the leading cause of unanalyzable experiments.

Lesson 2 • Pre-Experiment Checklist

Consolidates all design decisions into a launch-readiness checklist. Systematic review catches errors that would invalidate results after data collection.

Lesson 3 • Control and Variant Construction

Guides creation of control baselines and treatment variants that isolate one variable. Isolation is the core principle that makes causal inference possible.

Lesson 4 • Defining the Experiment Scope

Establishes target population, exposure surface, and exclusion criteria before launch. Scope decisions directly control internal validity and generalizability of results.

Lesson 1 • Interaction Effects Between Tests

Explains how simultaneous experiments can interfere and bias each other's results. Learners apply mutual exclusion and factorial designs to manage concurrent tests.

Lesson 2 • Launching an Experiment Safely

Covers staged rollouts, traffic ramping, and kill-switch protocols for safe launches. Gradual exposure limits user impact if a critical bug surfaces post-launch.

Lesson 3 • Data Quality Monitoring

Establishes ongoing checks for metric anomalies, logging gaps, and bot traffic. Continuous monitoring prevents silent data corruption from distorting final results.

Lesson 4 • Sample Ratio Mismatch Detection

Teaches how to identify when observed traffic splits deviate from intended ratios. SRM invalidates randomization and must be caught before analysis begins.

Lesson 1 • Interpreting Inconclusive Results

Provides a decision framework for null results: ship, iterate, or abandon. Inconclusive tests carry information and should not default to shipping the variant.

Lesson 2 • Choosing the Right Statistical Test

Maps metric types to appropriate tests: z-test, t-test, chi-square, and Mann-Whitney. Selecting the wrong test inflates error rates and produces misleading conclusions.

Lesson 3 • Variance Reduction Techniques

Introduces CUPED and stratified analysis to reduce noise and increase test sensitivity. Lower variance means the same sample size detects smaller, real effects.

Lesson 4 • Segmented Analysis

Covers breaking results by user segments to find heterogeneous treatment effects. Segment analysis reveals who benefits and who is harmed by a change.

Lesson 1 • Bandit Algorithms for Optimization

Explains epsilon-greedy, UCB, and Thompson sampling for explore-exploit trade-offs. Bandits maximize cumulative reward when learning speed outweighs causal inference needs.

Lesson 2 • Bayesian A/B Testing

Introduces prior distributions, posterior updates, and probability-of-being-best metrics. Bayesian framing aligns naturally with business decision language and risk tolerance.

Lesson 3 • Switchback and Interleaving Tests

Covers time-based switchback designs and interleaving for marketplace and ranking systems. Addresses network effects that make user-level randomization invalid.

Lesson 4 • Multivariate Testing (MVT)

Teaches full-factorial and fractional-factorial designs for testing multiple elements simultaneously. MVT reveals interaction effects invisible to isolated A/B tests.

Lesson 1 • Measuring Experimentation Program ROI

Quantifies the business value of the experimentation program to justify investment. ROI measurement secures executive sponsorship and resources for platform growth.

Lesson 2 • Scaling Experimentation Velocity

Covers strategies for increasing test throughput without sacrificing quality or coordination. High velocity requires standardized tooling, templates, and self-serve analytics.

Lesson 3 • Knowledge Management and Learnings

Designs experiment repositories and retrospective processes that compound organizational learning. Documented learnings prevent repeated mistakes and accelerate future hypothesis generation.

Lesson 4 • Experimentation Platform Architecture

Outlines the components of an internal experimentation platform: assignment, logging, and analysis. Platform maturity directly determines how many tests an organization can run.