A/B Testing & Experimentation — AI Tutorials

Why A/B Testing ML Models?

Offline evaluation (test set metrics) tells you how a model performs on historical data. But it cannot tell you:

How users will actually behave with the new model
Whether gains in ML metrics translate to business metrics (revenue, engagement)
Whether the model causes harm in unexpected ways

A/B testing (online controlled experiment) is the gold standard for measuring the causal impact of a model change on real users.

Statistical Foundations

Hypothesis Testing

Before running any test, define:

H₀ (null hypothesis): the new model has no effect (variants are equal)
H₁ (alternative hypothesis): the new model improves the business metric
α (significance level): the acceptable false positive rate (typically 0.05)
β (Type II error): the acceptable false negative rate
Power (1 − β): probability of detecting a true effect (typically 0.80 or 0.95)

Sample Size Calculation

The required sample size per variant depends on the expected effect size, the metric's variance, and your chosen α and power:

$n = \frac{2 \sigma^2 (z_{\alpha/2} + z_{\beta})^2}{\delta^2}$

Where δ is the minimum detectable effect (MDE) and σ² is the metric variance.

Sample Size and Power Calculationpython

from scipy import stats
import numpy as np

def required_sample_size(
    baseline_rate: float,
    mde: float,           # minimum detectable effect (relative)
    alpha: float = 0.05,
    power: float = 0.80,
) -> int:
    """
    Calculate required sample size per variant for a proportion metric
    (e.g., conversion rate, fraud rate).
    """
    p1 = baseline_rate
    p2 = baseline_rate * (1 + mde)

    z_alpha = stats.norm.ppf(1 - alpha / 2)  # two-sided
    z_beta = stats.norm.ppf(power)

    pooled = (p1 + p2) / 2
    n = (z_alpha + z_beta) ** 2 * 2 * pooled * (1 - pooled) / (p2 - p1) ** 2
    return int(np.ceil(n))

# Example: fraud detection rate = 1%, want to detect 10% relative improvement
baseline = 0.01
mde = 0.10  # detect a 10% relative lift (1% → 1.1%)

n = required_sample_size(baseline, mde)
print(f"Required per variant: {n:,} users")
print(f"Total: {2*n:,} users")
print(f"At 10k users/day: {2*n/10_000:.1f} days")

# Check achieved power for a given sample
def achieved_power(n, baseline_rate, mde, alpha=0.05):
    p1, p2 = baseline_rate, baseline_rate * (1 + mde)
    pooled = (p1 + p2) / 2
    z_alpha = stats.norm.ppf(1 - alpha / 2)
    se = np.sqrt(2 * pooled * (1 - pooled) / n)
    z = abs(p2 - p1) / se - z_alpha
    return stats.norm.cdf(z)

Designing a Proper A/B Test

Traffic Splitting

Randomly assign users (not sessions) to variants. User-level assignment prevents:

Novelty effect: users in treatment see something new and engage more initially
Carryover effect: a user sees both variants in the same session

Holdback Groups

A holdback group receives the old model while the rest of traffic migrates to the new model. This lets you measure the aggregate impact after the experiment ends.

Group	Traffic	Receives
Control	10%	Old model (Model A)
Treatment	80%	New model (Model B)
Holdback	10%	Old model (for post-launch measurement)

Guardrail Metrics

Alongside your primary metric, define guardrail metrics that must not degrade:

Latency p99 (serving performance must not get worse)
User error rate (the model must not break the product)
Fairness metrics by demographic subgroup

A/B Test Analysis: Two-Proportion Z-Testpython

import numpy as np
from scipy import stats
from dataclasses import dataclass

@dataclass
class ABTestResult:
    control_rate: float
    treatment_rate: float
    lift_pct: float
    p_value: float
    significant: bool
    confidence_interval: tuple[float, float]

def analyze_ab_test(
    control_conversions: int,
    control_n: int,
    treatment_conversions: int,
    treatment_n: int,
    alpha: float = 0.05,
) -> ABTestResult:
    p_c = control_conversions / control_n
    p_t = treatment_conversions / treatment_n

    # Two-proportion z-test
    p_pool = (control_conversions + treatment_conversions) / (control_n + treatment_n)
    se = np.sqrt(p_pool * (1 - p_pool) * (1/control_n + 1/treatment_n))
    z = (p_t - p_c) / se
    p_value = 2 * (1 - stats.norm.cdf(abs(z)))  # two-sided

    # 95% confidence interval on the lift
    se_diff = np.sqrt(p_c * (1 - p_c) / control_n + p_t * (1 - p_t) / treatment_n)
    z_crit = stats.norm.ppf(1 - alpha / 2)
    ci = (p_t - p_c - z_crit * se_diff, p_t - p_c + z_crit * se_diff)

    return ABTestResult(
        control_rate=p_c,
        treatment_rate=p_t,
        lift_pct=(p_t - p_c) / p_c * 100,
        p_value=p_value,
        significant=p_value < alpha,
        confidence_interval=ci,
    )

result = analyze_ab_test(
    control_conversions=950, control_n=100_000,
    treatment_conversions=1_050, treatment_n=100_000,
)
print(f"Lift: {result.lift_pct:.1f}% | p={result.p_value:.4f} | Significant: {result.significant}")

Multi-Armed Bandits

Traditional A/B tests fix traffic splits for the entire experiment duration, wasting traffic on underperforming variants. Multi-armed bandits adapt traffic allocation dynamically, sending more traffic to better-performing variants.

Thompson Sampling

Thompson Sampling is the most widely used bandit algorithm for conversion metrics. It maintains a Beta distribution over each variant's true conversion rate and samples from those distributions to make allocation decisions.

Thompson Sampling Multi-Armed Banditpython

import numpy as np
from dataclasses import dataclass, field

@dataclass
class BetaBandit:
    """Thompson Sampling bandit for binary conversion metrics."""
    n_arms: int
    alpha: list[float] = field(default_factory=list)  # successes + 1
    beta: list[float] = field(default_factory=list)   # failures + 1

    def __post_init__(self):
        self.alpha = [1.0] * self.n_arms
        self.beta = [1.0] * self.n_arms

    def select_arm(self) -> int:
        """Sample from each arm's Beta posterior and pick the best."""
        samples = [np.random.beta(self.alpha[i], self.beta[i]) for i in range(self.n_arms)]
        return int(np.argmax(samples))

    def update(self, arm: int, reward: int) -> None:
        """Update Beta posterior given a 0/1 reward."""
        self.alpha[arm] += reward
        self.beta[arm] += (1 - reward)

    def traffic_allocation(self) -> list[float]:
        """Estimate current traffic allocation via Monte Carlo."""
        wins = [0] * self.n_arms
        for _ in range(10_000):
            best = self.select_arm()
            wins[best] += 1
        return [w / 10_000 for w in wins]

# Simulate: 3 models with true conversion rates 0.01, 0.012, 0.011
bandit = BetaBandit(n_arms=3)
true_rates = [0.010, 0.012, 0.011]

for step in range(5_000):
    arm = bandit.select_arm()
    reward = int(np.random.random() < true_rates[arm])
    bandit.update(arm, reward)

print("Traffic allocation:", [f"{p:.1%}" for p in bandit.traffic_allocation()])
# → ~80% to arm 1 (the best), remaining split between arms 0 and 2

Interleaving for Ranking Models

For ranking models (search, recommendations), standard A/B testing requires large samples because per-user conversion signals are noisy. Interleaving dramatically increases sensitivity.

How Interleaving Works

Instead of showing a user a ranking from either Model A or Model B, you show a merged list from both models, then track which model's items get clicked.

Team Draft Interleaving:

Flip a coin to decide which model picks first (A or B)
Model A picks its top item → add to list, mark as A's
Model B picks its top unselected item → add to list, mark as B's
Repeat until list is full

After collecting clicks, count: did A's items or B's items get more clicks? This gives a signal 10–100× more sensitive than a standard A/B test.

Team Draft Interleavingpython

import random
from collections import Counter

def team_draft_interleave(ranking_a: list, ranking_b: list, k: int = 10) -> tuple[list, dict]:
    """
    Interleave two ranked lists using team draft method.
    Returns the interleaved list and a dict mapping item → model.
    """
    teams = {}  # item_id → 'A' or 'B'
    interleaved = []
    seen = set()
    a_idx, b_idx = 0, 0

    # Coin flip: which model picks first
    turn = random.choice(['A', 'B'])

    while len(interleaved) < k:
        if turn == 'A':
            while a_idx < len(ranking_a) and ranking_a[a_idx] in seen:
                a_idx += 1
            if a_idx < len(ranking_a):
                item = ranking_a[a_idx]
                interleaved.append(item)
                seen.add(item)
                if item not in teams:
                    teams[item] = 'A'
            turn = 'B'
        else:
            while b_idx < len(ranking_b) and ranking_b[b_idx] in seen:
                b_idx += 1
            if b_idx < len(ranking_b):
                item = ranking_b[b_idx]
                interleaved.append(item)
                seen.add(item)
                if item not in teams:
                    teams[item] = 'B'
            turn = 'A'

    return interleaved, teams

def tally_clicks(clicks: list[str], teams: dict) -> Counter:
    """Count which team's items got clicked."""
    tally = Counter()
    for item in clicks:
        if item in teams:
            tally[teams[item]] += 1
    return tally

Knowledge check

Why is Thompson Sampling preferable to a fixed-split A/B test when you need to minimize regret during an experiment?

Summary

Statistical power and sample size must be calculated before running any experiment — underpowered tests produce unreliable results
A/B tests require user-level randomization, guardrail metrics, and holdback groups for post-launch measurement
Multi-armed bandits (Thompson Sampling) adaptively allocate traffic to minimize regret during exploration
Interleaving provides 10–100× more sensitive evaluation for ranking models by showing merged results from both models to the same user
Always check for novelty effects, network effects, and Simpson's paradox before concluding an experiment

Next: cost optimization and infrastructure — GPU spot instances, auto-scaling, and cost-per-prediction.

← MLOps & AI Engineering