A/B Testing for ML Features: Measuring What Actually Matters

Your new model has an AUC of 0.88. Your old model has AUC of 0.83. That is a 6% relative improvement. Will it improve your product?

You do not know yet. AUC measures prediction accuracy on a holdout set. It does not measure whether better predictions lead to better user outcomes. A/B testing bridges that gap.

Why Offline Metrics Are Not Enough

A churn model with better AUC predicts which customers will churn more accurately. But the product impact depends on:

• What action is taken when a customer is predicted to churn?
• How much does that intervention help?
• Are there false positive costs (e.g., offering discounts to customers who were not going to churn anyway)?

A more accurate model with a poorly designed intervention could perform worse in production. An A/B test measures the full chain.

Experiment Design

Define the business metric first. Never run an A/B test without a primary metric and a secondary guardrail metric.

Example for a churn model:

pythonCopy
experiment_design = {
    "hypothesis": "Routing customers flagged by the new churn model to the retention team "
                  "will reduce 90-day churn rate compared to the current model's routing.",
    "primary_metric": "90_day_churn_rate",  # lower is better
    "secondary_metrics": [
        "discount_offer_rate",          # guardrail: don't increase discount spend too much
        "false_positive_rate",          # guardrail: don't waste retention team time
        "days_to_churn_prediction"      # leading indicator
    ],
    "allocation": {"control": 0.50, "treatment": 0.50},
    "min_sample_size": 5000,  # calculated from power analysis
    "max_duration_days": 28,
    "minimum_detectable_effect": 0.02  # 2 percentage point reduction in churn rate
}

Power analysis - calculate required sample size:

pythonCopy
from statsmodels.stats.power import TTestIndPower
import numpy as np

def calculate_sample_size(
    baseline_rate: float,
    minimum_detectable_effect: float,
    alpha: float = 0.05,
    power: float = 0.80
) -> int:
    """
    How many samples per arm do we need?
    baseline_rate: current churn rate (e.g., 0.12 for 12%)
    minimum_detectable_effect: smallest change worth detecting (e.g., 0.02 for 2pp)
    """
    # Convert to effect size (Cohen's d)
    control_rate = baseline_rate
    treatment_rate = baseline_rate - minimum_detectable_effect
    pooled_std = np.sqrt((control_rate * (1 - control_rate) +
                          treatment_rate * (1 - treatment_rate)) / 2)
    effect_size = minimum_detectable_effect / pooled_std

    analysis = TTestIndPower()
    n = analysis.solve_power(
        effect_size=effect_size,
        alpha=alpha,
        power=power
    )
    return int(np.ceil(n))

n_per_arm = calculate_sample_size(
    baseline_rate=0.12,      # 12% baseline churn
    minimum_detectable_effect=0.02  # want to detect 2pp reduction
)
print(f"Need {n_per_arm:,} users per arm ({n_per_arm * 2:,} total)")

Underpowered experiments are worse than no experiment - they produce false negatives (you conclude no effect when there is one) and drain team resources.

Assignment and Logging

pythonCopy
import hashlib

def get_experiment_assignment(user_id: int, experiment_name: str) -> str:
    """
    Deterministic assignment: same user always gets same variant.
    Prevents switching a user between variants mid-experiment.
    """
    hash_input = f"{experiment_name}:{user_id}".encode()
    hash_value = int(hashlib.md5(hash_input).hexdigest(), 16)
    bucket = hash_value % 100  # 0-99
    return "treatment" if bucket < 50 else "control"

def log_experiment_event(user_id: int, variant: str, event: str, value=None):
    """Log to your analytics system."""
    import time
    event_record = {
        "user_id": user_id,
        "experiment": "churn-model-v2",
        "variant": variant,
        "event": event,
        "value": value,
        "timestamp": time.time()
    }
    # Send to Kafka, Segment, BigQuery, etc.
    analytics_client.track(event_record)

Critically: log at decision time, not just outcome time. When you decide to send a customer to the retention team, log that event with the variant. Outcomes may arrive weeks later.

Statistical Analysis

pythonCopy
import pandas as pd
import numpy as np
from scipy import stats

def analyze_experiment(experiment_log_path: str, outcome_col: str):
    df = pd.read_parquet(experiment_log_path)

    control = df[df['variant'] == 'control'][outcome_col]
    treatment = df[df['variant'] == 'treatment'][outcome_col]

    # Means
    control_mean = control.mean()
    treatment_mean = treatment.mean()
    relative_lift = (treatment_mean - control_mean) / control_mean

    # Statistical test
    stat, p_value = stats.ttest_ind(treatment, control)
    is_significant = p_value < 0.05

    # Confidence interval
    se = np.sqrt(treatment.var() / len(treatment) + control.var() / len(control))
    ci_lower = (treatment_mean - control_mean) - 1.96 * se
    ci_upper = (treatment_mean - control_mean) + 1.96 * se

    print(f"Control {outcome_col}: {control_mean:.4f} (n={len(control):,})")
    print(f"Treatment {outcome_col}: {treatment_mean:.4f} (n={len(treatment):,})")
    print(f"Relative lift: {relative_lift:+.2%}")
    print(f"Absolute delta: {treatment_mean - control_mean:+.4f} "
          f"(95% CI: [{ci_lower:+.4f}, {ci_upper:+.4f}])")
    print(f"P-value: {p_value:.4f} ({'significant' if is_significant else 'not significant'})")

    return {
        "is_significant": is_significant,
        "relative_lift": relative_lift,
        "p_value": p_value,
        "ci": (ci_lower, ci_upper)
    }

results = analyze_experiment("data/churn_experiment_v2.parquet", "churned_90d")

Common Mistakes

Peeking: Checking results every day and stopping when p < 0.05. This inflates false positive rates. Pre-register your sample size and do not stop early unless you use a sequential testing framework (e.g., O'Brien-Fleming bounds).

Multiple comparisons: Testing 10 secondary metrics at α=0.05 means a 40% chance of at least one false positive. Apply Bonferroni correction or pre-specify that secondary metrics are exploratory only.

pythonCopy
# Bonferroni correction for multiple metrics
n_tests = len(secondary_metrics)
adjusted_alpha = 0.05 / n_tests
print(f"Adjusted significance threshold: {adjusted_alpha:.4f}")

Network effects: If users interact with each other (social features, shared resources), treatment can "contaminate" control. Cluster randomize (randomize by team, region, etc.) rather than by individual.

Novelty effect: Users behave differently toward new things just because they are new, not because they are better. Run experiments for at least 1–2 full business cycles.

Interpreting and Acting on Results

Result	What it means	Action
Significant positive lift	New model is better	Roll out fully
Significant negative lift	New model is worse	Rollback, diagnose
Not significant, small sample	Need more data	Extend experiment
Not significant, adequate sample	True null - models are equivalent	Keep champion (less operational risk)
Significant guardrail metric negative	Lifted primary but at a cost	Redesign intervention

The goal of an A/B test is not to prove your model is better - it is to make the right deployment decision with evidence. A null result that prevents a bad deploy is a successful experiment.

What to Practice Next

• Set up a mock A/B experiment in a notebook: define a null hypothesis, calculate the required sample size using a power calculator (e.g., statsmodels.stats.power), and simulate what happens when you stop early.
• Take a published ML paper that reports a metric improvement and check whether they reported p-values, confidence intervals, and sample sizes - identify what is missing.
• Sketch an experiment design for one feature in a product you use, including how you would split traffic, what metric is the primary outcome, and what guardrail metrics you would monitor.