Applied Stats for Engineering Decisions

Statistics has a reputation problem in engineering circles. It feels like a pile of theoretical definitions until the moment you ship a model, run an A/B test, and have to explain to a product manager why a 2% lift is not statistically meaningful. That moment happens to every ML engineer. This article gives you the statistical vocabulary and tools you need before it happens to you.

Hypothesis Testing for A/B Tests

An A/B test is a controlled experiment: you randomly assign users to a control group (A) and a treatment group (B), then measure whether the treatment produced a statistically meaningful difference in some metric.

pythonCopy
import numpy as np
from scipy import stats

# Simulated conversion rates
np.random.seed(42)
control = np.random.binomial(1, 0.10, size=5000)    # 10% baseline conversion
treatment = np.random.binomial(1, 0.115, size=5000)  # 11.5% treatment conversion

# Two-sample proportion z-test
from statsmodels.stats.proportion import proportions_ztest

count = np.array([treatment.sum(), control.sum()])
nobs = np.array([len(treatment), len(control)])

stat, p_value = proportions_ztest(count, nobs)
print(f"Z-statistic: {stat:.4f}")
print(f"P-value: {p_value:.4f}")
print(f"Significant at 0.05: {p_value < 0.05}")

The p-value answers: "If there were no real difference, how likely is it to see a difference this large by chance?" A p-value below 0.05 does not mean the treatment works - it means the result is unlikely under the null hypothesis of no effect.

The Confidence Interval Is More Useful Than the P-Value

A confidence interval tells you the range of plausible effect sizes, not just whether significance was achieved.

pythonCopy
from statsmodels.stats.proportion import proportion_confint

control_rate = control.mean()
treatment_rate = treatment.mean()
lift = treatment_rate - control_rate

# 95% CI for each group
ci_control = proportion_confint(control.sum(), len(control), alpha=0.05)
ci_treatment = proportion_confint(treatment.sum(), len(treatment), alpha=0.05)

print(f"Control: {control_rate:.4f} (95% CI: {ci_control[0]:.4f}–{ci_control[1]:.4f})")
print(f"Treatment: {treatment_rate:.4f} (95% CI: {ci_treatment[0]:.4f}–{ci_treatment[1]:.4f})")
print(f"Lift: {lift:.4f}")

A 95% confidence interval that does not include zero is significant. But more importantly, the width of the interval tells you how precise your estimate is. A significant result with a wide CI might not be actionable.

Effect Size: Is This Practically Significant?

Statistical significance and practical significance are different things. You can have a tiny effect that is highly significant because your sample is enormous.

Cohen's d for continuous outcomes:

pythonCopy
def cohens_d(group1, group2):
    pooled_std = np.sqrt((group1.std()**2 + group2.std()**2) / 2)
    return (group1.mean() - group2.mean()) / pooled_std

# For model error comparison
errors_model_a = np.random.normal(0.35, 0.08, 1000)
errors_model_b = np.random.normal(0.33, 0.08, 1000)

d = cohens_d(errors_model_a, errors_model_b)
print(f"Cohen's d: {d:.4f}")
# d < 0.2: negligible, 0.2–0.5: small, 0.5–0.8: medium, > 0.8: large

For an A/B test on conversion rates, Cohen's h is the appropriate effect size measure. The point is: report effect size alongside p-values.

Statistical Power and Sample Size Planning

Running an underpowered experiment is worse than not running one - you get a null result and do not know whether the treatment failed or your test was too small to detect the effect.

pythonCopy
from statsmodels.stats.power import NormalIndPower

analysis = NormalIndPower()

# How many samples do I need to detect a 1.5pp lift from 10% baseline?
baseline = 0.10
treatment_rate = 0.115
effect_size = (treatment_rate - baseline) / np.sqrt(baseline * (1 - baseline))

n = analysis.solve_power(
    effect_size=effect_size,
    alpha=0.05,       # Type I error rate
    power=0.80,       # 80% chance of detecting a real effect
    ratio=1.0         # equal group sizes
)
print(f"Required sample size per group: {int(np.ceil(n))}")

Plan your sample size before you run the experiment. Starting an A/B test and checking it daily until it is significant (p-hacking) inflates your false positive rate.

Applying Stats to Model Comparison Decisions

The same logic applies when comparing model versions offline. If Model B gets 0.812 F1 vs Model A's 0.808, is that meaningful?

pythonCopy
from scipy.stats import wilcoxon

# Cross-validation scores across folds
model_a_scores = np.array([0.80, 0.81, 0.82, 0.80, 0.81])
model_b_scores = np.array([0.82, 0.81, 0.83, 0.81, 0.82])

# Wilcoxon signed-rank test for paired comparison
stat, p = wilcoxon(model_b_scores, model_a_scores)
print(f"Wilcoxon p-value: {p:.4f}")

Use paired tests when the same folds are evaluated on both models. This accounts for fold-level variance and gives you a more sensitive comparison.

Common Mistakes

Peeking at results: Checking significance daily and stopping when p < 0.05 is not a valid experimental protocol. It increases your false positive rate substantially. Use sequential testing if you need early stopping.

Confusing p-value with effect size: A tiny p-value with a negligible Cohen's d means you have a massive sample, not a meaningful difference.

Ignoring statistical power: An underpowered test that finds no significant difference proves nothing.

Multiple comparisons without correction: Testing 20 metrics and reporting the one that is significant is not science. Apply Bonferroni or Benjamini-Hochberg correction.

Where to Go Next

• evaluation-metrics-error-analysis-systems - connect statistical thinking to model evaluation choices
• training-pipelines-experiment-strategy - structure experiments so statistical analysis is reproducible
• milestone-gate-1-foundations-validation - validate your ML foundations readiness

What to Practice Next

• Run a two-sample t-test and compute a 95% confidence interval on a real dataset (e.g., A/B experiment results from a public dataset on Kaggle) using scipy.stats - report both the p-value and the confidence interval, not just one.
• Find a blog post or engineering post-mortem that makes a data-driven claim without reporting uncertainty; write a one-paragraph critique identifying what is missing.
• Calculate the minimum detectable effect for an experiment with a fixed sample size using an online power calculator, then explain in plain English what that number means for decision-making.