What is A/B testing?
A/B testing compares two variants:
- A = control
- B = treatment
You measure a metric (conversion, revenue, retention) and test whether B improves it.
Choose the right metric
- Primary metric (one)
- Guardrail metrics (to prevent harm)
Examples:
- Primary: conversion rate
- Guardrail: refund rate, latency
Example: compare conversion rates
import numpy as np
# A: 120 conversions out of 4000
x1, n1 = 120, 4000
# B: 150 conversions out of 4100
x2, n2 = 150, 4100
p1 = x1 / n1
p2 = x2 / n2
p_pool = (x1 + x2) / (n1 + n2)
se = np.sqrt(p_pool * (1 - p_pool) * (1/n1 + 1/n2))
z = (p2 - p1) / se
# two-sided p-value using normal approximation
from math import erf, sqrt
def norm_cdf(z):
return 0.5 * (1 + erf(z / sqrt(2)))
p_value = 2 * (1 - norm_cdf(abs(z)))
print("p1:", p1)
print("p2:", p2)
print("z:", z)
print("p-value:", p_value)Practical guidance
- Run long enough to cover weekly seasonality.
- Don’t peek too often (increases false positives).
- Report effect size + CI.
The A/B testing lifecycle
flowchart LR A["Pick primary +
guardrail metrics"] --> B["Split traffic
A vs B"] B --> C["Run for planned
duration"] C --> D["Compute z-test
or t-test"] D --> E["Compare p to alpha
+ report CI"] E --> F["Ship, iterate,
or discard"]
Check your understanding
In the example, variant A has 120 conversions out of 4000 and B has 150 out of 4100. What are p1 and p2 approximately?
p1 = 120 / 4000 = 0.03 and p2 = 150 / 4100, about 0.0366.
Show answer
A — 0.030 and 0.037 — p1 = 120 / 4000 = 0.03 and p2 = 150 / 4100, about 0.0366.
Why does the two-proportion z-test use a pooled proportion p_pool?
The standard error is computed assuming H0, that A and B have the same rate, so the conversions are pooled.
Show answer
A — Under the null hypothesis both groups share one true conversion rate — The standard error is computed assuming H0, that A and B have the same rate, so the conversions are pooled.
What is the role of a guardrail metric such as refund rate or latency?
The lesson uses one primary metric and guardrail metrics to prevent harm.
Show answer
B — It checks that the change is not causing harm elsewhere — The lesson uses one primary metric and guardrail metrics to prevent harm.
Why should you not peek at the test results too often?
Repeated significance checks inflate false positives, so decide sample size and duration before starting.
Show answer
B — It increases false positives — Repeated significance checks inflate false positives, so decide sample size and duration before starting.
🧪 Try It Yourself
Exercise 1 – Compute conversion rates
Tab indents. Press Escape, then Tab, to leave the editor. Control plus Enter runs the code; Control plus Shift plus Enter submits it.
Press Run to see what your code prints.Exercise 2 – Two-proportion z-test
Tab indents. Press Escape, then Tab, to leave the editor. Control plus Enter runs the code; Control plus Shift plus Enter submits it.
Press Run to see what your code prints.Exercise 3 – Guardrail metric check
Tab indents. Press Escape, then Tab, to leave the editor. Control plus Enter runs the code; Control plus Shift plus Enter submits it.
Press Run to see what your code prints.Next
Continue to Statistical Power (intuition) to understand how sample size and effect size determine whether your A/B test can even detect a real change.