Power in one sentence
Power is the probability your test detects a real effect.
Where is the Type II error rate.
What controls power
- Sample size (n): larger n → higher power
- Effect size: bigger effect → easier to detect
- Noise/variance: more noise → lower power
- Significance level (alpha): smaller alpha → lower power
Why it matters in analytics
You might conclude “no effect” when:
- sample size was too small
- metric variance was high
Quick simulation idea
You can estimate power by simulation:
- simulate A/B outcomes under a known effect
- run the test repeatedly
- measure how often p < 0.05
# This is a conceptual sketch.
# For real experiments, use statsmodels or a dedicated power calculator.
import numpy as np
rng = np.random.default_rng(0)
def run_once(n=2000, p1=0.03, p2=0.031):
A = rng.binomial(1, p1, size=n)
B = rng.binomial(1, p2, size=n)
# compute a simple z-test here (see A/B page)
return A.mean(), B.mean()
# Repeat many times and track rejection rate.Practical guidance
- If you can’t increase n, reduce noise (better logging, stronger metric).
- Pre-register hypotheses and expected effect sizes.
What feeds into power
flowchart TD A["Sample size (n)"] --> E["Power
(1 - beta)"] B["Effect size"] --> E C["Noise / variance"] --> E D["Significance level (alpha)"] --> E
Visualize it
The null distribution (no real effect) and the true-effect distribution always overlap a little. That shaded overlap is where your test fails to detect the effect — a Type II error. Bigger effect sizes push the curves apart and shrink the overlap:
Check your understanding
What is statistical power?
Power = 1 - beta, where beta is the Type II error rate.
Show answer
A — The probability the test detects a real effect, 1 minus beta — Power = 1 - beta, where beta is the Type II error rate.
Which change lowers power?
Smaller alpha makes rejecting H0 harder. Larger n, bigger effects and less noise all raise power.
Show answer
A — A smaller significance level alpha — Smaller alpha makes rejecting H0 harder. Larger n, bigger effects and less noise all raise power.
Why might you wrongly conclude there is no effect?
An underpowered test often fails to detect a real effect.
Show answer
A — The sample was too small or the metric variance was high — An underpowered test often fails to detect a real effect.
How can you estimate power by simulation?
The rejection rate under a known true effect estimates power.
Show answer
A — Simulate A/B outcomes with a known effect, run the test repeatedly and count how often p is under 0.05 — The rejection rate under a known true effect estimates power.
🧪 Try It Yourself
Exercise 1 – Larger n narrows the noise band
# Task: show that standard error (a proxy for overlap width) shrinks with n
import numpy as np
std = 20
for n in [30, 120, 480]:
se = std / np.sqrt(n)
print(f"n={n}: SE={round(se, 2)}")
# Expected output:
# n=30: SE=3.65
# n=120: SE=1.83
# n=480: SE=0.91Exercise 2 – Bigger effect size is easier to detect
# Task: compare a small effect vs. a large effect using a standardized measure (Cohen's d)
mean_diff_small = 1
mean_diff_large = 8
pooled_std = 10
d_small = mean_diff_small / pooled_std
d_large = mean_diff_large / pooled_std
print("small effect size (d):", d_small)
print("large effect size (d):", d_large)
# Expected output:
# small effect size (d): 0.1
# large effect size (d): 0.8Exercise 3 – Estimate power by simulation
# Task: run many simulated A/B tests with a known real effect and count how often p < 0.05
import numpy as np
from scipy import stats
rng = np.random.default_rng(0)
significant = 0
trials = 500
for _ in range(trials):
A = rng.normal(loc=50, scale=10, size=40)
B = rng.normal(loc=54, scale=10, size=40) # true effect: +4
_, p = stats.ttest_ind(A, B)
if p < 0.05:
significant += 1
power_estimate = significant / trials
print("estimated power:", power_estimate)
# Expected output (will vary slightly by run):
# estimated power: <somewhere around 0.4-0.6 for this sample size and effect>Next
Continue to Statistics Mini Project (Analyze a Marketing Campaign) to apply power, confidence intervals, and hypothesis tests to a real end-to-end analysis.