Skip to content
Python Central Hub

Statistical Power (intuition)

Power in one sentence

Power is the probability your test detects a real effect.

[ \text{Power} = 1 - \beta ]

Where (\beta) is the Type II error rate.

What controls power

  1. Sample size (n): larger n → higher power
  2. Effect size: bigger effect → easier to detect
  3. Noise/variance: more noise → lower power
  4. Significance level (alpha): smaller alpha → lower power

Why it matters in analytics

You might conclude “no effect” when:

  • sample size was too small
  • metric variance was high

Quick simulation idea

You can estimate power by simulation:

  • simulate A/B outcomes under a known effect
  • run the test repeatedly
  • measure how often p < 0.05
Power simulation sketch
# This is a conceptual sketch.
# For real experiments, use statsmodels or a dedicated power calculator.
 
import numpy as np
 
rng = np.random.default_rng(0)
 
def run_once(n=2000, p1=0.03, p2=0.031):
    A = rng.binomial(1, p1, size=n)
    B = rng.binomial(1, p2, size=n)
    # compute a simple z-test here (see A/B page)
    return A.mean(), B.mean()
 
# Repeat many times and track rejection rate.
Power simulation sketch
# This is a conceptual sketch.
# For real experiments, use statsmodels or a dedicated power calculator.
 
import numpy as np
 
rng = np.random.default_rng(0)
 
def run_once(n=2000, p1=0.03, p2=0.031):
    A = rng.binomial(1, p1, size=n)
    B = rng.binomial(1, p2, size=n)
    # compute a simple z-test here (see A/B page)
    return A.mean(), B.mean()
 
# Repeat many times and track rejection rate.

Practical guidance

  • If you can’t increase n, reduce noise (better logging, stronger metric).
  • Pre-register hypotheses and expected effect sizes.

What feeds into power

diagram What determines statistical power mermaid
Sample size, effect size, noise, and alpha all combine to determine whether a test can detect a real effect.

Visualize it

The null distribution (no real effect) and the true-effect distribution always overlap a little. That shaded overlap is where your test fails to detect the effect — a Type II error. Bigger effect sizes push the curves apart and shrink the overlap:

sketch Overlap between null and true-effect distributions p5.js
The shaded overlap between two bell curves shows where a real effect goes undetected (low power).

🧪 Try It Yourself

Exercise 1 – Larger n narrows the noise band

python
# Task: show that standard error (a proxy for overlap width) shrinks with n
import numpy as np
 
std = 20
for n in [30, 120, 480]:
    se = std / np.sqrt(n)
    print(f"n={n}: SE={round(se, 2)}")
 
# Expected output:
# n=30: SE=3.65
# n=120: SE=1.83
# n=480: SE=0.91
python
# Task: show that standard error (a proxy for overlap width) shrinks with n
import numpy as np
 
std = 20
for n in [30, 120, 480]:
    se = std / np.sqrt(n)
    print(f"n={n}: SE={round(se, 2)}")
 
# Expected output:
# n=30: SE=3.65
# n=120: SE=1.83
# n=480: SE=0.91

Exercise 2 – Bigger effect size is easier to detect

python
# Task: compare a small effect vs. a large effect using a standardized measure (Cohen's d)
mean_diff_small = 1
mean_diff_large = 8
pooled_std = 10
 
d_small = mean_diff_small / pooled_std
d_large = mean_diff_large / pooled_std
 
print("small effect size (d):", d_small)
print("large effect size (d):", d_large)
 
# Expected output:
# small effect size (d): 0.1
# large effect size (d): 0.8
python
# Task: compare a small effect vs. a large effect using a standardized measure (Cohen's d)
mean_diff_small = 1
mean_diff_large = 8
pooled_std = 10
 
d_small = mean_diff_small / pooled_std
d_large = mean_diff_large / pooled_std
 
print("small effect size (d):", d_small)
print("large effect size (d):", d_large)
 
# Expected output:
# small effect size (d): 0.1
# large effect size (d): 0.8

Exercise 3 – Estimate power by simulation

python
# Task: run many simulated A/B tests with a known real effect and count how often p < 0.05
import numpy as np
from scipy import stats
 
rng = np.random.default_rng(0)
significant = 0
trials = 500
 
for _ in range(trials):
    A = rng.normal(loc=50, scale=10, size=40)
    B = rng.normal(loc=54, scale=10, size=40)  # true effect: +4
    _, p = stats.ttest_ind(A, B)
    if p < 0.05:
        significant += 1
 
power_estimate = significant / trials
print("estimated power:", power_estimate)
 
# Expected output (will vary slightly by run):
# estimated power: <somewhere around 0.4-0.6 for this sample size and effect>
python
# Task: run many simulated A/B tests with a known real effect and count how often p < 0.05
import numpy as np
from scipy import stats
 
rng = np.random.default_rng(0)
significant = 0
trials = 500
 
for _ in range(trials):
    A = rng.normal(loc=50, scale=10, size=40)
    B = rng.normal(loc=54, scale=10, size=40)  # true effect: +4
    _, p = stats.ttest_ind(A, B)
    if p < 0.05:
        significant += 1
 
power_estimate = significant / trials
print("estimated power:", power_estimate)
 
# Expected output (will vary slightly by run):
# estimated power: <somewhere around 0.4-0.6 for this sample size and effect>

Next

Continue to Statistics Mini Project (Analyze a Marketing Campaign) to apply power, confidence intervals, and hypothesis tests to a real end-to-end analysis.

If this helped you, consider buying me a coffee ☕

Buy me a coffee

Was this page helpful?

Let us know how we did