Hypothesis Testing (p-value, alpha, errors)

The basic workflow

1. Define null hypothesis (H0) and alternative (H1)
2. Choose a significance level (\alpha) (commonly 0.05)
3. Compute a test statistic and p-value
4. Decide whether to reject H0

P-value intuition

A p-value is:

• Probability of seeing results at least as extreme as your observation if H0 were true.

Low p-value means:

• Data is unlikely under H0 → evidence against H0

Errors

• Type I error: reject H0 when H0 is true (false positive). Probability ≈ (\alpha).
• Type II error: fail to reject H0 when H0 is false (false negative). Probability = (\beta).
• Power: (1 - \beta)

Example: one-sample t-test

Is the average delivery time different from 30 minutes?

import numpy as np
from scipy import stats
 
x = np.array([28, 32, 31, 29, 26, 30, 33, 27])
 
t_stat, p = stats.ttest_1samp(x, popmean=30)
print("t:", t_stat)
print("p:", p)

Good practice

• Report effect size + CI, not just p-value.
• Don’t equate “not significant” with “no difference”.
• Predefine your hypothesis; avoid testing many variants blindly.

Visualize it

This flowchart traces the decision path from stating your hypotheses to accepting or rejecting H0 based on the p-value.

  flowchart TD
  A["State H0 and H1"] --> B["Collect data"]
  B --> C["Compute p-value"]
  C --> D["Compare to alpha (0.05)"]
  D -->|"p < alpha"| E["Reject H0 (significant)"]
  D -->|"p >= alpha"| F["Fail to reject H0 (not significant)"]

Check your understanding

🧪 Try It Yourself

Exercise 1 – One-sample t-test

Press Run to see what your code prints.

Exercise 2 – Deciding based on alpha

Press Run to see what your code prints.

Exercise 3 – Type I vs. Type II error

Press Run to see what your code prints.

Next

Continue to t-test (independent and paired) to apply this decision framework to comparing two groups' means.