Hypothesis Testing (p-value, alpha, errors)

The basic workflow

Define null hypothesis (H0) and alternative (H1)
Choose a significance level (\alpha) (commonly 0.05)
Compute a test statistic and p-value
Decide whether to reject H0

P-value intuition

A p-value is:

Probability of seeing results at least as extreme as your observation if H0 were true.

Low p-value means:

Data is unlikely under H0 → evidence against H0

Errors

Type I error: reject H0 when H0 is true (false positive). Probability ≈ (\alpha).
Type II error: fail to reject H0 when H0 is false (false negative). Probability = (\beta).
Power: (1 - \beta)

Example: one-sample t-test

Is the average delivery time different from 30 minutes?

One-sample t-test

import numpy as np
from scipy import stats
 
x = np.array([28, 32, 31, 29, 26, 30, 33, 27])
 
t_stat, p = stats.ttest_1samp(x, popmean=30)
print("t:", t_stat)
print("p:", p)

Good practice

Report effect size + CI, not just p-value.
Don’t equate “not significant” with “no difference”.
Predefine your hypothesis; avoid testing many variants blindly.

Visualize it

This flowchart traces the decision path from stating your hypotheses to accepting or rejecting H0 based on the p-value.

diagram Hypothesis testing decision flow mermaid

  flowchart TD
  A["State H0 and H1"] --> B["Collect data"]
  B --> C["Compute p-value"]
  C --> D["Compare to alpha (0.05)"]
  D -->|"p < alpha"| E["Reject H0 (significant)"]
  D -->|"p >= alpha"| F["Fail to reject H0 (not significant)"]

From H0/H1 through the p-value comparison to a final decision

🧪 Try It Yourself

Exercise 1 – One-sample t-test

Exercise 2 – Deciding based on alpha

Exercise 3 – Type I vs. Type II error

Continue to t-test (independent and paired) to apply this decision framework to comparing two groups’ means.

pch.coffeeTagline

pch.coffeeCta

pch.feedbackHeading

pch.feedbackSubheading