What you’ll learn

• what the ROC curve plots, and how it differs from the precision-recall curve
• how AUC turns an entire curve into one comparable number
• why a great-looking ROC-AUC can still hide a mediocre classifier
• when to reach for ROC/AUC versus the precision-recall curve instead

What ROC is

ROC stands for Receiver Operating Characteristic — a curve that shows the trade-off between:

• TPR (True Positive Rate) = recall = TP / (TP + FN)TP / (TP + FN)
• FPR (False Positive Rate) = FP / (FP + TN)FP / (FP + TN) — the fraction of actual negatives incorrectly flagged as positive; equal to 1 - specificity1 - specificity

As you slide the decision threshold from high to low, both TPR and FPR climb from 0 toward 1. Plotting one against the other, across every possible threshold, produces the ROC curve.

  flowchart LR
  S["Model outputs scores / probabilities"] --> T["Sweep every threshold"]
  T --> R["Compute TPR and FPR at each one"]
  R --> C["Plot (FPR, TPR) points"]
  C --> Curve["ROC curve"]

How to interpret the curve

• A curve that hugs the top-left corner is better — high TPR (catches positives) at a low FPR (few false alarms).
• The diagonal line from (0,0) to (1,1) is what a purely random guesser would produce — any useful classifier should stay above it.

AUC

AUC is the Area Under the ROC Curve — a single number that summarizes the whole curve:

• AUC = 1.0AUC = 1.0 → perfect ranking: every positive scores higher than every negative
• AUC = 0.5AUC = 0.5 → no better than random guessing
• AUC < 0.5AUC < 0.5 → worse than random (the model is ranking backwards!)

AUC measures how well the model ranks positives above negatives, across every possible threshold at once — not whether any single threshold gives good predictions.

from sklearn.metrics import roc_curve, roc_auc_score
import numpy as np
 
y_true = np.array([0, 0, 0, 1, 1, 1, 1, 1, 1, 1])
y_scores = np.array([-3, -1, 0.5, 0.2, 1, 2, 3, 4, 5, 6])
 
fpr, tpr, thresholds = roc_curve(y_true, y_scores)
print(fpr)
# [0.         0.         0.         0.33333333 0.33333333 1.        ]
print(tpr)
# [0.         0.14285714 0.85714286 0.85714286 1.         1.        ]
 
auc = roc_auc_score(y_true, y_scores)
print("ROC-AUC:", round(auc, 4))
# ROC-AUC: 0.9524

from sklearn.metrics import roc_curve, roc_auc_score
import numpy as np
 
y_true = np.array([0, 0, 0, 1, 1, 1, 1, 1, 1, 1])
y_scores = np.array([-3, -1, 0.5, 0.2, 1, 2, 3, 4, 5, 6])
 
fpr, tpr, thresholds = roc_curve(y_true, y_scores)
print(fpr)
# [0.         0.         0.         0.33333333 0.33333333 1.        ]
print(tpr)
# [0.         0.14285714 0.85714286 0.85714286 1.         1.        ]
 
auc = roc_auc_score(y_true, y_scores)
print("ROC-AUC:", round(auc, 4))
# ROC-AUC: 0.9524

Compare that to a model whose scores barely relate to the true labels at all:

from sklearn.metrics import roc_auc_score
import numpy as np
 
np.random.seed(42)
y_true = np.random.randint(0, 2, 20)
y_scores = np.random.rand(20)   # scores unrelated to y_true
 
print("ROC-AUC:", round(roc_auc_score(y_true, y_scores), 4))
# ROC-AUC: 0.5934  -> barely better than a coin flip

from sklearn.metrics import roc_auc_score
import numpy as np
 
np.random.seed(42)
y_true = np.random.randint(0, 2, 20)
y_scores = np.random.rand(20)   # scores unrelated to y_true
 
print("ROC-AUC:", round(roc_auc_score(y_true, y_scores), 4))
# ROC-AUC: 0.5934  -> barely better than a coin flip

When ROC/AUC is useful

• comparing classifiers independent of any one chosen threshold
• when you care mainly about ranking quality — “does the model put true positives near the top?”
• Géron’s book uses this to compare an SGDClassifierSGDClassifier against a RandomForestClassifierRandomForestClassifier on MNIST’s 5-detector task: the Random Forest’s ROC curve hugs the top-left corner much more tightly, and its AUC (0.998) beats the SGD classifier’s (0.961) by a wide margin — a clear, single-number way to declare a winner.

When ROC can mislead

With highly imbalanced data, ROC-AUC can look deceptively great, because the FPR denominator (FP + TNFP + TN) is dominated by a huge number of true negatives. Precision-recall curves are more informative in that situation, since they never involve TNTN at all.

Mini-checkpoint

If a model has high accuracy but ROC-AUC close to 0.5, what’s likely happening?

• The model is probably predicting the majority class almost every time and isn’t actually learning to rank the classes at all — high accuracy alone hid that.

🧪 Try It Yourself

Exercise 1 – Compute FPR and TPR

Exercise 2 – Compute ROC-AUC

Exercise 3 – Spot a Near-Random Classifier

Next

Continue to Phase 5 - Ensemble Learning to see how combining many models — bagging, boosting, and stacking — pushes accuracy further than any single classifier from this phase.