What you’ll learn

• why Logistic Regression is a classifier, despite the name
• the sigmoid (logistic) function, and why its output can be read as a probability
• the log loss cost function, and why it’s convex (Gradient Descent always finds the minimum)
• decision boundaries in feature space
• Softmax Regression for more than two classes at once
• why the threshold matters, and how to tune it

Why it’s called “regression”

Logistic Regression is a classification algorithm. Like Linear Regression, it computes a weighted sum of the input features plus a bias term — but instead of outputting that sum directly, it squashes it through a sigmoid function so the result always lands between 0 and 1, which can be read as a probability.

The sigmoid function

  flowchart LR
  Z["Linear score z = w * x + b"] --> S["Sigmoid: sigma(z)"]
  S --> P["p = estimated probability"]
  P --> Y["Class via threshold (default 0.5)"]

The logistic function is:

σ(t) = 1 / (1 + exp(-t))σ(t) = 1 / (1 + exp(-t))

• large positive tt → σ(t)σ(t) close to 1
• large negative tt → σ(t)σ(t) close to 0
• t = 0t = 0 → σ(t) = 0.5σ(t) = 0.5, exactly on the boundary

Once the model estimates a probability p = σ(x·w + b)p = σ(x·w + b), it predicts:

• ŷ = 1ŷ = 1 if p ≥ 0.5p ≥ 0.5
• ŷ = 0ŷ = 0 if p < 0.5p < 0.5

Since σ(t) ≥ 0.5σ(t) ≥ 0.5 exactly when t ≥ 0t ≥ 0, a Logistic Regression model predicts class 1 whenever the raw weighted score is positive, and class 0 when it’s negative. That raw score tt is called the logit, or log-odds.

Training: log loss

There’s no closed-form solution for the parameters that minimize the cost — but the cost function (called log loss) is convex, so Gradient Descent is guaranteed to find the global minimum:

cost = -log(p)cost = -log(p) if y = 1y = 1, or -log(1 - p)-log(1 - p) if y = 0y = 0

This makes sense: -log(p)-log(p) explodes as p → 0p → 0, so the model is heavily penalized for being confidently wrong on a positive instance (and symmetrically for negatives).

Decision boundaries: the iris example

Géron’s book illustrates this with the iris dataset — detecting Iris virginica from petal width alone:

import numpy as np
from sklearn import datasets
from sklearn.linear_model import LogisticRegression
 
iris = datasets.load_iris()
X = iris["data"][:, 3:]              # petal width only
y = (iris["target"] == 2).astype(int)  # 1 if Iris virginica, else 0
 
log_reg = LogisticRegression()
log_reg.fit(X, y)
 
print(log_reg.predict([[1.7], [1.5]]))
# [1 0]  -> the decision boundary sits around 1.6 cm
 
print(log_reg.predict_proba([[2.0]]))
# [[0.187 0.813]]  -> 81.3% confident it's Iris virginica

import numpy as np
from sklearn import datasets
from sklearn.linear_model import LogisticRegression
 
iris = datasets.load_iris()
X = iris["data"][:, 3:]              # petal width only
y = (iris["target"] == 2).astype(int)  # 1 if Iris virginica, else 0
 
log_reg = LogisticRegression()
log_reg.fit(X, y)
 
print(log_reg.predict([[1.7], [1.5]]))
# [1 0]  -> the decision boundary sits around 1.6 cm
 
print(log_reg.predict_proba([[2.0]]))
# [[0.187 0.813]]  -> 81.3% confident it's Iris virginica

Above about 2 cm the model is highly confident it’s Iris virginica; below 1 cm it’s highly confident it’s not. In between, it’s genuinely unsure — the decision boundary sits wherever the estimated probability crosses 0.5.

Just like other linear models, Logistic Regression can be regularized. Scikit-Learn adds an ℓ2 penalty by default, controlled by CC — the inverse of regularization strength (a higher CC means less regularization, unlike alphaalpha elsewhere).

Multiclass: Softmax Regression

Logistic Regression generalizes directly to multiple classes without needing to combine several binary classifiers. This is called Softmax Regression (or Multinomial Logistic Regression).

For an instance xx, the model computes one score per class, then normalizes all the scores with the softmax function so they sum to 1:

p_k = exp(s_k(x)) / Σ_j exp(s_j(x))p_k = exp(s_k(x)) / Σ_j exp(s_j(x))

The predicted class is simply the one with the highest probability (equivalently, the highest raw score) — argmax_k p_kargmax_k p_k.

from sklearn import datasets
from sklearn.linear_model import LogisticRegression
 
iris = datasets.load_iris()
X = iris["data"][:, (2, 3)]  # petal length, petal width
y = iris["target"]           # all 3 classes: 0, 1, 2
 
softmax_reg = LogisticRegression(C=10, max_iter=1000)
softmax_reg.fit(X, y)
 
print(softmax_reg.predict([[5, 2]]))
# [2]  -> predicted class: Iris virginica
 
print(softmax_reg.predict_proba([[5, 2]]).round(3))
# [[0.    0.057 0.943]]  -> 94.3% confident

from sklearn import datasets
from sklearn.linear_model import LogisticRegression
 
iris = datasets.load_iris()
X = iris["data"][:, (2, 3)]  # petal length, petal width
y = iris["target"]           # all 3 classes: 0, 1, 2
 
softmax_reg = LogisticRegression(C=10, max_iter=1000)
softmax_reg.fit(X, y)
 
print(softmax_reg.predict([[5, 2]]))
# [2]  -> predicted class: Iris virginica
 
print(softmax_reg.predict_proba([[5, 2]]).round(3))
# [[0.    0.057 0.943]]  -> 94.3% confident

Softmax Regression predicts only one class per instance (it’s multiclass, not multioutput) — use it only for mutually exclusive classes. Its cost function is called cross entropy, which reduces to ordinary log loss when there are exactly 2 classes.

Threshold tuning matters

The default threshold is 0.5, but for imbalanced problems (fraud, disease screening) you’ll often move it deliberately:

• lower the threshold to catch more positives (increase recall)
• raise the threshold to reduce false alarms (increase precision)

The The ROC Curve and AUC and Precision, Recall, and F1-Score pages later in this phase cover exactly how to choose that threshold.

Mini-checkpoint

If missing fraud is expensive, do you optimize for precision or recall?

(Usually recall — you’d rather flag a few extra false alarms than miss real fraud.)

🧪 Try It Yourself

Exercise 1 – Compute the Sigmoid

Exercise 2 – Fit a LogisticRegression Classifier

Exercise 3 – Read Class Probabilities

Next

Continue to K-Nearest Neighbors (KNN) — a classifier with no training step at all, that decides purely by looking at the closest labeled points.