Logistic Regression — Module 5.2

How This Lesson Fits the Module

Linear regression predicts numbers; logistic regression predicts class probabilities. It is the workhorse for churn, fraud, and medical screening—fast, interpretable, and a natural bridge from linear models to decision trees and ensembles.

Despite the name, logistic regression is a classification algorithm. The “regression” refers to modeling log-odds as a linear function of features.

Learning Objectives

By the end of this lesson, students should be able to:

Explain sigmoid output as a probability and the log-loss objective.
Train LogisticRegression for binary and multiclass problems.
Choose thresholds using precision, recall, and ROC-AUC.
Handle class imbalance with class_weight or resampling.
Build sklearn pipelines with scaling and one-hot encoding for mixed data.

From Linear Scores to Probabilities

Logistic regression computes a linear score z = w·x + b, then passes it through the sigmoid: P(y=1) = 1 / (1 + e^−z). Training maximizes log-likelihood (equivalently minimizes log-loss). For more than two classes, sklearn uses multinomial or one-vs-rest logistic models.

Setting	Target	Key metric
Binary	0 or 1 (e.g., churned)	ROC-AUC, F1, precision@k
Multiclass	0…K−1 (e.g., plan tier)	Macro-F1, confusion matrix
Imbalanced	Rare positive class	Recall, PR-AUC (not accuracy alone)

sklearn Pipeline Example

The breast cancer dataset is a classic binary classification benchmark. Scale features and tune the decision threshold for your business cost of false negatives vs. false positives.

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import classification_report, roc_auc_score

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=42
)

clf = Pipeline([
    ("scale", StandardScaler()),
    ("model", LogisticRegression(max_iter=1000, class_weight="balanced")),
])
clf.fit(X_train, y_train)
proba = clf.predict_proba(X_test)[:, 1]

print("ROC-AUC:", round(roc_auc_score(y_test, proba), 3))
print(classification_report(y_test, clf.predict(X_test)))

Thresholds and Calibration

predict() uses a default threshold of 0.5. For fraud detection you may lower the threshold to catch more positives; for marketing you may raise it to limit wasted outreach. Inspect the precision–recall tradeoff with PrecisionRecallDisplay.

Engineering Habit — Report Probabilities

Downstream systems (ranking, A/B tests, cost-sensitive routing) need calibrated scores, not just hard labels. Store predict_proba outputs and document the threshold used in production.

Regularization and Multiclass

LogisticRegression supports L1 (penalty="l1"), L2 (default), and elastic net. Use multi_class="multinomial" when classes are mutually exclusive and you want a single softmax over all labels. Categorical features belong in a ColumnTransformer with OneHotEncoder—never as arbitrary integers unless ordinality is real.

Critical Mistake — Accuracy on Imbalanced Data

99% accuracy sounds great until you discover the model never predicts the rare fraud class. Always stratify splits, check per-class recall, and consider class_weight="balanced" or SMOTE inside a pipeline.

Knowledge Check

Short Answer: What does the sigmoid function bound? Answer: Output between 0 and 1, interpretable as P(y=1).
True/False: Logistic regression can natively predict three or more classes in sklearn. Answer: True, via multinomial or OvR strategies.
Multiple Choice: Fraud with 0.1% positives—best primary metric: (a) accuracy, (b) recall, (c) R². Answer: (b).
Short Answer: What does max_iter=1000 guard against? Answer: Convergence warnings when the solver needs more iterations.
Short Answer: Why stratify train_test_split? Answer: Preserves class proportions in train and test sets.
True/False: Logistic regression models P(y=1|x) as a linear function of x with no link. Answer: False—it uses the sigmoid / logit link.
Multiple Choice: Default sklearn LogisticRegression penalty is typically: (a) none, (b) L2, (c) dropout. Answer: (b).
Short Answer: Why scale features before logistic regression? Answer: Regularization and solvers assume comparable feature scales.
True/False: Accuracy is a reliable primary metric on 0.1% fraud. Answer: False—it can look high while missing all fraud.
Multiple Choice: A 0.5 probability threshold: (a) is always optimal, (b) should be tuned for FP vs FN cost, (c) maximizes R². Answer: (b).

Key Takeaways

Logistic regression outputs class probabilities via the sigmoid (or softmax).
Scale features and encode categoricals in a pipeline before fitting.
Choose metrics and thresholds for the business cost of errors.
Handle imbalance with weights, stratification, or resampling—not accuracy alone.
Next: Decision Trees for nonlinear, rule-based splits.

Trainer’s Guide

Hands-on idea: Students plot ROC and precision–recall curves, then pick two thresholds and justify each for “minimize missed cancer” vs. “minimize false alarms.”

Discussion prompt: When would a linear decision boundary fail? What visual pattern in a scatter plot hints you need trees?

Recap: Logistic regression maps linear scores through a sigmoid to class probabilities. Continue with Decision Trees.