Gradient Boosting

How This Lesson Fits the Module

Random forest builds trees in parallel and averages them. Gradient boosting builds trees sequentially, each one correcting the residual errors of the ensemble so far. sklearn’s GradientBoostingClassifier introduces the boosting pattern that XGBoost later optimizes for speed and scale.

Understand boosting here first—then the capstone library will feel like an upgrade, not a new paradigm.

Learning Objectives

By the end of this lesson, students should be able to:

Contrast bagging (random forest) with boosting (sequential error correction).
Train GradientBoostingClassifier and GradientBoostingRegressor.
Tune n_estimators, learning_rate, and max_depth jointly.
Use staged_predict and validation curves to spot early stopping points.
Explain why shallow trees (max_depth=3) are standard in boosting.

Boosting in Plain Language

Start with a weak prediction (often the mean or log-odds). Fit a small tree to the negative gradient of the loss—the direction that most reduces error. Add that tree to the ensemble, scaled by learning_rate. Repeat for n_estimators rounds. Later trees specialize in hard examples previous trees missed.

Hyperparameter	Role	Interaction
`n_estimators`	Number of boosting stages	More trees + low LR → smoother fit
`learning_rate`	Shrink each tree’s contribution	Typical range 0.01–0.2
`max_depth`	Complexity per tree	3–5 is common; deep trees overfit fast
`subsample`	Row fraction per stage (stochastic GB)	< 1.0 adds regularization

sklearn Gradient Boosting Example

Breast cancer classification with a deliberately conservative learning rate and shallow trees. Monitor staged performance to decide when extra trees stop helping.

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.metrics import roc_auc_score

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=42
)

gb = GradientBoostingClassifier(
    n_estimators=300,
    learning_rate=0.05,
    max_depth=3,
    subsample=0.8,
    random_state=42,
)
gb.fit(X_train, y_train)

# Track AUC as trees are added
staged_auc = [
    roc_auc_score(y_test, proba)
    for proba in gb.staged_predict_proba(X_test)[:, 1]
]
print("Best staged AUC:", round(max(staged_auc), 3))
print("Final AUC:", round(roc_auc_score(y_test, gb.predict_proba(X_test)[:, 1]), 3))

Learning Rate vs. Number of Trees

Lower learning_rate demands more n_estimators but often generalizes better. This tradeoff is not independent—tune both together with cross-validation or early stopping on a validation fold.

Random Forest

Parallel tree training
Averages independent models
Robust defaults, less tuning
Plateaus on some tabular tasks

Sequential, corrective trees
Often higher peak accuracy
Sensitive to hyperparameters
Risk of overfit without early stopping

Engineering Habit — Early Stopping

sklearn GB lacks built-in early stopping like XGBoost’s early_stopping_rounds. Use staged_predict on a validation set or switch to HistGradientBoostingClassifier for faster training and native early stopping.

Critical Mistake — Deep Trees in Boosting

Setting max_depth=15 in gradient boosting is not like random forest. Each stage can memorize residuals; the ensemble overfits quickly. Start with depth 3 and increase only with evidence from validation curves.

Knowledge Check

Short Answer: What does each new tree in boosting fit? Answer: The negative gradient of the loss (residual errors) from the current ensemble.
True/False: Boosting trees are trained in parallel like random forest. Answer: False—boosting is sequential.
Multiple Choice: Lower learning rate usually requires: (a) fewer trees, (b) more trees, (c) deeper trees. Answer: (b).
Short Answer: What does subsample=0.8 do? Answer: Each stage uses 80% of rows, adding stochastic regularization.
Short Answer: Why preview staged_predict? Answer: Find the tree count where validation metric peaks before overfitting.
True/False: max_depth in boosting is usually kept small (e.g. 2–4). Answer: True—shallow trees plus many stages.
Multiple Choice: Gradient boosting vs random forest: (a) RF is sequential, (b) boosting fits residuals sequentially, (c) identical. Answer: (b).
Short Answer: What is shrinkage in boosting? Answer: Scaling each new tree by a learning rate so updates are conservative.
True/False: Too many boosting rounds with a high learning rate overfits. Answer: True.
Multiple Choice: loss="log_loss" is for: (a) regression only, (b) classification, (c) clustering. Answer: (b).

Key Takeaways

Gradient boosting adds shallow trees that correct previous errors.
Learning rate and tree count must be tuned together.
Shallow trees and subsampling control overfitting.
Monitor staged metrics to approximate early stopping.
Next: XGBoost capstone—production-grade boosted trees.

Trainer’s Guide

Hands-on idea: Plot validation AUC vs. boosting iteration for learning rates 0.2, 0.05, and 0.01. Students mark where each curve peaks.

Discussion prompt: Why might boosting beat random forest on the same features yet lose in deployment (latency, maintenance, tuning cost)?

Recap: Gradient boosting adds shallow trees that correct residual errors; tune learning rate with tree count. Continue with XGBoost.