XGBoost — Module 5.2

How This Lesson Fits the Module

This capstone ties together the supervised learning arc: from linear models through trees, bagging, and gradient boosting. XGBoost (eXtreme Gradient Boosting) implements boosted trees with regularization, efficient histogram splits, parallel training, and built-in early stopping—the library many Kaggle tabular winners and production pipelines reach for after sklearn baselines.

You have seen the ideas in sklearn; XGBoost is where speed, scale, and tuning ergonomics meet on real datasets.

Learning Objectives

By the end of this lesson, students should be able to:

Explain how XGBoost extends gradient boosting with regularized objectives and efficient tree construction.
Train models with the xgboost Python API (XGBClassifier, XGBRegressor).
Use early_stopping_rounds, eval_set, and eval_metric to prevent overfitting.
Tune max_depth, learning_rate, n_estimators, subsample, and colsample_bytree.
Compare an end-to-end tabular workflow against sklearn random forest and gradient boosting baselines.

Why XGBoost After sklearn Boosting?

XGBoost adds L1/L2 penalties on leaf weights, handles missing values natively, supports column subsampling per tree, and uses histogram-based split finding for speed on large data. The sklearn-compatible API (XGBClassifier) drops into existing pipelines while exposing GPU training and categorical support in recent versions.

Feature	sklearn `GradientBoosting`	XGBoost
Early stopping	Manual via `staged_predict`	Built-in with `eval_set`
Missing values	Requires imputation upstream	Learned default split direction
Speed at scale	Slower on large n	Histogram algorithm, threading, optional GPU
Regularization	Tree depth, subsample	+`reg_alpha`, `reg_lambda`, `min_child_weight`

Capstone: Tabular Classification with Early Stopping

Train on breast cancer features with a validation fold watched during boosting. Stop when validation AUC stops improving—the production pattern for every boosted model.

from xgboost import XGBClassifier
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.metrics import roc_auc_score, classification_report

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=42
)
X_fit, X_val, y_fit, y_val = train_test_split(
    X_train, y_train, test_size=0.2, stratify=y_train, random_state=42
)

xgb = XGBClassifier(
    n_estimators=500,
    learning_rate=0.05,
    max_depth=4,
    subsample=0.8,
    colsample_bytree=0.8,
    reg_lambda=1.0,
    eval_metric="auc",
    random_state=42,
    n_jobs=-1,
)
xgb.fit(
    X_fit, y_fit,
    eval_set=[(X_val, y_val)],
    early_stopping_rounds=30,
    verbose=False,
)

proba = xgb.predict_proba(X_test)[:, 1]
print("Test AUC:", round(roc_auc_score(y_test, proba), 3))
print("Best iteration:", xgb.best_iteration)
print(classification_report(y_test, xgb.predict(X_test)))

Module-Wide Model Selection Checklist

Before declaring XGBoost the winner, run the same split and metrics across the module’s toolkit:

Start Simple

LogisticRegression / Ridge baseline
Fast, interpretable coefficients
Strong when relationships are linear

Scale Up

RandomForestClassifier — robust defaults
GradientBoostingClassifier — learn boosting mechanics
XGBClassifier — peak tabular accuracy + early stopping

Capstone Project — End-to-End Tabular Challenge

Pick a Kaggle-style dataset (credit default, telco churn, or house prices). Deliver: (1) EDA and feature pipeline from Volume 04, (2) at least three models from this module, (3) a metric table on a held-out test set, (4) saved best_model.json or pickle with documented hyperparameters.

Key Hyperparameters

Parameter	Purpose
`max_depth`	Tree depth per round; 4–8 typical for tabular
`min_child_weight`	Minimum sum of instance weight in a child; raises bar for splits
`colsample_bytree`	Feature fraction per tree; decorrelates learners
`scale_pos_weight`	Handles class imbalance (ratio of negatives to positives)

Critical Mistake — Tuning on the Test Set

Early stopping watches eval_set—that validation fold is not your final test set. Reserve X_test untouched until all model and hyperparameter choices are frozen, exactly as taught in train-test split.

Knowledge Check

Short Answer: What does early_stopping_rounds=30 do? Answer: Stops training if the eval metric does not improve for 30 consecutive boosting rounds.
True/False: XGBoost requires one-hot encoding for missing values. Answer: False—it learns optimal directions for missing entries.
Multiple Choice: First model to try on a new tabular set: (a) XGBoost with 500 trees, (b) logistic regression baseline, (c) k=1 KNN. Answer: (b).
Short Answer: What is best_iteration after early stopping? Answer: The boosting round that achieved the best validation metric.
Short Answer: When might you still pick random forest over XGBoost? Answer: When tuning budget is tiny, interpretability matters more, or marginal accuracy gains do not justify complexity.
True/False: colsample_bytree randomly samples features per tree. Answer: True.
Multiple Choice: eta / learning_rate in XGBoost is: (a) tree depth, (b) step size for each boosting update, (c) number of classes. Answer: (b).
Short Answer: Why use scale_pos_weight on imbalanced labels? Answer: It upweights the minority class in the loss.
True/False: You should early-stop on the final test set. Answer: False—use a validation eval_set.
Multiple Choice: After XGBoost, Module 5.3 begins with: (a) Ridge, (b) unsupervised learning, (c) RNNs. Answer: (b).

Key Takeaways

XGBoost is regularized gradient boosting optimized for speed, scale, and tabular accuracy.
Use validation eval_set and early stopping—never tune on the final test set.
Compare against simpler module models before adopting XGBoost in production.
The capstone workflow: clean features → baselines → tuned boosted model → frozen test evaluation.
Next module: 5.3 Unsupervised Learning — clustering and dimensionality reduction without labels.

Trainer’s Guide

Capstone deliverable: Teams submit a notebook with logistic regression, random forest, and XGBoost on the same split, plus a one-page memo recommending which model to ship and why.

Discussion prompt: Module 5.2 covered nine algorithms—what decision tree would you use to pick among them for a new problem?

Recap: XGBoost is regularized, high-performance gradient boosting for tabular data—always validate before shipping. Continue to Module 5.3 Unsupervised Learning.