Random Forest — Module 5.2

How This Lesson Fits the Module

A single decision tree is unstable—small data changes rewrite the whole structure. Random forest trains many decorrelated trees on bootstrap samples and random feature subsets, then averages their votes. It is often the first ensemble students should reach for on tabular data.

Random forests trade a bit of interpretability for robust accuracy with sensible defaults and minimal tuning.

Learning Objectives

By the end of this lesson, students should be able to:

Explain bagging (bootstrap aggregating) and feature randomization at each split.
Configure RandomForestClassifier and RandomForestRegressor.
Use n_estimators, max_depth, and max_features to balance bias and variance.
Rank features with feature_importances_ and sanity-check with permutation importance.
Compare forest performance to a single tree and logistic regression baseline.

Bagging Many Trees

For each of n_estimators trees, sklearn draws a bootstrap sample (sampling rows with replacement), grows a tree with only a random subset of features at each split, and aggregates predictions—majority vote for classification, mean for regression. Trees that overfit different noise patterns cancel out when averaged.

Parameter	Role	Practical note
`n_estimators`	Number of trees in the forest	More trees → smoother predictions; diminishing returns after ~200–500
`max_features`	Features tried per split	`"sqrt"` (classify) or `"log2"` decorrelates trees
`max_depth`	Depth cap per tree	`None` is common; shallow trees need more estimators
`n_jobs=-1`	Parallel training	Use all CPU cores for faster fits

sklearn Example on Digits

Handwritten digit recognition (8×8 pixel images) is a multiclass problem where random forests shine without neural networks.

from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.ensemble import RandomForestClassifier
from sklearn.inspection import permutation_importance

X, y = load_digits(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=42
)

rf = RandomForestClassifier(
    n_estimators=200,
    max_features="sqrt",
    random_state=42,
    n_jobs=-1,
)
rf.fit(X_train, y_train)
print("Test accuracy:", rf.score(X_test, y_test))

scores = cross_val_score(rf, X_train, y_train, cv=5)
print("CV mean:", round(scores.mean(), 3), "+/-", round(scores.std(), 3))

perm = permutation_importance(rf, X_test, y_test, n_repeats=10, random_state=42)
top_idx = perm.importances_mean.argsort()[-5:][::-1]
print("Top pixel indices:", top_idx.tolist())

Out-of-Bag (OOB) Error

Each bootstrap sample uses roughly 63% of unique rows; the rest are “out of bag.” With oob_score=True, sklearn evaluates each tree on its OOB rows and reports an internal validation estimate—handy when holdout data is scarce.

Engineering Habit — Permutation Over Default Importance

Mean decrease in impurity can favor high-cardinality features. Confirm rankings with permutation_importance on a held-out set before removing columns in production.

Critical Mistake — Treating Forests as Interpretable

Feature importances summarize global behavior but hide interaction effects and local rules. For regulatory explanations, pair forests with SHAP values or train a surrogate shallow tree on forest predictions.

Knowledge Check

Short Answer: What is bagging? Answer: Training multiple models on bootstrap samples and aggregating their predictions.
True/False: Random forests require feature scaling. Answer: False.
Multiple Choice: More n_estimators generally: (a) increases overfitting, (b) reduces variance, (c) has no effect. Answer: (b).
Short Answer: What does max_features="sqrt" do? Answer: Considers only √p features at each split, decorrelating trees.
Short Answer: When might boosting beat random forest? Answer: On large tabular datasets where sequential error correction yields higher accuracy (see gradient boosting lessons).
True/False: OOB score estimates generalization using rows left out of each bootstrap. Answer: True.
Multiple Choice: Random forest regression prediction is usually: (a) majority vote, (b) average of tree outputs, (c) max leaf. Answer: (b).
Short Answer: Why are RF trees decorrelated? Answer: Bootstrap samples plus random feature subsets at splits.
True/False: Increasing n_estimators typically increases variance. Answer: False—it usually reduces variance.
Multiple Choice: feature_importances_ in RF: (a) prove causality, (b) rank split usefulness, (c) replace SHAP always. Answer: (b).

Key Takeaways

Random forest bagging reduces variance by averaging many decorrelated trees.
Bootstrap samples and random feature subsets are the key randomizations.
OOB score and cross-validation guide tuning without wasting test data.
Validate feature importance with permutation tests, not impurity alone.
Next: Naive Bayes for fast probabilistic text and count models.

Trainer’s Guide

Hands-on idea: Compare a single DecisionTreeClassifier, logistic regression, and random forest on digits with a shared split. Students tabulate accuracy, fit time, and top-5 features.

Discussion prompt: Why does increasing n_estimators rarely hurt test performance the way deepening one tree does?

Recap: Random forests bag many decorrelated trees to cut variance without needing scaled features. Continue with Naive Bayes.