Decision Trees — Module 5.2

How This Lesson Fits the Module

Logistic regression draws a single linear boundary. Decision trees partition feature space with nested if–then rules—easy to explain to stakeholders and the building block for random forests and boosting.

Trees handle mixed feature types and interactions without explicit engineering, but they overfit aggressively unless you prune or ensemble them.

Learning Objectives

By the end of this lesson, students should be able to:

Describe how trees choose splits using impurity (Gini, entropy) or MSE.
Train DecisionTreeClassifier and DecisionTreeRegressor in sklearn.
Control complexity with max_depth, min_samples_leaf, and ccp_alpha.
Visualize and interpret trees with plot_tree and feature importances.
Explain why a single deep tree is a poor production choice without regularization.

How a Decision Tree Learns

Starting at the root, the algorithm tests every feature and threshold, picking the split that most reduces impurity (classification) or variance (regression). It repeats recursively until stopping rules trigger. Each leaf predicts the majority class or mean target of training samples that reach it.

Hyperparameter	Effect	Typical starting point
`max_depth`	Limits how deep the tree grows	3–8 for exploration
`min_samples_leaf`	Minimum samples per leaf; smooths predictions	5–20
`max_features`	Features considered per split	`"sqrt"` (classification)
`ccp_alpha`	Cost-complexity pruning after full growth	Tune via CV

sklearn Classification Example

The iris dataset separates three flower species with petal and sepal measurements. A shallow tree is often sufficient and far easier to explain than a black-box model.

from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier, plot_tree
from sklearn.metrics import accuracy_score
import matplotlib.pyplot as plt

X, y = load_iris(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, stratify=y, random_state=42
)

tree = DecisionTreeClassifier(
    max_depth=3,
    min_samples_leaf=5,
    random_state=42,
)
tree.fit(X_train, y_train)
print("Accuracy:", accuracy_score(y_test, tree.predict(X_test)))

plt.figure(figsize=(12, 6))
plot_tree(tree, feature_names=load_iris().feature_names, filled=True)
plt.savefig("iris_tree.png", dpi=120, bbox_inches="tight")

Regression Trees

DecisionTreeRegressor minimizes MSE at each split. Leaves predict the average target of contained samples. Without depth limits, a tree can memorize every training point (zero training error, terrible test error).

Engineering Habit — Rules for Humans

Export the top three levels of a shallow tree as business rules (“if tenure < 90 days and support_tickets > 5, flag churn risk”). Stakeholders often trust explicit rules more than a probability score alone.

Strengths

No feature scaling required
Handles nonlinear boundaries and interactions
Missing values can be routed with surrogate splits
Interpretable when depth is constrained

Weaknesses

High variance—small data changes alter structure
Axis-aligned splits miss diagonal boundaries
Extrapolates poorly beyond training range
Deep trees overfit without pruning or ensembles

Critical Mistake — Unlimited Depth

A default DecisionTreeClassifier() grows until leaves are pure. Training accuracy hits 100% while test performance collapses. Always set max_depth or min_samples_leaf and validate with cross-validation.

Knowledge Check

Short Answer: What does a leaf node predict for classification? Answer: The majority class of training samples in that leaf.
True/False: Decision trees require standardized numeric features. Answer: False—splits are rank-based on raw values.
Multiple Choice: Reduces overfitting most directly: (a) deeper tree, (b) min_samples_leaf=20, (c) more features. Answer: (b).
Short Answer: What is Gini impurity measuring? Answer: How often a random sample from the node would be misclassified if labeled by the node majority.
Short Answer: Why ensemble many trees instead of one deep tree? Answer: Averaging reduces variance and improves generalization.
True/False: Gini / information gain is evaluated at each candidate split. Answer: True.
Multiple Choice: max_depth=None with tiny leaves tends to: (a) underfit, (b) overfit, (c) regularize. Answer: (b).
Short Answer: What is CART? Answer: Classification and Regression Trees—the usual binary-split tree algorithm.
True/False: Some tree implementations can split categoricals without one-hot encoding. Answer: True.
Multiple Choice: plot_tree is most useful for: (a) 100-level models only, (b) interpreting shallow trees, (c) SVMs. Answer: (b).

Key Takeaways

Trees split features recursively to minimize impurity or MSE.
Depth and leaf-size limits are essential to control overfitting.
Shallow trees are interpretable; deep trees need ensembles or pruning.
No scaling needed, but axis-aligned splits have geometric limits.
Next: Random Forest to bag many trees and cut variance.

Trainer’s Guide

Hands-on idea: Students train trees at depths 2, 10, and unlimited on iris, plot train vs. test accuracy, and explain the gap.

Discussion prompt: Which iris rule from plot_tree would you show a botanist? Which would you hide because it overfits noise?

Recap: Decision trees recursively split features to reduce impurity, but deep trees overfit unless constrained. Continue with Random Forest.