Hyperparameter — Module 5.1

How This Lesson Fits the Module

Models have knobs you do not learn from data directly—hyperparameters like tree depth, regularization strength, and learning rate. Cross-validation scores each setting so you pick configurations without overfitting a single validation slice.

Engineers automate search with GridSearchCV and RandomizedSearchCV, log winners, and ship the same params in production configs.

Learning Objectives

By the end of this lesson, students should be able to:

Distinguish learned parameters from hyperparameters.
Define a search space and scoring metric aligned to business goals.
Run GridSearchCV and RandomizedSearchCV on pipelines.
Read best_params_, best_score_, and cv_results_.
Avoid tuning on the final test set.
Balance search cost vs marginal metric gains in production workflows.

Parameters vs Hyperparameters

Parameters are learned during fit (coefficients, split thresholds). Hyperparameters are set before training and control model capacity or training dynamics.

Model	Learned parameters	Common hyperparameters
Logistic regression	Coefficients, intercept	`C`, `penalty`, `class_weight`
Random forest	Tree structures	`n_estimators`, `max_depth`, `min_samples_leaf`
Gradient boosting	Ensemble of trees	`learning_rate`, `max_depth`, `subsample`

GridSearchCV on a Pipeline

Prefix hyperparameter names with step name and double underscore: clf__max_depth. Search runs CV for every combination in the grid.

from sklearn.model_selection import GridSearchCV
from sklearn.ensemble import RandomForestClassifier
from sklearn.pipeline import Pipeline

pipe = Pipeline([
    ("prep", preprocess),   # ColumnTransformer from Feature lesson
    ("clf", RandomForestClassifier(random_state=42)),
])

param_grid = {
    "clf__n_estimators": [100, 300],
    "clf__max_depth": [None, 8, 16],
    "clf__min_samples_leaf": [1, 5, 20],
}

search = GridSearchCV(
    pipe, param_grid,
    cv=5, scoring="roc_auc", n_jobs=-1, refit=True
)
search.fit(X_train, y_train)

print(search.best_params_)
print(search.best_score_)   # mean CV score — not test

RandomizedSearch for Large Spaces

When grids explode, sample random combinations with RandomizedSearchCV and n_iter. Often finds strong configs faster than exhaustive grid.

from scipy.stats import randint
from sklearn.model_selection import RandomizedSearchCV

param_dist = {
    "clf__n_estimators": randint(100, 500),
    "clf__max_depth": [None, 6, 10, 20],
    "clf__min_samples_leaf": randint(1, 30),
}

rand_search = RandomizedSearchCV(
    pipe, param_dist, n_iter=25, cv=5,
    scoring="roc_auc", random_state=42, n_jobs=-1
)
rand_search.fit(X_train, y_train)

Critical Mistake — Tuning on Test Data

Running grid search including X_test in fit or picking params because test improved invalidates your report. Tune on train/CV; evaluate once on test.

Engineering Habit — Search Budget

Cap wall-clock time per experiment. Log best CV score, params, and data snapshot. A 0.001 AUC gain after 200 trials may not justify serving complexity.

Knowledge Check

Short Answer: What is a hyperparameter? Answer: A setting fixed before training, not learned from data.
True/False: best_score_ is performance on the test set. Answer: False—mean CV score on training folds.
Multiple Choice: Pipeline param for tree depth: (a) max_depth, (b) clf__max_depth, (c) prep__max_depth. Answer: (b).
Short Answer: When prefer RandomizedSearch over GridSearch? Answer: Large search spaces where exhaustive grid is too slow.
Short Answer: What does refit=True do after search? Answer: Refits best estimator on full training data passed to fit.
True/False: Higher max_depth always improves generalization. Answer: False—can overfit.
Multiple Choice: Scoring should match: (a) easiest metric, (b) business cost, (c) train accuracy only. Answer: (b).
Short Answer: Why n_jobs=-1? Answer: Parallelize CV fits across CPU cores.
Short Answer: What is in cv_results_? Answer: Per-trial metrics, params, and timing from the search.
Multiple Choice: After tuning, final unbiased check uses: (a) validation fold only, (b) locked test set, (c) training set. Answer: (b).

Key Takeaways

Hyperparameters control capacity; tune them with CV-backed search.
Use pipeline-prefixed param names in GridSearchCV.
Randomized search scales better than huge grids.
Next: Pipeline—glue preprocessing and models for safe training.

Trainer’s Guide

Hands-on idea: Students run a small grid on random forest, plot max_depth vs mean CV AUC from cv_results_, and pick params with the elbow not the peak if variance is high.

Discussion prompt: Product wants maximum recall; how does that change your scoring argument in search?

Recap: Hyperparameters are settings chosen before training, tuned with CV rather than test peeking. Continue with Pipeline.