SVM — Module 5.2 | Codextroop

How This Lesson Fits the Module

KNN draws boundaries by local neighborhoods. Support Vector Machines (SVM) find a global boundary that maximizes the margin between classes—and kernels let that boundary bend in high dimensions without storing every training point at inference time (for the support vectors).

SVMs were dominant on small and medium tabular and text problems before deep learning; they remain strong when data is clean and well-scaled.

Learning Objectives

By the end of this lesson, students should be able to:

Explain the maximum-margin principle and the role of support vectors.
Train SVC and SVR with linear, RBF, and polynomial kernels.
Tune C (regularization) and gamma (RBF width) via grid search.
Scale features and use class_weight for imbalanced problems.
Choose between linear SVM and kernel SVM based on dataset size and nonlinearity.

Margin, Slack, and Kernels

Linear SVM seeks the hyperplane with largest gap between classes. Soft-margin SVM (parameter C) allows some misclassifications to handle noise. The RBF kernel K(x, x') = exp(−γ||x − x'||²) maps points into an implicit high-dimensional space where nonlinear separation becomes linear—controlled by gamma.

Hyperparameter	Effect when increased	Symptom if wrong
`C`	Harder margin, fewer slack violations	Low C → underfit; high C → overfit
`gamma` (RBF)	Tighter influence per support vector	High gamma → wiggly boundary, overfit
`kernel="linear"`	Fast on high-dimensional sparse text	Underfits strongly nonlinear data
`kernel="rbf"`	Flexible nonlinear boundaries	Slow on very large n

RBF SVM with Grid Search

Digits classification again—this time with scaling and a small grid over C and gamma. Always search on training folds, then report once on the test set.

from sklearn.datasets import load_digits
from sklearn.model_selection import GridSearchCV, train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
from sklearn.svm import SVC

X, y = load_digits(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=42
)

svm_pipe = Pipeline([
    ("scale", StandardScaler()),
    ("svc", SVC(kernel="rbf")),
])

param_grid = {
    "svc__C": [0.1, 1, 10],
    "svc__gamma": ["scale", 0.01, 0.001],
}
search = GridSearchCV(svm_pipe, param_grid, cv=3, n_jobs=-1)
search.fit(X_train, y_train)
print("Best params:", search.best_params_)
print("Test accuracy:", round(search.score(X_test, y_test), 3))

Linear SVM for Text

High-dimensional sparse TF–IDF vectors often work best with LinearSVC or SVC(kernel="linear")—faster than RBF and less prone to overfitting when p >> n. Pair with TfidfVectorizer as in the Naive Bayes lesson.

Engineering Habit — Support Vector Count

After fitting, inspect n_support_. If nearly every training point is a support vector, the model may be overfitting or C is too high. A compact set of support vectors suggests a stable margin.

Critical Mistake — RBF on Unscaled Data

RBF distance depends on squared Euclidean norm. Features with large ranges dominate gamma’s effect. Pipeline StandardScaler (or normalize sparse text appropriately) before any kernel SVM.

Knowledge Check

Short Answer: What is a support vector? Answer: A training point that lies on or defines the margin boundary; predictions depend on these points.
True/False: Larger C always improves test accuracy. Answer: False—too large C overfits noise.
Multiple Choice: Best kernel for 50k sparse text features: (a) RBF, (b) linear, (c) polynomial degree 5. Answer: (b).
Short Answer: What does high gamma do in RBF SVM? Answer: Each point influences a smaller region—more complex, local boundaries.
Short Answer: Why is SVM slow on 2 million rows? Answer: Training complexity scales poorly; kernel matrix and QP solve grow with sample size.
True/False: Soft-margin SVM allows some points inside or across the margin. Answer: True.
Multiple Choice: The kernel trick lets SVM: (a) skip scaling, (b) fit nonlinear boundaries without explicit feature maps, (c) reduce n. Answer: (b).
Short Answer: When prefer LinearSVC over kernel SVC? Answer: Large sparse / high-dimensional data where a linear margin is faster and enough.
True/False: Feature scaling is optional for RBF SVM. Answer: False—distance-based kernels need scaled features.
Multiple Choice: probability=True in SVC: (a) is free, (b) uses extra calibration (Platt) and extra cost, (c) changes the kernel. Answer: (b).

Key Takeaways

SVM maximizes the margin between classes; support vectors define the boundary.
C trades fit quality for margin width; gamma controls RBF locality.
Scale features; use linear kernels for high-dimensional sparse text.
Grid-search hyperparameters with cross-validation, not the test set.
Next: Gradient Boosting for sequential tree ensembles.

Trainer’s Guide

Hands-on idea: Students plot decision regions of an RBF SVM on 2-D iris (two features only) while sweeping gamma to visualize overfitting.

Discussion prompt: When would you pick LinearSVC over RandomForestClassifier for a fraud model with 500 numeric features?

Recap: SVMs maximize the class margin; tune C and kernel, and scale features. Continue with Gradient Boosting.