PCA — Module 5.3 | Codextroop

How This Lesson Fits the Module

Clustering lectures assume meaningful distances in feature space. PCA (Principal Component Analysis) projects high-dimensional data onto orthogonal directions of maximum variance—compressing noise, speeding up models, and making structure visible.

Unlike t-SNE and UMAP, PCA is linear and invertible in principle: it belongs in preprocessing pipelines, not just plots.

Learning Objectives

By the end of this lesson, students should be able to:

Explain PCA as variance-maximizing orthogonal projection.
Read scree plots and cumulative explained-variance ratios.
Apply PCA inside sklearn pipelines before clustering or regression.
Distinguish PCA for modeling vs t-SNE/UMAP for visualization.
Scale features before PCA on mixed-magnitude columns.
Choose component count by variance threshold or downstream CV performance.

What PCA Does

PCA finds new axes (principal components) that are linear combinations of original features. PC1 captures the largest spread; PC2 is orthogonal to PC1 and captures the next largest, and so on. Projecting onto the first k components yields a lower-dimensional representation with minimal reconstruction error (in the L2 sense).

Output	Interpretation
`explained_variance_ratio_`	Fraction of total variance each component explains
`components_`	Weight vectors showing feature contributions
Transformed coordinates	Scores used as inputs to downstream models

sklearn Implementation

import numpy as np
from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
from sklearn.cluster import KMeans

# Always scale before PCA when features differ in units
prep = Pipeline([
    ("scale", StandardScaler()),
    ("pca", PCA(n_components=0.95, random_state=42)),  # keep 95% variance
])
X_reduced = prep.fit_transform(X)

print("Components kept:", prep.named_steps["pca"].n_components_)
print("Variance ratios:", prep.named_steps["pca"].explained_variance_ratio_[:5])

# PCA then cluster — common pattern
cluster_pipe = Pipeline([
    ("prep", prep),
    ("kmeans", KMeans(n_clusters=4, n_init=20, random_state=42)),
])
labels = cluster_pipe.fit_predict(X)

Engineering Habit — Scree Plot

Plot cumulative explained variance vs component index. The elbow suggests how many dimensions preserve most signal. Cross-validate downstream task performance rather than chasing 100% variance.

Choosing How Many Components

Variance Threshold

PCA(n_components=0.95)
Simple, data-driven compression
Component count adapts to dataset

Fixed k + CV

Try k ∈ {10, 20, 50} in a pipeline
Pick k by validation metric
Best when PCA feeds a supervised model

When to Use PCA

Use PCA when…	Prefer t-SNE / UMAP when…
You need a preprocessing step inside a pipeline	The goal is a 2D plot for humans
Linear correlations dominate	Nonlinear manifold structure matters
You want interpretable variance ratios	Exact global distances are less important
Large n and p (randomized PCA scales)	Small/medium n for exploration only

Critical Mistake — Clustering on PCA Fit on Train+Test

Fit StandardScaler and PCA on training data only, then transform validation and test sets. Refitting PCA on the full dataset leaks distribution information into unsupervised steps used for modeling decisions.

PCA in the Clustering Workflow

Before K-Means or DBSCAN on hundreds of correlated features, PCA can denoise and make distances meaningful. After clustering, use t-SNE or UMAP on a sample to visualize segments—color points by cluster label, but keep production labels from the original feature space or PCA pipeline.

Knowledge Check

Short Answer: What does the first principal component maximize? Answer: Variance along a single linear direction.
True/False: PCA is ideal for nonlinear Swiss-roll manifolds. Answer: False—nonlinear methods like UMAP visualize those better.
Multiple Choice: n_components=0.90 means: (a) 90 components, (b) 90% variance retained, (c) 90% rows dropped. Answer: (b).
Short Answer: Why scale before PCA? Answer: So high-magnitude columns do not dominate components.
Short Answer: Can you deploy PCA transforms on new rows? Answer: Yes—fit on train, transform new data with the same components.
True/False: Principal components are orthogonal by construction. Answer: True.
Multiple Choice: explained_variance_ratio_ sums to: (a) 1 if all components kept, (b) k always, (c) accuracy. Answer: (a).
Short Answer: What is a scree plot? Answer: Variance explained per component, used to choose how many to keep.
True/False: PCA loadings can help interpret which original features drive a component. Answer: True.
Multiple Choice: Clustering on t-SNE coordinates is: (a) recommended for production, (b) risky / usually wrong, (c) required. Answer: (b).

Key Takeaways

PCA is linear compression that maximizes retained variance.
Scale features; choose components by variance or downstream CV.
Use PCA in pipelines for clustering and supervised models.
t-SNE and UMAP are for visualization—PCA is for reproducible transforms.
Next: t-SNE—nonlinear 2D embeddings for exploration.

Trainer’s Guide

Hands-on idea: On MNIST digits (flattened pixels), PCA to 50 components then K-Means. Compare silhouette to clustering on raw pixels and discuss speed gains.

Discussion prompt: A teammate clusters on t-SNE coordinates. What is wrong with that workflow?

Recap: PCA is linear dimensionality reduction that keeps directions of maximum variance—scale, then transform new rows. Continue with t-SNE.