Often you have lots of data and no labels. Clustering groups similar items; PCA finds the few directions that matter most.
4 minute read
In short
Unsupervised learning finds structure without labels: clusters and important directions.
PCA keeps the directions with the most variance; eigenvalues say how much each keeps.
Standardise before PCA or k-means, and inspect clusters by looking at real examples.
In plain words
Unsupervised learning looks for patterns in data without any right answers to learn from. Two tools cover most uses. Clustering, like k-means, groups similar points together: customer segments, topics in documents. Dimensionality reduction, like PCA, compresses many features into a few while keeping as much of the variation as possible: for visualising data, removing noise or speeding up other models.
Worked example: PCA on five points
Five points: (2, 1), (3, 2), (4, 3), (5, 4), (6, 6). They lie almost on a line. PCA finds that line.
1
Centre the data
Subtract the mean of each feature.
mean = (4, 3.2)
2
Covariance matrix
How the two features vary together.
[[2.5, 3.0],
[3.0, 3.7]]
3
Eigenvectors and eigenvalues
The eigenvectors are the principal directions; each eigenvalue is how much variance lies along that direction.
direction 1 = (0.634, 0.773), variance 6.159
direction 2 = (−0.773, 0.634), variance 0.041
4
How much does one direction keep?
6.159 ÷ (6.159 + 0.041) = 99.3%
5
Compress to one number per point
Project each centred point onto direction 1. Two numbers become one, losing only 0.7% of the variation.
−2.97, −1.56, −0.15, 1.25, 3.43
Real uses look the same at a larger scale: 300 features squeezed into 20 that keep 95% of the variance, or 768-dimensional embeddings projected to 2 for a scatter plot.
Clustering in one paragraph
k-means, covered in full in the ML from scratch module, places k centres, assigns each point to its nearest centre, moves each centre to the average of its points, and repeats. You choose k, often by plotting the total distance for several values and looking for the 'elbow' where adding clusters stops helping much. Scale your features first, or the largest-valued feature decides the clusters.
Your turn
PCA's first direction keeps 99.3% of the variance in a dataset. What does that tell you?
What to remember
PCA is linear and sensitive to scale, so standardise first. It finds directions of variance, which aren't necessarily the directions that matter for your task. Clusters are a starting point for understanding data, not ground truth: always look at examples from each cluster to see if they make sense.
Common mistakes
Running PCA or k-means on unscaled features, so the largest-valued feature dominates.
Assuming the top principal components are the most useful for prediction; they're just the most varied.
Treating clusters as real categories without checking examples from each.
Check yourself
Answer in your own words, out loud or on paper, before you open the model answer.
Explain PCA and when you'd use it.
PCA finds the directions in the data along which it varies the most. You centre the data, compute the covariance matrix, and take its eigenvectors, sorted by eigenvalue. The first eigenvector is the direction of greatest variance, the second is the next, at right angles to the first, and so on. Each eigenvalue tells you how much variance that direction holds, so you can keep just enough components to retain, say, 95%. I'd use it to visualise high-dimensional data in two dimensions, to reduce noise, or to compress features before another model. I'd standardise first, and remember it's linear and ignores labels.