Dimensionality reduction

Principal component analysis finds new axes — combinations of your existing columns — ordered by how much variance each captures. PC1 points along the direction of greatest spread, PC2 along the greatest remaining spread at right angles to it, and so on. Keep the first few and you have fewer columns carrying most of the information.

from sklearn.decomposition import PCA

pca = PCA(n_components=5).fit(X_scaled)
pca.explained_variance_ratio_
pca.components_

What a component is

Each is a weighted mix of the originals. Ours loads PC1 like this:

prep_min       0.698
items          0.683
hour           0.196
distance_km    0.046

Almost entirely prep_min plus items, which makes sense — bigger orders take longer to prepare, so those two columns measure the same underlying thing twice.

That's what PCA is finding: redundancy. When two columns move together it collapses them onto one axis, and the next axis gets whatever is left.

Scaling is mandatory

PCA maximises variance, and variance is measured in the column's own units squared. Measure distance in metres instead of kilometres and its variance goes up a millionfold — so it becomes PC1 for no reason at all.

Unscaled PCA is PCA of your unit choices. Always scale first.

Explained variance

0.2081  0.1455  0.1303  0.1241  0.1122  0.0511  0.0433  0.0371 …

cumulative:  0.208  0.354  0.484  0.608  0.720  0.771  0.815  0.852

Remarkably flat. Two components hold only 35%, and it takes ten of eighteen to reach 90%.

On a dataset with genuinely redundant columns the first two or three bars tower over the rest and the cumulative line shoots up. Ours doesn't, and that flatness is itself the finding: our columns are mostly independent, so there's very little redundancy to squeeze out.

Which is a useful thing to learn about your data in one line, even though it means PCA has nothing to offer here.

And it costs accuracy

logistic on the raw 18 features       0.8786
logistic on 5 principal components    0.8091

We threw away 0.07 AUC. The reason is worth understanding properly:

PCA keeps the directions with the most variance, and variance is not the same thing as usefulness for predicting y. A low-variance direction can carry all the signal. PCA is unsupervised — it has never seen the target — which is exactly why it can discard the part that mattered.

What it is genuinely for

• Visualising high-dimensional data — plot PC1 against PC2 to see structure you couldn't otherwise.
• Hundreds of correlated columns — sensor arrays, spectra, pixels. There the redundancy is real and large.
• Removing collinearity before a linear model; components are orthogonal by construction.
• Speeding up a model that's too slow on wide data.

And not: as a default preprocessing step, or as a way of picking which features to keep. For that use feature selection, which at least knows about y.

The cost you always pay

Interpretability. Every component is a weighted mix of all your columns, so no coefficient afterwards maps onto anything a person recognises. "Heavy rain adds 3.9 minutes" is actionable; "PC1 increased by 2" is not.

On our data that trade bought a 0.07 AUC loss. PCA is a genuinely useful tool with a much narrower range of application than its popularity suggests — and knowing when not to reach for something is most of what separates a practitioner from a tutorial.

Relatives worth knowing

• TruncatedSVD — the same idea for sparse matrices, where centring would destroy the sparsity.
• t-SNE and UMAP — much better for visualisation, and unusable as a preprocessing step because they can't transform new data consistently.
• SelectKBest and friends — supervised feature selection, which is usually what people actually want when they reach for PCA.