Ridge and lasso

Ordinary least squares minimises error and nothing else. Give it many features, or two correlated ones, and it will happily use enormous opposing coefficients to chase noise:

distance_km    +412.7
distance_m     −412.4
prep_min         +2.98

Two columns measuring the same thing, and the model picks +412 and −412 because their difference happens to track some randomness in the training set. The predictions look fine; the model is nonsense; and it had no reason not to, because minimising error was the only thing it was asked to do.

Add a price

minimise:   error  +  α × (size of the coefficients)

Now a coefficient has to earn its size. alpha is the exchange rate: zero means ordinary least squares, and large enough means every coefficient is zero.

Scaling is mandatory here, and this is the reason from lesson 10 that actually bites. The penalty acts on raw coefficient magnitude, so the same column measured in metres rather than kilometres gets penalised a thousand times harder. Unscaled regularisation penalises your units.

Ridge shrinks (L2)

from sklearn.linear_model import Ridge
# penalty: α · Σ wᵢ²

Squaring means large weights are punished hardest, so everything gets pulled towards zero — and nothing quite reaches it. You keep every feature, all a bit quieter.

Ridge is especially good with correlated columns: rather than picking one arbitrarily, it shares the effect between them. That's what fixes the ±412 pathology above.

Lasso deletes (L1)

from sklearn.linear_model import Lasso
# penalty: α · Σ |wᵢ|

The absolute value has a corner at zero, and that corner pushes weights exactly to zero rather than merely near it. Watch what happens to our 18 features as alpha rises:

alpha     ridge R²    lasso R²    lasso coefficients left
 0.01       0.5785      0.5790         16 / 18
 0.10       0.5785      0.5732         12 / 18
 1.00       0.5788      0.3724          3 / 18
10.00       0.5799     −0.0055          0 / 18
  100       0.5547     −0.0055          0 / 18
 1000       0.3550     −0.0055          0 / 18

Lasso is feature selection. At alpha=1 only three features survive, and at alpha=10 none do — so the model predicts the mean and scores R² −0.0055, precisely what DummyRegressor gets. Regularisation is a dial between fitting and not fitting, and both ends are bad.

What ours says

Ridge is essentially flat from alpha 0.01 to 10, gaining a whole 0.0014, and only degrades at 100.

That's the right answer, not a disappointment. Lesson 9 measured a 0.003 train/test gap: this model was never overfitting, so there was nothing for a penalty to rein in. Regularisation is a cure for variance. Apply it to a model whose problem is bias and you make things slowly worse while feeling diligent.

Choosing alpha

from sklearn.linear_model import RidgeCV
import numpy as np

model = Pipeline([("pre", pre), ("m", RidgeCV(alphas=np.logspace(-2, 3, 20)))])
model.fit(X_train, y_train)
model.named_steps["m"].alpha_      # 4.28

By cross-validation, always — never by looking at the test set. RidgeCV and LassoCV do it internally and store the winner in alpha_.

Search on a log scale: alpha's effect is multiplicative, so 0.01, 0.1, 1, 10, 100 explores vastly more ground than 1, 2, 3, 4, 5. And if the best value lands at the edge of your range, widen the range rather than accepting it — the optimum may be outside what you searched.

Which one

• Ridge — the default. Keeps everything, handles correlated features gracefully.
• Lasso — when you want a shorter model, for interpretability or for cost. Be aware it picks arbitrarily among correlated columns.
• ElasticNet — both, mixed by l1_ratio. Good when you have many correlated features and still want selection.

And a thing you've been doing all along

LogisticRegression is regularised by default, and its knob is C, which is 1/alpha — so smaller C means more penalty. Every logistic regression in this track has been quietly running at C=1.0.

That inversion catches people constantly, and it's why sweeping C feels backwards the first time you do it.