Polynomials and the bias-variance trade-off

Linear regression can only draw straight lines. But you can hand it curved features and it stays linear in those — all the curvature lives in the columns.

from sklearn.preprocessing import PolynomialFeatures

Pipeline([("pre", pre),
          ("poly", PolynomialFeatures(degree=2, include_bias=False)),
          ("m", LinearRegression())])

Interactions are the real reason

Squares are the obvious part; the products matter more. Degree 2 includes distance × rain_heavy, which lets the model finally say "rain costs more on long trips".

That's an interaction, and plain linear regression cannot express it at all — lesson 14's additivity assumption said effects simply add up. This is how you relax that, and "rain matters more when the trip is long" is a genuinely plausible claim about deliveries.

If you want only the products and not the powers, interaction_only=True.

And the cost

degree 1     18 features
degree 2    189 features
degree 3  1,329 features

It's combinatorial — every pair of columns produces a product. At degree 3 you have roughly twice as many features as training rows, which is a model capable of fitting essentially anything, including all the noise.

What actually happens

degree   features   train R²   test R²   5-fold R²
     1         18     0.6065   +0.5948     +0.5785
     2        189     0.6800   +0.5504     +0.4989
     3       1329     0.8872   −1.0152     −2.1076

Training R² climbs to 0.887 — exactly as lesson 9 promised it always would, because more capacity always fits the visible rows better.

Cross-validated R² goes the other way and ends at −2.11. That minus sign is not a bug: negative R² means worse than predicting the mean. Degree 3 is beaten by DummyRegressor.

Bias and variance

This is the trade-off with names attached, and it's the central idea of the module:

• Bias — error from a model too rigid to represent the truth. A straight line on a genuinely curved relationship. High bias = underfitting.
• Variance — error from a model so flexible that it changes shape with the particular sample you drew. Degree 3 on 675 rows. High variance = overfitting.
• Irreducible noise — the part no model removes, because the world isn't deterministic.

Total error is roughly bias² + variance + noise. Bias falls as capacity rises; variance climbs. Their sum is U-shaped, and the bottom of that U is the model you want.

Every "how flexible" hyperparameter you've met is a position on that axis: max_depth, alpha, C, polynomial degree, n_neighbors. And you find the bottom by cross-validation, because the total is the only one of the three curves you can actually measure.

Rescuing degree 2

You don't have to choose between 189 features and none — regularisation buys back the variance:

degree 2, no penalty       0.4989
degree 2 + ridge(1)        0.5055
degree 2 + ridge(10)       0.5300
degree 2 + ridge(100)      0.5626

Expand the features, then regularise hard. That pairing is the standard recipe when you suspect interactions, and it recovers most of the damage.

But still not better

And 0.5626 is still below plain degree 1's 0.5785. That's the honest ending: on this data the curvature genuinely isn't there. minutes is a weighted sum, and no amount of machinery improves on modelling it as one.

Which is worth knowing you can establish in four lines. Trying the flexible model and measuring it properly is how you learn that your problem is simple — and that's a real result, not a failure.

The practical summary

• Degree 2 is occasionally worth trying, especially with few features.
• Degree 3+ is almost never worth it — use a tree-based model instead.
• If you use polynomials, regularise, and scale first (the powers explode otherwise).
• A negative cross-validated score is not a bug. It's the model telling you it's worse than nothing.