Categorical features

area, rain and rider have been sitting in the frame since lesson 4, doing nothing, because a model needs numbers and those are text. And lesson 9 told us the model wasn't overfitting — it was out of information.

One-hot encoding

from sklearn.preprocessing import OneHotEncoder

enc = OneHotEncoder(handle_unknown="ignore").fit(X_train[CATEGORICAL])
enc.categories_
# [['Campus', 'Industrial', 'Old Town', 'Riverside', 'Station'],
#  ['heavy', 'light', 'none'],
#  ['Asha', 'Bilal', 'Chen', 'Dev']]

enc.transform(X_train[CATEGORICAL]).shape[1]     # 12

Each category becomes its own 0/1 column, with exactly one 1 per row per original column — which is where the name comes from. Our three columns become 12, and crucially no ordering is implied: no category is closer to any other.

Note that get_feature_names_out() gives you area_Campus, rain_heavy and so on, which is what keeps coefficients interpretable after the transform.

The payoff

numeric only     5-fold AUC 0.8142 ± 0.0154
+ categoricals   5-fold AUC 0.8590 ± 0.0299

+0.045 — three times the fold-to-fold spread, so by lesson 8's standard this is a real improvement rather than noise. Test recall moves from 0.34 to 0.42 too. Still not good; considerably less bad.

This is exactly what lesson 9 predicted. The model was at its ceiling with five columns, so more regularisation would have done nothing and more rows would have done nothing. It needed better features, and three of them were already in the frame.

Ordinal encoding, and its trap

from sklearn.preprocessing import OrdinalEncoder
# Campus → 0, Industrial → 1, Old Town → 2, Riverside → 3, Station → 4

One column instead of five. And for a linear model it now asserts that Riverside − Old Town = 1, that Industrial sits halfway between Campus and Old Town, and that Station is four times… something. None of that means anything.

But here is the honest measurement:

one-hot + logistic     0.8590
ordinal + logistic     0.8600
ordinal + forest       0.8567

They're the same. The trap is real and it did not fire. Five arbitrary categories can only do so much damage, and the alphabetical ordering happened not to hurt.

That's the actual lesson, and it's more useful than a rule. You cannot detect this problem from the score. Sometimes an invented ordering costs you a great deal; sometimes nothing; and the difference is invisible from the outside. So you follow the rule rather than checking — one-hot unless the order is real.

Ordinal is right when there's a genuine order, and then you must state it:

OrdinalEncoder(categories=[["small", "medium", "large"]])

Without categories= the default is alphabetical, which would give you large < medium < small. Silently backwards.

Trees are also far more tolerant of ordinal encoding than linear models, because they can carve an arbitrary ordering into pieces with successive splits. It costs them depth, not correctness.

ColumnTransformer

Different columns need different treatment, and this is how you say so:

from sklearn.compose import ColumnTransformer

pre = ColumnTransformer([
    ("num", Pipeline([("impute", SimpleImputer(strategy="median")),
                      ("scale", StandardScaler())]), NUMERIC),
    ("cat", OneHotEncoder(handle_unknown="ignore"), CATEGORICAL),
])

model = Pipeline([("pre", pre), ("clf", LogisticRegression(max_iter=1000))])

Numeric columns get imputed and scaled; text columns get one-hot encoded; the whole thing is one estimator that knows which is which. remainder="drop" is the default — anything you didn't name is discarded, which is usually what you want and occasionally a surprise. remainder="passthrough" keeps it untouched.

handle_unknown is not optional

OneHotEncoder()                          # strict — raises on a new category
OneHotEncoder(handle_unknown="ignore")   # unknown row gets all zeros

Six months in, a new rider joins. Without that flag your service raises ValueError: Found unknown categories at predict time — in production, long after you stopped thinking about it.

And notice this failure never appears in cross-validation, because every fold is drawn from the same fixed set of categories. It waits for the real world.

When there are many categories

One-hot on 5,000 postcodes gives you 5,000 mostly-zero columns. Options:

• min_frequency=0.01 — rare categories collapse into one "infrequent" column.
• max_categories=20 — keep the top 20, bucket the rest.
• Target encoding — replace the category with its mean target. Powerful and leaks trivially; use scikit-learn's TargetEncoder, which cross-fits.
• Drop it. A column with one level per row is a row identifier, not a feature. It can only be memorised — lesson 1's lookup table in disguise.