Capstone: diagnose a broken model

A colleague sends you this and says it's ready to ship:

X_all = deliveries[FEATURES + ["minutes"]].fillna(0)
X_all = StandardScaler().fit_transform(pd.get_dummies(X_all))

Xtr, Xte, ytr, yte = train_test_split(X_all, y, test_size=0.25, random_state=0)

best = 0
for C in np.logspace(-3, 3, 20):
    m = LogisticRegression(C=C, max_iter=1000).fit(Xtr, ytr)
    if m.score(Xte, yte) > best:
        best, best_C = m.score(Xte, yte), C

tree = DecisionTreeClassifier().fit(Xtr, ytr)
print("accuracy", tree.score(Xtr, ytr))     # 1.0
print("AUC", roc_auc_score(yte, ...))        # 0.9989

Your job isn't to build anything. It's to find what's wrong — and there are six things, in rough order of how much they matter.

1. A leaked column — AUC 0.9989

"minutes" is in the feature list, and late is minutes > 30. The model learned to compare a number to 30. At prediction time that column doesn't exist, because the delivery hasn't happened.

Honest AUC: 0.859. The tell was the score itself — a number that good should produce suspicion, not celebration (lesson 13).

2. No baseline

"94% accurate" is meaningless without the floor. DummyClassifier(strategy="most_frequent") scores 0.7778 here, so any accuracy has to be read against that. Report a number without its baseline and you have reported a decoration (lesson 5).

3. The wrong metric entirely

Accuracy on a 22% target measures how good you are at the boring case. The colleague never looked at recall — which for a plain logistic regression here is 0.44, and for a depth-3 tree is 0.08.

Worse, this poisons the model selection: a grid search under scoring="accuracy" picks class_weight=None and quietly abandons the rare class (lesson 23).

4. An unstratified split

train_test_split(X, y, test_size=0.25, random_state=0)
# test set is 19.1% late; the data is 22.2%

The evaluation happens on a test set easier than reality, and the result gets reported as if it described reality (lesson 3).

5. Tuned against the test set, then reported that score

The loop picks C by m.score(X_test, y_test). The test set is now spent — it's a validation set with a misleading name (lesson 3).

And separately: reporting best_score_ from a search instead of a held-out score would be 0.8952 against an honest 0.8465 here (lesson 24). Nearly five points of difference, in the direction that flatters.

6. A memorising tree

DecisionTreeClassifier()      # no max_depth
train 1.0000     test 0.7333

A perfect training score and the worst test score of any model in the track. The colleague printed the training accuracy, which is the one number that can never tell you anything (lesson 9).

And now the honest part: not all leaks are equal

Two more things in that notebook are textbook mistakes. On this data one of them is unmeasurable:

scaling before the split       leaked 0.8141    correct 0.8141    diff 0.0000
tuning C on the test set       "best" 0.8471    honest  0.8465    diff 0.0006

Both are genuine leaks and both move roughly nothing. It's worth understanding why rather than concluding the rules don't matter.

The size of a leak depends on how much the leaking step learns about y. A StandardScaler learns a mean and a standard deviation — and a mean computed from 80% of your rows is nearly identical to one computed from 100%. It never sees y at all. So the leak is real and tiny, at any sample size we tried.

Now the same mistake with a step that does look at the target:

# 200 columns of PURE NOISE. There is no signal. At all.
X_noise = rng.normal(size=(900, 200))

# select the 5 "best" features using all of y, then cross-validate
X_sel = SelectKBest(f_classif, k=5).fit_transform(X_noise, y)
cross_val_score(LogisticRegression(), X_sel, y, cv=5, scoring="roc_auc")   # 0.6062

# the same selection, inside the pipeline
Pipeline([("sel", SelectKBest(f_classif, k=5)), ("m", LogisticRegression())])
cross_val_score(pipe, X_noise, y, cv=5, scoring="roc_auc")                  # 0.5222

0.6062 from data containing no signal whatsoever. The selector went looking through 200 noise columns for the five that happened to correlate with this y, and cross-validation then scored those columns on the very rows that chose them. Done properly it reads 0.5222 — close to the honest 0.5.

So the rule to carry away is sharper than "avoid leakage":

• Steps that never see y — scalers, most imputers — leak a little. Fix them because it's free, not because it's urgent.
• Steps that use y — feature selection, target encoding, resampling — leak catastrophically. These are the ones that turn noise into a promising result.

And the defence is the same for both, which is the point: put every fitted step in a Pipeline and you never have to make the judgement.

It cuts the other way too. Don't reject a colleague's model because you spotted a theoretical leak — measure it. Some are fatal, some are rounding, and the difference is knowable in about four lines.

The review checklist

1. Is any score suspiciously high? Look for a leaked column before anything else.
2. Where is the baseline? No dummy, no result.
3. Does the metric match the decision? Accuracy on an uneven target almost never does.
4. Was the split stratified? And were the rows independent — no time order, no groups?
5. How many times was the test set touched? Once is the only right answer.
6. Is the reported number a test score, or best_score_, or a training score?
7. What's the train/test gap? Zero means underfitting; huge means memorising.
8. Is all preprocessing inside the pipeline?

Eight questions. They will catch most broken models in a few minutes, and every one of them is something this track measured rather than asserted.

Where the track ends

Thirty lessons, and the recurring finding is worth stating plainly: on this data logistic regression beat every ensemble, PCA lost information, polynomials made it worse, and the biggest single improvement came from add_indicator=True — a keyword about missing data.

The modelling was never the hard part. Framing the question, preparing the data honestly, and measuring the result without fooling yourself — that's the job, and it's why two of the eight modules were about evaluation.