Capstone: a classification project

Everything assembled, on the question the stall actually has: which deliveries will be late? Every stage is something from an earlier lesson; the value is in doing them in the right order and not skipping the two that people skip.

1. Frame it before you model

What decision does this serve? The stall wants to send an apologetic text when an order is going to be late. So:

• A miss costs about ₹200 — a surprised customer, a refund, some goodwill.
• A false alarm costs about ₹5 — one unnecessary text.

Those two numbers determine the metric and the threshold, and writing them down first is what stops you choosing them later to flatter whatever you built. Recall matters roughly forty times more than precision here.

2. Split first

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, stratify=y, random_state=0)

Before you look at a distribution or fill a gap. stratify=y because the target is uneven — without it the test set comes out 19.1% late instead of 22.2% (lesson 3).

3. Baseline

DummyClassifier(strategy="most_frequent").score(X_test, y_test)   # 0.7778

One line, and every number after it is now readable.

4. Pipeline

num = Pipeline([("impute", SimpleImputer(strategy="median", add_indicator=True)),
                ("scale", StandardScaler())])

pre = ColumnTransformer([("num", num, NUMERIC),
                         ("cat", OneHotEncoder(handle_unknown="ignore"), CATEGORICAL)])

pipe = Pipeline([("pre", pre), ("clf", LogisticRegression(max_iter=1000))])

add_indicator=True because the missingness is informative (lesson 12, +0.020 AUC). handle_unknown="ignore" because a new rider will eventually join (lesson 11). And everything inside the pipeline so no fold can leak (lesson 13).

5. Search

GridSearchCV(pipe,
             {"clf__C": [0.01, 0.1, 1, 10],
              "clf__class_weight": [None, "balanced"]},
             cv=5, scoring="roc_auc").fit(X_train, y_train)

best_params_   {'clf__C': 1, 'clf__class_weight': 'balanced'}
best_score_    0.8952

scoring="roc_auc" deliberately — under "accuracy" the same grid picks class_weight=None and quietly gives up on the late deliveries (lesson 23).

6. Choose the threshold from money

This is the step that separates a working project from an exercise. Sweep the cut and price each one:

cost = missed × 200 + false_alarms × 5

threshold 0.50   →  11 missed, 40 alarms   →  ₹2,400
threshold 0.10   →   0 missed, 100 alarms  →  ₹500

The cheapest cut is 0.10, and it catches all 50 late deliveries — recall 1.0, precision 0.33. That's what a 40:1 cost ratio implies, and no default would have found it.

And now a judgement the numbers cannot make: at that threshold you are texting 150 of 225 customers. Do that and people stop reading the texts, at which point your ₹5 estimate is wrong. The model tells you the trade; you decide whether the trade is real.

7. Report honestly

best_score_ (training CV)   0.8952      ← do NOT report this
TEST AUC                     0.8465      ← report this
dummy accuracy               0.7778

at threshold 0.10:  [[ 75  100]
                     [  0   50]]

Note the 0.049 gap between best_score_ and the test score. Most of that is ordinary train/test variance rather than selection optimism (lesson 24) — which is precisely why you keep a test set instead of trying to reason about which it is.

The checklist

1. Split before anything else, stratified.
2. Fit a dummy.
3. All preprocessing inside a pipeline.
4. Choose the metric from costs, before searching.
5. Cross-validate every decision; never touch the test set.
6. Tune the threshold from costs, on validation data.
7. Report the test score and the confusion matrix — not accuracy, not best_score_.
8. Say what the model can't decide for you.