Thresholds, ROC and PR

Here is something predict() doesn't tell you: the model never made a yes/no decision. It produced a probability, and predict cut it at 0.5.

proba = model.predict_proba(X_test)[:, 1]     # the number underneath
pred  = model.predict(X_test)

((proba >= 0.5).astype(int) == pred).all()    # True — that IS predict()

That 0.5 is a default, not a fact about your problem. It's the single easiest improvement available in most classification projects, and it costs no retraining at all.

One caution: predict_proba returns a column per class, ordered by model.classes_. Column 1 is the positive class here. Take the wrong column and you get a perfectly functioning, perfectly inverted model.

Moving the cut

Same model, same predictions, different threshold:

threshold   accuracy  precision  recall   flagged
    0.50      0.796      0.567     0.340       30
    0.40      0.796      0.545     0.480       44
    0.30      0.747      0.444     0.560       63
    0.25      0.733      0.438     0.700       80
    0.20      0.711      0.418     0.760       91
    0.15      0.658      0.374     0.800      107

Look at the 0.40 row.

Accuracy is identical to the default — 0.796 either way — and recall goes from 0.34 to 0.48. Seven more late deliveries caught out of fifty, for free, by changing a number nobody told you was a choice. No new features, no better algorithm, no retraining.

Below that, you start paying: at 0.25 you catch 70% of late deliveries and accuracy falls to 0.733. Whether that's a good trade is a question about the business, not about the model.

Choosing the threshold from costs

Put numbers on the mistakes and the choice becomes arithmetic:

# a missed late delivery costs ₹200 in refunds and goodwill
# a false alarm costs ₹5 — one apologetic text

cost = misses * 200 + false_alarms * 5

With that ratio the cheapest threshold is far below 0.5, because misses are forty times more expensive than alarms. With the ratio reversed — say each alarm dispatches a rider — you'd want a threshold well above 0.5.

Tune the threshold on validation data or by cross-validation, never on the test set. It's a decision like any other, and lesson 3's golden rule applies to it in full.

ROC curves

Rather than picking one threshold, sweep all of them. The ROC curve plots true-positive rate against false-positive rate at every possible cut:

from sklearn.metrics import roc_curve, roc_auc_score

fpr, tpr, thresholds = roc_curve(y_test, proba)
roc_auc_score(y_test, proba)      # 0.793

AUC is the area underneath, and it has a genuinely elegant interpretation: the probability that the model scores a randomly chosen late delivery above a randomly chosen on-time one. It is a statement about ranking. 0.5 is a coin flip; 1.0 is perfect separation.

Because it needs no threshold, AUC is the right metric when:

• you'll tune the cut later, and want to compare models before that;
• you'll use the scores to prioritise a queue rather than to decide yes/no;
• the class balance might shift, since AUC is insensitive to it.

And the wrong one when you won't. A model with a fine AUC can be useless at every threshold you'd actually deploy — the ranking can be good in a region nobody operates in.

Why PR curves are better here

ROC has a specific weakness with rare positives, and it's worth understanding rather than memorising. The false-positive rate is FP / (FP + TN), and TN is huge — 175 of our 225 test rows. So a hundred false alarms move the x axis barely at all, and the curve stays flattering.

Precision has no such cushion: TP / (TP + FP) has no large denominator to hide in, so a hundred false alarms wreck it immediately.

from sklearn.metrics import precision_recall_curve, average_precision_score

precision, recall, thresholds = precision_recall_curve(y_test, proba)
average_precision_score(y_test, proba)    # 0.534
y_test.mean()                              # 0.222 — the no-skill floor

And note the floors differ, which is the detail people miss. ROC's no-skill line is always 0.5. A PR curve's floor is the base rate — so our 0.534 has to be read against 0.222, not against zero. Quoting an average precision without the base rate beside it is the same error as quoting an accuracy without its baseline.

Which to reach for

• Roughly balanced classes, ranking matters → ROC AUC.
• Rare positives, you care about the positive class → PR curve and average precision.
• You must ship a yes/no decision → pick the threshold from costs, then report precision and recall at that threshold. The curves are for choosing; the confusion matrix is for reporting.