The confusion matrix

Two possible truths, two possible predictions, four boxes. Learn the box and the metrics stop being definitions to memorise and become consequences you can derive.

from sklearn.metrics import confusion_matrix
confusion_matrix(y_test, model.predict(X_test))

#              predicted
#            on time  late
#  on time   [ 162     13 ]     <- 175 actually on time
#     late   [  33     17 ]     <-  50 actually late
#              actual

Rows are truth, columns are prediction. The diagonal is what you got right; the off-diagonal holds the two different ways of being wrong. scikit-learn always orders classes by model.classes_, so the negative class comes first — check that before reading anyone's matrix, including your own, because a transposed matrix looks entirely plausible.

The four names

• TN = 162 — said on time, was on time. The easy case, and the bulk of the accuracy.
• FP = 13 — said late, arrived on time. A false alarm. Cost: an unnecessary apology text.
• FN = 33 — said on time, arrived late. A miss. Cost: a surprised, annoyed customer.
• TP = 17 — said late, was late. The whole point of the exercise.

"False positive" and "false negative" describe the prediction, not the truth: a false positive is a positive prediction that was false. Getting this backwards is astonishingly common, and it survives into production dashboards.

Precision — when it cries wolf, is there a wolf?

precision = TP / (TP + FP) = 17 / 30 = 0.567

Of the 30 deliveries we flagged, 17 really were late. The denominator is everything you predicted positive. This is the metric when acting on a flag is expensive or annoying — sending a technician, blocking a card, waking someone at 3am.

Recall — of the real wolves, how many did we catch?

recall = TP / (TP + FN) = 17 / 50 = 0.340

Of the 50 genuinely late deliveries, we found 17. The denominator is everything that was actually positive. This is the metric when missing one is expensive: disease screening, fraud, equipment failure — and our stall, which would much rather send a needless apology than surprise someone.

The mnemonic that sticks: precision is about your predictions, recall is about reality. Look at which denominator you're dividing by and it's unambiguous.

They pull against each other

Always, and it isn't a flaw. Flag every single row and recall is 1.0 while precision falls to the base rate (0.222). Flag only the one row you're most sure about and precision is 1.0 while recall is 0.02. You can move freely along that curve without improving the model at all — which is the next lesson.

So a claim of high precision and high recall is a claim about the model being genuinely good. It cannot be arranged by choosing cleverly.

F1 — one number, when you need one

f1 = 2 * (precision * recall) / (precision + recall) = 0.425

The harmonic mean, and the choice of mean is the point. An ordinary average of precision 1.0 and recall 0.0 is a respectable-looking 0.5; the harmonic mean is 0.0. F1 refuses to let one good number cover for one terrible one.

Use it when you must optimise a single value and both errors matter roughly equally. Do not use it as a default without thinking — it bakes in "precision and recall are equally important", which is a claim about your problem. fbeta_score lets you weight them: beta=2 favours recall, beta=0.5 favours precision.

The rest of the family

• Balanced accuracy (0.633) — the mean of the recall of each class. The rare class counts as much as the common one, so it can't be gamed by ignoring it.
• Specificity — recall for the negative class, TN / (TN + FP) = 0.926. Common in medicine, rare in scikit-learn's API.
• Matthews correlation (MCC) — a single number that uses all four cells and stays honest under imbalance. Underused.

classification_report

from sklearn.metrics import classification_report
print(classification_report(y_test, pred, target_names=["on time", "late"]))

              precision  recall  f1-score  support
     on time       0.83    0.93      0.88      175
        late       0.57    0.34      0.42       50
    accuracy                         0.80      225
   macro avg       0.70    0.63      0.65      225
weighted avg       0.77    0.80      0.78      225

Read the two class rows, not the summary. "on time" scores 0.88 and "late" scores 0.42 — the model is good at the easy class and bad at the one it exists for.

And know the difference between the averages: macro is an unweighted mean across classes, so the rare class counts fully; weighted multiplies by support, which hands the common class back its dominance. With an imbalanced target, macro is the honest one.

Choosing

There is no universally correct metric — only one that matches your costs.

Write the choice down before you fit anything. Choose afterwards and you will pick whichever metric your model happens to do well on, and you will believe you were being objective.