Why accuracy lies

The model from lesson 4 is 79.6% accurate. On a slide that reads like a working system. This lesson is about why the sentence is close to content-free, and it is the most important lesson in the track.

What accuracy actually counts

accuracy = rows you got right / all rows

Every row counts the same. That is an assumption, not a neutral choice, and here it's false: missing a late delivery costs a complaint and a refund; a false alarm costs a text message saying "running slightly behind". A metric that treats those as equal is measuring something nobody asked about.

The floor

77.8% of our deliveries are on time. So a model that ignores every column and always says "on time" is 77.8% accurate:

from sklearn.dummy import DummyClassifier

dummy = DummyClassifier(strategy="most_frequent").fit(X_train, y_train)
dummy.score(X_test, y_test)      # 0.7778

Accuracy's floor is not zero. It's the share of the majority class, and it moves with your data. Reporting an accuracy without that number beside it is like reporting a temperature without saying which scale.

Four dummies

DummyClassifier offers several ways to be deliberately useless, and comparing them is instructive:

strategy="most_frequent"   acc 0.7778   recall 0.00
strategy="prior"            acc 0.7778   recall 0.00
strategy="stratified"       acc 0.6978   recall 0.28
strategy="uniform"          acc 0.5333   recall 0.56

our model                   acc 0.7956   recall 0.34

Read the recall column again.

uniform is a coin flip. It ignores the data completely, and it catches 56% of the late deliveries. Our trained model, with five features and a pipeline, catches 34%.

On the only question the stall asked — which deliveries will be late? — the model is worse than random. And its accuracy is 26 points higher, which is why nobody noticed.

Why this happens

Nothing is broken. The model is optimising the thing it was told to optimise, and accuracy is dominated by the common class. Predicting "on time" is correct 78% of the time, so the training process rewards reluctance to say "late". The model learned to be cautious because caution scored well.

This is the general shape of the problem: a model becomes whatever your metric measures. Choose the metric carelessly and you will get exactly what you asked for, which is not the same as what you wanted.

And it gets worse as the class gets rarer

• Late deliveries at 22% → a do-nothing model is 78% accurate.
• Loan defaults at 5% → 95%.
• Card fraud at 1% → 99%.
• A rare disease at 0.1% → 99.9%.

A 99.9%-accurate model that has never once said "positive" is not a hypothetical; it is what you get by default on rare-class problems. And those are precisely the problems worth solving — the rarity is usually why they matter. So accuracy is least informative exactly where you most need a number you can trust.

The same trap in regression

DummyRegressor(strategy="mean") predicts the average every time. Its R² is exactly 0 by construction — which is a small mercy, because it means R² comes with its baseline built in. An R² of 0.05 is visibly nearly worthless in a way that "95% accurate" never looks.

Watch for the reverse trap though: R² can go negative, which means your model is worse than predicting the mean. People often assume the range is 0 to 1 and misread a negative as a bug.

What to do instead

1. Fit a dummy first. One line, before any real model, every time.
2. Never report an accuracy without its floor. "0.796 against a 0.778 baseline" is a result; "79.6% accurate" is a decoration.
3. Look at the rare class on its own. The average hides it.
4. Choose the metric before you model, from what the mistakes cost — otherwise you will unconsciously choose the one that flatters what you built.

Doing (3) properly needs the confusion matrix, which is the next lesson.