← Machine Learning·Lesson 34·14 min·0/3 exercises

Diabetes — a low ceiling, reported honestly

R² 0.48 is the answer, and saying so is the deliverable

load_diabetespre-scaled featuresR² ceilingssplit variancereporting limits

Watch it happen

Play it through, or step back and forth yourself.

diabetesEfron et al., 2004

rows 442features 10task regression

DummyRegressor(mean) = R² 0 by construction; MAE 62.2

from sklearn.datasets import load_diabetes

d = load_diabetes(as_frame=True)
d.data.columns
# age sex bmi bp s1 s2 s3 s4 s5 s6
d.target.describe()   # 25 to 346, mean 152

The target is a quantitative measure of disease progression one year after baseline. Higher is worse. Ten routine measurements — age, sex, BMI, blood pressure and six blood serum values.

442 diabetes patients, ten measurements at baseline, and a target: disease progression a year later. It is a genuinely hard prediction, and that turns out to be the point.

The idea

442 diabetes patients, ten measurements taken at baseline, and a target: a quantitative measure of disease progression one year later. Higher is worse.

d = load_diabetes(as_frame=True)
d.data.columns    # age sex bmi bp s1 s2 s3 s4 s5 s6
d.target          # 25 to 346, mean 152

It is a genuinely hard prediction, and that turns out to be the point of the lesson.

Already scaled

d.data.mean()   # every column: -0.000000
d.data.std()    # every column:  0.0476

The columns arrive centred and scaled. Convenient — coefficients are comparable out of the box, and ridge's penalty is applied fairly (lesson 15).

And genuinely lossy: the raw units are gone. You cannot say "each extra point of BMI adds three units of progression", because there are no BMI points any more. A pre-processed dataset is a set of decisions somebody else made — check for it, and prefer the raw version when you can get it.

R² 0.482

plain linear         0.4823 ± 0.0493
ridge (alpha=1)      0.4822 ± 0.0489
random forest (200)  0.4239 ± 0.0501
dummy (mean)         0.0000

That is not a bad model. It is most of what ten routine measurements can tell you about where a disease will be in a year. The instinct to treat "R² below 0.5" as failure is a habit picked up from textbook datasets, where the answer is usually engineered to be findable.

And that really is the ceiling

The forest scores worse. So the missing 0.52 is not a modelling failure you can fix with more flexibility — it's irreducible noise plus information these ten columns simply do not contain: diet, adherence, genetics, everything that happened during the intervening year.

This is the fourth dataset in a row where the linear model wins. That isn't a coincidence about this track — it's what tabular data usually looks like, and it's the reason lesson 21 said to fit a linear model and a boosted one on every problem.

Hand-drawn notes showing three models stopping at the same score with the most flexible one lowest, so the ceiling rather than the model is the finding.

Report it in something people can use

guessing the mean    MAE 62.2
our model            MAE 45.2

target sd 77.1, range 25 to 346

"R² 0.48" tells a technical reader something. "Typical error drops from 62 to 45 units" tells everybody something — about a 27% cut in typical error.

Lesson 17's rule applied: report R² and an error in real units. Here the target's units are arbitrary, so the ratio does the work.

One honest warning about small data

5-fold cross-validated R²    0.482
single held-out test R²      0.358

Same data, same model, same code. 0.12 of R² from nothing but which 110 rows landed in the test set.

Lesson 3 measured four accuracy points of seed luck on 900 rows. This is the same phenomenon on 442, where it is much worse — and it's why the cross-validated 0.482 ± 0.049 is the number to trust. On small data a single held-out score is close to anecdote.

The deliverable is the ceiling

The most useful sentence you can write about this dataset isn't about a model at all:

"These ten measurements cap out around R² 0.48. Linear and ridge reach it; a random forest does worse. Effort should go into collecting better measurements, not into trying another algorithm."

That's a finding, and it saves whoever reads it a month. To support it you need:

a flexible model that does no better — evidence it's not a capacity problem;
a learning curve (lesson 9) — evidence more rows wouldn't help either;
the cross-validated number with its spread, not one split.

Saying "I got it to 0.48 and here is why that's the limit" is a much stronger result than "I got it to 0.48" — and a far more honest one than quietly trying algorithms until the test set flatters you.

Practice

Write it yourself. The answer is there when you want it.

Putting the kettle on…

Starting up…

Write it yourself

not graded

Print the shape, the target's range and spread, and check the claim that the columns arrive already centred and scaled. Cross-validate linear regression, ridge, a 200-tree forest and a dummy for R², and say what it means that the most flexible model does worst. Print the model's MAE beside the MAE of guessing the mean — the version anyone can read. Finish by printing the cross-validated R² against a single held-out split, and how much of the difference is nothing but which rows landed where.

Write something and press Run — the output appears here.

Your turn

3 exercises. Write the code yourself, then press Check — a nudge and the answer are there if you want them.

Exercise 1

Confirm the columns arrive pre-scaled. Return [mean_of_column_means, mean_of_column_stds], rounded to 6 and 4 places.

your answer

Exercise 2

Show it's a ceiling. Return [ridge_r2, forest_r2] — 5-fold R² for Ridge(alpha=1) and a 200-tree forest, rounded to 4 places. The forest should be lower.

your answer

Exercise 3

Show how noisy one split is on 442 rows. Return [cross_validated_r2, single_split_test_r2] at random_state=0, rounded to 4 places.

your answer