Reading what the model learned

"Which columns mattered?" is the question everybody asks once a model works. It has at least three answers, they disagree with each other, and knowing why is more useful than any one of them.

Impurity importance — free with any tree

rf.named_steps["clf"].feature_importances_

distance_km    0.2104
prep_min       0.1956
temp           0.1157
hour           0.0876
items          0.0814

Accumulated while fitting: how much each column reduced impurity across every split it was used for, weighted by the rows passing through. Fast, always available — and quietly biased.

Notice temp and hour in third and fourth place. Both are high-cardinality continuous columns, and impurity importance systematically favours columns with many possible split points — more chances to split means more chances to reduce impurity somewhere. It's measuring opportunity, not usefulness.

It's also computed on training data, so a column the model overfitted to scores highly for precisely the wrong reason.

Permutation importance — the honest one

from sklearn.inspection import permutation_importance

pi = permutation_importance(model, X_test, y_test,
                            n_repeats=10, scoring="roc_auc", random_state=0)

Shuffle one column and see how much the score drops. If it doesn't drop, the model wasn't using it. Model-agnostic — it works on kNN, on a neural net, on anything with predict — and measured on held-out data in the units of your chosen metric.

prep_min             +0.1085 ± 0.0177
area_Campus          +0.0633 ± 0.0170
distance_km          +0.0591 ± 0.0127
items                +0.0093 ± 0.0091
prep_min_missing     +0.0040 ± 0.0080

A completely different ranking. prep_min first, and area_Campus second — which impurity importance never mentioned at all. And temp and hour have vanished: shuffling them costs the model nothing, so it was never really using them.

Note the error bars, too. ±0.018 on the top value means the ranking below the top three isn't distinguishable — a caveat impurity importance never gives you.

Coefficients — a third answer

rain_heavy     +1.766
prep_min       +1.371
distance_km    +1.355

The logistic model says rain_heavy first. Also legitimate, and answering a subtly different question. The three measure:

• coefficient — effect size, holding everything else fixed;
• permutation — how much the model relies on the column;
• impurity — how often the column got used while splitting.

Heavy rain has a large effect and is rare — about 9% of rows. Permuting a column that's constant for 91% of rows barely moves the score. Both facts are true at once, and neither method is wrong.

The trap all three share

distance_km      importance 0.06
distance_miles   importance 0.05

Two columns carrying the same information. Permute one and the model simply reads the other, so the score doesn't move — and both look unimportant, while distance is the most important thing in the dataset. Impurity importance splits the credit and reaches the same wrong conclusion.

Importance is a property of the model, not of the world. It tells you what this model uses; a different model on the same data would answer differently. Check correlations before you interpret, and consider permuting correlated columns as a group.

What none of it can tell you

• Causation. Importance is association. Only an experiment gives you cause, and no amount of model inspection substitutes.
• Direction. Impurity and permutation are unsigned — they say "this matters" without saying "up or down".
• The shape of the effect. Linear? A threshold? Importance can't tell you.
• What happens if you drop the column. Removing it changes what the model learns; the other columns take up the slack.

Partial dependence, for the shape

from sklearn.inspection import PartialDependenceDisplay
PartialDependenceDisplay.from_estimator(model, X_test, ["prep_min", "distance_km"])

Shows how the prediction moves as one column varies, averaging over the rest. It answers how, where importance only answers how much — and it will show you a threshold effect that no single number could.

The practical recipe

1. Fit a linear model and read the coefficients — direction and effect size, for free.
2. Run permutation_importance on the test set for reliance, with error bars.
3. Check correlations before believing either.
4. Plot partial dependence for anything that matters.
5. Say "associated with", not "causes".