Searching hyperparameters

You have been sweeping one parameter at a time and reading the numbers off a printout. GridSearchCV does that properly: every combination, cross-validated, with the winner refitted on all the training data.

from sklearn.model_selection import GridSearchCV

search = GridSearchCV(
    pipe,
    {"clf__C": [0.01, 0.1, 1, 10],
     "clf__class_weight": [None, "balanced"]},
    cv=5,
    scoring="roc_auc",
)
search.fit(X_train, y_train)

search.best_params_    # {'clf__C': 1, 'clf__class_weight': 'balanced'}
search.best_score_     # 0.8952
search.predict(X_test)  # already refitted on all of X_train

The search object is the model — best_estimator_ is refitted automatically, so you can call predict straight on it. And cv_results_ holds every combination's score, which is worth reading rather than just taking the winner.

You can tune anything in the pipeline

Because a pipeline is an estimator, the double-underscore path from lesson 13 addresses any step:

"clf__C"                            the classifier's regularisation
"clf__class_weight"                 how it treats the rare class
"pre__num__impute__strategy"        median vs mean, three levels down
"pre__num__impute__add_indicator"   whether to keep the missingness
"pre__cat__min_frequency"           how rare a category can be

Every one of those is a decision you've been making by hand, and preprocessing choices are hyperparameters too — often where the gains actually are. When you can't remember a path, pipe.get_params().keys() lists them all.

Count the fits

4 values of C  ×  2 class weights  ×  5 folds  =  40 fits  (+1 refit)

The multiplication is the thing to watch. Three parameters with five values each is 125 combinations — 625 fits at cv=5. n_jobs=-1 parallelises across cores and verbose=1 tells you how far along it is, which matters when a search runs for twenty minutes.

scoring decides the winner

Same grid, same data, three metrics:

scoring="roc_auc"    → class_weight="balanced"     0.8952
scoring="f1"          → class_weight="balanced"     0.6692
scoring="accuracy"    → class_weight=None           0.8459

Accuracy picks the model that ignores the rare class, exactly as lesson 5 predicted — and the search does it silently, dressed as an objective procedure. That's the most dangerous form the problem takes, because a grid search looks like rigour.

Which is why the metric has to be decided from costs before you search, not picked afterwards from whichever gives the nicest number. If you leave scoring unset you get the estimator's default — accuracy for classifiers — and you will not be told.

Randomized search

from sklearn.model_selection import RandomizedSearchCV
from scipy.stats import loguniform

RandomizedSearchCV(pipe, {"clf__C": loguniform(1e-3, 1e3)},
                   n_iter=25, cv=5, random_state=0)

A grid over two parameters with 5 values each spends 25 fits and tries only 5 distinct values of each. Random sampling spends the same 25 and tries 25 distinct values of each.

That matters because usually only one or two parameters actually matter and you don't know which in advance — so a grid wastes most of its budget resolving parameters that don't move the score. n_iter also gives you a compute budget you set directly rather than one that emerges from multiplication.

Searching well

• Log scales for C and alpha — their effect is multiplicative, so 0.01, 0.1, 1, 10, 100 covers far more than 1, 2, 3, 4, 5.
• Widen if the winner is at an edge. The optimum may be outside what you searched.
• Coarse first, then refine. Two cheap searches beat one enormous one.
• Fewer, better-chosen candidates. A huge grid costs more than compute, and the next lesson explains why.

And keep the golden rule in view: all of this happens on the training data. The test set is opened once, at the end, and its score — not best_score_ — is what you report.