Searching for Good Settings

GridSearchCV tries every combination of the settings you list, scores each by cross-validation on the training rows, keeps the best in best_params_ and best_score_ with every result in cv_results_, and refits the best pipeline on all the training rows, ready for one final test score; a pipeline's settings are named step__setting, and RandomizedSearchCV tries a sample when the grid is too big.

Freelancer · AI Basics · Python · Updated

By

After this lesson you can

  • Search a pipeline's settings with GridSearchCV, naming each setting step__setting, fitting it on the training rows only
  • Read best_params_, best_score_ and cv_results_, and explain how the search refits the best settings before one final test score

Key takeaways

  • GridSearchCV(pipeline, grid, cv=folds) tries every combination in the grid and scores each one by cross-validation.
  • A pipeline's settings are named after the step, two underscores, then the setting, such as model__max_depth.
  • best_params_ holds the winning settings, best_score_ their mean cross-validated score, and cv_results_ every setting's scores.
  • After the search, the best settings are refit on all the training rows, so the search itself can predict and score.
  • Fit the search on the training rows only, and score the test rows once at the end.

Hyperparameters: Settings You Choose tried four depths for the Harbor View Inn's cancellation tree with a loop. Now add a second setting with three values, and the loop is 21 combinations, each scored on 5 folds: 105 trees to train, keep track of and compare. You'll meet this every time you tune a model, and scikit-learn has a tool that writes the loop for you, keeps every score and hands back the winner.

The example is Harbor View Inn's bookings.csv, from Exercise: Fair Model Showdown. Each code block is one notebook cell, run in order: the first one loads the data, and the others use what it made.

Naming settings inside a pipeline

This time the features include channel, how the booking was made: direct, phone or travel_site. A tree can't read words, so it goes through a one-hot encoder first, inside a pipeline, as Different Steps for Different Columns shows. The steps are named prep and model:

import pandas as pd
from sklearn.compose import make_column_transformer
from sklearn.model_selection import GridSearchCV, StratifiedKFold, train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder
from sklearn.tree import DecisionTreeClassifier

bookings = pd.read_csv("bookings.csv")
features = ["lead_time_days", "nights", "guests", "deposit", "channel", "repeat_guest", "special_requests"]
X_train, X_test, y_train, y_test = train_test_split(
    bookings[features], bookings["canceled"], test_size=0.2, stratify=bookings["canceled"], random_state=42
)
prep = make_column_transformer((OneHotEncoder(handle_unknown="ignore"), ["channel"]), remainder="passthrough")
pipeline = Pipeline([("prep", prep), ("model", DecisionTreeClassifier(random_state=0))])
folds = StratifiedKFold(n_splits=5, shuffle=True, random_state=0)

A pipeline's settings include every step's settings. Each one's name is the step's name, two underscores, then the setting: model__max_depth, read "model, double underscore, max depth", is the max_depth of the step called model.

print(pipeline.get_params()["model__max_depth"])
pipeline.set_params(model__max_depth=4)
print(pipeline.named_steps["model"])
None
DecisionTreeClassifier(max_depth=4, random_state=0)

The grid and the search

A grid is a dictionary: each setting's name, with a list of values to try. GridSearchCV (grid search with cross-validation) tries every combination, the way a table's rows and columns make every pairing:

grid = {
    "model__max_depth": [2, 3, 4, 5, 6, 8, None],
    "model__min_samples_leaf": [1, 10, 30],
}
search = GridSearchCV(pipeline, grid, cv=folds, scoring="accuracy")
search.fit(X_train, y_train)
print("Settings tried:", len(search.cv_results_["params"]))
print("Best settings:", search.best_params_)
print("Best mean accuracy:", round(search.best_score_, 3))
Settings tried: 21
Best settings: {'model__max_depth': 4, 'model__min_samples_leaf': 1}
Best mean accuracy: 0.845

A grid of 21 squares, seven rows for the depths 2, 3, 4, 5, 6, 8 and None and three columns for the leaf sizes 1, 10 and 30, each square one setting the search scores on all five folds; the square for depth 4 and leaf size 1 is marked as the winner, 0.845, and the square for depth None and leaf size 1 is marked as the lowest, 0.765

min_samples_leaf is the fewest training bookings a leaf may hold. Seven depths times three leaf sizes is 21 settings. search.fit gets the training rows only: the search cuts its own folds out of them, so the test bookings stay unseen. scoring="accuracy" names the score it ranks by; for a classifier, that's also what it uses when you leave scoring out. best_params_ holds the winner's settings, and best_score_ its mean accuracy over the five folds.

A name the pipeline doesn't have stops the search before it starts:

wrong = GridSearchCV(pipeline, {"max_depth": [2, 4]}, cv=folds)
wrong.fit(X_train, y_train)
ValueError: Invalid parameter 'max_depth' for estimator Pipeline(steps=[('prep',
                 ColumnTransformer(remainder='passthrough',
                                   transformers=[('onehotencoder',
                                                  OneHotEncoder(handle_unknown='ignore'),
                                                  ['channel'])])),
                ('model', DecisionTreeClassifier(max_depth=4, random_state=0))]). Valid parameters are: ['memory', 'steps', 'transform_input', 'verbose'].

The pipeline itself has no max_depth; its model step does. The fix is model__max_depth.

Reading every result

cv_results_ keeps every setting's scores, and turns into a table with pandas. Here are the top four and the bottom three, by rank:

results = pd.DataFrame(search.cv_results_)
columns = ["param_model__max_depth", "param_model__min_samples_leaf", "mean_test_score", "rank_test_score"]
ranked = results[columns].sort_values("rank_test_score").round(3)
print(pd.concat([ranked.head(4), ranked.tail(3)]).to_string(index=False))
param_model__max_depth  param_model__min_samples_leaf  mean_test_score  rank_test_score
                     4                              1            0.845                1
                     4                             10            0.845                1
                     3                              1            0.843                3
                     3                             30            0.843                3
                  None                             10            0.828               19
                     8                              1            0.812               20
                  None                              1            0.765               21

Read two things from it. First, mean_test_score is the score on each fold's held-back rows, averaged: "test" here means the folds, not your test set. Second, the top is crowded: two settings tie at rank 1, and the next ones are a tenth of a point behind. Here best_params_ holds the first of the two the search tried. Differences that small are no reason to believe one setting is really better; Comparing Models Fairly says what to do about that. The bottom is clear, though: the tree with no limit and leaves of one booking is far behind.

Refit, then one test score

When the search finishes, it trains the best settings once more on all 2,400 training bookings: refitting (refit=True is the default). That model is best_estimator_, and the search's own predict and score use it:

best = search.best_estimator_
print(best.named_steps["model"])
print("Trained on:", len(X_train), "bookings")
print("Guesses for 3 new bookings:", search.predict(X_test.head(3)))
DecisionTreeClassifier(max_depth=4, random_state=0)
Trained on: 2400 bookings
Guesses for 3 new bookings: [0 0 0]

Then, once, the test bookings:

print("Test accuracy:", round(search.score(X_test, y_test), 3))
Test accuracy: 0.837

A little below the best cross-validated score, and that's normal: 0.845 was the best of 21 tries, so luck helped it. The test score is the honest one; report it as it is, and don't go back to change the grid because of it.

A grid of five settings with ten values each is 100,000 combinations. RandomizedSearchCV tries a fixed number of them instead, n_iter, picked at random from the lists you give it (set its random_state too), and is the usual choice when a grid gets that big.

Check yourself

2 questions to try on the lesson page.

So what

A grid search does the tuning loop for you, scored by cross-validation on the training rows, and hands back the best settings already refit, so the test set is looked at once, at the end.

Sources

  • scikit-learn, Tuning the hyper-parameters of an estimator, User Guide: GridSearchCV exhaustively generates candidates from param_grid, evaluates every combination when fit and retains the best; parameters of composite estimators such as a Pipeline are searched with the <estimator>__<parameter> syntax; by default the search uses the estimator's score (accuracy for classification), and scoring chooses another; evaluate the result on held-out samples not seen during the search; RandomizedSearchCV samples settings, with a budget set by n_iter.
  • scikit-learn 1.9.1, GridSearchCV, API reference: refit=True by default, refitting an estimator with the best found parameters on the whole dataset as best_estimator_, so predict works on the search; best_params_, best_score_ (the mean cross-validated score of the best estimator) and cv_results_ (with mean_test_score, std_test_score, rank_test_score and params, importable into a pandas DataFrame); score uses the best estimator; cv accepts a splitter.
  • scikit-learn, Pipelines and composite estimators, User Guide, Access to nested parameters: a pipeline's step parameters are reached with the <estimator>__<parameter> syntax, such as set_params(clf__C=10), which matters most for grid searches.
  • scikit-learn 1.9.1, RandomizedSearchCV, API reference: not all parameter values are tried; a fixed number of settings, n_iter, is sampled; lists are sampled uniformly; random_state makes the sampling reproducible.

Key terms

Related