Searching for Good Settings
GridSearchCV tries every combination of the settings you list, scores each by cross-validation on the training rows, keeps the best in best_params_ and best_score_ with every result in cv_results_, and refits the best pipeline on all the training rows, ready for one final test score; a pipeline's settings are named step__setting, and RandomizedSearchCV tries a sample when the grid is too big.
Freelancer · AI Basics · Python · Updated
After this lesson you can
- Search a pipeline's settings with GridSearchCV, naming each setting step__setting, fitting it on the training rows only
- Read best_params_, best_score_ and cv_results_, and explain how the search refits the best settings before one final test score
Key takeaways
- GridSearchCV(pipeline, grid, cv=folds) tries every combination in the grid and scores each one by cross-validation.
- A pipeline's settings are named after the step, two underscores, then the setting, such as model__max_depth.
- best_params_ holds the winning settings, best_score_ their mean cross-validated score, and cv_results_ every setting's scores.
- After the search, the best settings are refit on all the training rows, so the search itself can predict and score.
- Fit the search on the training rows only, and score the test rows once at the end.
Hyperparameters: Settings You Choose tried four depths for the Harbor View Inn's cancellation tree with a loop. Now add a second setting with three values, and the loop is 21 combinations, each scored on 5 folds: 105 trees to train, keep track of and compare. You'll meet this every time you tune a model, and scikit-learn has a tool that writes the loop for you, keeps every score and hands back the winner.
The example is Harbor View Inn's bookings.csv, from Exercise: Fair Model Showdown. Each code block is one notebook cell, run in order: the first one loads the data, and the others use what it made.
Naming settings inside a pipeline
This time the features include channel, how the booking was made: direct, phone or travel_site. A tree can't read words, so it goes through a one-hot encoder first, inside a pipeline, as Different Steps for Different Columns shows. The steps are named prep and model:
import pandas as pd
from sklearn.compose import make_column_transformer
from sklearn.model_selection import GridSearchCV, StratifiedKFold, train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder
from sklearn.tree import DecisionTreeClassifier
bookings = pd.read_csv("bookings.csv")
features = ["lead_time_days", "nights", "guests", "deposit", "channel", "repeat_guest", "special_requests"]
X_train, X_test, y_train, y_test = train_test_split(
bookings[features], bookings["canceled"], test_size=0.2, stratify=bookings["canceled"], random_state=42
)
prep = make_column_transformer((OneHotEncoder(handle_unknown="ignore"), ["channel"]), remainder="passthrough")
pipeline = Pipeline([("prep", prep), ("model", DecisionTreeClassifier(random_state=0))])
folds = StratifiedKFold(n_splits=5, shuffle=True, random_state=0)
A pipeline's settings include every step's settings. Each one's name is the step's name, two underscores, then the setting: model__max_depth, read "model, double underscore, max depth", is the max_depth of the step called model.
print(pipeline.get_params()["model__max_depth"])
pipeline.set_params(model__max_depth=4)
print(pipeline.named_steps["model"])
None
DecisionTreeClassifier(max_depth=4, random_state=0)
The grid and the search
A grid is a dictionary: each setting's name, with a list of values to try. GridSearchCV (grid search with cross-validation) tries every combination, the way a table's rows and columns make every pairing:
grid = {
"model__max_depth": [2, 3, 4, 5, 6, 8, None],
"model__min_samples_leaf": [1, 10, 30],
}
search = GridSearchCV(pipeline, grid, cv=folds, scoring="accuracy")
search.fit(X_train, y_train)
print("Settings tried:", len(search.cv_results_["params"]))
print("Best settings:", search.best_params_)
print("Best mean accuracy:", round(search.best_score_, 3))
Settings tried: 21
Best settings: {'model__max_depth': 4, 'model__min_samples_leaf': 1}
Best mean accuracy: 0.845

min_samples_leaf is the fewest training bookings a leaf may hold. Seven depths times three leaf sizes is 21 settings. search.fit gets the training rows only: the search cuts its own folds out of them, so the test bookings stay unseen. scoring="accuracy" names the score it ranks by; for a classifier, that's also what it uses when you leave scoring out. best_params_ holds the winner's settings, and best_score_ its mean accuracy over the five folds.
A name the pipeline doesn't have stops the search before it starts:
wrong = GridSearchCV(pipeline, {"max_depth": [2, 4]}, cv=folds)
wrong.fit(X_train, y_train)
ValueError: Invalid parameter 'max_depth' for estimator Pipeline(steps=[('prep',
ColumnTransformer(remainder='passthrough',
transformers=[('onehotencoder',
OneHotEncoder(handle_unknown='ignore'),
['channel'])])),
('model', DecisionTreeClassifier(max_depth=4, random_state=0))]). Valid parameters are: ['memory', 'steps', 'transform_input', 'verbose'].
The pipeline itself has no max_depth; its model step does. The fix is model__max_depth.
Reading every result
cv_results_ keeps every setting's scores, and turns into a table with pandas. Here are the top four and the bottom three, by rank:
results = pd.DataFrame(search.cv_results_)
columns = ["param_model__max_depth", "param_model__min_samples_leaf", "mean_test_score", "rank_test_score"]
ranked = results[columns].sort_values("rank_test_score").round(3)
print(pd.concat([ranked.head(4), ranked.tail(3)]).to_string(index=False))
param_model__max_depth param_model__min_samples_leaf mean_test_score rank_test_score
4 1 0.845 1
4 10 0.845 1
3 1 0.843 3
3 30 0.843 3
None 10 0.828 19
8 1 0.812 20
None 1 0.765 21
Read two things from it. First, mean_test_score is the score on each fold's held-back rows, averaged: "test" here means the folds, not your test set. Second, the top is crowded: two settings tie at rank 1, and the next ones are a tenth of a point behind. Here best_params_ holds the first of the two the search tried. Differences that small are no reason to believe one setting is really better; Comparing Models Fairly says what to do about that. The bottom is clear, though: the tree with no limit and leaves of one booking is far behind.
Refit, then one test score
When the search finishes, it trains the best settings once more on all 2,400 training bookings: refitting (refit=True is the default). That model is best_estimator_, and the search's own predict and score use it:
best = search.best_estimator_
print(best.named_steps["model"])
print("Trained on:", len(X_train), "bookings")
print("Guesses for 3 new bookings:", search.predict(X_test.head(3)))
DecisionTreeClassifier(max_depth=4, random_state=0)
Trained on: 2400 bookings
Guesses for 3 new bookings: [0 0 0]
Then, once, the test bookings:
print("Test accuracy:", round(search.score(X_test, y_test), 3))
Test accuracy: 0.837
A little below the best cross-validated score, and that's normal: 0.845 was the best of 21 tries, so luck helped it. The test score is the honest one; report it as it is, and don't go back to change the grid because of it.
A grid of five settings with ten values each is 100,000 combinations. RandomizedSearchCV tries a fixed number of them instead, n_iter, picked at random from the lists you give it (set its random_state too), and is the usual choice when a grid gets that big.
Check yourself
2 questions to try on the lesson page.
So what
A grid search does the tuning loop for you, scored by cross-validation on the training rows, and hands back the best settings already refit, so the test set is looked at once, at the end.
Sources
- scikit-learn, Tuning the hyper-parameters of an estimator, User Guide:
GridSearchCVexhaustively generates candidates fromparam_grid, evaluates every combination when fit and retains the best; parameters of composite estimators such as aPipelineare searched with the<estimator>__<parameter>syntax; by default the search uses the estimator'sscore(accuracy for classification), andscoringchooses another; evaluate the result on held-out samples not seen during the search;RandomizedSearchCVsamples settings, with a budget set byn_iter. - scikit-learn 1.9.1, GridSearchCV, API reference:
refit=Trueby default, refitting an estimator with the best found parameters on the whole dataset asbest_estimator_, sopredictworks on the search;best_params_,best_score_(the mean cross-validated score of the best estimator) andcv_results_(withmean_test_score,std_test_score,rank_test_scoreandparams, importable into a pandas DataFrame);scoreuses the best estimator;cvaccepts a splitter. - scikit-learn, Pipelines and composite estimators, User Guide, Access to nested parameters: a pipeline's step parameters are reached with the
<estimator>__<parameter>syntax, such asset_params(clf__C=10), which matters most for grid searches. - scikit-learn 1.9.1, RandomizedSearchCV, API reference: not all parameter values are tried; a fixed number of settings,
n_iter, is sampled; lists are sampled uniformly;random_statemakes the sampling reproducible.