# Searching for Good Settings

GridSearchCV tries every combination of the settings you list, scores each by cross-validation on the training rows, keeps the best in best_params\_ and best_score\_ with every result in cv_results\_, and refits the best pipeline on all the training rows, ready for one final test score; a pipeline's settings are named step\_\_setting, and RandomizedSearchCV tries a sample when the grid is too big.

Freelancer · AI Basics · Python · Updated November 19, 2026

By Gregor Dzierzon

## After this lesson you can

- Search a pipeline's settings with GridSearchCV, naming each setting step\_\_setting, fitting it on the training rows only
- Read best_params\_, best_score\_ and cv_results\_, and explain how the search refits the best settings before one final test score

## Key takeaways

- GridSearchCV(pipeline, grid, cv=folds) tries every combination in the grid and scores each one by cross-validation.
- A pipeline's settings are named after the step, two underscores, then the setting, such as model__max_depth.
- best_params_ holds the winning settings, best_score_ their mean cross-validated score, and cv_results_ every setting's scores.
- After the search, the best settings are refit on all the training rows, so the search itself can predict and score.
- Fit the search on the training rows only, and score the test rows once at the end.

*Hyperparameters: Settings You Choose* tried four depths for the Harbor View Inn's cancellation tree with a loop. Now add a second setting with three values, and the loop is 21 combinations, each scored on 5 folds: 105 trees to train, keep track of and compare. You'll meet this every time you tune a model, and scikit-learn has a tool that writes the loop for you, keeps every score and hands back the winner.

The example is Harbor View Inn's `bookings.csv`, from *Exercise: Fair Model Showdown*. Each code block is one notebook cell, run in order: the first one loads the data, and the others use what it made.

## Naming settings inside a pipeline

This time the features include `channel`, how the booking was made: `direct`, `phone` or `travel_site`. A tree can't read words, so it goes through a one-hot encoder first, inside a pipeline, as *Different Steps for Different Columns* shows. The steps are named `prep` and `model`:

```python
import pandas as pd
from sklearn.compose import make_column_transformer
from sklearn.model_selection import GridSearchCV, StratifiedKFold, train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder
from sklearn.tree import DecisionTreeClassifier

bookings = pd.read_csv("bookings.csv")
features = ["lead_time_days", "nights", "guests", "deposit", "channel", "repeat_guest", "special_requests"]
X_train, X_test, y_train, y_test = train_test_split(
    bookings[features], bookings["canceled"], test_size=0.2, stratify=bookings["canceled"], random_state=42
)
prep = make_column_transformer((OneHotEncoder(handle_unknown="ignore"), ["channel"]), remainder="passthrough")
pipeline = Pipeline([("prep", prep), ("model", DecisionTreeClassifier(random_state=0))])
folds = StratifiedKFold(n_splits=5, shuffle=True, random_state=0)
```

A pipeline's settings include every step's settings. Each one's name is the step's name, two underscores, then the setting: `model__max_depth`, read "model, double underscore, max depth", is the `max_depth` of the step called `model`.

```python
print(pipeline.get_params()["model__max_depth"])
pipeline.set_params(model__max_depth=4)
print(pipeline.named_steps["model"])
```

```text
None
DecisionTreeClassifier(max_depth=4, random_state=0)
```

## The grid and the search

A **grid** is a dictionary: each setting's name, with a list of values to try. `GridSearchCV` (grid search with cross-validation) tries every combination, the way a table's rows and columns make every pairing:

```python
grid = {
    "model__max_depth": [2, 3, 4, 5, 6, 8, None],
    "model__min_samples_leaf": [1, 10, 30],
}
search = GridSearchCV(pipeline, grid, cv=folds, scoring="accuracy")
search.fit(X_train, y_train)
print("Settings tried:", len(search.cv_results_["params"]))
print("Best settings:", search.best_params_)
print("Best mean accuracy:", round(search.best_score_, 3))
```

```text
Settings tried: 21
Best settings: {'model__max_depth': 4, 'model__min_samples_leaf': 1}
Best mean accuracy: 0.845
```

![A grid of 21 squares, seven rows for the depths 2, 3, 4, 5, 6, 8 and None and three columns for the leaf sizes 1, 10 and 30, each square one setting the search scores on all five folds; the square for depth 4 and leaf size 1 is marked as the winner, 0.845, and the square for depth None and leaf size 1 is marked as the lowest, 0.765](https://fsdtrainingmedia.blob.core.windows.net/public/images/2026/11/c9feb5f3f30748a1a818a24ee28bc8b0.webp)

`min_samples_leaf` is the fewest training bookings a leaf may hold. Seven depths times three leaf sizes is 21 settings. `search.fit` gets the training rows only: the search cuts its own folds out of them, so the test bookings stay unseen. `scoring="accuracy"` names the score it ranks by; for a classifier, that's also what it uses when you leave `scoring` out. `best_params_` holds the winner's settings, and `best_score_` its mean accuracy over the five folds.

A name the pipeline doesn't have stops the search before it starts:

```python
wrong = GridSearchCV(pipeline, {"max_depth": [2, 4]}, cv=folds)
wrong.fit(X_train, y_train)
```

```text
ValueError: Invalid parameter 'max_depth' for estimator Pipeline(steps=[('prep',
                 ColumnTransformer(remainder='passthrough',
                                   transformers=[('onehotencoder',
                                                  OneHotEncoder(handle_unknown='ignore'),
                                                  ['channel'])])),
                ('model', DecisionTreeClassifier(max_depth=4, random_state=0))]). Valid parameters are: ['memory', 'steps', 'transform_input', 'verbose'].
```

The pipeline itself has no `max_depth`; its `model` step does. The fix is `model__max_depth`.

## Reading every result

`cv_results_` keeps every setting's scores, and turns into a table with pandas. Here are the top four and the bottom three, by rank:

```python
results = pd.DataFrame(search.cv_results_)
columns = ["param_model__max_depth", "param_model__min_samples_leaf", "mean_test_score", "rank_test_score"]
ranked = results[columns].sort_values("rank_test_score").round(3)
print(pd.concat([ranked.head(4), ranked.tail(3)]).to_string(index=False))
```

```text
param_model__max_depth  param_model__min_samples_leaf  mean_test_score  rank_test_score
                     4                              1            0.845                1
                     4                             10            0.845                1
                     3                              1            0.843                3
                     3                             30            0.843                3
                  None                             10            0.828               19
                     8                              1            0.812               20
                  None                              1            0.765               21
```

Read two things from it. First, `mean_test_score` is the score on each fold's held-back rows, averaged: "test" here means the folds, not your test set. Second, the top is crowded: two settings tie at rank 1, and the next ones are a tenth of a point behind. Here `best_params_` holds the first of the two the search tried. Differences that small are no reason to believe one setting is really better; *Comparing Models Fairly* says what to do about that. The bottom is clear, though: the tree with no limit and leaves of one booking is far behind.

## Refit, then one test score

When the search finishes, it trains the best settings once more on all 2,400 training bookings: **refitting** (`refit=True` is the default). That model is `best_estimator_`, and the search's own `predict` and `score` use it:

```python
best = search.best_estimator_
print(best.named_steps["model"])
print("Trained on:", len(X_train), "bookings")
print("Guesses for 3 new bookings:", search.predict(X_test.head(3)))
```

```text
DecisionTreeClassifier(max_depth=4, random_state=0)
Trained on: 2400 bookings
Guesses for 3 new bookings: [0 0 0]
```

Then, once, the test bookings:

```python
print("Test accuracy:", round(search.score(X_test, y_test), 3))
```

```text
Test accuracy: 0.837
```

A little below the best cross-validated score, and that's normal: 0.845 was the best of 21 tries, so luck helped it. The test score is the honest one; report it as it is, and don't go back to change the grid because of it.

A grid of five settings with ten values each is 100,000 combinations. `RandomizedSearchCV` tries a fixed number of them instead, `n_iter`, picked at random from the lists you give it (set its `random_state` too), and is the usual choice when a grid gets that big.

## Check yourself

2 questions to try on the lesson page.

## So what

A grid search does the tuning loop for you, scored by cross-validation on the training rows, and hands back the best settings already refit, so the test set is looked at once, at the end.

## Sources

- scikit-learn, [Tuning the hyper-parameters of an estimator](https://scikit-learn.org/stable/modules/grid_search.html), User Guide: `GridSearchCV` exhaustively generates candidates from `param_grid`, evaluates every combination when fit and retains the best; parameters of composite estimators such as a `Pipeline` are searched with the `<estimator>__<parameter>` syntax; by default the search uses the estimator's `score` (accuracy for classification), and `scoring` chooses another; evaluate the result on held-out samples not seen during the search; `RandomizedSearchCV` samples settings, with a budget set by `n_iter`.
- scikit-learn 1.9.1, [GridSearchCV](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html), API reference: `refit=True` by default, refitting an estimator with the best found parameters on the whole dataset as `best_estimator_`, so `predict` works on the search; `best_params_`, `best_score_` (the mean cross-validated score of the best estimator) and `cv_results_` (with `mean_test_score`, `std_test_score`, `rank_test_score` and `params`, importable into a pandas DataFrame); `score` uses the best estimator; `cv` accepts a splitter.
- scikit-learn, [Pipelines and composite estimators](https://scikit-learn.org/stable/modules/compose.html), User Guide, *Access to nested parameters*: a pipeline's step parameters are reached with the `<estimator>__<parameter>` syntax, such as `set_params(clf__C=10)`, which matters most for grid searches.
- scikit-learn 1.9.1, [RandomizedSearchCV](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.RandomizedSearchCV.html), API reference: not all parameter values are tried; a fixed number of settings, `n_iter`, is sampled; lists are sampled uniformly; `random_state` makes the sampling reproducible.

## Key terms

- [Grid Search](https://www.ask-codey.com/glossary/grid-search)
- [GridSearchCV](https://www.ask-codey.com/glossary/gridsearchcv)
- [Hyperparameter](https://www.ask-codey.com/glossary/hyperparameter)
- [Cross-Validation](https://www.ask-codey.com/glossary/cross-validation)
- [Pipeline](https://www.ask-codey.com/glossary/pipeline-python)

## Related

- [Hyperparameters: Settings You Choose](https://www.ask-codey.com/lessons/ml-hyperparameters)
- [Pipelines: Every Step, Every Time](https://www.ask-codey.com/lessons/ml-pipelines)
- [Comparing Models Fairly](https://www.ask-codey.com/lessons/ml-comparing-models-fairly)
- [Exercise: Fair Model Showdown](https://www.ask-codey.com/lessons/exercise-ml-fair-model-showdown)

Source: https://www.ask-codey.com/lessons/ml-grid-search
