# Exercise: Fair Model Showdown

Write four functions that build a tree's settings grid with step names, set up a grid search scored by accuracy on the folds you're given, compare models on the same folds with each one's mean and spread, and score the chosen model once on the test bookings, and let pytest check the grid, the shared folds and that the test rows are never learned from.

Freelancer · AI Basics · Python · Updated November 19, 2026

By Gregor Dzierzon

The Harbor View Inn's manager wants one model to flag the bookings likely to cancel, and wants to trust the choice. You'll write the four functions behind a fair showdown: one that names a tree's settings for a grid search, one that sets the search up, one that scores every model on the same folds, and one that gives the chosen model its single look at the test bookings. Tests call each function and check what it gives back, and a notebook comes with the exercise for looking at the data while you work.

## Get the exercise code

Open this lesson's GitHub repository, [github.com/ask-codey/exercise-ml-fair-model-showdown](https://github.com/ask-codey/exercise-ml-fair-model-showdown), also linked from the **Exercise** box beside the lesson. Clone it into your `python-course` folder. In VS Code's terminal:

```bash
cd ~/python-course
git clone https://github.com/ask-codey/exercise-ml-fair-model-showdown.git
```

Git makes a folder, `exercise-ml-fair-model-showdown`, with the exercise code inside. Open that folder in VS Code (**File > Open Folder**), open its terminal with **Terminal > New Terminal**, and make the exercise's virtual environment. On Windows (PowerShell):

```powershell
python -m venv .venv
.venv\Scripts\Activate.ps1
pip install -r requirements.txt
```

On a Mac (zsh):

```bash
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
```

The prompt starts with `(.venv)` once the virtual environment is on. The exercise's README says what each command does, and what to do if PowerShell won't run the second one.

The folder holds:

- `bookings.csv`: the 3,000 bookings, our own made-up data, the same file the chapter's lessons use.
- `seed_data.py`: the script that made the CSV, with the pattern it planted. You don't need to run it: the committed CSV is what keeps everyone's numbers the same.
- `showdown.py`: the four functions, for you to finish, with `FEATURES`, `LABEL`, `split_bookings` (holds back a fifth of the bookings for the final test, as the lessons do) and `build_pipeline` (a pipeline whose steps are named `prep` and `model`), ready to use.
- `explore.ipynb`: a notebook for looking at the bookings and trying your code. Choose the `.venv` kernel with **Select Kernel** (top right) before you run it. The tests don't read it.
- `report.py`: runs the whole showdown with your four functions and prints what each step gave. Ready to use.
- `tests/`: the tests, which call each function and check what comes back, and `conftest.py`, which gives each test a fresh copy of the bookings. You don't need to change them.
- `requirements.txt` and `pytest.ini`: the packages the exercise needs, at exact versions, and where pytest finds the tests and your code.

## What to do

Every function in `showdown.py` is already declared with its name and parameters, and for now it stops with a `NotImplementedError` that says *Not built yet.* The docstring under each one says what it should do, with an example:

```python
def tree_grid(depths, leaf_sizes):
    """Return the grid for a grid search over a pipeline's decision tree, the step named "model".

    The grid is a dictionary with two keys: "model__max_depth", holding the depths to try,
    and "model__min_samples_leaf", holding the leaf sizes to try, each as a list in the
    order it was given.

    tree_grid([2, 4], [1, 10]) -> {"model__max_depth": [2, 4], "model__min_samples_leaf": [1, 10]}
    """
    raise NotImplementedError("Not built yet.")
```

In each function, replace the `raise NotImplementedError("Not built yet.")` line with your own code:

1. **`tree_grid(depths, leaf_sizes)`** returns a dictionary with the keys `"model__max_depth"` and `"model__min_samples_leaf"`, each holding its values as a list, in the order given. `tree_grid([2, 4], [1, 10])` gives `{"model__max_depth": [2, 4], "model__min_samples_leaf": [1, 10]}`, and `tree_grid([8, None, 3], [30, 1])` keeps `None` and that order.
2. **`make_search(pipeline, grid, folds)`** returns a `GridSearchCV` made with the pipeline and the grid it's given, `cv=folds` and `scoring="accuracy"`, every other setting left at its default, and **not fitted**. The tests check those four things and that it has no `best_params_` yet. Fitted on the 2,400 training bookings with `tree_grid([2, 3, 4, 5, 6, 8, None], [1, 10, 30])`, it tries 21 settings and picks `{"model__max_depth": 4, "model__min_samples_leaf": 1}`.
3. **`compare_models(models, X, y, folds)`** takes a dictionary of names to models, gets each model's fold scores with `cross_val_score(model, X, y, cv=folds, scoring="accuracy")`, and returns a dictionary of the same names, each to `{"mean": ..., "std": ...}`: the scores' `.mean()` and `.std()`, as plain floats, not rounded. Worked through: ten bookings labeled 0, 0, 0, 1, 0, 0, 1, 1, 0, 0, the folds `KFold(n_splits=5)` (each fold is two bookings in a row), and one model, `"baseline"`, a `DummyClassifier(strategy="most_frequent")`. Each time, the eight training bookings hold more 0s than 1s, so the baseline guesses 0. The five folds' accuracies are 1.0, 0.5, 1.0, 0.0 and 1.0: their mean is 0.7, and their standard deviation is 0.4. The answer is `{"baseline": {"mean": 0.7, "std": 0.4}}`. The tests also check a tree and a baseline on the real bookings against `cross_val_score`, and that both were trained on exactly the folds given.
4. **`final_report(search, X_test, y_test)`** takes a search that's already fitted on the training rows, fits nothing, and returns `{"best_params": ..., "cv_accuracy": ..., "test_accuracy": ...}`: the search's `best_params_`, its `best_score_` as a plain float, and `search.score(X_test, y_test)` as a plain float, nothing rounded. The docstring's example, a search over depths 1 and 2 on eight bookings, gives `{"best_params": {"model__max_depth": 1}, "cv_accuracy": 1.0, "test_accuracy": 0.75}`.

The last test runs the whole showdown, search, comparison and report, with a step that writes down the bookings every model is fitted on, and checks that none of the 600 test bookings is ever among them.

## Check your work

Run the tests from the exercise folder, with the virtual environment on:

```bash
python -m pytest
```

At first all 14 tests fail. That's expected: each function you finish turns its tests green. A failing test's message says what it called, what it expected and what came back, for example *tree_grid([2, 4], [1, 10]) should return {'model__max_depth': [2, 4], 'model__min_samples_leaf': [1, 10]}, but it returned {'max_depth': [2, 4], 'min_samples_leaf': [1, 10]}.* No test checks how accurate a model is: the tests check the grid, the search's settings, the shared folds, and that the test rows are only scored.

To see your functions at work on the whole file, run `python report.py`.

## Hints

- **Look before you write.** Run the notebook's first cells: the share of canceled bookings and each channel's cancellation rate.
- **Step, two underscores, setting.** *Searching for Good Settings* names the tree's depth inside a pipeline `model__max_depth`.
- **Make it, don't fit it.** `make_search` is one line: `GridSearchCV(...)` with the pipeline, the grid, `cv=` and `scoring=`.
- **One splitter for everyone.** In `compare_models`, every model gets the same `folds`, as *Comparing Models Fairly* does. A `for name, model in models.items():` loop builds the answer.
- **The search already refit.** `search.score` uses the model the search trained on all the training rows, as *Searching for Good Settings* shows. Fitting again on the test rows would teach the model the answers it's about to be graded on.

## When you're done

Every test is green: pytest's last line says `14 passed`. And `python report.py` prints:

```text
1) bookings: 3000
2) training bookings: 2400
3) test bookings: 600
4) settings tried: 21
5) best settings: {'model__max_depth': 4, 'model__min_samples_leaf': 1}
6) baseline (most frequent): mean 0.758, spread 0.001
7) logistic regression: mean 0.826, spread 0.011
8) tuned tree: mean 0.845, spread 0.013
9) forest, depth 6: mean 0.846, spread 0.012
10) chosen settings: {'model__max_depth': 4, 'model__min_samples_leaf': 1}
11) cross-validated accuracy: 0.845
12) test accuracy, scored once: 0.837
```

Lines 8 and 9 differ by less than either spread, so the tuned tree is kept: it's simpler, and you can read it. Now write two sentences for the inn's manager in a Markdown cell at the end of the notebook: how often the model is right on bookings it never saw, and how that compares with always guessing "not canceled". In the notebook, change the folds' `random_state` from 0 to 1 and compare again: which lines move, and does the order change?

## Submit your exercise

The exercise is how you finish this lesson. When your work is done, put it on GitHub in a repo of your own, then submit that repo's address here. *Hand In Your Work with Git* walks through each step.

1. On GitHub, click **+**, then **New repository**. Name it (for example `exercise-ml-fair-model-showdown`), choose **Public**, and click **Create repository**. Leave out the README: your folder already has one.
2. In VS Code's terminal, in your exercise folder, send your work to your new repo. Point your copy at your own repo, then commit and push:

   ```bash
   git remote set-url origin https://github.com/your-name/exercise-ml-fair-model-showdown.git
   git add .
   git commit -m "My solution"
   git push -u origin main
   ```

3. Paste your repo's address, such as `https://github.com/your-name/exercise-ml-fair-model-showdown`, into **Submit your exercise** on this page. The site checks that the repo is public and holds your commits, and then the lesson is complete.

Changed something after you submitted? Push it, and submit again: your latest submission is the one that counts.

## So what

A settings grid with the right names, a search fitted on the training rows, one set of folds for every model and one look at the test bookings: that's a model choice you can defend, and each step is small enough to test.

## Key terms

- [GridSearchCV](https://www.ask-codey.com/glossary/gridsearchcv)
- [Cross-Validation](https://www.ask-codey.com/glossary/cross-validation)
- [Hyperparameter](https://www.ask-codey.com/glossary/hyperparameter)
- [Baseline](https://www.ask-codey.com/glossary/baseline-ai)
- [pytest](https://www.ask-codey.com/glossary/pytest)

## Related

- [Hyperparameters: Settings You Choose](https://www.ask-codey.com/lessons/ml-hyperparameters)
- [Searching for Good Settings](https://www.ask-codey.com/lessons/ml-grid-search)
- [Random Forests](https://www.ask-codey.com/lessons/ml-random-forests)
- [Comparing Models Fairly](https://www.ask-codey.com/lessons/ml-comparing-models-fairly)

Source: https://www.ask-codey.com/lessons/exercise-ml-fair-model-showdown
