Exercise: Honest Scores

Write three functions that score a model with shuffled, seeded cross-validation, put its training score beside its validation score, and choose a tree's depth by cross-validation without ever touching the test showings, on Lakeside Cinema's 900 showings, and let pytest check them.

Journeyman · AI Basics · Python · Updated

By

Lakeside Cinema wants to guess how many tickets each showing will sell, and before it relies on a model it wants a score it can trust. You'll write the three functions behind an honest score: one that scores a model with shuffled cross-validation, one that puts a model's training score beside its validation score to spot memorizing, and one that chooses a tree's depth by cross-validation, using only the rows it's given. The test showings are put away before any of them runs. Tests call each function and check what it gives back, and a notebook comes with the exercise for looking at the data while you work.

Get the exercise code

Open this lesson's GitHub repository, github.com/ask-codey/exercise-ml-honest-scores, also linked from the Exercise box beside the lesson. Clone it into your python-course folder. In VS Code's terminal:

cd ~/python-course
git clone https://github.com/ask-codey/exercise-ml-honest-scores.git

Git makes a folder, exercise-ml-honest-scores, with the exercise code inside. Open that folder in VS Code (File > Open Folder), open its terminal with Terminal > New Terminal, and make the exercise's virtual environment. On Windows (PowerShell):

python -m venv .venv
.venv\Scripts\Activate.ps1
pip install -r requirements.txt

On a Mac (zsh):

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

The prompt starts with (.venv) once the virtual environment is on. The exercise's README says what each command does, and what to do if PowerShell won't run the second one.

The folder holds:

  • showings.csv: the 900 showings, our own made-up data, in date order, the same file the chapter's lessons use.
  • seed_data.py: the script that made the CSV, with the pattern it planted. You don't need to run it: the committed CSV is what keeps everyone's numbers the same.
  • honest_scores.py: the three functions, for you to finish, with FEATURES (the five number columns the models look at) and LABEL (tickets_sold), ready to use.
  • explore.ipynb: a notebook for looking at the showings and trying your code. Choose the .venv kernel with Select Kernel (top right) before you run it. The tests don't read it.
  • report.py: puts 180 showings away as the test set, then scores trees on the other 720 with your three functions. It never scores the test showings. Ready to use.
  • tests/: the tests, which call each function and check what comes back, and conftest.py, which gives each test a fresh copy of the showings and the same 720 training showings report.py uses. You don't need to change them.
  • requirements.txt and pytest.ini: the packages the exercise needs, at exact versions, and where pytest finds the tests and your code.

What to do

Every function in honest_scores.py is already declared with its name and parameters, and for now it stops with a NotImplementedError that says Not built yet. The docstring under each one says what it should do, with examples. In each function, replace the raise NotImplementedError("Not built yet.") line with your own code:

  1. fold_scores(model, X, y, k, seed) makes the folds with KFold(n_splits=k, shuffle=True, random_state=seed), scores the model on them with cross_val_score, and returns the k scores as a plain list (.tolist()). Worked through: X holds 1 to 10 and y is 2 × x + 1. A LinearRegression learns that line exactly from any 8 of the 10 rows and guesses the other 2 perfectly, so R² is 1.0 in every fold: fold_scores(LinearRegression(), X, y, 5, 0) returns [1.0, 1.0, 1.0, 1.0, 1.0]. That example is one of the tests.
  2. train_and_validation_scores(model, X, y, seed) splits the rows with train_test_split(X, y, test_size=0.25, random_state=seed), fits the model on the bigger part, and returns its score on both parts as {"train": ..., "validation": ...}. On the 720 training showings, that's 540 rows to learn from and 180 for validation. A DummyRegressor() always guesses the average of the rows it learned from, and R² compares a model with exactly that guess, so its "train" score is 0.0.
  3. best_depth(depths, X, y, k, seed) goes through depths in order. For each one it makes a DecisionTreeRegressor(max_depth=depth, random_state=seed), scores it with your fold_scores(tree, X, y, k, seed), and keeps the mean of those k scores. It returns two things: the depth with the highest mean, and a dictionary of each depth to its mean, in the order depths gave them. If two depths tie, it returns the one listed first: on a table where one yes-or-no question gives every answer, depths 2, 1 and 3 all score 1.0, so best_depth([2, 1, 3], ...) returns (2, {2: 1.0, 1: 1.0, 3: 1.0}).

best_depth is never given the test showings, and it doesn't need them: it chooses from the training rows alone, the way Validation Sets: Keep the Test Set for Last does.

Check your work

Run the tests from the exercise folder, with the virtual environment on:

python -m pytest

At first all 14 tests fail. That's expected: each function you finish turns its tests green. A failing test's message says what it called, what it expected and what came back, for example best_depth([2, 1, 3], ...) on a three-way tie should return 2, the first listed, but it returned 1. The tests check:

  • fold_scores: the worked example, that it returns a list of k plain numbers, that its scores are the shuffled KFold's for k 5 with seed 0 and for k 3 with seed 7, and that the same seed gives the same scores twice;
  • train_and_validation_scores: an exact line scoring 1.0 on both parts, the average guess scoring 0.0 on its own rows, and that both scores come from the 75% / 25% split for seeds 42 and 3;
  • best_depth: the tie example, a mean for depths 1, 5 and 10 in order, that each mean is the mean of that depth's seeded fold scores for seeds 0 and 7, and that the depth it returns has the highest of its own means.

No test checks how good a model is: every random step has a fixed seed, so each test compares your numbers with the same calls made in the test.

To see your functions at work on the whole file, run python report.py.

Hints

  • Look before you write. Run the notebook's first cells: the averages of the first and last 150 showings show the slow rise.
  • Folds go in the cv. Cross-Validation has cross_val_score(model, X, y, cv=folds) with a seeded KFold.
  • Fit on one part, score both. Overfitting and Underfitting puts score on the training rows beside score on held-back rows.
  • Reuse your own function. best_depth can call fold_scores for each depth, and sum(scores) / len(scores) is the mean.
  • Don't round. The tests compare your scores with scikit-learn's own, to many decimal places.

When you're done

Every test is green: pytest's last line says 14 passed. And python report.py prints:

1) training showings: 720
2) test showings, put away: 180
3) max_depth 5: training R² 0.78, validation R² 0.71
4) max_depth None: training R² 1.00, validation R² 0.51
5) fold scores, max_depth 5: [0.69, 0.76, 0.67, 0.67, 0.7]
6) mean 0.70, spread (standard deviation) 0.03
7) mean R² by depth: {1: 0.33, 2: 0.56, 3: 0.62, 4: 0.66, 5: 0.7, 6: 0.69, 7: 0.64, 8: 0.59, 9: 0.56, 10: 0.52}
8) best depth: 5

Lines 3 and 4 are the memorizer again: a perfect training score beside a validation score of 0.51. Now try a few changes in the notebook: best_depth with seeds 1 and 2, and fold_scores with k set to 10. Does the best depth stay the same, and how does the spread change?

Submit your exercise

The exercise is how you finish this lesson. When your work is done, put it on GitHub in a repo of your own, then submit that repo's address here. Hand In Your Work with Git walks through each step.

  1. On GitHub, click +, then New repository. Name it (for example exercise-ml-honest-scores), choose Public, and click Create repository. Leave out the README: your folder already has one.

  2. In VS Code's terminal, in your exercise folder, send your work to your new repo. Point your copy at your own repo, then commit and push:

    git remote set-url origin https://github.com/your-name/exercise-ml-honest-scores.git
    git add .
    git commit -m "My solution"
    git push -u origin main
    
  3. Paste your repo's address, such as https://github.com/your-name/exercise-ml-honest-scores, into Submit your exercise on this page. The site checks that the repo is public and holds your commits, and then the lesson is complete.

Changed something after you submitted? Push it, and submit again: your latest submission is the one that counts.

So what

Seeded folds, a training score beside a validation score, and a choice made without the test set: that's how you get a score you can trust, and each piece is small enough to test.

Key terms

Related