Training and Testing: Never Grade Your Own Homework
A model scored on the examples it learned from can look perfect and still be useless, so some examples are held back as test data that it never sees while it trains, and when the data is in time order, the most recent days are the ones held back.
Apprentice · AI Basics · Python · Updated
After this lesson you can
- Explain why a model must be scored on test data it never saw while it trained, and not on its training data
- Hold back the most recent examples as test data when the data is in time order, and measure a model's average miss on them
Key takeaways
- A model can score perfectly on the examples it learned from by memorizing them, and still have nothing to say about a new day.
- Training data is what a model learns from. Test data is held back, never seen during training, and used only to score its guesses.
- The score that tells you how a model will do is its score on the test data, the examples it never saw.
- When the examples are in time order, hold back the most recent ones, so the model is tested on days that came after everything it learned from.
The weekday averages from Guessing from Examples give Sunrise Bakery's owner a number for every day of the week. Before the owner bakes by them, there's an obvious question: how good are they? Every model you'll ever build or use, a sales forecast at work or a project for a class, has to answer it, and there's a wrong way to answer it that looks right.
The example is the bakery's bakery_days.csv again, from Exercise: How Many Loaves?.
The perfect score that means nothing
A student who memorized last year's answer key scores 100% on last year's test, and might know nothing about this year's. A model can do the same. Here's a "model" that memorizes: it stores each past day's sales by day number, and its guess for a day is whatever that day sold. Scored on the days it learned from:
past = {1: 74, 2: 96, 3: 93, 4: 98, 5: 96, 6: 143, 7: 117}
def memorizer(day_number):
return past.get(day_number)
misses = [abs(past[number] - memorizer(number)) for number in past]
print("Average miss on the days it learned from:", sum(misses) / len(misses))
print("Guess for day 8:", memorizer(8))
Average miss on the days it learned from: 0.0
Guess for day 8: None
A miss is how far a guess is from what really happened, counted as a positive number: abs() (read "absolute value") drops the minus sign, so being 5 loaves too high and 5 too low are both a miss of 5. The memorizer misses by 0.0 on its own days: a perfect score. On day 8, the first day it hasn't seen, it has no guess at all.
Grading a model on its own examples is grading your own homework. A model that learns its examples too closely, wobbles and all, is overfitting: it does well on the examples it learned from and worse on new ones. Most models don't overfit as plainly as the memorizer, which is exactly why you can't see it without a test.
Training data and test data
The fix is to set some examples aside before the model learns anything. The examples the model learns from are the training data. The ones set aside are the test data: the model never sees them while it learns, and they're used only to score its guesses. The score on the test data is the one that tells you how the model will do on days to come.
The bakery's days are in time order: day 1, then day 2, up to day 120. So the days held back are the most recent ones, here the last two weeks:
import csv
with open("bakery_days.csv", newline="") as file:
days = list(csv.DictReader(file))
train = days[:-14]
test = days[-14:]
print("Training days:", len(train))
print("Test days:", len(test), "from day", test[0]["day"], "to day", test[-1]["day"])
Training days: 106
Test days: 14 from day 107 to day 120
days[:-14] is every day except the last 14, and days[-14:] is the last 14: negative positions count from the end of the list.

Why not pick 14 days from all over the file? In real life, tonight's guess is about tomorrow, and nothing from tomorrow is known yet. A model trained on day 115 and tested on day 110 got to peek at the future, which no real guess ever can. Holding back the most recent days keeps the test honest: every test day comes after every training day, as tomorrow does.
Scoring the weekday averages honestly
Now the weekday averages are learned from the 106 training days only, and each test day gets the average for its weekday as its guess. The average miss adds up the misses and divides by how many there are:
def weekday_averages(rows):
totals = {}
counts = {}
for row in rows:
weekday = row["weekday"]
totals[weekday] = totals.get(weekday, 0) + int(row["loaves_sold"])
counts[weekday] = counts.get(weekday, 0) + 1
return {weekday: totals[weekday] / counts[weekday] for weekday in totals}
model = weekday_averages(train)
def average_miss(rows):
misses = [abs(int(row["loaves_sold"]) - model[row["weekday"]]) for row in rows]
return round(sum(misses) / len(misses), 1)
print("Average miss on the 106 training days:", average_miss(train))
print("Average miss on the 14 test days:", average_miss(test))
Average miss on the 106 training days: 8.1
Average miss on the 14 test days: 6.9
On days it never saw, a guess is off by about 7 loaves on average: some days more, some less. Its worst test day was day 113, a holiday Monday, when it guessed about 94 loaves and the bakery sold 117: the weekday averages know nothing about holidays.
This time the test score is a little better than the training score. That's luck of these 14 days, not a sign the model improved: a small test set wobbles. A model this simple can't memorize, so its two scores land close together; a big gap, like the memorizer's, is what overfitting looks like. Your textbook calls the average miss the mean absolute error; Errors and the Mean Absolute Error takes it further, and Splitting Data with train_test_split does the holding back with scikit-learn when the rows aren't in time order.
Do it: hold back the last days and score a guess
The simplest model of all guesses the same number every day: the average of the training days. The owner wants to know how far off it is on the most recent days, held back as test data.
The inputs:
loaves: the loaves sold on each day, a list in time order, oldest first.n: how many of the most recent days to hold back as test data, at least 1 and fewer than the number of days.
For example:
loaves = [90, 96, 84, 135, 120, 92, 88]
n = 2
| Step | Work | Result |
|---|---|---|
| Training days | the first 5: 90, 96, 84, 135, 120 | |
| The guess | (90 + 96 + 84 + 135 + 120) ÷ 5 = 525 ÷ 5 | 105.0 |
| Test days | the last 2: 92 and 88 | |
| Misses | 92 − 105.0 is −13.0, a miss of 13.0; 88 − 105.0 is −17.0, a miss of 17.0 | 13.0 and 17.0 |
| Average miss | (13.0 + 17.0) ÷ 2 | 15.0 |
So average_guess_miss(loaves, 2) returns 15.0.
Your job: write average_guess_miss(loaves, n).
- The test days are the last
ndays ofloaves; the training days are all the days before them. - The guess is the average of the training days only. The test days never go into it.
- Return the average miss of that guess on the test days, where a day's miss is the distance between its loaves and the guess, as a positive number, rounded to 2 decimal places with
round(average, 2).
Practiced live on the lesson page: write the Python right there and check it.
Check yourself
2 questions to try on the lesson page.
So what
A score on the examples a model learned from can be perfect and mean nothing; hold examples back first, the most recent ones when the data is in time order, and trust only the score on those.
Sources
- scikit-learn, Cross-validation: evaluating estimator performance, User Guide, its opening: learning a model and testing it on the same data is a mistake, because a model that repeats the labels it has seen would score perfectly and predict nothing useful on unseen data (overfitting), so part of the data is held out as a test set.
- scikit-learn, Cross-validation: Cross validation of time series data, User Guide: observations near in time are related, so a model for time-ordered data should be evaluated on the "future" observations, after the ones it was trained on, which
TimeSeriesSplitdoes. - scikit-learn 1.9.1, Getting Started, Model evaluation: fitting a model to some data does not mean it will predict well on unseen data, and that has to be evaluated directly.
- Python documentation, abs(): the absolute value of a number.