The Simplest Guess: Baselines
A baseline is the simplest guess there is, always the training data's average for a number or its most common answer for a category, and every model has to beat it on the test data by enough to be worth the trouble.
Apprentice · AI Basics · Python · Updated
After this lesson you can
- Work out a baseline guess from the training data, the average for a number or the most common answer for a category
- Compare a model's score with its baseline's on the same test data, and decide whether the model is worth using
Key takeaways
- A baseline is the simplest possible guess, worked out from the training data alone, the average for a number and the most common answer for a category.
- A model is only useful if it beats the baseline on the test data, and by enough to be worth building and looking after.
- A baseline can look good on its own. Always guessing not busy is right on 11 of 14 test days, yet it never warns about a single busy day.
- scikit-learn's DummyRegressor and DummyClassifier make these baselines, and they ignore the features completely.
Training and Testing: Never Grade Your Own Homework found that the weekday averages miss Sunrise Bakery's test days by about 7 loaves on average. Is that good? A number on its own can't say. Seven loaves could be a great score for a busy bakery or a poor one for a small cart. Whether you're judging a model at work or your own model in a class project, you need something to compare it with, and the fairest thing is the laziest guess anyone could make.
The example is the bakery's bakery_days.csv, from Exercise: How Many Loaves?, split as before: the first 106 days for training and the last 14 for testing.
The laziest guess for a number
Imagine guessing tomorrow's loaves without looking at anything about tomorrow: not the weekday, not the weather. The best you can do is the usual amount, the average of the training days. That guess is a baseline: the simplest possible guess, the score any real model has to beat.
import csv
with open("bakery_days.csv", newline="") as file:
days = list(csv.DictReader(file))
train = days[:-14]
test = days[-14:]
train_loaves = [int(day["loaves_sold"]) for day in train]
baseline = sum(train_loaves) / len(train_loaves)
print("Always guess:", round(baseline, 1))
misses = [abs(int(day["loaves_sold"]) - baseline) for day in test]
print("Average miss on the 14 test days:", round(sum(misses) / len(misses), 1))
Always guess: 100.0
Average miss on the 14 test days: 16.6
The baseline guesses 100 loaves every day and misses by 16.6 on average. The weekday averages missed by 6.9 on the same 14 days. Now the 6.9 means something: knowing the day of the week cuts the average miss by more than half.
Notice where the baseline came from: the training days only, the same as the model. Working it out from all 120 days would let the test days help make the guess they're scored on, the same mistake as grading your own homework.
The laziest guess for a category
Say the owner only wants to know whether tomorrow will be busy, 120 loaves or more, so there's time to call in a second baker. That's a category, and the average of "busy" and "not busy" makes no sense. The baseline for a category is the most common answer in the training data. Python's Counter, from the collections module, counts how often each answer appears, and most_common() lists them from most to least:
from collections import Counter
answers = ["busy" if int(day["loaves_sold"]) >= 120 else "not busy" for day in train]
print(Counter(answers).most_common())
test_answers = ["busy" if int(day["loaves_sold"]) >= 120 else "not busy" for day in test]
right = sum(1 for answer in test_answers if answer == "not busy")
print("Always guessing not busy is right on", right, "of", len(test_answers), "test days")
[('not busy', 83), ('busy', 23)]
Always guessing not busy is right on 11 of 14 test days
Right on 11 of 14 days, about 79%, sounds respectable. But this baseline never once says "busy", so it misses every busy day, which is the whole reason the owner asked. A model has to beat this, and Accuracy in Module 3 shows why a score like 79% can flatter a model that's no help at all.

Better than the baseline, and by how much
Every model must beat its baseline on the test data. A model that doesn't is worse than no model: it's more work, more code to look after, and its guesses are no better than "the usual". When a carefully built model scores no better than the baseline, something is wrong, such as features that don't hold the answer, or a mistake in the code.
Beating it isn't enough on its own, either. A model that misses by 16.2 loaves instead of 16.6 has won, but four-tenths of a loaf a day won't pay for the time it takes to build, check and keep up to date. How much better is worth the trouble depends on what a miss costs: a few stale loaves are cheap, and a hospital's wrong guess is not. That's a judgment, made in the label's own units, and it belongs in any write-up of a model.
In your textbook, a baseline may be called a naive model, a null model or a benchmark. scikit-learn makes these guesses with DummyRegressor, which by default always guesses the training data's average, and DummyClassifier, which by default always guesses its most common answer. Both ignore the features completely. Baselines and Residual Plots in Module 2 uses them.
Do it: work out both baselines
The owner wants both baselines ready for any stretch of training days: a number of loaves, and an answer to "busy or not?".
The inputs:
train_loaves: the loaves sold on each training day, a list of whole numbers.train_answers: the answer on each training day, a list of strings such as"busy"and"not busy".
For example, with four days' loaves and three days' answers:
train_loaves = [91, 92, 85, 96]
train_answers = ["not busy", "busy", "not busy"]
The loaves add up to 91 + 92 + 85 + 96 = 364, and 364 ÷ 4 = 91.0, so average_baseline(train_loaves) returns 91.0. "not busy" appears twice and "busy" once, so most_common_baseline(train_answers) returns "not busy".
Your job: write two functions.
average_baseline(train_loaves)returns the average oftrain_loaves, rounded to 1 decimal place withround(average, 1).most_common_baseline(train_answers)returns the answer that appears most often intrain_answers.- If two answers appear equally often,
most_common_baselinereturns the one that appears first in the list.
Practiced live on the lesson page: write the Python right there and check it.
Check yourself
2 questions to try on the lesson page.
So what
A model's score means something only next to its baseline's, on the same test data: the simplest guess sets the bar, and a model has to clear it by enough to be worth keeping.
Sources
- scikit-learn, Metrics and scoring: Dummy estimators, User Guide: comparing a model with simple rules of thumb is a sanity check;
most_frequentalways predicts the most frequent label in the training set,meanalways predicts the training targets' mean, and in every strategypredictignores the input data; a classifier whose accuracy is close to a dummy's probably means something went wrong. - scikit-learn 1.9.1, DummyRegressor, API reference: a simple baseline to compare with other regressors, not for real problems;
strategydefaults to"mean", the mean of the training set. - scikit-learn 1.9.1, DummyClassifier, API reference: a simple baseline that ignores the input features;
strategydefaults to"prior", whosepredictreturns the most frequent class in theypassed tofit, as"most_frequent"does. - Python documentation, collections.Counter:
most_common()lists elements from the most common to the least, and elements with equal counts are ordered in the order first encountered.