Comparing Models Fairly

A fair comparison scores every model on the same training rows with one seeded splitter, one metric and a baseline, puts each model's mean and spread side by side, treats a difference smaller than the spread as a tie that goes to the simpler model, and ends with one test-set score for the chosen model, reported as it is.

Freelancer · AI Basics · Python · Updated

By

After this lesson you can

  • Compare models fairly with one seeded splitter, one metric and a baseline, putting each model's mean and spread side by side
  • Decide when a difference is smaller than the spread, choose the simpler model on a tie, and report one final test score as it is

Key takeaways

  • Score every model on the same folds, from one splitter with a fixed random_state, with the same metric, on the training rows.
  • Put a baseline in the race. A model that can't beat always guessing the most common answer has learned nothing useful.
  • Report each model's mean and spread. A difference smaller than the spread may be luck in which rows fell in which fold.
  • When scores tie, keep the simpler model, then score it once on the test set and report that number as it is.

The Harbor View Inn's manager asks the question every model ends with: which one should we use? You have a logistic regression, a tuned tree and a forest. A comparison can look scientific and not be: score one model on lucky folds and another on hard ones, pick a different metric for each, or keep checking the test set until a favorite wins. In a project write-up or at work, the comparison is the part people trust you on.

The example is Harbor View Inn's bookings.csv, from Exercise: Fair Model Showdown. Each code block is one notebook cell, run in order: the first one loads the data, and the others use what it made.

One starting line for every runner

A fair race has one track, one stopwatch and one starting line. For models, that's the same training rows, one metric, and the same folds. The first cell sets up the data and two kinds of preparation: logistic regression cares about scale, so its numbers are standardized (Scaling Features); trees don't, so theirs pass through.

import pandas as pd
from sklearn.compose import make_column_transformer
from sklearn.dummy import DummyClassifier
from sklearn.ensemble import RandomForestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import StratifiedKFold, cross_val_score, train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler
from sklearn.tree import DecisionTreeClassifier

bookings = pd.read_csv("bookings.csv")
features = ["lead_time_days", "nights", "guests", "deposit", "channel", "repeat_guest", "special_requests"]
numbers = ["lead_time_days", "nights", "guests", "deposit", "repeat_guest", "special_requests"]
X_train, X_test, y_train, y_test = train_test_split(
    bookings[features], bookings["canceled"], test_size=0.2, stratify=bookings["canceled"], random_state=42
)
encode = make_column_transformer((OneHotEncoder(handle_unknown="ignore"), ["channel"]), remainder="passthrough")
encode_and_scale = make_column_transformer(
    (OneHotEncoder(handle_unknown="ignore"), ["channel"]), (StandardScaler(), numbers)
)

The race has four runners. The tree's depth and the forest's depth are the ones their grid searches chose (Searching for Good Settings). The fourth is the baseline: DummyClassifier(strategy="most_frequent") ignores the features and always guesses the training rows' most common answer, "not canceled". One splitter object, folds, made once with a fixed random_state, cuts the same five folds for every model:

models = {
    "baseline (most frequent)": Pipeline([("prep", encode), ("model", DummyClassifier(strategy="most_frequent"))]),
    "logistic regression": Pipeline([("prep", encode_and_scale), ("model", LogisticRegression())]),
    "tree, depth 4": Pipeline([("prep", encode), ("model", DecisionTreeClassifier(max_depth=4, random_state=0))]),
    "forest, depth 6": Pipeline([("prep", encode), ("model", RandomForestClassifier(n_estimators=200, max_depth=6, random_state=0))]),
}
folds = StratifiedKFold(n_splits=5, shuffle=True, random_state=0)
rows = []
fold_scores = {}
for name, model in models.items():
    scores = cross_val_score(model, X_train, y_train, cv=folds, scoring="accuracy")
    fold_scores[name] = scores
    rows.append({"model": name, "mean": scores.mean(), "spread": scores.std()})
print(pd.DataFrame(rows).set_index("model").round(3))
                           mean  spread
model
baseline (most frequent)  0.758   0.001
logistic regression       0.826   0.011
tree, depth 4             0.845   0.013
forest, depth 6           0.846   0.012

A chart of the four models' mean accuracy over the same five folds, each a dot with a bar one spread wide on each side: the baseline at 0.758, plus or minus 0.001; logistic regression at 0.826, plus or minus 0.011; the depth-4 tree at 0.845, plus or minus 0.013; and the depth-6 forest at 0.846, plus or minus 0.012, its bar lined up almost exactly with the tree's

The spread is the standard deviation of the five fold scores. Start with the baseline: 0.758 is the share of bookings not canceled. An accuracy of 80% sounds good until you see that guessing "not canceled" every time gets almost 76%. Every model beats it, so every model learned something.

Fold by fold

Because every model met the same folds, you can line them up fold by fold:

print(pd.DataFrame(fold_scores).round(3).to_string(index=False))
 baseline (most frequent)  logistic regression  tree, depth 4  forest, depth 6
                    0.758                0.844          0.844            0.852
                    0.758                0.821          0.844            0.854
                    0.758                0.821          0.831            0.829
                    0.756                0.833          0.869            0.860
                    0.756                0.812          0.835            0.833

Each row is one fold. The logistic regression never beats the tree: they tie on the first fold, and the tree wins the other four, by about two points on average, more than the spread. That gap has a reason: the seed script planted a pattern where a long lead time and no deposit together raise the chance of canceling more than each does alone, and a plain logistic regression adds up each feature's effect on its own. The tree and the forest are another story: the forest wins two folds and the tree three, and their means differ by 0.001.

Smaller than the spread

Is the forest better? Cut the folds with other seeds and look again:

for seed in [0, 1, 2, 3]:
    other_folds = StratifiedKFold(n_splits=5, shuffle=True, random_state=seed)
    tree_mean = cross_val_score(models["tree, depth 4"], X_train, y_train, cv=other_folds).mean()
    forest_mean = cross_val_score(models["forest, depth 6"], X_train, y_train, cv=other_folds).mean()
    print("folds seed", seed, "| tree", round(tree_mean, 3), "| forest", round(forest_mean, 3))
folds seed 0 | tree 0.845 | forest 0.846
folds seed 1 | tree 0.845 | forest 0.846
folds seed 2 | tree 0.842 | forest 0.847
folds seed 3 | tree 0.844 | forest 0.848

The forest stays a hair ahead, by 0.001 to 0.005: at most half a point, 3 bookings in 600, and less than half the spread. A difference that small is a tie for any practical purpose. And notice what a different seed for each model would do: score the tree on seed 2's folds and the forest on seed 3's, and the forest's lead grows to 0.006, bigger than on any folds they shared, for a reason that has nothing to do with the models. That's why one splitter, made once, scores them all.

When two models tie, keep the simpler one. The depth-4 tree trains in a moment, and you can print its questions and show them to the manager; the forest is 200 trees nobody can read. If the forest's lead were bigger than the spread, and held on every seed, it would earn its extra cost.

One honest caution: the tree's 0.845 and the forest's 0.846 each won a grid search on these same training rows, the best of several tries, so luck has lifted them a little. Textbooks call the fix nested cross-validation, a search inside each fold. The final test score is the simpler check.

One test score, reported as it is

The choice is made, so the test bookings come out, once:

chosen = models["tree, depth 4"]
chosen.fit(X_train, y_train)
print("Test accuracy:", round(chosen.score(X_test, y_test), 3))
Test accuracy: 0.837

A little below 0.845, as expected. Report 0.837, beside the baseline and the spread: "about 84% of bookings guessed right, against 76% for always guessing 'not canceled'." If the test score disappoints you, don't go back and pick another model: the test set has done its one job.

Check yourself

2 questions to try on the lesson page.

So what

A comparison is fair when every model runs the same race (the same rows, the same folds, the same metric, and a baseline), when a lead smaller than the spread counts as a tie that goes to the simpler model, and when the winner gets one test score, reported as it is.

Sources

  • scikit-learn, Common pitfalls and recommended practices, Controlling randomness: with an integer random_state, calling split again gives the same splits; comparing estimators on different splits is a mistake; for comparable fold-to-fold results, pass an integer to the splitter, such as KFold(shuffle=True, random_state=0); test data should never be used to make choices about the model.
  • scikit-learn 1.9.1, DummyClassifier, API reference: makes predictions that ignore the input features, as a simple baseline to compare other classifiers against; with strategy="most_frequent", predict always returns the most frequent class label seen in fit.
  • scikit-learn, Cross-validation: evaluating estimator performance, User Guide: cross_val_score gives one score per fold, reported as the mean with its standard deviation; a test set should still be held out for final evaluation.
  • scikit-learn, Tuning the hyper-parameters of an estimator, User Guide, Model selection: development and evaluation: evaluate the chosen model on held-out samples not seen during the search; nested cross-validation is the best practice for evaluating a model chosen by grid search.
  • scikit-learn 1.9.1, StratifiedKFold, API reference: folds that keep the percentage of samples of each class; random_state gives the same folds across calls when shuffle is true.

Key terms

Related