Cross-Validation

Cross-validation splits the rows into k folds and tests the model on each fold in turn, learning from the others, so every row is tested exactly once; cross_val_score gives the k scores, you report their mean and their spread, and KFold with shuffle=True and a random_state makes the folds random and repeatable.

Journeyman · AI Basics · Python · Updated

By

After this lesson you can

  • Assign rows to k folds the way KFold does, and explain how every row is tested exactly once
  • Score a model with cross_val_score and a shuffled, seeded KFold, and report the mean and the spread of the fold scores

Key takeaways

  • One split is one roll of the dice. Other rows held back give another score.
  • In k-fold cross-validation, each of the k folds takes one turn as the rows the model is tested on, while it learns from the rest, so every row is tested exactly once.
  • cross_val_score(model, X, y, cv=5) gives five scores. Report their mean and their spread, the standard deviation.
  • A number for cv makes folds in the rows' order, without shuffling, and for a classifier the folds keep each class's share (stratified).
  • KFold(n_splits=5, shuffle=True, random_state=0) shuffles before cutting the folds, and the seed makes the same folds on every run.

You built a model, held back a fifth of the rows, and it scored 0.69. A classmate makes the same model, holds back a different fifth, and gets 0.59. Which number is right? Both are honest, and neither is the whole story: each came from one particular set of held-back rows. When you report how good a model is, in a project write-up or to a manager, you want a number that doesn't hang on which rows happened to be held back, and how much it moves.

The example is Lakeside Cinema's showings.csv, from Exercise: Honest Scores. Each code block is one notebook cell, run in order: the first one loads the data, and the others use what it made.

One split is one roll of the dice

The 180 test showings stay put away. From the 720 training showings, here is the same depth-5 tree, held back on five different fifths:

import pandas as pd
from sklearn.model_selection import KFold, cross_val_score, train_test_split
from sklearn.tree import DecisionTreeRegressor

showings = pd.read_csv("showings.csv")
features = ["day_number", "weekday", "start_hour", "is_new_release", "screen_seats"]
X_train, X_test, y_train, y_test = train_test_split(
    showings[features], showings["tickets_sold"], test_size=0.2, random_state=42
)
tree = DecisionTreeRegressor(max_depth=5, random_state=0)

for seed in range(5):
    X_fit, X_check, y_fit, y_check = train_test_split(X_train, y_train, test_size=0.2, random_state=seed)
    tree.fit(X_fit, y_fit)
    print("random_state", seed, "| R² on the held-back showings:", round(tree.score(X_check, y_check), 2))
random_state 0 | R² on the held-back showings: 0.69
random_state 1 | R² on the held-back showings: 0.64
random_state 2 | R² on the held-back showings: 0.59
random_state 3 | R² on the held-back showings: 0.64
random_state 4 | R² on the held-back showings: 0.68

Same model, same showings, scores from 0.59 to 0.69. Some held-back showings are easier to guess than others.

Every row gets a turn

Picture a teacher grading a class's practice tests in five piles. Each pile takes one turn as the "test" while the other four are what was studied, until every pile has had its turn. That's k-fold cross-validation: cut the rows into k parts, called folds; train on k − 1 of them and score on the one left out; repeat k times, so every row is tested exactly once; average the k scores.

Without shuffling, the folds are cut in order. With 10 rows and 3 folds, the first fold gets the extra row:

for number, (fit_rows, check_rows) in enumerate(KFold(n_splits=3).split(range(10)), start=1):
    print("Fold", number, "| tested on rows", check_rows.tolist(), "| learns from rows", fit_rows.tolist())
Fold 1 | tested on rows [0, 1, 2, 3] | learns from rows [4, 5, 6, 7, 8, 9]
Fold 2 | tested on rows [4, 5, 6] | learns from rows [0, 1, 2, 3, 7, 8, 9]
Fold 3 | tested on rows [7, 8, 9] | learns from rows [0, 1, 2, 3, 4, 5, 6]

KFold gives the first (rows ÷ k's remainder) folds one row more than the others: 10 rows in 3 folds is 3 each with 1 left over, so fold 1 gets 4.

The 720 training showings drawn as a long bar cut into five folds of 144, shown five times, one row per round: in each round a different fold is shaded as the 144 showings tested, and the other four, 576 showings, are what the tree learns from; every fold is tested once, and the five scores are averaged

cross_val_score: the mean and the spread

cross_val_score does every round: it makes a fresh copy of the model for each fold, fits it on the other folds, and scores it with the model's own score, R² for a regression model.

scores = cross_val_score(tree, X_train, y_train, cv=5)
print("Scores:", scores.round(2))
print("Mean:", round(scores.mean(), 2))
print("Spread (standard deviation):", round(scores.std(), 2))
Scores: [0.73 0.67 0.7  0.64 0.72]
Mean: 0.69
Spread (standard deviation): 0.04

Report both numbers: "R² 0.69, give or take 0.04 from fold to fold". The standard deviation is the typical distance of a fold's score from the mean. A small spread says the score doesn't hang much on which rows were tested; a big one says one split could have told you almost anything.

cv=5 makes the folds without shuffling, cut in the rows' order. Here that's harmless, because train_test_split already shuffled the 720 training showings. And when the model is a classifier, such as one guessing whether a showing sells out, a number for cv makes stratified folds (StratifiedKFold), each keeping the classes' shares, as Stratified Splits explains.

Shuffled, seeded folds

To shuffle whatever order the rows come in, and get the same folds every run, hand cv a KFold:

folds = KFold(n_splits=5, shuffle=True, random_state=0)
scores = cross_val_score(tree, X_train, y_train, cv=folds)
print("Scores:", scores.round(2))
print("Mean:", round(scores.mean(), 2))
print("Spread (standard deviation):", round(scores.std(), 2))
Scores: [0.69 0.76 0.67 0.67 0.7 ]
Mean: 0.7
Spread (standard deviation): 0.03

Other folds, other scores; the mean barely moves, 0.69 to 0.70. random_state=0 is the shuffle's seed: leave it out, and each run makes different folds. Each round learns from 576 showings and is tested on 144.

Do it: deal showings into folds

Lakeside Cinema's analyst wants to check a cross-validation by hand, before trusting KFold. Write the folds the way KFold makes them without shuffling.

The inputs:

  • row_count: how many showings there are, numbered from 0.
  • k: how many folds to make.
  • folds: a list of folds, each a list of row numbers, as make_folds returns.
  • fold_index: the position of one fold in folds, from 0.

For example, 10 showings in 3 folds. 10 ÷ 3 is 3, with 1 left over, so the first fold gets one extra:

Fold Size Rows
0 3 + 1 = 4 0, 1, 2, 3
1 3 4, 5, 6
2 3 7, 8, 9

So make_folds(10, 3) returns [[0, 1, 2, 3], [4, 5, 6], [7, 8, 9]], and training_rows(make_folds(10, 3), 1) returns [0, 1, 2, 3, 7, 8, 9]: every row except fold 1's.

Your job: write two functions.

  • make_folds(row_count, k) returns k lists of row numbers. Every fold gets row_count // k rows, and the first row_count % k folds get one more. The rows go in order: fold 0 gets the first ones, then fold 1, and so on.
  • training_rows(folds, fold_index) returns every row number in the other folds, as one list, from the lowest to the highest.

Practiced live on the lesson page: write the Python right there and check it.

Check yourself

2 questions to try on the lesson page.

So what

One split gives one score that could have been a little different; cross-validation tests every row once and gives a mean with its spread, a number you can report and repeat.

Sources

  • scikit-learn, Cross-validation: evaluating estimator performance, User Guide: k-fold cross-validation splits the training set into k folds, trains on k − 1 and validates on the remaining one, and reports the average; with an integer cv, cross_val_score uses KFold, or StratifiedKFold for a classifier; the default score is the estimator's score method; the guide reports the scores' mean and standard deviation; with shuffle=True and no random_state the folds differ every time.
  • scikit-learn 1.9.1, cross_val_score, API reference: for an integer cv, StratifiedKFold when the estimator is a classifier and y is binary or multiclass, otherwise KFold, both made with shuffle=False; it returns an array of one score per fold.
  • scikit-learn 1.9.1, KFold, API reference: consecutive folds without shuffling by default; random_state controls the shuffle when shuffle is true; the first n_samples % n_splits folds have size n_samples // n_splits + 1, the others n_samples // n_splits.
  • scikit-learn, Common pitfalls and recommended practices, Controlling randomness: an integer random_state gives the same splits every time split is called, and passing an integer to the splitter, such as KFold(shuffle=True, random_state=0), gives comparable fold-to-fold results.

Key terms

Related