Cross-Validation
Cross-validation splits the rows into k folds and tests the model on each fold in turn, learning from the others, so every row is tested exactly once; cross_val_score gives the k scores, you report their mean and their spread, and KFold with shuffle=True and a random_state makes the folds random and repeatable.
Journeyman · AI Basics · Python · Updated
After this lesson you can
- Assign rows to k folds the way KFold does, and explain how every row is tested exactly once
- Score a model with cross_val_score and a shuffled, seeded KFold, and report the mean and the spread of the fold scores
Key takeaways
- One split is one roll of the dice. Other rows held back give another score.
- In k-fold cross-validation, each of the k folds takes one turn as the rows the model is tested on, while it learns from the rest, so every row is tested exactly once.
- cross_val_score(model, X, y, cv=5) gives five scores. Report their mean and their spread, the standard deviation.
- A number for cv makes folds in the rows' order, without shuffling, and for a classifier the folds keep each class's share (stratified).
- KFold(n_splits=5, shuffle=True, random_state=0) shuffles before cutting the folds, and the seed makes the same folds on every run.
You built a model, held back a fifth of the rows, and it scored 0.69. A classmate makes the same model, holds back a different fifth, and gets 0.59. Which number is right? Both are honest, and neither is the whole story: each came from one particular set of held-back rows. When you report how good a model is, in a project write-up or to a manager, you want a number that doesn't hang on which rows happened to be held back, and how much it moves.
The example is Lakeside Cinema's showings.csv, from Exercise: Honest Scores. Each code block is one notebook cell, run in order: the first one loads the data, and the others use what it made.
One split is one roll of the dice
The 180 test showings stay put away. From the 720 training showings, here is the same depth-5 tree, held back on five different fifths:
import pandas as pd
from sklearn.model_selection import KFold, cross_val_score, train_test_split
from sklearn.tree import DecisionTreeRegressor
showings = pd.read_csv("showings.csv")
features = ["day_number", "weekday", "start_hour", "is_new_release", "screen_seats"]
X_train, X_test, y_train, y_test = train_test_split(
showings[features], showings["tickets_sold"], test_size=0.2, random_state=42
)
tree = DecisionTreeRegressor(max_depth=5, random_state=0)
for seed in range(5):
X_fit, X_check, y_fit, y_check = train_test_split(X_train, y_train, test_size=0.2, random_state=seed)
tree.fit(X_fit, y_fit)
print("random_state", seed, "| R² on the held-back showings:", round(tree.score(X_check, y_check), 2))
random_state 0 | R² on the held-back showings: 0.69
random_state 1 | R² on the held-back showings: 0.64
random_state 2 | R² on the held-back showings: 0.59
random_state 3 | R² on the held-back showings: 0.64
random_state 4 | R² on the held-back showings: 0.68
Same model, same showings, scores from 0.59 to 0.69. Some held-back showings are easier to guess than others.
Every row gets a turn
Picture a teacher grading a class's practice tests in five piles. Each pile takes one turn as the "test" while the other four are what was studied, until every pile has had its turn. That's k-fold cross-validation: cut the rows into k parts, called folds; train on k − 1 of them and score on the one left out; repeat k times, so every row is tested exactly once; average the k scores.
Without shuffling, the folds are cut in order. With 10 rows and 3 folds, the first fold gets the extra row:
for number, (fit_rows, check_rows) in enumerate(KFold(n_splits=3).split(range(10)), start=1):
print("Fold", number, "| tested on rows", check_rows.tolist(), "| learns from rows", fit_rows.tolist())
Fold 1 | tested on rows [0, 1, 2, 3] | learns from rows [4, 5, 6, 7, 8, 9]
Fold 2 | tested on rows [4, 5, 6] | learns from rows [0, 1, 2, 3, 7, 8, 9]
Fold 3 | tested on rows [7, 8, 9] | learns from rows [0, 1, 2, 3, 4, 5, 6]
KFold gives the first (rows ÷ k's remainder) folds one row more than the others: 10 rows in 3 folds is 3 each with 1 left over, so fold 1 gets 4.

cross_val_score: the mean and the spread
cross_val_score does every round: it makes a fresh copy of the model for each fold, fits it on the other folds, and scores it with the model's own score, R² for a regression model.
scores = cross_val_score(tree, X_train, y_train, cv=5)
print("Scores:", scores.round(2))
print("Mean:", round(scores.mean(), 2))
print("Spread (standard deviation):", round(scores.std(), 2))
Scores: [0.73 0.67 0.7 0.64 0.72]
Mean: 0.69
Spread (standard deviation): 0.04
Report both numbers: "R² 0.69, give or take 0.04 from fold to fold". The standard deviation is the typical distance of a fold's score from the mean. A small spread says the score doesn't hang much on which rows were tested; a big one says one split could have told you almost anything.
cv=5 makes the folds without shuffling, cut in the rows' order. Here that's harmless, because train_test_split already shuffled the 720 training showings. And when the model is a classifier, such as one guessing whether a showing sells out, a number for cv makes stratified folds (StratifiedKFold), each keeping the classes' shares, as Stratified Splits explains.
Shuffled, seeded folds
To shuffle whatever order the rows come in, and get the same folds every run, hand cv a KFold:
folds = KFold(n_splits=5, shuffle=True, random_state=0)
scores = cross_val_score(tree, X_train, y_train, cv=folds)
print("Scores:", scores.round(2))
print("Mean:", round(scores.mean(), 2))
print("Spread (standard deviation):", round(scores.std(), 2))
Scores: [0.69 0.76 0.67 0.67 0.7 ]
Mean: 0.7
Spread (standard deviation): 0.03
Other folds, other scores; the mean barely moves, 0.69 to 0.70. random_state=0 is the shuffle's seed: leave it out, and each run makes different folds. Each round learns from 576 showings and is tested on 144.
Do it: deal showings into folds
Lakeside Cinema's analyst wants to check a cross-validation by hand, before trusting KFold. Write the folds the way KFold makes them without shuffling.
The inputs:
row_count: how many showings there are, numbered from 0.k: how many folds to make.folds: a list of folds, each a list of row numbers, asmake_foldsreturns.fold_index: the position of one fold infolds, from 0.
For example, 10 showings in 3 folds. 10 ÷ 3 is 3, with 1 left over, so the first fold gets one extra:
| Fold | Size | Rows |
|---|---|---|
| 0 | 3 + 1 = 4 | 0, 1, 2, 3 |
| 1 | 3 | 4, 5, 6 |
| 2 | 3 | 7, 8, 9 |
So make_folds(10, 3) returns [[0, 1, 2, 3], [4, 5, 6], [7, 8, 9]], and training_rows(make_folds(10, 3), 1) returns [0, 1, 2, 3, 7, 8, 9]: every row except fold 1's.
Your job: write two functions.
make_folds(row_count, k)returnsklists of row numbers. Every fold getsrow_count // krows, and the firstrow_count % kfolds get one more. The rows go in order: fold 0 gets the first ones, then fold 1, and so on.training_rows(folds, fold_index)returns every row number in the other folds, as one list, from the lowest to the highest.
Practiced live on the lesson page: write the Python right there and check it.
Check yourself
2 questions to try on the lesson page.
So what
One split gives one score that could have been a little different; cross-validation tests every row once and gives a mean with its spread, a number you can report and repeat.
Sources
- scikit-learn, Cross-validation: evaluating estimator performance, User Guide: k-fold cross-validation splits the training set into k folds, trains on k − 1 and validates on the remaining one, and reports the average; with an integer
cv,cross_val_scoreusesKFold, orStratifiedKFoldfor a classifier; the default score is the estimator'sscoremethod; the guide reports the scores' mean and standard deviation; withshuffle=Trueand norandom_statethe folds differ every time. - scikit-learn 1.9.1, cross_val_score, API reference: for an integer
cv,StratifiedKFoldwhen the estimator is a classifier andyis binary or multiclass, otherwiseKFold, both made withshuffle=False; it returns an array of one score per fold. - scikit-learn 1.9.1, KFold, API reference: consecutive folds without shuffling by default;
random_statecontrols the shuffle whenshuffleis true; the firstn_samples % n_splitsfolds have sizen_samples // n_splits + 1, the othersn_samples // n_splits. - scikit-learn, Common pitfalls and recommended practices, Controlling randomness: an integer
random_stategives the same splits every timesplitis called, and passing an integer to the splitter, such asKFold(shuffle=True, random_state=0), gives comparable fold-to-fold results.