# Cross-Validation

Cross-validation splits the rows into k folds and tests the model on each fold in turn, learning from the others, so every row is tested exactly once; cross_val_score gives the k scores, you report their mean and their spread, and KFold with shuffle=True and a random_state makes the folds random and repeatable.

Journeyman · AI Basics · Python · Updated November 19, 2026

By Gregor Dzierzon

## After this lesson you can

- Assign rows to k folds the way KFold does, and explain how every row is tested exactly once
- Score a model with cross_val_score and a shuffled, seeded KFold, and report the mean and the spread of the fold scores

## Key takeaways

- One split is one roll of the dice. Other rows held back give another score.
- In k-fold cross-validation, each of the k folds takes one turn as the rows the model is tested on, while it learns from the rest, so every row is tested exactly once.
- cross_val_score(model, X, y, cv=5) gives five scores. Report their mean and their spread, the standard deviation.
- A number for cv makes folds in the rows' order, without shuffling, and for a classifier the folds keep each class's share (stratified).
- KFold(n_splits=5, shuffle=True, random_state=0) shuffles before cutting the folds, and the seed makes the same folds on every run.

You built a model, held back a fifth of the rows, and it scored 0.69. A classmate makes the same model, holds back a different fifth, and gets 0.59. Which number is right? Both are honest, and neither is the whole story: each came from one particular set of held-back rows. When you report how good a model is, in a project write-up or to a manager, you want a number that doesn't hang on which rows happened to be held back, and how much it moves.

The example is Lakeside Cinema's `showings.csv`, from *Exercise: Honest Scores*. Each code block is one notebook cell, run in order: the first one loads the data, and the others use what it made.

## One split is one roll of the dice

The 180 test showings stay put away. From the 720 training showings, here is the same depth-5 tree, held back on five different fifths:

```python
import pandas as pd
from sklearn.model_selection import KFold, cross_val_score, train_test_split
from sklearn.tree import DecisionTreeRegressor

showings = pd.read_csv("showings.csv")
features = ["day_number", "weekday", "start_hour", "is_new_release", "screen_seats"]
X_train, X_test, y_train, y_test = train_test_split(
    showings[features], showings["tickets_sold"], test_size=0.2, random_state=42
)
tree = DecisionTreeRegressor(max_depth=5, random_state=0)

for seed in range(5):
    X_fit, X_check, y_fit, y_check = train_test_split(X_train, y_train, test_size=0.2, random_state=seed)
    tree.fit(X_fit, y_fit)
    print("random_state", seed, "| R² on the held-back showings:", round(tree.score(X_check, y_check), 2))
```

```text
random_state 0 | R² on the held-back showings: 0.69
random_state 1 | R² on the held-back showings: 0.64
random_state 2 | R² on the held-back showings: 0.59
random_state 3 | R² on the held-back showings: 0.64
random_state 4 | R² on the held-back showings: 0.68
```

Same model, same showings, scores from 0.59 to 0.69. Some held-back showings are easier to guess than others.

## Every row gets a turn

Picture a teacher grading a class's practice tests in five piles. Each pile takes one turn as the "test" while the other four are what was studied, until every pile has had its turn. That's **k-fold cross-validation**: cut the rows into k parts, called **folds**; train on k − 1 of them and score on the one left out; repeat k times, so every row is tested exactly once; average the k scores.

Without shuffling, the folds are cut in order. With 10 rows and 3 folds, the first fold gets the extra row:

```python
for number, (fit_rows, check_rows) in enumerate(KFold(n_splits=3).split(range(10)), start=1):
    print("Fold", number, "| tested on rows", check_rows.tolist(), "| learns from rows", fit_rows.tolist())
```

```text
Fold 1 | tested on rows [0, 1, 2, 3] | learns from rows [4, 5, 6, 7, 8, 9]
Fold 2 | tested on rows [4, 5, 6] | learns from rows [0, 1, 2, 3, 7, 8, 9]
Fold 3 | tested on rows [7, 8, 9] | learns from rows [0, 1, 2, 3, 4, 5, 6]
```

`KFold` gives the first (rows ÷ k's remainder) folds one row more than the others: 10 rows in 3 folds is 3 each with 1 left over, so fold 1 gets 4.

![The 720 training showings drawn as a long bar cut into five folds of 144, shown five times, one row per round: in each round a different fold is shaded as the 144 showings tested, and the other four, 576 showings, are what the tree learns from; every fold is tested once, and the five scores are averaged](https://fsdtrainingmedia.blob.core.windows.net/public/images/2026/11/457e24a1d517454c8c8cb44a7abe693e.webp)

## cross_val_score: the mean and the spread

`cross_val_score` does every round: it makes a fresh copy of the model for each fold, fits it on the other folds, and scores it with the model's own `score`, R² for a regression model.

```python
scores = cross_val_score(tree, X_train, y_train, cv=5)
print("Scores:", scores.round(2))
print("Mean:", round(scores.mean(), 2))
print("Spread (standard deviation):", round(scores.std(), 2))
```

```text
Scores: [0.73 0.67 0.7  0.64 0.72]
Mean: 0.69
Spread (standard deviation): 0.04
```

Report both numbers: "R² 0.69, give or take 0.04 from fold to fold". The **standard deviation** is the typical distance of a fold's score from the mean. A small spread says the score doesn't hang much on which rows were tested; a big one says one split could have told you almost anything.

`cv=5` makes the folds without shuffling, cut in the rows' order. Here that's harmless, because `train_test_split` already shuffled the 720 training showings. And when the model is a classifier, such as one guessing whether a showing sells out, a number for `cv` makes stratified folds (`StratifiedKFold`), each keeping the classes' shares, as *Stratified Splits* explains.

## Shuffled, seeded folds

To shuffle whatever order the rows come in, and get the same folds every run, hand `cv` a `KFold`:

```python
folds = KFold(n_splits=5, shuffle=True, random_state=0)
scores = cross_val_score(tree, X_train, y_train, cv=folds)
print("Scores:", scores.round(2))
print("Mean:", round(scores.mean(), 2))
print("Spread (standard deviation):", round(scores.std(), 2))
```

```text
Scores: [0.69 0.76 0.67 0.67 0.7 ]
Mean: 0.7
Spread (standard deviation): 0.03
```

Other folds, other scores; the mean barely moves, 0.69 to 0.70. `random_state=0` is the shuffle's seed: leave it out, and each run makes different folds. Each round learns from 576 showings and is tested on 144.

## Do it: deal showings into folds

Lakeside Cinema's analyst wants to check a cross-validation by hand, before trusting `KFold`. Write the folds the way `KFold` makes them without shuffling.

The inputs:

- `row_count`: how many showings there are, numbered from 0.
- `k`: how many folds to make.
- `folds`: a list of folds, each a list of row numbers, as `make_folds` returns.
- `fold_index`: the position of one fold in `folds`, from 0.

For example, 10 showings in 3 folds. 10 ÷ 3 is 3, with 1 left over, so the first fold gets one extra:

| Fold | Size | Rows |
|---|---|---|
| 0 | 3 + 1 = 4 | 0, 1, 2, 3 |
| 1 | 3 | 4, 5, 6 |
| 2 | 3 | 7, 8, 9 |

So `make_folds(10, 3)` returns `[[0, 1, 2, 3], [4, 5, 6], [7, 8, 9]]`, and `training_rows(make_folds(10, 3), 1)` returns `[0, 1, 2, 3, 7, 8, 9]`: every row except fold 1's.

**Your job:** write two functions.

- `make_folds(row_count, k)` returns `k` lists of row numbers. Every fold gets `row_count // k` rows, and the first `row_count % k` folds get one more. The rows go in order: fold 0 gets the first ones, then fold 1, and so on.
- `training_rows(folds, fold_index)` returns every row number in the other folds, as one list, from the lowest to the highest.

Practiced live on the lesson page: write the Python right there and check it.

## Check yourself

2 questions to try on the lesson page.

## So what

One split gives one score that could have been a little different; cross-validation tests every row once and gives a mean with its spread, a number you can report and repeat.

## Sources

- scikit-learn, [Cross-validation: evaluating estimator performance](https://scikit-learn.org/stable/modules/cross_validation.html), User Guide: k-fold cross-validation splits the training set into k folds, trains on k − 1 and validates on the remaining one, and reports the average; with an integer `cv`, `cross_val_score` uses `KFold`, or `StratifiedKFold` for a classifier; the default score is the estimator's `score` method; the guide reports the scores' mean and standard deviation; with `shuffle=True` and no `random_state` the folds differ every time.
- scikit-learn 1.9.1, [cross_val_score](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.cross_val_score.html), API reference: for an integer `cv`, `StratifiedKFold` when the estimator is a classifier and `y` is binary or multiclass, otherwise `KFold`, both made with `shuffle=False`; it returns an array of one score per fold.
- scikit-learn 1.9.1, [KFold](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.KFold.html), API reference: consecutive folds without shuffling by default; `random_state` controls the shuffle when `shuffle` is true; the first `n_samples % n_splits` folds have size `n_samples // n_splits + 1`, the others `n_samples // n_splits`.
- scikit-learn, [Common pitfalls and recommended practices](https://scikit-learn.org/stable/common_pitfalls.html), *Controlling randomness*: an integer `random_state` gives the same splits every time `split` is called, and passing an integer to the splitter, such as `KFold(shuffle=True, random_state=0)`, gives comparable fold-to-fold results.

## Key terms

- [Cross-Validation](https://www.ask-codey.com/glossary/cross-validation)
- [K-Fold Cross-Validation](https://www.ask-codey.com/glossary/k-fold-cross-validation)
- [Standard Deviation](https://www.ask-codey.com/glossary/standard-deviation)
- [Random Seed](https://www.ask-codey.com/glossary/random-seed)

## Related

- [Validation Sets: Keep the Test Set for Last](https://www.ask-codey.com/lessons/ml-validation-and-test-sets)
- [Splits for Time and Groups](https://www.ask-codey.com/lessons/ml-splits-for-time-and-groups)
- [Exercise: Honest Scores](https://www.ask-codey.com/lessons/exercise-ml-honest-scores)

Source: https://www.ask-codey.com/lessons/ml-cross-validation
