# Stratified Splits

A random split can leave more or fewer of one class in the test part than in the data, and stratify=y makes train_test_split keep each class's share the same in both parts, which you check with value_counts(normalize=True).

Journeyman · AI Basics · Python · Updated November 19, 2026

By Gregor Dzierzon

## After this lesson you can

- Split a classification dataset with train_test_split and stratify=y, and check each part's class shares with value_counts(normalize=True)
- Explain why a random split can give the two parts different class shares, and when a stratified split matters most

## Key takeaways

- A random split can tip the balance. With 299 spam emails in 1,000, five random test parts of 200 held from 55 to 67 spam.
- train_test_split(X, y, test_size=0.2, random_state=42, stratify=y) keeps each class's share in both parts about the same as in the whole data.
- value_counts(normalize=True) shows each class's share, so you can check both parts.
- Stratifying doesn't change the data's own mix, and it can't be used with shuffle=False. It matters most when a class is rare.

Every classifier you test is tested on a handful of held-back examples, and the score you report depends on what's in that handful. If Inbox Guard's test emails happen to hold more spam than its inbox really gets, or less, the score describes a mix that doesn't exist. Whenever you split data whose label is a category (spam or not, renew or not, fraud or not), you'll want both parts to look like the whole.

The example is Inbox Guard's `emails.csv`, from *Exercise: Spam or Not*. Each code block is one notebook cell, run in order: the first one loads the data.

## A shuffle can tip the balance

First, the share of each class in all 1,000 emails. `value_counts(normalize=True)` gives shares instead of counts:

```python
import pandas as pd
from sklearn.model_selection import train_test_split

emails = pd.read_csv("emails.csv")
FEATURES = ["num_links", "num_exclamations", "mentions_free", "sender_in_contacts", "subject_all_caps"]
X = emails[FEATURES]
y = emails["is_spam"]
print(y.value_counts(normalize=True))
```

```text
is_spam
0    0.701
1    0.299
Name: proportion, dtype: float64
```

29.9% spam. A fifth of the emails, 200, held back at random should hold about 60 spam. Here's what five different shuffles give, with the split from *Splitting Data with train_test_split*:

```python
for seed in range(5):
    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=seed)
    print("random_state", seed, "test emails that are spam:", y_test.sum(), "of", len(y_test))
```

```text
random_state 0 test emails that are spam: 67 of 200
random_state 1 test emails that are spam: 61 of 200
random_state 2 test emails that are spam: 55 of 200
random_state 3 test emails that are spam: 65 of 200
random_state 4 test emails that are spam: 57 of 200
```

`y_test.sum()` counts the spam, because spam is 1 and everything else is 0. Anywhere from 55 to 67: 27.5% to 33.5%. That moves every score you'd report. A filter that never says spam would be right about 145 of the 200 test emails with `random_state=2` (72.5%) but only 133 with `random_state=0` (66.5%), and the model's own score moves the same way, though nothing about the model changed.

## stratify=y keeps the shares

A **stratified split** deals out each class separately. Think of two decks of cards, the spam and the rest: deal a fifth of each deck to the test pile, and the test pile has the same mix as the two decks together. *Stratum* is Latin for a layer, and each class is one layer. In scikit-learn it's one more setting, `stratify`, given the labels to keep in step.

![Two decks of email cards: 299 spam cards, each with a small flag, and 701 plain cards. A fifth of each deck is dealt to the test pile, 60 spam and 140 plain, and the rest to the training pile, 239 spam and 561 plain, so both piles are about 30% spam, like the whole](https://fsdtrainingmedia.blob.core.windows.net/public/images/2026/11/85646c6509b5442ebe528a1b7a783e33.webp)

```python
for seed in range(5):
    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=seed, stratify=y)
    print("random_state", seed, "test emails that are spam:", y_test.sum(), "of", len(y_test))
```

```text
random_state 0 test emails that are spam: 60 of 200
random_state 1 test emails that are spam: 60 of 200
random_state 2 test emails that are spam: 60 of 200
random_state 3 test emails that are spam: 60 of 200
random_state 4 test emails that are spam: 60 of 200
```

Every seed now holds back 60 spam emails. The seed still decides *which* 60, so `random_state` keeps the split repeatable as before. Check both parts the same way you checked the whole:

```python
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42, stratify=y)
print(y_train.value_counts(normalize=True))
print(y_test.value_counts(normalize=True))
```

```text
is_spam
0    0.70125
1    0.29875
Name: proportion, dtype: float64
is_spam
0    0.7
1    0.3
Name: proportion, dtype: float64
```

Training 29.875% spam (239 of 800), test 30% (60 of 200), the whole 29.9%: as close as whole emails allow. scikit-learn promises shares that are about the same, not exactly the same, because 299 spam can't be split into exact fifths.

## When it matters most

With 30% spam and 1,000 emails, a random split is off by a few emails. The rarer a class and the smaller the data, the worse a random split can do. A credit union looking for fraud in 2% of its payments could get a test part with far fewer fraud cases than its share, or none at all, and a score with nothing to find in it says nothing about finding fraud. scikit-learn's guide gives exactly that reason, rare classes such as fraud detection, for its stratified splitters. *When the Answer Is Rare* comes back to it.

## What stratifying doesn't do

It doesn't change the data's own mix: the emails are still 70% not spam, and *Accuracy* shows why that matters for scores. It doesn't make the model better; it makes the test fairer. And it works only with shuffling, because dealing out each class means picking rows from all over the file:

```python
train_test_split(X, y, test_size=0.2, shuffle=False, stratify=y)
```

```text
ValueError: Stratified train/test split is not implemented for shuffle=False
```

Stratify on the label, the category you're predicting. For data in time order, such as daily sales, *Splitting Data with train_test_split* holds back the most recent rows instead of shuffling; *Splits for Time and Groups* goes further.

## Check yourself

2 questions to try on the lesson page.

## So what

When the label is a category, split with `stratify=y` and check both parts with `value_counts(normalize=True)`, so the test emails have the same mix as the rest and the score describes data you'll really see.

## Sources

- scikit-learn 1.9.1, [train_test_split](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html), API reference: `stratify` (default None), if not None, splits the data in a stratified fashion, using it as the class labels; `shuffle` (default True), and if `shuffle=False` then `stratify` must be None.
- scikit-learn, [Cross-validation: Cross-validation iterators with stratification based on class labels](https://scikit-learn.org/stable/modules/cross_validation.html#cross-validation-iterators-with-stratification-based-on-class-labels), User Guide: some classification tasks have rare classes (medical screening, fraud detection), so a split can end up without any examples of a class; stratified sampling keeps the relative class frequencies approximately the same in each part; `train_test_split` allows stratified splitting with the class labels.
- pandas 3.1, [Series.value_counts](https://pandas.pydata.org/docs/reference/api/pandas.Series.value_counts.html), API reference: `normalize=True` returns the relative frequencies of the unique values instead of counts.

## Key terms

- [Stratified Split](https://www.ask-codey.com/glossary/stratified-split)
- [value_counts()](https://www.ask-codey.com/glossary/value-counts)
- [train_test_split()](https://www.ask-codey.com/glossary/train-test-split)
- [Test Data](https://www.ask-codey.com/glossary/test-data)

## Related

- [Splitting Data with train_test_split](https://www.ask-codey.com/lessons/ml-train-test-split)
- [What Classification Is](https://www.ask-codey.com/lessons/ml-what-classification-is)
- [Exercise: Spam or Not](https://www.ask-codey.com/lessons/exercise-ml-spam-or-not)

Source: https://www.ask-codey.com/lessons/ml-stratified-splits
