Leaks in Preparing the Data
A preparation step fit on the whole table before the split, a scaler, an imputer or a step that chooses columns, learns from the rows it will be tested on, which barely moved StreamBox's score for a scaler but turned a worse model into one that looked better when columns were chosen against the label, and putting every step that learns in a pipeline that cross-validation fits inside each fold closes the leak.
Freelancer · AI Basics · Python · Updated
After this lesson you can
- Explain how a scaler, an imputer or a column-choosing step fit before the split leaks the test rows into training
- Keep every learned step inside a pipeline that cross_val_score fits inside each fold, and compare the leaky score with the honest one
Key takeaways
- Any step that learns from data, a scaler, an imputer or a step that chooses columns, leaks when it's fit on the whole table before the split or before cross-validation.
- A scaler fit on all of StreamBox's rows scored the same as an honest one, 0.8735, as its averages hardly moved. A small leak is still a leak, and you can't tell ahead which are small.
- Choosing 50 columns out of 1,004 on the whole table scored 0.883, better than the 0.874 of four real columns. Chosen inside each fold, the same plan scored 0.852, worse.
- Put every step that learns in a pipeline, and give the pipeline to cross-validation, so each fold fits every step on its training rows only.
Almost every model gets its data prepared first: numbers rescaled, blanks filled in, the most useful columns picked out. It's tempting to do that once, to the whole table, and then split or cross-validate. That order is a leak, and it's one of the easiest to make by accident, because each step looks like tidying up rather than learning. You'll use what this lesson shows every time a model has a preparation step, which is almost every time.
The example is StreamBox's subscribers.csv, from Exercise: Plug the Leak. Each code block is one notebook cell, run in order: the first one loads the data. Every score here comes from five folds that keep each subscriber's rows together (GroupKFold, from Splits for Time and Groups), because some subscribers are listed twice; Leaky Features and Duplicate Rows shows why that matters.
Every step that learns
A scaler learns something: the average and the spread of each column, which it then uses to rescale every row (Scaling Features). An imputer learns the value it fills blanks with. A step that chooses columns learns which ones line up best with the label. Fit any of them on the whole table, and the test rows help decide what it learns. The rule from scikit-learn is short: never call fit on the test data, for the model or for any step before it.
Here is the leaky order, scaling all 2,000 rows and then cross-validating:
import pandas as pd
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import GroupKFold, cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
subscribers = pd.read_csv("subscribers.csv")
features = ["months_subscribed", "hours_last_month", "profiles", "support_tickets"]
X = subscribers[features]
y = subscribers["canceled"]
groups = subscribers["subscriber_id"]
folds = GroupKFold(n_splits=5, shuffle=True, random_state=0)
X_scaled = StandardScaler().fit_transform(X)
scaled_first = cross_val_score(LogisticRegression(), X_scaled, y, cv=folds, groups=groups)
print("Scaled before the split:", round(scaled_first.mean(), 4))
Scaled before the split: 0.8735
shuffle=True with random_state=0 mixes the subscribers before they're dealt into folds, the same way on every run. And the honest order, with the scaler inside a pipeline, so each fold fits it on that fold's training rows:
model = make_pipeline(StandardScaler(), LogisticRegression())
scaled_inside = cross_val_score(model, X, y, cv=folds, groups=groups)
print("Scaled inside each fold:", round(scaled_inside.mean(), 4))
Scaled inside each fold: 0.8735
A small leak is still a leak
The same score, to four places. That's honest to report: here, the leak changed nothing you can see, because the averages hardly move when a fifth of the rows is added:
fit_rows, check_rows = next(folds.split(X, y, groups))
print("Averages from all 2,000 rows:", StandardScaler().fit(X).mean_.round(2))
print("Averages from one fold's training rows:", StandardScaler().fit(X.iloc[fit_rows]).mean_.round(2))
Averages from all 2,000 rows: [24.34 21.44 2.36 0.88]
Averages from one fold's training rows: [24.34 21.47 2.37 0.88]
So why bother? Because a scaler is the mildest step there is. It only looks at the features. A step that looks at the label leaks far more, and from the outside you can't tell a mild leak from a big one.
Choosing columns on the whole table
Picking the most useful columns is called feature selection. scikit-learn's SelectKBest(k=50) scores each column by how strongly it lines up with the label and keeps the best 50. To see what it does with a leak, give StreamBox's four real columns 1,000 extra yes-or-no columns that are pure coin flips: we made them up with a seeded random generator, so we know they carry nothing about canceling.
import numpy as np
from sklearn.feature_selection import SelectKBest
rng = np.random.default_rng(0)
flags = pd.DataFrame(rng.integers(0, 2, size=(len(subscribers), 1000)),
columns=[f"flag_{n}" for n in range(1, 1001)])
X_wide = pd.concat([X, flags], axis=1)
chosen_first = SelectKBest(k=50).fit_transform(X_wide, y)
leaky = cross_val_score(model, chosen_first, y, cv=folds, groups=groups)
print("Columns to choose from:", X_wide.shape[1])
print("Four real columns:", round(scaled_inside.mean(), 3))
print("50 columns chosen before the split:", round(leaky.mean(), 3))
Columns to choose from: 1004
Four real columns: 0.874
50 columns chosen before the split: 0.883
The leaky run says the coin flips helped. They can't have: among 1,000 random columns, some line up with canceling by luck, and choosing them with every row's label, the test rows' included, picks exactly the lucky ones for those test rows.
The fix: a pipeline inside cross-validation
Put the choosing step in the pipeline too:
honest = make_pipeline(SelectKBest(k=50), StandardScaler(), LogisticRegression())
chosen_inside = cross_val_score(honest, X_wide, y, cv=folds, groups=groups)
print("50 columns chosen inside each fold:", round(chosen_inside.mean(), 3))
50 columns chosen inside each fold: 0.852
0.852: worse than the four real columns alone. That's the truth the leak hid. As Pipelines: Every Step, Every Time showed, cross_val_score fits a fresh copy of the whole pipeline in every fold, on that fold's training rows only, and then scores it on the rows it held out. No step ever learns from the rows it's scored on.

The same goes for filling blanks: a SimpleImputer fit on the whole table learns its median from the test rows. Every step that learns goes in the pipeline, and the pipeline goes into cross_val_score (or is fit on the training part only, then scored once on the test part).
Check yourself
2 questions to try on the lesson page.
So what
A preparation step learns from data like a model does, so it belongs inside the pipeline that cross-validation refits in every fold, where it can't peek at the rows it's scored on.
Sources
- scikit-learn, Common pitfalls and recommended practices, Data leakage, How to avoid data leakage and Data leakage during pre-processing: split first, never include test data in
fitorfit_transform, transformations are learned from the training data only, the risk applies to almost all transformations (StandardScaler,SimpleImputerand feature selection among them), and a pipeline, alone or insidecross_val_score, makes sure the right rows are used. - scikit-learn, Cross-validation: evaluating estimator performance, User Guide, Computing cross-validated metrics: preprocessing such as standardization and feature selection should be learned from a training set and applied to held-out data, which a
Pipelinedoes under cross-validation. - scikit-learn, Pipelines and composite estimators, User Guide: a pipeline avoids leaking statistics from the test data into the trained model in cross-validation by making sure the same samples train the transformers and the predictor.
- scikit-learn 1.9.1, SelectKBest, API reference: keeps the
kfeatures with the highest scores, scored by default withf_classif, which works for classification. - scikit-learn 1.9.1, GroupKFold, API reference: each group appears in the test set of exactly one fold;
shuffleshuffles the groups before they're split, andrandom_statemakes that repeatable (both added in version 1.6).