Leaky Features and Duplicate Rows

Asking "when is this known?" of every column finds the ones filled in only after the outcome, and a table that lists the same subscriber twice puts them on both sides of a row-by-row split, where a tree recognized them and scored 0.843 instead of an honest 0.814, so leaky columns are dropped and the rows are split by subscriber with GroupKFold or GroupShuffleSplit.

Freelancer · AI Basics · Python · Updated

By

After this lesson you can

  • Decide for each column whether it's known at prediction time, and drop the ones known only after the outcome
  • Find the IDs that sit on both sides of a split, and split by ID with GroupKFold or GroupShuffleSplit so none does

Key takeaways

  • Ask "when is this known?" of every column. A column filled in only after the outcome, or one that stands in for it like a changed account number, can't be a feature.
  • A subscriber listed twice can land on both sides of a split by row. A model that recognizes them looks better than it will on new subscribers.
  • On StreamBox's table, a tree scored 0.843 on folds by row and 0.814 on folds by subscriber.
  • Split by the ID, not by the row, with GroupKFold for cross-validation or GroupShuffleSplit for one test set, and check that no ID is on both sides.

What Data Leakage Is found one column that gave the answer away because the tree happened to ask about it first. Most leaky columns hide better than that, and so do repeated rows, which no tree prints. Both come from how the data was collected, so the place to catch them is before training: a question for every column, and a check for every split. You'll do both at the start of any project that predicts what people will do.

The example is StreamBox's subscribers.csv, from Exercise: Plug the Leak. Each code block is one notebook cell, run in order: the first one loads the data.

When is this known?

The model will be used on a subscriber today, to guess whether they'll cancel later. So every feature has to be something StreamBox knows today. Go through the columns one by one:

Column When StreamBox knows it A feature?
months_subscribed, hours_last_month, profiles, plan, support_tickets Before anyone cancels Yes
exit_survey_sent Only after a cancellation No: it's the answer in disguise
subscriber_id Before, but it's a name tag, not a fact about behavior No
canceled After: it's the label It's what the model guesses

IDs and dates need a second look. A number that means nothing on its own can still carry the answer: if accounts were renumbered when they closed, the new numbers would mark the canceled. A date can be the day an account was closed, or a "last updated" time that changes when someone leaves. For each one, ask when it was filled in, and by what.

The same subscriber twice

The second leak is in the rows. StreamBox moved its accounts from an old billing system, and some were copied over twice:

import pandas as pd
from sklearn.model_selection import GroupKFold, GroupShuffleSplit, KFold, cross_val_score, train_test_split
from sklearn.tree import DecisionTreeClassifier

subscribers = pd.read_csv("subscribers.csv")
print("Rows:", len(subscribers))
print("Subscribers:", subscribers["subscriber_id"].nunique())
print("Rows that repeat an earlier row exactly:", subscribers.duplicated().sum())
print("Rows whose subscriber_id came earlier:", subscribers.duplicated("subscriber_id").sum())
Rows: 2000
Subscribers: 1700
Rows that repeat an earlier row exactly: 22
Rows whose subscriber_id came earlier: 300

duplicated() marks a row that repeats one above it in every column; given a column, it compares only that one. It finds 22 exact copies, but 300 subscribers are listed twice. Here are two:

twice = subscribers[subscribers.duplicated("subscriber_id", keep=False)]
twice[["subscriber_id", "hours_last_month", "support_tickets", "canceled"]].head(4)
   subscriber_id  hours_last_month  support_tickets  canceled
16      SB-10017              31.8                1         0
17      SB-10017              32.0                1         0
19      SB-10019              23.3                1         0
20      SB-10019              23.0                1         0

The old system rounded hours to whole numbers, so most copies aren't exact, and dropping exact duplicates would miss them. The ID is what ties them together. (keep=False marks every row of a repeated ID, the first one too.)

What a copy on both sides does

A split by row knows nothing about IDs, so one copy can land in training and the other in testing. With no max_depth, a DecisionTreeClassifier keeps splitting until every leaf holds one class or is too small to split, so its leaves come close to single rows, and it can recognize the copy:

features = ["months_subscribed", "hours_last_month", "profiles", "support_tickets"]
X = subscribers[features]
y = subscribers["canceled"]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

train_ids = set(subscribers.loc[X_train.index, "subscriber_id"])
seen = subscribers.loc[X_test.index, "subscriber_id"].isin(train_ids)
tree = DecisionTreeClassifier(random_state=0).fit(X_train, y_train)
right = tree.predict(X_test) == y_test
print("Test rows whose subscriber is also in training:", seen.sum(), "of", len(X_test))
print("Right on those rows:", round(right[seen].mean(), 3))
print("Right on the other rows:", round(right[~seen].mean(), 3))
Test rows whose subscriber is also in training: 85 of 400
Right on those rows: 0.918
Right on the other rows: 0.854

On subscribers it had already met, the tree was right more often. New subscribers never get that help. Across five folds, the gap shows in the score itself:

groups = subscribers["subscriber_id"]
by_row = cross_val_score(tree, X, y, cv=KFold(n_splits=5, shuffle=True, random_state=0))
by_subscriber = cross_val_score(tree, X, y, cv=GroupKFold(n_splits=5, shuffle=True, random_state=0), groups=groups)
print("Folds by row:", round(by_row.mean(), 3))
print("Folds by subscriber:", round(by_subscriber.mean(), 3))
Folds by row: 0.843
Folds by subscriber: 0.814

Two side-by-side panels of StreamBox subscriber cards. Left, "Split by row": subscriber SB-10017's two cards are torn apart, one in the training pile and one in the test pile, with a dotted line between them labeled "the model has met this subscriber". Right, "Split by subscriber": both SB-10017 cards sit together in the training pile, and the test pile holds only subscribers the model never saw

Splitting by subscriber

For one held-back test set, GroupShuffleSplit is train_test_split by group: test_size=0.2 holds back a fifth of the subscribers, with every row of each:

splitter = GroupShuffleSplit(n_splits=1, test_size=0.2, random_state=42)
train_rows, test_rows = next(splitter.split(subscribers, groups=groups))
train = subscribers.iloc[train_rows]
test = subscribers.iloc[test_rows]
print("Training rows:", len(train), "| test rows:", len(test))
print("Test subscribers:", test["subscriber_id"].nunique(), "of", groups.nunique())
print("Subscribers on both sides:", len(set(train["subscriber_id"]) & set(test["subscriber_id"])))
Training rows: 1589 | test rows: 411
Test subscribers: 340 of 1700
Subscribers on both sides: 0

split gives back row positions, so .iloc turns them into rows, and next takes the one split it makes. The last line is the check to run after any split: no ID on both sides.

Do it: find the subscribers on both sides

StreamBox's analyst split some rows into training and test, and wants to know which subscribers ended up on both sides.

The inputs:

  • train_ids: the subscriber_id of each training row, in row order. An ID can appear more than once.
  • test_ids: the subscriber_id of each test row, in the same form.

For example:

train_ids = ["SB-10001", "SB-10002", "SB-10002", "SB-10005"]
test_ids = ["SB-10005", "SB-10003", "SB-10002"]
ID In training? In test? On both sides?
SB-10001 yes no no
SB-10002 yes (twice) yes yes
SB-10003 no yes no
SB-10005 yes yes yes

So ids_on_both_sides(train_ids, test_ids) returns ["SB-10002", "SB-10005"], and is_leak_free(train_ids, test_ids) returns False.

Your job: write two functions.

  • ids_on_both_sides(train_ids, test_ids) returns every ID that is in both lists, each one once, sorted from smallest to largest (sorted() puts these IDs in order).
  • If no ID is in both lists, it returns an empty list, [].
  • is_leak_free(train_ids, test_ids) returns True when no ID is in both lists, and False when at least one is.

Practiced live on the lesson page: write the Python right there and check it.

Check yourself

2 questions to try on the lesson page.

So what

A feature has to be known when the model is used, and a test has to be on strangers, so ask "when is this known?" of every column and split by the ID, never by the row.

Sources

  • scikit-learn, Common pitfalls and recommended practices, Data leakage: information that would not be available at prediction time must not be used when building the model; test data should never be used to make choices about the model.
  • scikit-learn, Cross-validation: evaluating estimator performance, User Guide, Cross-validation iterators for grouped data, GroupKFold and GroupShuffleSplit: several samples from one subject depend on that subject; a model flexible enough to learn subject-specific features can fail on new subjects, and GroupKFold makes sure the same group is never in both the training and test sets, which detects it.
  • scikit-learn 1.9.1, GroupShuffleSplit, API reference: random train/test splits by group; test_size is a share of the groups, not the samples (rounded up), and an integer random_state makes the split repeatable.
  • scikit-learn 1.9.1, GroupKFold, API reference: each group is in the test set of exactly one fold; shuffle and random_state added in version 1.6.
  • pandas, DataFrame.duplicated, API reference: marks duplicate rows, optionally comparing only the columns given in subset; keep=False marks every duplicate.

Key terms

Related