# Leaky Features and Duplicate Rows

Asking "when is this known?" of every column finds the ones filled in only after the outcome, and a table that lists the same subscriber twice puts them on both sides of a row-by-row split, where a tree recognized them and scored 0.843 instead of an honest 0.814, so leaky columns are dropped and the rows are split by subscriber with GroupKFold or GroupShuffleSplit.

Freelancer · AI Basics · Python · Updated November 19, 2026

By Gregor Dzierzon

## After this lesson you can

- Decide for each column whether it's known at prediction time, and drop the ones known only after the outcome
- Find the IDs that sit on both sides of a split, and split by ID with GroupKFold or GroupShuffleSplit so none does

## Key takeaways

- Ask "when is this known?" of every column. A column filled in only after the outcome, or one that stands in for it like a changed account number, can't be a feature.
- A subscriber listed twice can land on both sides of a split by row. A model that recognizes them looks better than it will on new subscribers.
- On StreamBox's table, a tree scored 0.843 on folds by row and 0.814 on folds by subscriber.
- Split by the ID, not by the row, with GroupKFold for cross-validation or GroupShuffleSplit for one test set, and check that no ID is on both sides.

*What Data Leakage Is* found one column that gave the answer away because the tree happened to ask about it first. Most leaky columns hide better than that, and so do repeated rows, which no tree prints. Both come from how the data was collected, so the place to catch them is before training: a question for every column, and a check for every split. You'll do both at the start of any project that predicts what people will do.

The example is StreamBox's `subscribers.csv`, from *Exercise: Plug the Leak*. Each code block is one notebook cell, run in order: the first one loads the data.

## When is this known?

The model will be used on a subscriber today, to guess whether they'll cancel later. So every feature has to be something StreamBox knows *today*. Go through the columns one by one:

| Column | When StreamBox knows it | A feature? |
|---|---|---|
| `months_subscribed`, `hours_last_month`, `profiles`, `plan`, `support_tickets` | Before anyone cancels | Yes |
| `exit_survey_sent` | Only after a cancellation | No: it's the answer in disguise |
| `subscriber_id` | Before, but it's a name tag, not a fact about behavior | No |
| `canceled` | After: it's the label | It's what the model guesses |

IDs and dates need a second look. A number that means nothing on its own can still carry the answer: if accounts were renumbered when they closed, the new numbers would mark the canceled. A date can be the day an account was closed, or a "last updated" time that changes when someone leaves. For each one, ask when it was filled in, and by what.

## The same subscriber twice

The second leak is in the rows. StreamBox moved its accounts from an old billing system, and some were copied over twice:

```python
import pandas as pd
from sklearn.model_selection import GroupKFold, GroupShuffleSplit, KFold, cross_val_score, train_test_split
from sklearn.tree import DecisionTreeClassifier

subscribers = pd.read_csv("subscribers.csv")
print("Rows:", len(subscribers))
print("Subscribers:", subscribers["subscriber_id"].nunique())
print("Rows that repeat an earlier row exactly:", subscribers.duplicated().sum())
print("Rows whose subscriber_id came earlier:", subscribers.duplicated("subscriber_id").sum())
```

```text
Rows: 2000
Subscribers: 1700
Rows that repeat an earlier row exactly: 22
Rows whose subscriber_id came earlier: 300
```

`duplicated()` marks a row that repeats one above it in every column; given a column, it compares only that one. It finds 22 exact copies, but 300 subscribers are listed twice. Here are two:

```python
twice = subscribers[subscribers.duplicated("subscriber_id", keep=False)]
twice[["subscriber_id", "hours_last_month", "support_tickets", "canceled"]].head(4)
```

```text
   subscriber_id  hours_last_month  support_tickets  canceled
16      SB-10017              31.8                1         0
17      SB-10017              32.0                1         0
19      SB-10019              23.3                1         0
20      SB-10019              23.0                1         0
```

The old system rounded hours to whole numbers, so most copies aren't exact, and dropping exact duplicates would miss them. The ID is what ties them together. (`keep=False` marks every row of a repeated ID, the first one too.)

## What a copy on both sides does

A split by row knows nothing about IDs, so one copy can land in training and the other in testing. With no `max_depth`, a `DecisionTreeClassifier` keeps splitting until every leaf holds one class or is too small to split, so its leaves come close to single rows, and it can recognize the copy:

```python
features = ["months_subscribed", "hours_last_month", "profiles", "support_tickets"]
X = subscribers[features]
y = subscribers["canceled"]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

train_ids = set(subscribers.loc[X_train.index, "subscriber_id"])
seen = subscribers.loc[X_test.index, "subscriber_id"].isin(train_ids)
tree = DecisionTreeClassifier(random_state=0).fit(X_train, y_train)
right = tree.predict(X_test) == y_test
print("Test rows whose subscriber is also in training:", seen.sum(), "of", len(X_test))
print("Right on those rows:", round(right[seen].mean(), 3))
print("Right on the other rows:", round(right[~seen].mean(), 3))
```

```text
Test rows whose subscriber is also in training: 85 of 400
Right on those rows: 0.918
Right on the other rows: 0.854
```

On subscribers it had already met, the tree was right more often. New subscribers never get that help. Across five folds, the gap shows in the score itself:

```python
groups = subscribers["subscriber_id"]
by_row = cross_val_score(tree, X, y, cv=KFold(n_splits=5, shuffle=True, random_state=0))
by_subscriber = cross_val_score(tree, X, y, cv=GroupKFold(n_splits=5, shuffle=True, random_state=0), groups=groups)
print("Folds by row:", round(by_row.mean(), 3))
print("Folds by subscriber:", round(by_subscriber.mean(), 3))
```

```text
Folds by row: 0.843
Folds by subscriber: 0.814
```

![Two side-by-side panels of StreamBox subscriber cards. Left, "Split by row": subscriber SB-10017's two cards are torn apart, one in the training pile and one in the test pile, with a dotted line between them labeled "the model has met this subscriber". Right, "Split by subscriber": both SB-10017 cards sit together in the training pile, and the test pile holds only subscribers the model never saw](https://fsdtrainingmedia.blob.core.windows.net/public/images/2026/11/3b842f8ffc57421e8161f8b49bd7e56f.webp)

## Splitting by subscriber

For one held-back test set, `GroupShuffleSplit` is `train_test_split` by group: `test_size=0.2` holds back a fifth of the *subscribers*, with every row of each:

```python
splitter = GroupShuffleSplit(n_splits=1, test_size=0.2, random_state=42)
train_rows, test_rows = next(splitter.split(subscribers, groups=groups))
train = subscribers.iloc[train_rows]
test = subscribers.iloc[test_rows]
print("Training rows:", len(train), "| test rows:", len(test))
print("Test subscribers:", test["subscriber_id"].nunique(), "of", groups.nunique())
print("Subscribers on both sides:", len(set(train["subscriber_id"]) & set(test["subscriber_id"])))
```

```text
Training rows: 1589 | test rows: 411
Test subscribers: 340 of 1700
Subscribers on both sides: 0
```

`split` gives back row positions, so `.iloc` turns them into rows, and `next` takes the one split it makes. The last line is the check to run after any split: no ID on both sides.

## Do it: find the subscribers on both sides

StreamBox's analyst split some rows into training and test, and wants to know which subscribers ended up on both sides.

The inputs:

- `train_ids`: the `subscriber_id` of each training row, in row order. An ID can appear more than once.
- `test_ids`: the `subscriber_id` of each test row, in the same form.

For example:

```python
train_ids = ["SB-10001", "SB-10002", "SB-10002", "SB-10005"]
test_ids = ["SB-10005", "SB-10003", "SB-10002"]
```

| ID | In training? | In test? | On both sides? |
|---|---|---|---|
| SB-10001 | yes | no | no |
| SB-10002 | yes (twice) | yes | yes |
| SB-10003 | no | yes | no |
| SB-10005 | yes | yes | yes |

So `ids_on_both_sides(train_ids, test_ids)` returns `["SB-10002", "SB-10005"]`, and `is_leak_free(train_ids, test_ids)` returns `False`.

**Your job:** write two functions.

- `ids_on_both_sides(train_ids, test_ids)` returns every ID that is in both lists, each one once, sorted from smallest to largest (`sorted()` puts these IDs in order).
- If no ID is in both lists, it returns an empty list, `[]`.
- `is_leak_free(train_ids, test_ids)` returns `True` when no ID is in both lists, and `False` when at least one is.

Practiced live on the lesson page: write the Python right there and check it.

## Check yourself

2 questions to try on the lesson page.

## So what

A feature has to be known when the model is used, and a test has to be on strangers, so ask "when is this known?" of every column and split by the ID, never by the row.

## Sources

- scikit-learn, [Common pitfalls and recommended practices](https://scikit-learn.org/stable/common_pitfalls.html), *Data leakage*: information that would not be available at prediction time must not be used when building the model; test data should never be used to make choices about the model.
- scikit-learn, [Cross-validation: evaluating estimator performance](https://scikit-learn.org/stable/modules/cross_validation.html), User Guide, *Cross-validation iterators for grouped data*, *GroupKFold* and *GroupShuffleSplit*: several samples from one subject depend on that subject; a model flexible enough to learn subject-specific features can fail on new subjects, and `GroupKFold` makes sure the same group is never in both the training and test sets, which detects it.
- scikit-learn 1.9.1, [GroupShuffleSplit](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GroupShuffleSplit.html), API reference: random train/test splits by group; `test_size` is a share of the groups, not the samples (rounded up), and an integer `random_state` makes the split repeatable.
- scikit-learn 1.9.1, [GroupKFold](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GroupKFold.html), API reference: each group is in the test set of exactly one fold; `shuffle` and `random_state` added in version 1.6.
- pandas, [DataFrame.duplicated](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.duplicated.html), API reference: marks duplicate rows, optionally comparing only the columns given in `subset`; `keep=False` marks every duplicate.

## Key terms

- [Target Leakage](https://www.ask-codey.com/glossary/target-leakage)
- [Duplicate Row](https://www.ask-codey.com/glossary/duplicate-row)
- [duplicated()](https://www.ask-codey.com/glossary/duplicated)
- [GroupShuffleSplit](https://www.ask-codey.com/glossary/groupshufflesplit)

## Related

- [What Data Leakage Is](https://www.ask-codey.com/lessons/ml-what-data-leakage-is)
- [Leaks in Preparing the Data](https://www.ask-codey.com/lessons/ml-leakage-in-preprocessing)
- [Splits for Time and Groups](https://www.ask-codey.com/lessons/ml-splits-for-time-and-groups)
- [Exercise: Plug the Leak](https://www.ask-codey.com/lessons/exercise-ml-plug-the-leak)

Source: https://www.ask-codey.com/lessons/ml-leaky-features-and-duplicates
