# Exercise: Plug the Leak

Write three functions that close the leaks in StreamBox's 2,000 subscriber rows, dropping the column known only after a cancellation, splitting so no subscriber is on both sides, and cross-validating a scaled model with the scaler inside a pipeline, and let pytest check that nothing leaks.

Freelancer · AI Basics · Python · Updated November 19, 2026

By Gregor Dzierzon

StreamBox's first model of who will cancel scored 99%, and it only seemed to work: a column in the table is the answer in disguise, and 300 subscribers are listed twice. You'll write the three functions that plug each leak: one drops the columns StreamBox only knows after a cancellation, one splits the rows so no subscriber is on both sides, and one cross-validates a scaled model with the scaling done inside each fold. The tests don't check how accurate the model is; they check that nothing leaks, and one of them watches every row the scaler learns from.

## Get the exercise code

Open this lesson's GitHub repository, [github.com/ask-codey/exercise-ml-plug-the-leak](https://github.com/ask-codey/exercise-ml-plug-the-leak), also linked from the **Exercise** box beside the lesson. Clone it into your `python-course` folder. In VS Code's terminal:

```bash
cd ~/python-course
git clone https://github.com/ask-codey/exercise-ml-plug-the-leak.git
```

Git makes a folder, `exercise-ml-plug-the-leak`, with the exercise code inside. Open that folder in VS Code (**File > Open Folder**), open its terminal with **Terminal > New Terminal**, and make the exercise's virtual environment. On Windows (PowerShell):

```powershell
python -m venv .venv
.venv\Scripts\Activate.ps1
pip install -r requirements.txt
```

On a Mac (zsh):

```bash
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
```

The prompt starts with `(.venv)` once the virtual environment is on. The exercise's README says what each command does, and what to do if PowerShell won't run the second one.

The folder holds:

- `subscribers.csv`: the 2,000 rows for 1,700 subscribers, our own made-up data, the same file the chapter's lessons use.
- `seed_data.py`: the script that made the CSV, with the pattern it planted and the two leaks planted with it. You don't need to run it: the committed CSV is what keeps everyone's numbers the same.
- `churn_model.py`: the three functions, for you to finish, with `FEATURES` (the four columns the model looks at), `LABEL` (`canceled`), `GROUP` (`subscriber_id`) and `KNOWN_LATER` (`["exit_survey_sent"]`), ready to use.
- `explore.ipynb`: a notebook for looking at the subscribers and trying your code. Choose the `.venv` kernel with **Select Kernel** (top right) before you run it. The tests don't read it.
- `report.py`: plugs the leaks with your three functions and prints what each step gave, next to a leaky score it makes the wrong way on purpose. Ready to use.
- `tests/`: the tests, which call each function and check what comes back, and `conftest.py`, which gives each test a fresh copy of the subscribers. You don't need to change them.
- `requirements.txt` and `pytest.ini`: the packages the exercise needs, at exact versions, and where pytest finds the tests and your code.

## What to do

Every function in `churn_model.py` is already declared with its name and parameters, and for now it stops with a `NotImplementedError` that says *Not built yet.* The docstring under each one says what it should do, with examples:

```python
def split_by_subscriber(subscribers, seed):
    """Split the rows into a training table and a test table, with no subscriber in both.

    Use GroupShuffleSplit(n_splits=1, test_size=0.2, random_state=seed), with the
    GROUP column as the groups, so a fifth of the subscribers (not of the rows) go
    to the test table and every row of a subscriber lands on the same side.
    Pass seed on as random_state, so the same seed always gives the same split.

    Return two tables, training first: train, test. Each keeps every column.

    With the 2,000 rows in subscribers.csv: 340 of the 1,700 subscribers in the
    test table, and no subscriber_id in both tables.
    """
    raise NotImplementedError("Not built yet.")
```

In each function, replace the `raise NotImplementedError("Not built yet.")` line with your own code:

1. **`drop_after_the_fact(subscribers, known_later)`** returns a new table without the columns named in `known_later`, keeping every row and every other column in its order, and leaving the table it was given unchanged. With `["exit_survey_sent"]`: 2,000 rows and seven columns, `canceled` among them.
2. **`split_by_subscriber(subscribers, seed)`** splits the rows with `GroupShuffleSplit(n_splits=1, test_size=0.2, random_state=seed)`, grouped by `subscriber_id`, and returns `train, test`. With seed 42: 340 of the 1,700 subscribers in the test table, and none in both.
3. **`honest_cv_score(subscribers, seed)`** cross-validates `make_pipeline(StandardScaler(), LogisticRegression())` on the `FEATURES` columns and the `LABEL` column, with `GroupKFold(n_splits=5, shuffle=True, random_state=seed)` and the `subscriber_id` column as `groups`, and returns the five accuracies `cross_val_score` gives, as a list of floats.

Nothing is fit on the whole table first: the scaler lives in the pipeline, and `cross_val_score` fits a fresh copy in each fold. That's what the recording test checks.

## Check your work

Run the tests from the exercise folder, with the virtual environment on:

```bash
python -m pytest
```

At first all 17 tests fail. That's expected: each function you finish turns its tests green. A failing test's message says what it called, what it expected and what came back, for example *how many different subscribers the test table holds should return 340, but it returned 400.* Four of the tests swap in a scaler that writes down every row it learns from and every row it scales: they fail if the scaler ever learns from all 2,000 rows, if a row is only ever scaled by a scaler that learned from it, if a subscriber is in a fold's training rows and its test rows at once, or if the scaler is given any column but the four features.

To see your functions at work on the whole file, run `python report.py`.

## Hints

- **Look before you write.** The notebook's cells show the survey column against `canceled`, and the subscribers listed twice.
- **A new table, not a changed one.** `drop(columns=...)` gives back a new table and leaves the old one as it was; *What Data Leakage Is* uses it to set the features apart.
- **Positions, not rows.** A splitter's `split` gives back row positions: `next(...)` takes its one split, and `.iloc` turns positions into rows. *Leaky Features and Duplicate Rows* has the whole pattern.
- **Groups go to the splitter.** `GroupShuffleSplit` and `GroupKFold` keep a subscriber together only when they're given `groups=`.
- **Everything that learns goes in the pipeline.** *Leaks in Preparing the Data* shows `cross_val_score` with a pipeline, raw features and `groups=`.

## When you're done

Every test is green: pytest's last line says `17 passed`. And `python report.py` prints:

```text
1) rows: 2000
2) subscribers: 1700
3) columns left: ['subscriber_id', 'months_subscribed', 'hours_last_month', 'profiles', 'plan', 'support_tickets', 'canceled']
4) training rows: 1589 test rows: 411
5) subscribers on both sides: 0
6) fold accuracies: [0.903, 0.848, 0.898, 0.872, 0.863]
7) honest mean accuracy: 0.877
8) leaky mean accuracy, for comparison: 0.983
```

Line 7 is the score StreamBox can expect from new subscribers; line 8 is the score that only seemed to work. In the notebook, run `honest_cv_score` with seeds 1 and 2 as well: how much do the five accuracies move, and does the mean stay well above the 0.795 of always guessing "stays"?

## Submit your exercise

The exercise is how you finish this lesson. When your work is done, put it on GitHub in a repo of your own, then submit that repo's address here. *Hand In Your Work with Git* walks through each step.

1. On GitHub, click **+**, then **New repository**. Name it (for example `exercise-ml-plug-the-leak`), choose **Public**, and click **Create repository**. Leave out the README: your folder already has one.
2. In VS Code's terminal, in your exercise folder, send your work to your new repo. Point your copy at your own repo, then commit and push:

   ```bash
   git remote set-url origin https://github.com/your-name/exercise-ml-plug-the-leak.git
   git add .
   git commit -m "My solution"
   git push -u origin main
   ```

3. Paste your repo's address, such as `https://github.com/your-name/exercise-ml-plug-the-leak`, into **Submit your exercise** on this page. The site checks that the repo is public and holds your commits, and then the lesson is complete.

Changed something after you submitted? Push it, and submit again: your latest submission is the one that counts.

## So what

A model's score is only honest when every column is known in time, every subscriber is a stranger to the folds that test them, and every step that learns is refit in each fold: three small functions, each one a leak plugged.

## Key terms

- [Data Leakage](https://www.ask-codey.com/glossary/data-leakage)
- [Target Leakage](https://www.ask-codey.com/glossary/target-leakage)
- [GroupShuffleSplit](https://www.ask-codey.com/glossary/groupshufflesplit)
- [Pipeline](https://www.ask-codey.com/glossary/pipeline-python)
- [Cross-Validation](https://www.ask-codey.com/glossary/cross-validation)
- [pytest](https://www.ask-codey.com/glossary/pytest)

## Related

- [What Data Leakage Is](https://www.ask-codey.com/lessons/ml-what-data-leakage-is)
- [Leaks in Preparing the Data](https://www.ask-codey.com/lessons/ml-leakage-in-preprocessing)
- [Leaky Features and Duplicate Rows](https://www.ask-codey.com/lessons/ml-leaky-features-and-duplicates)

Source: https://www.ask-codey.com/lessons/exercise-ml-plug-the-leak
