Exercise: Plug the Leak

Write three functions that close the leaks in StreamBox's 2,000 subscriber rows, dropping the column known only after a cancellation, splitting so no subscriber is on both sides, and cross-validating a scaled model with the scaler inside a pipeline, and let pytest check that nothing leaks.

Freelancer · AI Basics · Python · Updated

By

StreamBox's first model of who will cancel scored 99%, and it only seemed to work: a column in the table is the answer in disguise, and 300 subscribers are listed twice. You'll write the three functions that plug each leak: one drops the columns StreamBox only knows after a cancellation, one splits the rows so no subscriber is on both sides, and one cross-validates a scaled model with the scaling done inside each fold. The tests don't check how accurate the model is; they check that nothing leaks, and one of them watches every row the scaler learns from.

Get the exercise code

Open this lesson's GitHub repository, github.com/ask-codey/exercise-ml-plug-the-leak, also linked from the Exercise box beside the lesson. Clone it into your python-course folder. In VS Code's terminal:

cd ~/python-course
git clone https://github.com/ask-codey/exercise-ml-plug-the-leak.git

Git makes a folder, exercise-ml-plug-the-leak, with the exercise code inside. Open that folder in VS Code (File > Open Folder), open its terminal with Terminal > New Terminal, and make the exercise's virtual environment. On Windows (PowerShell):

python -m venv .venv
.venv\Scripts\Activate.ps1
pip install -r requirements.txt

On a Mac (zsh):

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

The prompt starts with (.venv) once the virtual environment is on. The exercise's README says what each command does, and what to do if PowerShell won't run the second one.

The folder holds:

  • subscribers.csv: the 2,000 rows for 1,700 subscribers, our own made-up data, the same file the chapter's lessons use.
  • seed_data.py: the script that made the CSV, with the pattern it planted and the two leaks planted with it. You don't need to run it: the committed CSV is what keeps everyone's numbers the same.
  • churn_model.py: the three functions, for you to finish, with FEATURES (the four columns the model looks at), LABEL (canceled), GROUP (subscriber_id) and KNOWN_LATER (["exit_survey_sent"]), ready to use.
  • explore.ipynb: a notebook for looking at the subscribers and trying your code. Choose the .venv kernel with Select Kernel (top right) before you run it. The tests don't read it.
  • report.py: plugs the leaks with your three functions and prints what each step gave, next to a leaky score it makes the wrong way on purpose. Ready to use.
  • tests/: the tests, which call each function and check what comes back, and conftest.py, which gives each test a fresh copy of the subscribers. You don't need to change them.
  • requirements.txt and pytest.ini: the packages the exercise needs, at exact versions, and where pytest finds the tests and your code.

What to do

Every function in churn_model.py is already declared with its name and parameters, and for now it stops with a NotImplementedError that says Not built yet. The docstring under each one says what it should do, with examples:

def split_by_subscriber(subscribers, seed):
    """Split the rows into a training table and a test table, with no subscriber in both.

    Use GroupShuffleSplit(n_splits=1, test_size=0.2, random_state=seed), with the
    GROUP column as the groups, so a fifth of the subscribers (not of the rows) go
    to the test table and every row of a subscriber lands on the same side.
    Pass seed on as random_state, so the same seed always gives the same split.

    Return two tables, training first: train, test. Each keeps every column.

    With the 2,000 rows in subscribers.csv: 340 of the 1,700 subscribers in the
    test table, and no subscriber_id in both tables.
    """
    raise NotImplementedError("Not built yet.")

In each function, replace the raise NotImplementedError("Not built yet.") line with your own code:

  1. drop_after_the_fact(subscribers, known_later) returns a new table without the columns named in known_later, keeping every row and every other column in its order, and leaving the table it was given unchanged. With ["exit_survey_sent"]: 2,000 rows and seven columns, canceled among them.
  2. split_by_subscriber(subscribers, seed) splits the rows with GroupShuffleSplit(n_splits=1, test_size=0.2, random_state=seed), grouped by subscriber_id, and returns train, test. With seed 42: 340 of the 1,700 subscribers in the test table, and none in both.
  3. honest_cv_score(subscribers, seed) cross-validates make_pipeline(StandardScaler(), LogisticRegression()) on the FEATURES columns and the LABEL column, with GroupKFold(n_splits=5, shuffle=True, random_state=seed) and the subscriber_id column as groups, and returns the five accuracies cross_val_score gives, as a list of floats.

Nothing is fit on the whole table first: the scaler lives in the pipeline, and cross_val_score fits a fresh copy in each fold. That's what the recording test checks.

Check your work

Run the tests from the exercise folder, with the virtual environment on:

python -m pytest

At first all 17 tests fail. That's expected: each function you finish turns its tests green. A failing test's message says what it called, what it expected and what came back, for example how many different subscribers the test table holds should return 340, but it returned 400. Four of the tests swap in a scaler that writes down every row it learns from and every row it scales: they fail if the scaler ever learns from all 2,000 rows, if a row is only ever scaled by a scaler that learned from it, if a subscriber is in a fold's training rows and its test rows at once, or if the scaler is given any column but the four features.

To see your functions at work on the whole file, run python report.py.

Hints

  • Look before you write. The notebook's cells show the survey column against canceled, and the subscribers listed twice.
  • A new table, not a changed one. drop(columns=...) gives back a new table and leaves the old one as it was; What Data Leakage Is uses it to set the features apart.
  • Positions, not rows. A splitter's split gives back row positions: next(...) takes its one split, and .iloc turns positions into rows. Leaky Features and Duplicate Rows has the whole pattern.
  • Groups go to the splitter. GroupShuffleSplit and GroupKFold keep a subscriber together only when they're given groups=.
  • Everything that learns goes in the pipeline. Leaks in Preparing the Data shows cross_val_score with a pipeline, raw features and groups=.

When you're done

Every test is green: pytest's last line says 17 passed. And python report.py prints:

1) rows: 2000
2) subscribers: 1700
3) columns left: ['subscriber_id', 'months_subscribed', 'hours_last_month', 'profiles', 'plan', 'support_tickets', 'canceled']
4) training rows: 1589 test rows: 411
5) subscribers on both sides: 0
6) fold accuracies: [0.903, 0.848, 0.898, 0.872, 0.863]
7) honest mean accuracy: 0.877
8) leaky mean accuracy, for comparison: 0.983

Line 7 is the score StreamBox can expect from new subscribers; line 8 is the score that only seemed to work. In the notebook, run honest_cv_score with seeds 1 and 2 as well: how much do the five accuracies move, and does the mean stay well above the 0.795 of always guessing "stays"?

Submit your exercise

The exercise is how you finish this lesson. When your work is done, put it on GitHub in a repo of your own, then submit that repo's address here. Hand In Your Work with Git walks through each step.

  1. On GitHub, click +, then New repository. Name it (for example exercise-ml-plug-the-leak), choose Public, and click Create repository. Leave out the README: your folder already has one.

  2. In VS Code's terminal, in your exercise folder, send your work to your new repo. Point your copy at your own repo, then commit and push:

    git remote set-url origin https://github.com/your-name/exercise-ml-plug-the-leak.git
    git add .
    git commit -m "My solution"
    git push -u origin main
    
  3. Paste your repo's address, such as https://github.com/your-name/exercise-ml-plug-the-leak, into Submit your exercise on this page. The site checks that the repo is public and holds your commits, and then the lesson is complete.

Changed something after you submitted? Push it, and submit again: your latest submission is the one that counts.

So what

A model's score is only honest when every column is known in time, every subscriber is a stranger to the folds that test them, and every step that learns is refit in each fold: three small functions, each one a leak plugged.

Key terms

Related