Exercise: Model Check-Up
Give Pine Ridge Delivery's year-old model its check-up in Python, saving the whole pipeline with joblib and loading it back only when the scikit-learn version matches, reporting each column's share of new orders outside the training range and its change in mean, and building a model card that refuses an empty section, with pytest checking each step.
Journeyman · AI Basics · Python · Updated
A year after Pine Ridge Delivery's app started using its model, the team wants a check-up they can run any time: save the model safely and load it back, see whether this year's orders still look like the ones it learned from, and keep a model card that says what it's for and where it must not be used. You'll write the four functions behind that check-up, the way When the World Changes, Saving and Loading a Model and Model Cards worked them out. Tests check each one, so you'll know the saved model comes back the same and your drift numbers are exact.
Get the exercise code
Open this lesson's GitHub repository, github.com/ask-codey/exercise-ml-model-check-up, also linked from the Exercise box beside the lesson. Clone it into your python-course folder. In VS Code's terminal:
cd ~/python-course
git clone https://github.com/ask-codey/exercise-ml-model-check-up.git
Git makes a folder, exercise-ml-model-check-up, with the exercise code inside. Open that folder in VS Code (File > Open Folder), open its terminal with Terminal > New Terminal, and make the exercise's virtual environment. On Windows (PowerShell):
python -m venv .venv
.venv\Scripts\Activate.ps1
pip install -r requirements.txt
On a Mac (zsh):
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
The prompt starts with (.venv) once the virtual environment is on. The exercise's README says what each command does, and what to do if PowerShell won't run the second one.
The folder holds:
deliveries.csvanddeliveries_new.csv: last year's 500 orders and this year's 300, our own made-up data, the same files the chapter's lessons use.seed_data.py: the script that made both files, with what changed in a year. You don't need to run it: the committed CSVs are what keep everyone's numbers the same.check_up.py: the four functions, for you to finish, andSECTIONS, the card's seven sections.explore.ipynb: a notebook for comparing the two years' orders, drawing their distances, and trying your functions. Choose the.venvkernel with Select Kernel (top right) before you run it. The tests don't read it.report.py: trains the model, saves and loads it, checks this year's orders and builds the card with your functions. Ready to use.tests/: the tests, which call each function and check what comes back, andconftest.py, which gives a test fresh copies of the orders and the trained model. You don't need to change them.requirements.txtandpytest.ini: the packages the exercise needs, at exact versions, and where pytest finds the tests and your code.
What to do
Every function in check_up.py is already declared with its name and parameters, and for now it stops with a NotImplementedError that says Not built yet. The docstring under each one says what it should do, with examples:
def load_model(path):
"""Load a file that save_model wrote, and return the pipeline inside it.
Read the dictionary with joblib.load. If its "sklearn_version" isn't the version
running now (sklearn.__version__), don't return the model: raise a ValueError whose
message names both versions, in this shape:
Saved with scikit-learn 1.8.0, but this is scikit-learn 1.9.1.
Only ever load files you made yourself: loading a joblib file can run code inside it.
model = load_model("delivery_model.joblib") -> the pipeline, ready to predict
"""
raise NotImplementedError("Not built yet.")
In each function, replace the raise NotImplementedError("Not built yet.") line with your own code.
save_model(pipeline, path)saves one dictionary topathwithjoblib.dump:"model"holds the whole pipeline, scaler and all, and"sklearn_version"holdssklearn.__version__, the version running now. It returns nothing.load_model(path)reads that dictionary back withjoblib.loadand returns the pipeline. If the file's"sklearn_version"isn't the version running now, it raises aValueErrorwith the message Saved with scikit-learn 1.8.0, but this is scikit-learn 1.9.1., with the file's version first and the running version second.drift_report(train, new, columns)returns a dictionary with one entry per column incolumns, in that order and no others. Each entry is a dictionary with"share_outside", the share ofnew's values belowtrain's smallest or above its largest (a value equal to either is inside), and"mean_change",new's mean minustrain's. With training distances[1.0, 2.0, 4.0, 5.0]and new ones[2.0, 6.0, 7.0, 5.0], the distance entry is{"share_outside": 0.5, "mean_change": 2.0}: 6.0 and 7.0 are above 5.0, 2 of 4, and the mean went from 3.0 to 5.0. It doesn't change either table.model_card(model, date_trained, intended_use, not_for, data, performance, limits)returns a dictionary whose keys are the sevenSECTIONS, in order,"Model"holdingmodel,"Date trained"holdingdate_trained, and so on. It raises aValueErrorwhose message starts with the section's name whenmodel,date_trained,intended_useordatais empty or only spaces, whennot_fororlimitsis an empty list, or whenperformancedoesn't have an"overall"entry and at least one group beside it.
Check your work
Run the tests from the exercise folder, with the virtual environment on:
python -m pytest
At first all 19 tests fail. That's expected: each function you finish turns its tests green. A failing test's message says what it called, what it expected and what came back. The save-and-load tests write their files in tmp_path, a temporary folder pytest makes for each test and removes later, so they never touch your exercise folder; they check the loaded pipeline makes exactly the same guesses for all 100 test orders, that it's the whole pipeline, and that a file saved by another version is refused. The drift tests use small tables written in the test file, so every number can be worked out by hand, and then the two CSV files. No test checks how good the model is.
To see your functions at work on the whole files, run python report.py.
Hints
- One dictionary in the file.
save_modelandload_modelagree on the dictionary's two keys;load_modelcompares"sklearn_version"before it returns anything. - The range comes from the training rows.
train[column].min()andtrain[column].max()are the edges;new[column] < lowgives oneTrueorFalseper row, and.sum()counts theTrueones. - Strictly outside. A value equal to the smallest or the largest is inside, so use
<and>, not<=and>=. - Build the card, then check it.
" ".strip()is"", which counts as empty;len(not_for) == 0finds an empty list;"overall" in performancechecks for the overall entry. - Stop with a message.
raise ValueError(f"{section} is empty.")stops the function there and shows the message after the error's type; each docstring gives its message's exact words.
When you're done
Every test is green: pytest's last line says 19 passed. And python report.py prints:
1) same guesses after loading: True
2) distance_miles: outside the training range 0.25, mean change +1.87
3) stops_before: outside the training range 0.00, mean change -0.05
4) is_raining: outside the training range 0.00, mean change +0.02
5) model card sections: Model, Date trained, Intended use, Not for, Data, Performance, Limits
6) MAE in minutes (overall, dry, raining): 3.63 3.91 2.33
The drift report says the distances moved and the stops didn't. But When the World Changes found the stops cost more time this year. Line 3 can't show that: what can? In the notebook, run the cell that averages this year's errors by the number of stops. Then write one more item for the card's Limits, in words a delivery manager would understand, as a Markdown cell at the end of the notebook. There's no test for that one: when the manager would know what to watch for, it works.
Submit your exercise
The exercise is how you finish this lesson. When your work is done, put it on GitHub in a repo of your own, then submit that repo's address here. Hand In Your Work with Git walks through each step.
On GitHub, click +, then New repository. Name it (for example
exercise-ml-model-check-up), choose Public, and click Create repository. Leave out the README: your folder already has one.In VS Code's terminal, in your exercise folder, send your work to your new repo. Point your copy at your own repo, then commit and push:
git remote set-url origin https://github.com/your-name/exercise-ml-model-check-up.git git add . git commit -m "My solution" git push -u origin mainPaste your repo's address, such as
https://github.com/your-name/exercise-ml-model-check-up, into Submit your exercise on this page. The site checks that the repo is public and holds your commits, and then the lesson is complete.
Changed something after you submitted? Push it, and submit again: your latest submission is the one that counts.
So what
A model you can save safely, check against new data and describe honestly is a model other people can rely on, and know when not to.