Exercise: Tidy the Shelter Intake Log
Write five cleaning functions for an animal shelter's messy intake log, reading hidden blanks as missing, turning weights and fees into numbers, filling missing ages honestly with a flag, and dropping double entries, and let pytest check the table each one returns.
Apprentice · Python · Updated
Happy Tails Animal Shelter wants a monthly report on the animals it takes in, and its intake log isn't ready for one: volunteers typed ages as unknown, weights as 59 lbs or 5.5, fees as $75 or 75.00, and entered some animals twice. You'll write the five functions that clean it, each one step of the cleaning, and tests call each function and check the table it hands back. A notebook comes with the exercise for looking at the data while you work.
Get the exercise code
Open this lesson's GitHub repository, github.com/ask-codey/exercise-data-shelter-intake, also linked from the Exercise box beside the lesson. Clone it into your python-course folder. In VS Code's terminal:
cd ~/python-course
git clone https://github.com/ask-codey/exercise-data-shelter-intake.git
Git makes a folder, exercise-data-shelter-intake, with the exercise code inside. Get the Exercise Code with Git walks through cloning step by step, if you'd like a reminder. Open that folder in VS Code (File > Open Folder), open its terminal with Terminal > New Terminal, and make the exercise's virtual environment. On Windows (PowerShell):
python -m venv .venv
.venv\Scripts\Activate.ps1
pip install -r requirements.txt
On a Mac (zsh):
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
The prompt starts with (.venv) once the virtual environment is on. The exercise's README says what each command does, and what to do if PowerShell won't run the second one. Then make the data, once:
python seed.py
Wrote intake.csv: 800 rows.
The folder holds:
seed.py: makesintake.csv, the shelter's log of 800 arrivals, our own made-up data. Its fixed random seed means everyone gets the same rows.intake.py: the five functions, for you to finish.explore.ipynb: a notebook for looking at the log and trying your code. Choose the.venvkernel with Select Kernel (top right) before you run it. The tests don't read it.report.py: cleans the log with your five functions and prints what each step changed. Ready to use.tests/: the tests, which call each function and check what comes back, andconftest.py, which gives every test its own fresh copy of the log. You don't need to change them.requirements.txtandpytest.ini: the packages the exercise needs, at exact versions, and where pytest finds the tests and your code.
What to do
Every function in intake.py is already declared with its name and parameter, and for now it stops with a NotImplementedError that says Not built yet. The docstring under each one says what it should do, with examples:
def fix_weights(df):
"""Return a new DataFrame whose weight_lbs column holds numbers (float64), not text.
Take ' lbs' off each weight, then convert the column with pd.to_numeric, so a weight
that still isn't a number, such as 'about 15', becomes NaN. A missing weight stays NaN.
'59 lbs' -> 59.0
'5.5' -> 5.5
'about 15' -> NaN
"""
raise NotImplementedError("Not built yet.")
In each function, replace the raise NotImplementedError("Not built yet.") line with your own code:
read_intake(path)reads the CSV file atpath, withunknown,n/aand-read as missing, as empty cells are. On the seeded log, 117 ages come out missing.fix_weights(df)returns a new table whoseweight_lbsholds numbers:59 lbsbecomes59.0, and a value that still isn't a number, such asabout 15, becomesNaN.fix_fees(df)does the same foradoption_fee, taking off$and,:$75and75.00both become75.0.fill_missing_ages(df)adds anage_estimatedcolumn (Truewhere the age was missing), then fills each missing age with the median age of its species. With dogs aged 2, 4, missing and 10, the missing age becomes 4.0.drop_double_entries(df)removes every row with the sameanimal_idandintake_dateas a row above it, keeping the first. The same animal on two different dates is two real visits, and both rows stay.
Each function returns a new table and leaves the one it was given unchanged; the tests check that too. When a function changes a column, start it with result = df.copy(), change result, and return it.
Check your work
Run the tests from the exercise folder, with the virtual environment on:
python -m pytest
At first all 19 tests fail. That's expected: each function you finish turns its tests green. A failing test's message says what it called, what it expected and what came back, for example fix_weights(df)['weight_lbs'] for '59 lbs', '5.5', '42' should give [59.0, 5.5, 42.0], but it gave ['59', '5.5', '42'].
To see your functions at work on the whole log, run python report.py.
Hints
- Look before you write. Run the notebook's cells:
raw.isna().sum()andvalue_counts()show the gaps and the odd spellings, as Finding Missing Values: NaN and isna does. - One argument does
read_intake's whole job.read_csvtakes a list of the words that mean missing in this file. - Two steps for every number in text.
.str.replacetakes the unit or symbol off, andpd.to_numeric(..., errors="coerce")converts what's left, as Turning Text into Numbers: to_numeric shows. Then check which values becameNaN. - Flag first, then fill. Once the gaps are filled, nothing shows where they were. Dropping or Filling Missing Values fills each age with its species' median using
groupby("species")["age_years"].transform("median"). - Choose the columns that make two rows the same event.
drop_duplicateswithsubset=, as Finding and Removing Duplicates shows; the animal alone isn't enough.
When you're done
Every test is green: pytest's last line says 19 passed. And python report.py prints:
1) rows read: 800
2) ages missing: 117
3) rows after dropping double entries: 762
4) different animals: 750
5) weights that became missing: 4
6) average weight (lbs): 28.7
7) fees missing: 47
8) total of the fees: 75300.0
9) ages estimated: 113
10) average age (years): 4.8
Now look at lines 2 and 9: 117 ages were missing before the double entries were dropped, and 113 after. Which order of steps is right for a report, and why does it matter for the medians your function used?
Submit your exercise
The exercise is how you finish this lesson. When your work is done, put it on GitHub in a repo of your own, then submit that repo's address here. Hand In Your Work with Git walks through each step.
On GitHub, click +, then New repository. Name it (for example
exercise-data-shelter-intake), choose Public, and click Create repository. Leave out the README: your folder already has one.In VS Code's terminal, in your exercise folder, send your work to your new repo. Point your copy at your own repo, then commit and push:
git remote set-url origin https://github.com/your-name/exercise-data-shelter-intake.git git add . git commit -m "My solution" git push -u origin mainPaste your repo's address, such as
https://github.com/your-name/exercise-data-shelter-intake, into Submit your exercise on this page. The site checks that the repo is public and holds your commits, and then the lesson is complete.
Changed something after you submitted? Push it, and submit again: your latest submission is the one that counts.
So what
Cleaning is a set of small, honest decisions, each one a function you can test: what counts as missing, what a number really is, what a guess looks like, and when two rows are one event. Written down as code, every report after this one gets the same careful answer.