Text and Dates
Text typed by people and dates written a dozen ways are the hardest part of cleaning. Using a fun run's registrations and results, tidy names, make t-shirt sizes and hometowns match, split and extract parts of text, parse dates, pull out the weekday and hour, and turn finish times into minutes per mile.
Updated
Tidying Text
- Tidying Text with .str (Apprentice): Text that people type differs in spaces and capital letters, so the same name fails to match itself; the .str tools strip, lower and title tidy a whole column at once, and len, contains and startswith test every value so you can find the ones that look wrong.
- Making Categories Match: replace and map (Apprentice): value_counts shows every spelling a column holds, replace with a dictionary fixes the few misspellings you name and leaves every other value alone, map turns every spelling into one standard value and leaves missing whatever it doesn't know, which you count before moving on, and the category type stores a column with a fixed list of values.
- Splitting and Extracting Text (Journeyman): str.split with expand=True turns one text column into several, with n=1 to split at the first space only, and str.extract with the short regular expression (\d+), read 'one or more digits, kept', pulls a number out of text typed in many different ways, where a plain replace would need a call for every wording.
Dates and Times
- Parsing Dates with to_datetime (Apprentice): Dates read from a file are text that sorts and compares wrongly, so pd.to_datetime turns them into real dates, format= says how they were written instead of letting pandas guess, '03/04/2026' is March 4 or April 3 depending on dayfirst, errors='coerce' turns a date that can't exist into NaT for you to list, and parse_dates= does it while loading.
- Parts of a Date: .dt (Apprentice): Once a column holds dates, .dt gives its parts for every row at once, such as year, month, day_name(), hour and date, so you can count by weekday or hour, and subtracting two dates gives a Timedelta, the gap between them, whose .dt.days counts whole days.
- Durations: Finish Times with Timedelta (Journeyman): Finish times typed as text such as '0:27:15' sort wrongly and can't be averaged, so pd.to_timedelta turns them into durations, the ones it can't read, such as '27:15', are found with errors='coerce' and fixed, dt.total_seconds() turns a duration into minutes and a pace per mile, and whole minutes and seconds turn it back into text people can read.
Exercise: Hillcrest 5K Results
- Exercise: Hillcrest 5K Results (Apprentice): Write six functions for a fun run's messy registrations and results, tidying names, standardizing t-shirt sizes, splitting names, counting sign-ups by weekday, and turning typed finish times into minutes and a pace per mile, and let pytest check what each one returns.