# Impossible Values and a Cleaning Routine

Some values can't be right: a temperature of -999, humidity above 100%. Using a school weather station's readings, turn stand-in values into missing ones, flag what looks wrong instead of deleting it, check your data with assert, and put every cleaning step into one function that turns raw data into a clean file the same way every time.

Updated: November 19, 2026

## [Values That Can't Be Right](https://www.ask-codey.com/courses/python-data-analysis/topics/data-impossible-values)

1. [Values That Can't Be Right: Sentinels and Outliers](https://www.ask-codey.com/lessons/pandas-sentinels-and-outliers) (Apprentice): A stand-in such as -999 for "no reading" and a sensor that reads 152 °F in the sun both throw off an average, so replace turns stand-ins into NaN, describe and between find the values that can't be true, and a suspect column flags them before they're blanked, while real but unusual readings stay.
2. [Checking Your Data with assert](https://www.ask-codey.com/lessons/pandas-checking-data-with-assert) (Journeyman): assert stops the code with an AssertionError and your message whenever a condition is False, so checks on ranges with between and all, on repeated timestamps and on missing values, gathered in one check_readings function, catch a cleaning step that went wrong every time the cleaning runs.

## [A Cleaning Routine](https://www.ask-codey.com/courses/python-data-analysis/topics/data-a-cleaning-routine)

1. [A Cleaning Function, Step by Step](https://www.ask-codey.com/lessons/pandas-cleaning-function) (Journeyman): Cleaning steps scattered across notebook cells are hard to run again the same way, so each step becomes a function that starts with result = df.copy() and returns a new table, one clean(raw) function runs them all in order, method chaining with pipe, assign and rename writes it as one readable recipe, and in pandas 3 a single value is changed with .loc.
2. [Saving the Clean Data: to_csv and to_excel](https://www.ask-codey.com/lessons/pandas-saving-data) (Apprentice): A clean table lives only as long as the notebook's kernel, so to_csv with index=False saves it as a file the next notebook can load, to_excel saves a workbook for people who use Excel, and keeping data/raw and data/clean folders, with a save function that refuses the raw folder, means the raw file is never written over.

## [Exercise: Weather Station Cleanup](https://www.ask-codey.com/courses/python-data-analysis/topics/exercise-data-weather-station-cleanup)

1. [Exercise: Weather Station Cleanup](https://www.ask-codey.com/lessons/exercise-data-weather-station-cleanup) (Journeyman): Write a cleaning routine for a school weather station's messy year of readings, turning -999 into missing values, flagging and blanking readings that can't be true, checking the result with assert, cleaning in one function that leaves the raw table alone, and saving a clean file without touching the raw one, and let pytest check each step.

## [Up Next: Answering Questions](https://www.ask-codey.com/courses/python-data-analysis/topics/data-up-next-questions)

1. [Up Next: Answering Questions](https://www.ask-codey.com/lessons/data-up-next-answering-questions) (Apprentice): Look back at what Module 2 taught, finding and handling missing values, fixing types, removing duplicates, tidying text and dates, catching impossible values and putting it all in a cleaning function, try a cleaning log of your own, and see what Module 3 adds, filtering, sorting, grouping, pivot tables, joins and reshaping.

Source: https://www.ask-codey.com/courses/python-data-analysis/chapters/data-cleaning-routine
