A Cleaning Function, Step by Step
Cleaning steps scattered across notebook cells are hard to run again the same way, so each step becomes a function that starts with result = df.copy() and returns a new table, one clean(raw) function runs them all in order, method chaining with pipe, assign and rename writes it as one readable recipe, and in pandas 3 a single value is changed with .loc.
Journeyman · Python · Updated
After this lesson you can
- Write a step function that returns a new DataFrame with result = df.copy(), and change one value with .loc in pandas 3
- Write a clean(raw) function that runs every step in order, using method chaining with pipe, assign and rename
Key takeaways
- A step function that sets a column on the table it was given changes the caller's table too. Starting with result = df.copy() and returning result leaves the original as it was.
- In pandas 3 a copy is cheap (copy-on-write), and a chained assignment such as df["temp_f"][0] = 61.8 changes nothing. Change one value with .loc[row, column].
- One clean(raw) function runs every step in order and ends with the checks, so the raw table is never changed and every run cleans the same way.
- Method chaining puts one step per line. pipe() calls your own step function, assign() adds a column, and rename() tidies column names.
So far the weather station's cleaning lives in notebook cells, run in an order you have to remember. When June's readings arrive, or next year's, every step has to run again, in the same order, with nothing skipped. A cleaning function does that: you write the steps down once, as a recipe, and every run follows it. It's how a class project's analysis, a lab's data and a team's monthly report stay repeatable.
The example is the Ridgeview School weather station's data/raw/readings.csv (timestamp, temp_f, humidity_pct, rain_in, wind_mph, battery_v), with -999 where its sensor dropped out.
A step that changes what it's given
Here are two steps as functions. The science classes work in Celsius, so the second adds a temp_c column:
import numpy as np
import pandas as pd
raw = pd.read_csv("data/raw/readings.csv")
def replace_sentinels(df):
return df.replace(-999, np.nan)
def add_celsius(df):
df["temp_c"] = ((df["temp_f"] - 32) * 5 / 9).round(1)
return df
readings = replace_sentinels(raw)
with_celsius = add_celsius(readings)
print("readings has temp_c:", "temp_c" in readings.columns)
readings has temp_c: True
readings changed too. replace hands back a new table, as most pandas methods do, but df["temp_c"] = ... changes the table itself, and df is the caller's table under another name. Write a step like this on the raw table and the raw data is gone. The fix is to work on a copy:
def add_celsius(df):
result = df.copy()
result["temp_c"] = ((result["temp_f"] - 32) * 5 / 9).round(1)
return result
readings = replace_sentinels(raw)
with_celsius = add_celsius(readings)
print("readings has temp_c:", "temp_c" in readings.columns)
readings has temp_c: False
result = df.copy(), then change result and return it: every step that sets a column starts this way. In pandas 3 a copy is cheap, because of copy-on-write: the copy shares the data until one of them changes.
One value: .loc
Copy-on-write has a second effect. Say a teacher's paper log shows the first reading should be 61.8:
fixed = with_celsius.copy()
fixed["temp_f"][0] = 61.8
print("chained:", fixed["temp_f"][0])
fixed.loc[0, "temp_f"] = 61.8
print("with .loc:", fixed["temp_f"][0])
chained: 62.3
with .loc: 61.8
fixed["temp_f"][0] = 61.8 is a chained assignment: it picks a column, then a value in it. In pandas 3 it changes nothing and prints a ChainedAssignmentError warning (older tutorials show it working). .loc[row, column] = value changes the value in one step.
One function, every step in order
The steps from Values That Can't Be Right: Sentinels and Outliers and Checking Your Data with assert become functions the same way, and one clean runs them:
def flag_suspect(df):
result = df.copy()
temp, humidity = result["temp_f"], result["humidity_pct"]
impossible_temp = (temp < -30) | (temp > 120)
impossible_humidity = (humidity < 0) | (humidity > 100)
result["suspect"] = impossible_temp | impossible_humidity
result["temp_f"] = temp.mask(impossible_temp)
result["humidity_pct"] = humidity.mask(impossible_humidity)
return result
def check_readings(df):
assert df["temp_f"].dropna().between(-30, 120).all(), "temp_f out of range"
assert df["humidity_pct"].dropna().between(0, 100).all(), "humidity_pct out of range"
assert not df["timestamp"].duplicated().any(), "a timestamp appears twice"
def clean(raw):
readings = replace_sentinels(raw)
readings = flag_suspect(readings)
readings = readings.drop_duplicates(subset=["timestamp"])
check_readings(readings)
return readings
result = clean(raw)
print("raw rows:", len(raw), "clean rows:", len(result))
print("-999s still in raw:", (raw["temp_f"] == -999).sum())
raw rows: 13344 clean rows: 13296
-999s still in raw: 61
The order is part of the recipe: run flag_suspect before replace_sentinels and it flags 132 rows, because every -999 is "below -30". And raw still has all 61 sentinels, so you can always compare before and after.
Method chaining: pipe, assign and rename
Method chaining writes the same recipe as one expression, a step per line, inside parentheses. pipe(step) calls your own function with the table so far, assign(temp_c=...) adds a column to a new table, and lambda df: ... (read "a function of df that gives…") is a one-line function with no name, which assign calls with the table at that point:
def clean(raw):
result = (
raw.pipe(replace_sentinels)
.pipe(flag_suspect)
.drop_duplicates(subset=["timestamp"])
.assign(temp_c=lambda df: ((df["temp_f"] - 32) * 5 / 9).round(1))
)
check_readings(result)
return result
clean(raw)[["timestamp", "temp_f", "temp_c", "suspect"]].head(3)
timestamp temp_f temp_c suspect
0 2025-09-02 00:00 62.3 16.8 False
1 2025-09-02 00:30 61.2 16.2 False
2 2025-09-02 01:00 62.9 17.2 False
rename(columns={old: new}) belongs at the front of a chain when a file names its columns differently. The school's older station, on the gym roof, writes Temp (F):
gym = pd.DataFrame({"Timestamp": ["2026-03-14 16:00"], "Temp (F)": [46.9], "Humidity (%)": [97]})
names = {"Timestamp": "timestamp", "Temp (F)": "temp_f", "Humidity (%)": "humidity_pct"}
list(gym.rename(columns=names).columns)
['timestamp', 'temp_f', 'humidity_pct']
With .rename(columns=names) as the first step of the gym's chain, every step after it finds the names it expects.
Check yourself
2 questions to try on the lesson page.
So what
A cleaning function turns a list of cells you have to remember into a recipe that runs the same way every time: each step returns a new table, the checks run at the end, and the raw data is never touched.