# A Cleaning Function, Step by Step

Cleaning steps scattered across notebook cells are hard to run again the same way, so each step becomes a function that starts with result = df.copy() and returns a new table, one clean(raw) function runs them all in order, method chaining with pipe, assign and rename writes it as one readable recipe, and in pandas 3 a single value is changed with .loc.

Journeyman · Python · Updated November 19, 2026

By Gregor Dzierzon

## After this lesson you can

- Write a step function that returns a new DataFrame with result = df.copy(), and change one value with .loc in pandas 3
- Write a clean(raw) function that runs every step in order, using method chaining with pipe, assign and rename

## Key takeaways

- A step function that sets a column on the table it was given changes the caller's table too. Starting with result = df.copy() and returning result leaves the original as it was.
- In pandas 3 a copy is cheap (copy-on-write), and a chained assignment such as df["temp_f"][0] = 61.8 changes nothing. Change one value with .loc[row, column].
- One clean(raw) function runs every step in order and ends with the checks, so the raw table is never changed and every run cleans the same way.
- Method chaining puts one step per line. pipe() calls your own step function, assign() adds a column, and rename() tidies column names.

So far the weather station's cleaning lives in notebook cells, run in an order you have to remember. When June's readings arrive, or next year's, every step has to run again, in the same order, with nothing skipped. A cleaning function does that: you write the steps down once, as a recipe, and every run follows it. It's how a class project's analysis, a lab's data and a team's monthly report stay repeatable.

The example is the Ridgeview School weather station's `data/raw/readings.csv` (`timestamp`, `temp_f`, `humidity_pct`, `rain_in`, `wind_mph`, `battery_v`), with -999 where its sensor dropped out.

## A step that changes what it's given

Here are two steps as functions. The science classes work in Celsius, so the second adds a `temp_c` column:

```python
import numpy as np
import pandas as pd

raw = pd.read_csv("data/raw/readings.csv")

def replace_sentinels(df):
    return df.replace(-999, np.nan)

def add_celsius(df):
    df["temp_c"] = ((df["temp_f"] - 32) * 5 / 9).round(1)
    return df

readings = replace_sentinels(raw)
with_celsius = add_celsius(readings)
print("readings has temp_c:", "temp_c" in readings.columns)
```

```text
readings has temp_c: True
```

`readings` changed too. `replace` hands back a new table, as most pandas methods do, but `df["temp_c"] = ...` changes the table itself, and `df` is the caller's table under another name. Write a step like this on the raw table and the raw data is gone. The fix is to work on a copy:

```python
def add_celsius(df):
    result = df.copy()
    result["temp_c"] = ((result["temp_f"] - 32) * 5 / 9).round(1)
    return result

readings = replace_sentinels(raw)
with_celsius = add_celsius(readings)
print("readings has temp_c:", "temp_c" in readings.columns)
```

```text
readings has temp_c: False
```

`result = df.copy()`, then change `result` and return it: every step that sets a column starts this way. In pandas 3 a copy is cheap, because of **copy-on-write**: the copy shares the data until one of them changes.

## One value: .loc

Copy-on-write has a second effect. Say a teacher's paper log shows the first reading should be 61.8:

```python
fixed = with_celsius.copy()
fixed["temp_f"][0] = 61.8
print("chained:", fixed["temp_f"][0])
fixed.loc[0, "temp_f"] = 61.8
print("with .loc:", fixed["temp_f"][0])
```

```text
chained: 62.3
with .loc: 61.8
```

`fixed["temp_f"][0] = 61.8` is a *chained assignment*: it picks a column, then a value in it. In pandas 3 it changes nothing and prints a `ChainedAssignmentError` warning (older tutorials show it working). **`.loc[row, column] = value`** changes the value in one step.

## One function, every step in order

The steps from *Values That Can't Be Right: Sentinels and Outliers* and *Checking Your Data with assert* become functions the same way, and one `clean` runs them:

```python
def flag_suspect(df):
    result = df.copy()
    temp, humidity = result["temp_f"], result["humidity_pct"]
    impossible_temp = (temp < -30) | (temp > 120)
    impossible_humidity = (humidity < 0) | (humidity > 100)
    result["suspect"] = impossible_temp | impossible_humidity
    result["temp_f"] = temp.mask(impossible_temp)
    result["humidity_pct"] = humidity.mask(impossible_humidity)
    return result

def check_readings(df):
    assert df["temp_f"].dropna().between(-30, 120).all(), "temp_f out of range"
    assert df["humidity_pct"].dropna().between(0, 100).all(), "humidity_pct out of range"
    assert not df["timestamp"].duplicated().any(), "a timestamp appears twice"

def clean(raw):
    readings = replace_sentinels(raw)
    readings = flag_suspect(readings)
    readings = readings.drop_duplicates(subset=["timestamp"])
    check_readings(readings)
    return readings

result = clean(raw)
print("raw rows:", len(raw), "clean rows:", len(result))
print("-999s still in raw:", (raw["temp_f"] == -999).sum())
```

```text
raw rows: 13344 clean rows: 13296
-999s still in raw: 61
```

The order is part of the recipe: run `flag_suspect` before `replace_sentinels` and it flags 132 rows, because every -999 is "below -30". And `raw` still has all 61 sentinels, so you can always compare before and after.

## Method chaining: pipe, assign and rename

**Method chaining** writes the same recipe as one expression, a step per line, inside parentheses. **`pipe(step)`** calls your own function with the table so far, **`assign(temp_c=...)`** adds a column to a new table, and `lambda df: ...` (read "a function of df that gives…") is a one-line function with no name, which `assign` calls with the table at that point:

```python
def clean(raw):
    result = (
        raw.pipe(replace_sentinels)
        .pipe(flag_suspect)
        .drop_duplicates(subset=["timestamp"])
        .assign(temp_c=lambda df: ((df["temp_f"] - 32) * 5 / 9).round(1))
    )
    check_readings(result)
    return result

clean(raw)[["timestamp", "temp_f", "temp_c", "suspect"]].head(3)
```

```text
          timestamp  temp_f  temp_c  suspect
0  2025-09-02 00:00    62.3    16.8    False
1  2025-09-02 00:30    61.2    16.2    False
2  2025-09-02 01:00    62.9    17.2    False
```

**`rename(columns={old: new})`** belongs at the front of a chain when a file names its columns differently. The school's older station, on the gym roof, writes `Temp (F)`:

```python
gym = pd.DataFrame({"Timestamp": ["2026-03-14 16:00"], "Temp (F)": [46.9], "Humidity (%)": [97]})
names = {"Timestamp": "timestamp", "Temp (F)": "temp_f", "Humidity (%)": "humidity_pct"}
list(gym.rename(columns=names).columns)
```

```text
['timestamp', 'temp_f', 'humidity_pct']
```

With `.rename(columns=names)` as the first step of the gym's chain, every step after it finds the names it expects.

## Check yourself

2 questions to try on the lesson page.

## So what

A cleaning function turns a list of cells you have to remember into a recipe that runs the same way every time: each step returns a new table, the checks run at the end, and the raw data is never touched.

## Key terms

- [Method Chaining](https://www.ask-codey.com/glossary/method-chaining-python)
- [Copy-on-Write](https://www.ask-codey.com/glossary/copy-on-write)
- [pipe()](https://www.ask-codey.com/glossary/pipe-python)
- [assign()](https://www.ask-codey.com/glossary/assign)
- [Lambda Function](https://www.ask-codey.com/glossary/lambda-function)

## Related

- [Checking Your Data with assert](https://www.ask-codey.com/lessons/pandas-checking-data-with-assert)
- [Saving the Clean Data: to_csv and to_excel](https://www.ask-codey.com/lessons/pandas-saving-data)

Source: https://www.ask-codey.com/lessons/pandas-cleaning-function
