# Exercise: Fairness Audit

Write two functions that audit the Summit Scholarship Fund's interview screen, one that works out each district's selection rate, true positive rate and false positive rate from its own counts, reporting a rate that would divide by 0 as None, and one that finds the largest gap between districts, and let pytest check the exact rates.

Freelancer · AI Basics · Python · Updated November 19, 2026

By Gregor Dzierzon

The Summit Scholarship Fund's board wants an audit of its interview screen: for each school district, how often the screen invites applicants, how often it invites the qualified ones, and how often it invites the ones who don't meet the standard. You'll write the two functions behind that audit, the same ones you'd reach for whenever a model's results need checking group by group. Tests call each function and check what it gives back, and a notebook comes with the exercise for looking at the data while you work.

## Get the exercise code

Open this lesson's GitHub repository, [github.com/ask-codey/exercise-ml-fairness-audit](https://github.com/ask-codey/exercise-ml-fairness-audit), also linked from the **Exercise** box beside the lesson. Clone it into your `python-course` folder. In VS Code's terminal:

```bash
cd ~/python-course
git clone https://github.com/ask-codey/exercise-ml-fairness-audit.git
```

Git makes a folder, `exercise-ml-fairness-audit`, with the exercise code inside. Open that folder in VS Code (**File > Open Folder**), open its terminal with **Terminal > New Terminal**, and make the exercise's virtual environment. On Windows (PowerShell):

```powershell
python -m venv .venv
.venv\Scripts\Activate.ps1
pip install -r requirements.txt
```

On a Mac (zsh):

```bash
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
```

The prompt starts with `(.venv)` once the virtual environment is on. The exercise's README says what each command does, and what to do if PowerShell won't run the second one.

The folder holds:

- `applicants.csv`: the 2,000 applicants, our own made-up data, the same file the chapter's lessons use.
- `seed_data.py`: the script that made the CSV, with the gap it planted. You don't need to run it: the committed CSV is what keeps everyone's numbers the same.
- `fairness_audit.py`: the two functions, for you to finish, with `MEASURES` (the names of the three rates: `selection_rate`, `true_positive_rate` and `false_positive_rate`), ready to use.
- `explore.ipynb`: a notebook for looking at the applicants and trying your code. Choose the `.venv` kernel with **Select Kernel** (top right) before you run it. The tests don't read it.
- `report.py`: trains the fund's screen the way *Measuring Results Group by Group* does, audits it and the old reviewers' decisions with your two functions, and prints what they gave. Ready to use.
- `tests/`: the tests, which call each function and check what comes back, and `conftest.py`, which gives a test a fresh copy of the applicants. You don't need to change them.
- `requirements.txt` and `pytest.ini`: the packages the exercise needs, at exact versions, and where pytest finds the tests and your code.

## What to do

Both functions in `fairness_audit.py` are already declared with their names and parameters, and for now each stops with a `NotImplementedError` that says *Not built yet.* The docstring under each one says what it should do, with examples:

```python
def largest_gap(rates, measure):
    """Return the largest rate minus the smallest, across the groups, for one measure.

    rates is a dictionary like the one rates_by_group returns, and measure is one of
    MEASURES. Skip any group whose rate for that measure is None. If fewer than two
    groups have a rate for it, there's no gap to measure: return None.

    Example: with North's selection rate 0.6, South's 0.333... and East's 0.333...,
    largest_gap(rates, "selection_rate") returns 0.6 - 0.333... = 0.266...
    With East's true positive rate None, North's 0.666... and South's 0.5,
    largest_gap(rates, "true_positive_rate") returns 0.666... - 0.5 = 0.166...
    """
    raise NotImplementedError("Not built yet.")
```

In each function, replace the `raise NotImplementedError("Not built yet.")` line with your own code:

1. **`rates_by_group(table, group, actual, predicted)`** takes a table and three column names: the column that says each row's group, the right answers (1 for qualified, 0 for not) and the screen's guesses (1 for invited, 0 for not). For each group, it counts the group's true positives, false positives, false negatives and true negatives, and works out its three rates as *Measuring Results Group by Group* does: the selection rate (TP + FP) ÷ the group's rows, the true positive rate TP ÷ (TP + FN), and the false positive rate FP ÷ (FP + TN). It returns a dictionary with one entry per group, in sorted order, each a dictionary of the three rates, not rounded. For North in the tests' small table (TP 2, FN 1, FP 1, TN 1) that's `{"selection_rate": 0.6, "true_positive_rate": 0.666..., "false_positive_rate": 0.5}`.
2. **A rate that would divide by 0 is missing, not zero.** The tests' small East district has no qualified applicants at all, so TP + FN is 0 and East has no true positive rate: `rates_by_group` reports it as Python's `None`, not `0` and not NaN. A rate of 0 would say "the screen turned away every qualified East applicant", which never happened. A group with no unqualified applicants reports its false positive rate as `None` the same way.
3. **`largest_gap(rates, measure)`** returns the largest group's rate minus the smallest for one of the three measures, skipping any group whose rate is `None`, and returns `None` when fewer than two groups have that rate. For the selection rate that gap is Fairlearn's demographic parity difference; the larger of the two other gaps is its equalized odds difference.

Work out each group's rates from its own rows, and leave the table as it was: add no columns to it.

## Check your work

Run the tests from the exercise folder, with the virtual environment on:

```bash
python -m pytest
```

At first all 13 tests fail. That's expected: each function you finish turns its tests green. A failing test's message says what it called, what it expected and what came back, for example *rates_by_group(TABLE, ...)["East"]['true_positive_rate'] should be None (missing, not 0), but it was 0.* Most tests use a small table of 14 applicants written in the test file, with each district's counts in a comment, so you can work out every rate by hand; one checks the old reviewers' decisions in `applicants.csv` against each district's exact counts. No test checks how accurate the screen is: the tests check your arithmetic.

To see your functions at work on the whole file, run `python report.py`.

## Hints

- **Count first, divide second.** Each row lands in exactly one of the four counts, as *The Confusion Matrix* sorts them. Get the four counts for one group right, and the rates follow.
- **One group at a time.** `table.groupby(group)` gives each group's name and its rows in a `for` loop, as the lessons' `test.groupby("district")` does.
- **Missing has its own value.** A conditional expression, `a / b if b > 0 else None`, gives `None` instead of dividing by 0.
- **Sorted keys.** `dict(sorted(rates.items()))` puts a dictionary's entries in order of their keys.
- **Skip what's missing.** In `largest_gap`, a list comprehension with `if value is not None` keeps only the rates that exist; `max` and `min` do the rest.

## When you're done

Every test is green: pytest's last line says `13 passed`. And `python report.py` prints:

```text
1) applicants: 2000
2) test applicants: 500
3) old reviewers, all applicants, North: selection_rate 0.532, true_positive_rate 0.929, false_positive_rate 0.23
4) old reviewers, all applicants, South: selection_rate 0.294, true_positive_rate 0.692, false_positive_rate 0.02
5) old reviewers, all applicants, selection_rate gap: 0.238
6) old reviewers, all applicants, true_positive_rate gap: 0.237
7) old reviewers, all applicants, false_positive_rate gap: 0.21
8) screen, test applicants, North: selection_rate 0.454, true_positive_rate 0.89, false_positive_rate 0.188
9) screen, test applicants, South: selection_rate 0.342, true_positive_rate 0.768, false_positive_rate 0.02
10) screen, test applicants, selection_rate gap: 0.112
11) screen, test applicants, true_positive_rate gap: 0.122
12) screen, test applicants, false_positive_rate gap: 0.168
```

Lines 3 to 7 audit the old reviewers, and lines 8 to 12 the screen trained to copy them, with the district left out. Compare the two: which gaps did the screen keep, and which did it shrink? Then write two sentences for the fund's board, in plain words, about what a qualified South applicant can expect from this screen, using the rates from line 9. Which fairness goal would you ask the board to choose, and who would pay for the gaps that are left?

## Submit your exercise

The exercise is how you finish this lesson. When your work is done, put it on GitHub in a repo of your own, then submit that repo's address here. *Hand In Your Work with Git* walks through each step.

1. On GitHub, click **+**, then **New repository**. Name it (for example `exercise-ml-fairness-audit`), choose **Public**, and click **Create repository**. Leave out the README: your folder already has one.
2. In VS Code's terminal, in your exercise folder, send your work to your new repo. Point your copy at your own repo, then commit and push:

   ```bash
   git remote set-url origin https://github.com/your-name/exercise-ml-fairness-audit.git
   git add .
   git commit -m "My solution"
   git push -u origin main
   ```

3. Paste your repo's address, such as `https://github.com/your-name/exercise-ml-fairness-audit`, into **Submit your exercise** on this page. The site checks that the repo is public and holds your commits, and then the lesson is complete.

Changed something after you submitted? Push it, and submit again: your latest submission is the one that counts.

## So what

An audit is a few honest counts per group and the gaps between them, with "missing" kept apart from zero, and it's the evidence every conversation about a model's fairness has to start from.

## Key terms

- [Selection Rate](https://www.ask-codey.com/glossary/selection-rate)
- [Recall](https://www.ask-codey.com/glossary/recall-ai)
- [False Positive Rate](https://www.ask-codey.com/glossary/false-positive-rate)
- [Demographic Parity](https://www.ask-codey.com/glossary/demographic-parity)
- [Equalized Odds](https://www.ask-codey.com/glossary/equalized-odds)
- [pytest](https://www.ask-codey.com/glossary/pytest)

## Related

- [Where Bias Comes From](https://www.ask-codey.com/lessons/ml-where-bias-comes-from)
- [Measuring Results Group by Group](https://www.ask-codey.com/lessons/ml-fairness-by-group)
- [Fairness Trade-offs](https://www.ask-codey.com/lessons/ml-fairness-tradeoffs)

Source: https://www.ask-codey.com/lessons/exercise-ml-fairness-audit
