# Where Bias Comes From

A model trained on past decisions learns the old decision-makers' habits along with their judgment, and bias also gets in through groups missing from the data, groups measured less well, proxies and feedback loops, so leaving the sensitive column out of the features doesn't take the gap out of the model.

Journeyman · AI Basics · Python · Updated November 19, 2026

By Gregor Dzierzon

## After this lesson you can

- Name the ways bias gets into a model's data, such as past decisions used as labels, missing groups, uneven measurement and feedback loops
- Explain how proxy columns let a model keep a gap between groups after the group's column is dropped, and check for one by guessing the group

## Key takeaways

- A label made of past decisions teaches the model those decisions, unfair parts included. The model copies what people did, not what was right.
- Bias also gets in through groups with few examples in the data, groups measured less well, and feedback loops, where the model's own choices become the next model's data.
- A proxy is a column that stands in for another. Leave the district out, and work hours and activities still tell the districts apart, so the old gap comes back through them.
- Check a suspected proxy by training a model to guess the group from the other columns. If it guesses well, the group is still in the data.

The Summit Scholarship Fund gets 2,000 applications a year, and its volunteer reviewers invite the strongest applicants to an interview. Reading every application takes them weeks, so the fund asks for a model that screens the applications first, trained on the reviewers' past invitations. Before a model like this decides who gets a chance, someone has to ask whether it treats applicants from different places the same way. You'll ask that question whenever a model's guesses reach people: an interview, a loan, a price, a second look.

The example for the whole chapter is `applicants.csv`: 2,000 made-up applicants from two school districts, North and South. It comes with *Exercise: Fairness Audit*, beside the seed script that made it. The groups here are school districts. The arithmetic in this chapter is the same for any groups of people a model affects.

## The labels are old decisions

Each applicant has a GPA, an essay score, a number of activities, a recommendation from 1 to 5 and the hours they work each week. Two columns are answers. `invited_past` is what the old reviewers decided. `qualified` says whether the applicant meets the fund's own written standard, which looks at GPA, the essay and the recommendation only. Made-up data can know that; real data almost never has a column that says who truly deserved the invitation, and that's part of why fairness is hard to measure.

Each code block is one notebook cell, run in order. The first one loads the file:

```python
import pandas as pd

applicants = pd.read_csv("applicants.csv")
print("Applicants:", len(applicants))
print(applicants.groupby("district")[["qualified", "invited_past"]].mean().round(3))
```

```text
Applicants: 2000
          qualified  invited_past
district
North         0.432         0.532
South         0.408         0.294
```

The mean of a column of 1s and 0s is the share of 1s. About the same share of each district meets the standard, 43.2% and 40.8%. But the old reviewers invited 53.2% of North and 29.4% of South. Something besides the standard moved their choices.

A model trained to copy `invited_past` learns whatever moved them. That's the first and most common way bias gets in: **past decisions used as labels**. The model can't tell the reviewers' good judgment from their habits; both are in the label it's told to match. A model whose results are unfair to a group in this way is said to have **algorithmic bias**.

## Other ways bias gets in

- **Who's missing.** If the fund had advertised only in North's schools, few South students would have applied, and the model would have few South examples to learn from. Its guesses for South would rest on very little.
- **Measured less well.** A recommendation from a school where each counselor writes for 400 students may be shorter than one from a school with 100 students per counselor. The same student gets a lower number, because the measurement is worse, not the student.
- **Feedback loops.** Suppose this year's model invitations become next year's training labels. South applicants the model turns away never get the interview where they could show what they'd do, so next year's data says the same thing again. The model's choices feed the next model.

## Leaving the column out

A first idea for a fix: don't give the model the `district` column. This cell trains the screen on the other five columns only, then counts the invitations it would send to the 500 test applicants. `district` is a **sensitive feature**: Fairlearn's word for the column that names the groups you want to check.

```python
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

features = ["gpa", "essay_score", "activities", "recommendation", "work_hours_per_week"]
train, test = train_test_split(applicants, test_size=0.25, random_state=42)
screen = make_pipeline(StandardScaler(), LogisticRegression())
screen.fit(train[features], train["invited_past"])
test = test.assign(invited=screen.predict(test[features]))
print(test.groupby("district")["invited"].mean().round(3))
```

```text
district
North    0.454
South    0.342
Name: invited, dtype: float64
```

The model never saw a district, and it still invites 45.4% of North's test applicants and 34.2% of South's. Its weights show how:

```python
weights = screen.named_steps["logisticregression"].coef_[0]
for name, weight in zip(features, weights):
    print(f"{name}: {weight:.2f}")
```

```text
gpa: 4.30
essay_score: 2.75
activities: 2.22
recommendation: 2.35
work_hours_per_week: -0.54
```

Activities count for a lot, as they did for the old reviewers. And work hours get a negative weight, though neither the standard nor the old reviewers looked at them. Here are the two districts' averages on those columns:

```python
print(applicants.groupby("district")[["activities", "work_hours_per_week"]].mean().round(1))
```

```text
          activities  work_hours_per_week
district
North            3.9                  6.3
South            2.2                 16.0
```

South applicants work more hours and list fewer activities, so these two columns point to the district. A column that stands in for another like this is a **proxy**. The model learned the old North bonus through them.

## Checking for a proxy

To check whether the group is still hiding in the data, try to guess it from the other columns. If a model can tell the districts apart, so can the screen:

```python
clues = ["activities", "work_hours_per_week"]
guesser = make_pipeline(StandardScaler(), LogisticRegression())
guesser.fit(train[clues], train["district"])
print("District guessed right:", round(guesser.score(test[clues], test["district"]), 3))
```

```text
District guessed right: 0.844
```

From two columns, the district is right for 84.4% of the test applicants. That's why dropping the sensitive column isn't enough: with many columns, the group can usually be pieced back together from the rest. A guess this good doesn't prove the screen is unfair; it says the gap has a way in, and the only way to know what the screen does is to measure its results group by group, which *Measuring Results Group by Group* does. The weights describe what the model learned from these applicants. They don't say why South students work more hours, and they don't make work hours a sign of merit.

![A table of applicants with the district column crossed out and set aside, while two other columns, activities and work hours per week, are joined to it by dotted lines, and a model reading the table still sends 45.4% of North's applicants and 34.2% of South's to interview](https://fsdtrainingmedia.blob.core.windows.net/public/images/2026/11/ab0b9e290ffa45529233a90c2a6ce005.webp)

## Check yourself

2 questions to try on the lesson page.

## So what

A model learns what its data shows, including old unfairness and the columns that carry it, so a fair-looking feature list proves nothing until you measure what the model does to each group.

## Sources

- Solon Barocas, Moritz Hardt and Arvind Narayanan, *Fairness and Machine Learning: Limitations and Opportunities* (MIT Press, 2023), [Chapter 1, Introduction](https://fairmlbook.org/introduction.html): historical examples of outcomes "will almost always reflect historical prejudices"; groups underrepresented in the data, with fewer data points about them; measurement "fraught with subjective decisions and technical difficulties"; and predictions that reinforce outcomes in feedback loops.
- Barocas, Hardt and Narayanan, [Chapter 3, Classification](https://fairmlbook.org/classification.html): removing the sensitive attribute "can be ineffective and even harmful", because sensitive attributes are generally redundant given the other features and a classifier finds a redundant encoding of them; target variables can reflect existing inequality.
- Fairlearn, [Fairness in Machine Learning](https://fairlearn.org/main/user_guide/fairness_in_machine_learning.html), User Guide: machine learning systems are sociotechnical; allocation harms, when a system extends or withholds opportunities; sensitive features name the groups being checked, and "sensitive" doesn't mean the feature must be left out of predictions.
- scikit-learn 1.9.1, [LogisticRegression](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html), API reference: `coef_` holds one weight per feature; `score` returns the mean accuracy on the given data and labels.

## Key terms

- [Algorithmic Bias](https://www.ask-codey.com/glossary/algorithmic-bias)
- [Proxy](https://www.ask-codey.com/glossary/proxy-ai)
- [Sensitive Feature](https://www.ask-codey.com/glossary/sensitive-feature)
- [Feedback Loop](https://www.ask-codey.com/glossary/feedback-loop)

## Related

- [Measuring Results Group by Group](https://www.ask-codey.com/lessons/ml-fairness-by-group)
- [Fairness Trade-offs](https://www.ask-codey.com/lessons/ml-fairness-tradeoffs)
- [Exercise: Fairness Audit](https://www.ask-codey.com/lessons/exercise-ml-fairness-audit)

Source: https://www.ask-codey.com/lessons/ml-where-bias-comes-from
