When the Answer Is Rare
When the class you're looking for is rare, accuracy rewards ignoring it, so you score a DummyClassifier that always guesses the most common class as the baseline to beat, read recall and balanced accuracy (the average of each class's recall) instead, and know that class_weight='balanced' moves the model toward the rare class while only more examples of it teach the model more.
Journeyman · AI Basics · Python · Updated
After this lesson you can
- Explain why accuracy misleads when one class is rare, and score a DummyClassifier that always guesses the most common class as the baseline to beat
- Compute balanced accuracy as the average of each class's recall, by hand and with balanced_accuracy_score
Key takeaways
- With 2% fraud, a check that never says fraud is right 98% of the time and catches nothing. That's the accuracy trap of imbalanced classes.
- DummyClassifier(strategy='most_frequent') always guesses the training data's most common class. It's the honest baseline a real model has to beat.
- Balanced accuracy is the average of each class's recall. Doing nothing scores 0.5 on two classes, however rare one of them is.
- class_weight='balanced' makes each rare example count more in training, so the model catches more fraud and raises more false alarms. Only more real examples of fraud give it more to learn.
Most of the things people build classifiers to find are rare: fraud among card payments, a failing part among thousands of good ones, a disease in a screening test, a customer about to leave. That's the whole reason a model is worth building: nobody can check every payment by hand to find the 2 in 100. And it's exactly when the usual scores stop working. You'll meet this in nearly every real classification project, and the first sign is a score that looks too good.
The example is still Northside Credit Union's transactions.csv, from Exercise: Fraud Check Scorecard: 5,000 card payments, 100 of them fraud.
The accuracy trap
Accuracy found the trap already: on the 1,000 test payments, a "check" that never says fraud scores 98%, and the trained model scores 98.4%. When one class is much more common than the other, the classes are imbalanced (your textbook may say class imbalance or skewed classes), and accuracy mostly measures how well a model gets the common class right. Getting the common class right takes no skill at all: say it every time.
So the first thing to do with an imbalanced problem is to put a number on doing nothing. scikit-learn's DummyClassifier is a classifier that ignores the features completely. With strategy="most_frequent" it learns one thing in fit, the most common class in the training labels, and guesses it for every row:
import pandas as pd
from sklearn.dummy import DummyClassifier
from sklearn.metrics import accuracy_score, recall_score
from sklearn.model_selection import train_test_split
payments = pd.read_csv("transactions.csv")
features = ["amount", "hour", "miles_from_home", "new_merchant", "online"]
train, test = train_test_split(
payments, test_size=0.2, random_state=42, stratify=payments["is_fraud"]
)
baseline = DummyClassifier(strategy="most_frequent", random_state=0)
baseline.fit(train[features], train["is_fraud"])
baseline_guesses = baseline.predict(test[features])
print("Baseline accuracy:", accuracy_score(test["is_fraud"], baseline_guesses))
print("Baseline recall:", recall_score(test["is_fraud"], baseline_guesses))
Baseline accuracy: 0.98
Baseline recall: 0.0
It's the classifier version of DummyRegressor from Baselines and Residual Plots: the same make, fit, predict, the same job. Its random_state matters only for the strategies that guess at random; "most_frequent" gives the same answer every time, and the setting is there out of habit.
Beating the baseline where it counts
Now the real model, on the same test payments:
from sklearn.linear_model import LogisticRegression
model = LogisticRegression()
model.fit(train[features], train["is_fraud"])
guesses = model.predict(test[features])
print("Model accuracy:", accuracy_score(test["is_fraud"], guesses))
print("Model recall:", recall_score(test["is_fraud"], guesses))
Model accuracy: 0.984
Model recall: 0.4
On accuracy the model is 0.004 ahead of a rule that does nothing, which sounds like no progress. On recall it's 0.4 against 0.0: it catches 8 frauds the baseline never would. The model is useful; accuracy was the wrong ruler. With a rare class, report the recall (and precision) of that class beside the baseline's, never accuracy alone.
Balanced accuracy
Sometimes you do want one number that doesn't fall for the trap. Balanced accuracy is the average of each class's recall: how much of the honest payments the check called honest, and how much of the fraud it called fraud, each counted once, however big its class is.
honest_recall = 976 / 980
fraud_recall = 8 / 20
print("Honest payments let through:", round(honest_recall, 3))
print("Fraud caught:", fraud_recall)
print("Balanced accuracy:", round((honest_recall + fraud_recall) / 2, 3))
Honest payments let through: 0.996
Fraud caught: 0.4
Balanced accuracy: 0.698
The counts are the confusion matrix's: 976 of the 980 honest payments, 8 of the 20 frauds. balanced_accuracy_score gives the same, and shows the baseline for what it is:
from sklearn.metrics import balanced_accuracy_score
print("Model:", round(balanced_accuracy_score(test["is_fraud"], guesses), 3))
print("Baseline:", balanced_accuracy_score(test["is_fraud"], baseline_guesses))
Model: 0.698
Baseline: 0.5
The baseline's recall is 1.0 on honest payments and 0.0 on fraud, so its balanced accuracy is 0.5, the same as a coin flip's, whatever the share of fraud. It's the macro-averaged recall from the classification report, under its own name.
Weighting the rare class, and why data beats tricks
Training sees 4,000 payments and only 80 frauds, so a model learns a lot about honest payments and little about fraud. LogisticRegression(class_weight="balanced") makes each fraud count more while the model learns: each class is weighted by the number of rows divided by (2 × that class's count), so here each fraud counts 49 times as much as an honest payment.
from sklearn.metrics import confusion_matrix, precision_score
weighted = LogisticRegression(class_weight="balanced")
weighted.fit(train[features], train["is_fraud"])
weighted_guesses = weighted.predict(test[features])
print(confusion_matrix(test["is_fraud"], weighted_guesses))
print("Precision:", round(precision_score(test["is_fraud"], weighted_guesses), 3))
print("Recall:", recall_score(test["is_fraud"], weighted_guesses))
[[912 68]
[ 2 18]]
Precision: 0.209
Recall: 0.9
It catches 18 of the 20 frauds, and pauses 68 honest payments to do it. That's a trade, not a free win: the model saw the same 80 frauds and no new facts about them; counting each fraud more heavily changed its weights so that it calls far more payments fraud, much as a lower line would. Moving the Threshold makes that trade on purpose. What does teach a model more about a rare class is more real examples of it: more months of payments, more confirmed frauds. When you can collect them, that beats any setting.
Do it: work out balanced accuracy
The fraud team wants a score for each week's check that can't be fooled by how rare fraud is: balanced accuracy.
The inputs:
actual: what each payment really was, 1 for fraud and 0 for honest.predicted: what the check guessed for the same payments, in the same order, 1 or 0.
For example, with ten payments:
actual = [0, 0, 0, 0, 0, 0, 1, 1, 0, 0]
predicted = [0, 0, 1, 0, 0, 0, 1, 0, 0, 0]
| Class | How many | Guessed right | Recall |
|---|---|---|---|
| Honest (0) | 8 | 7 (payment 3 was flagged) | 7 / 8 = 0.875 |
| Fraud (1) | 2 | 1 (payment 8 was missed) | 1 / 2 = 0.5 |
Balanced accuracy is (0.875 + 0.5) / 2, so balanced_accuracy(actual, predicted) returns 0.6875.
Your job: write one function.
balanced_accuracy(actual, predicted)returns the average of two recalls: the share of honest payments (actual 0) guessed 0, and the share of frauds (actual 1) guessed 1.- The lists always hold at least one honest payment and at least one fraud.
Practiced live on the lesson page: write the Python right there and check it.
Check yourself
2 questions to try on the lesson page.
So what
When the answer you're looking for is rare, a high accuracy is the easiest score in the world to get; put a DummyClassifier's score beside every model, and judge the model by how much of the rare class it finds.
Sources
- scikit-learn, Metrics and scoring: Dummy estimators, User Guide: comparing an estimator against simple rules of thumb is a sanity check;
most_frequentalways predicts the most frequent label in the training set,predictignores the input data with every strategy, and an accuracy too close to a dummy's can mean the classifier is suffering from class imbalance. - scikit-learn, Metrics and scoring: Balanced accuracy score, User Guide: balanced accuracy is the macro-average of the recall scores per class; when a classifier's ordinary accuracy is above chance only because it takes advantage of an imbalanced test set, its balanced accuracy drops to 1 divided by the number of classes.
- scikit-learn 1.9.1, DummyClassifier, API reference: a classifier that makes predictions ignoring the input features, as a simple baseline to compare against other classifiers.
- scikit-learn 1.9.1, LogisticRegression, API reference:
class_weight="balanced"weights the classes inversely proportional to how often they appear in the training labels, as n_samples / (n_classes × np.bincount(y)).