# Moving the Threshold

Lowering a classifier's threshold below 0.5 flags more payments, so recall goes up and precision usually goes down, precision_recall_curve and PrecisionRecallDisplay show the whole trade, and you choose the threshold from what each mistake costs, on validation data set aside for choosing, then score the test data once.

Freelancer · AI Basics · Python · Updated November 19, 2026

By Gregor Dzierzon

## After this lesson you can

- Turn predict_proba's chances into labels at a threshold, and explain how lowering the threshold trades precision for recall
- Choose a threshold from what each kind of mistake costs, on validation data, and score the test data only once with it

## Key takeaways

- The threshold is the line on predict_proba's chance where a payment gets flagged. predict uses 0.5, and you can use any other.
- Lowering the threshold flags more payments, so more fraud is caught (recall up) and more honest payments are flagged (precision usually down).
- precision_recall_curve gives precision and recall at every threshold, and PrecisionRecallDisplay draws them as a curve.
- Choose the threshold from what each mistake costs, on validation data. The test data is for the final check only, or its score stops being honest.
- TunedThresholdClassifierCV does the same search with cross-validation, scoring each threshold with the metric you give it.

Northside Credit Union's fraud check catches 8 of every 20 frauds, and *Precision and Recall* worked out what the other 12 cost: $1,661.51 on the test payments. The finance manager asks the natural question: "Can't it flag more?" It can, without training anything new. Every classifier that gives a probability has a dial, and deciding where to set it is one of the most practical decisions in machine learning: you'll make it for fraud checks, spam filters, screening tests and any model whose two kinds of mistakes cost different amounts.

The example is still `transactions.csv`, from *Exercise: Fraud Check Scorecard*.

## The line on the chance

*Probabilities and the 0.5 Line* showed that `predict_proba` gives each row a chance of each class, and that `predict` calls a row the positive class when its chance is above 0.5. That 0.5 is the **threshold**, and nothing about fraud says it should be 0.5. A payment with a 30% chance of fraud is worth a phone call when a missed fraud costs hundreds of dollars.

But choosing the threshold is choosing, and *Training and Testing: Never Grade Your Own Homework* has a rule for that: anything chosen by looking at the test data makes the test score look better than it will be. So this lesson splits the payments three ways. **Validation data** is held back from training for choosing, here the threshold; the test data stays untouched for one final check. (*Validation Sets: Keep the Test Set for Last*, in Module 4, goes further.)

```python
import pandas as pd
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split

payments = pd.read_csv("transactions.csv")
features = ["amount", "hour", "miles_from_home", "new_merchant", "online"]
rest, test = train_test_split(
    payments, test_size=0.2, random_state=42, stratify=payments["is_fraud"]
)
train, valid = train_test_split(
    rest, test_size=0.25, random_state=42, stratify=rest["is_fraud"]
)
print("Training:", len(train))
print("Validation:", len(valid))
print("Test:", len(test))

model = LogisticRegression()
model.fit(train[features], train["is_fraud"])
chances = model.predict_proba(valid[features])[:, 1]
```

```text
Training: 3000
Validation: 1000
Test: 1000
```

The test part is the same 1,000 payments as the rest of the chapter; the other 4,000 are split into 3,000 for training and 1,000 for validation, each with 20 frauds (a quarter of 4,000 is 1,000). Column 1 of `predict_proba` is the chance of fraud, because `classes_` is `[0, 1]`.

## The trade

Here's the validation data at five thresholds. A payment is flagged when its chance is at or above the line:

```python
from sklearn.metrics import precision_score, recall_score

for threshold in [0.5, 0.2, 0.1, 0.05, 0.02]:
    flags = (chances >= threshold).astype(int)
    p = precision_score(valid["is_fraud"], flags)
    r = recall_score(valid["is_fraud"], flags)
    print(f"{threshold}: {flags.sum()} flagged, precision {p:.2f}, recall {r:.2f}")
```

```text
0.5: 11 flagged, precision 0.91, recall 0.50
0.2: 15 flagged, precision 0.73, recall 0.55
0.1: 28 flagged, precision 0.50, recall 0.70
0.05: 38 flagged, precision 0.42, recall 0.80
0.02: 75 flagged, precision 0.24, recall 0.90
```

Lowering the line flags more payments. Recall can only go up, because every fraud flagged at 0.5 is still flagged at 0.2, plus some new ones. Precision usually goes down, because the payments a little above a low line are mostly honest. Neither end is free: at 0.5 the check misses half the fraud, and at 0.02 three flags in four are false alarms.

`precision_recall_curve(y_true, chances)` works out precision and recall at every threshold the chances allow, and `PrecisionRecallDisplay` draws them:

```python
import matplotlib.pyplot as plt
from sklearn.metrics import PrecisionRecallDisplay

PrecisionRecallDisplay.from_predictions(valid["is_fraud"], chances)
plt.show()
```

![The fraud check's precision-recall curve on 1,000 validation payments: recall across the bottom from 0 to 1 and precision up the side from 0 to 1, a line that stays flat at precision 1.0 until recall 0.5, then steps down to the right, with five marked points: threshold 0.5 at recall 0.50 and precision 0.91, threshold 0.2 at recall 0.55 and precision 0.73, threshold 0.1 at recall 0.70 and precision 0.50, threshold 0.05 at recall 0.80 and precision 0.42, and threshold 0.02 at recall 0.90 and precision 0.24](https://fsdtrainingmedia.blob.core.windows.net/public/images/2026/11/2a999027258944cb87377d0814dec799.webp)

Each point on the curve is one threshold (the picture marks the five from the table; your own plot shows the curve without the marks): moving right catches more fraud, and moving down means more false alarms per flag. The legend's *AP*, average precision, sums the whole curve up in one number; this course reads the curve instead. You'll also meet the **ROC curve**, which plots the true positive rate against the false positive rate; with fraud this rare, the precision-recall curve shows the false alarms more plainly, so it's the one used here.

## Choosing by cost

The curve shows the choices; the costs make the choice. Say the fraud team estimates that a missed fraud costs about $200 (the refund) and a false alarm about $20 (a staff member's call, and an annoyed member). The cost at a threshold is $200 for each fraud missed plus $20 for each false alarm:

```python
from sklearn.metrics import confusion_matrix

for threshold in [0.5, 0.2, 0.1, 0.05, 0.02]:
    flags = (chances >= threshold).astype(int)
    tn, fp, fn, tp = confusion_matrix(valid["is_fraud"], flags).ravel()
    cost = 200 * fn + 20 * fp
    print(f"{threshold}: missed {fn}, false alarms {fp}, cost ${cost:,}")
```

```text
0.5: missed 10, false alarms 1, cost $2,020
0.2: missed 9, false alarms 4, cost $1,880
0.1: missed 6, false alarms 14, cost $1,480
0.05: missed 4, false alarms 22, cost $1,240
0.02: missed 2, false alarms 57, cost $1,540
```

0.05 is the cheapest on the validation payments. Different costs pick a different line: if false alarms cost more, the best threshold moves up. Now, once, the test payments:

```python
test_chances = model.predict_proba(test[features])[:, 1]
test_flags = (test_chances >= 0.05).astype(int)
print(confusion_matrix(test["is_fraud"], test_flags))
print("Precision:", round(precision_score(test["is_fraud"], test_flags), 3))
print("Recall:", recall_score(test["is_fraud"], test_flags))
```

```text
[[948  32]
 [  3  17]]
Precision: 0.347
Recall: 0.85
```

17 of the 20 test frauds caught instead of 8, for 32 false alarms. That's the honest score to report, because nothing was chosen by looking at it. If you didn't like it, going back to try another threshold on the test data would make it dishonest.

scikit-learn can run this search for you: `TunedThresholdClassifierCV` tries thresholds with cross-validation (Module 4) and keeps the one with the best score on the metric you give it, balanced accuracy unless you say otherwise. Doing it by hand first shows you what it does.

## Do it: find the cheapest threshold

The fraud team has its model's chances of fraud for some validation payments, and it wants the threshold that costs the least.

The inputs:

- `chances`: the model's chance of fraud for each payment, between 0 and 1.
- `actual`: what each payment really was, 1 for fraud and 0 for honest, in the same order.
- `threshold`: the line. A payment is flagged when its chance is at or above it.
- `miss_cost`: what one missed fraud costs, and `alarm_cost`: what one false alarm costs.
- `thresholds`: the lines to try, in the order given.

For example:

```python
chances = [0.9, 0.6, 0.35, 0.3, 0.2, 0.05]
actual = [1, 0, 1, 0, 1, 0]
```

with a missed fraud at `100` and a false alarm at `10`:

| Threshold | Flagged | Frauds missed | False alarms | Cost |
|---|---|---|---|---|
| 0.5 | 0.9, 0.6 | 2 (0.35 and 0.2) | 1 (0.6) | 2 × 100 + 1 × 10 = 210 |
| 0.3 | 0.9, 0.6, 0.35, 0.3 | 1 (0.2) | 2 (0.6 and 0.3) | 1 × 100 + 2 × 10 = 120 |
| 0.1 | all but 0.05 | 0 | 2 (0.6 and 0.3) | 0 × 100 + 2 × 10 = 20 |

So `cost_at(chances, actual, 0.3, 100, 10)` returns `120`, and `best_threshold(chances, actual, [0.5, 0.3, 0.1], 100, 10)` returns `0.1`.

**Your job:** write two functions.

- `cost_at(chances, actual, threshold, miss_cost, alarm_cost)` returns `miss_cost` times the frauds not flagged, plus `alarm_cost` times the honest payments flagged.
- A payment is flagged when its chance is at or above the threshold (`>=`).
- `best_threshold(chances, actual, thresholds, miss_cost, alarm_cost)` returns the threshold from `thresholds` whose cost is lowest.
- If two thresholds tie for the lowest cost, `best_threshold` returns the one that comes first in `thresholds`.

Practiced live on the lesson page: write the Python right there and check it.

## Check yourself

2 questions to try on the lesson page.

## So what

The 0.5 line is only scikit-learn's default; the right threshold comes from what each mistake costs, chosen on validation data, so the test score you report is still one the model hasn't been tuned to.

## Sources

- scikit-learn, [Tuning the decision threshold for class prediction](https://scikit-learn.org/stable/modules/classification_threshold.html), User Guide: for binary classification, `predict` calls the positive class when the probability from `predict_proba` is greater than 0.5, which is not ideal for most uses; when a missed positive costs much more than a false positive, a threshold well below 0.5 can be better, at the price of more false positives; `TunedThresholdClassifierCV` tunes the threshold with an internal cross-validation to maximize a given metric, balanced accuracy by default; the same data should never be used both to train the classifier and to tune its threshold.
- scikit-learn 1.9.1, [precision_recall_curve](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.precision_recall_curve.html), API reference: precision and recall for predictions with a score greater than or equal to each threshold, with the thresholds in increasing order and recall decreasing.
- scikit-learn 1.9.1, [PrecisionRecallDisplay](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.PrecisionRecallDisplay.html), API reference: `from_predictions` plots a precision-recall curve from the true labels and the predicted scores, and shows the average precision in its legend.
- scikit-learn, [Metrics and scoring: Precision, recall and F-measures](https://scikit-learn.org/stable/modules/model_evaluation.html#precision-recall-and-f-measures), User Guide: `precision_recall_curve` computes a precision-recall curve by varying a decision threshold; `roc_curve` computes the receiver operating characteristic (ROC) curve.

## Key terms

- [Decision Threshold](https://www.ask-codey.com/glossary/decision-threshold)
- [Precision-Recall Curve](https://www.ask-codey.com/glossary/precision-recall-curve)
- [Validation Set](https://www.ask-codey.com/glossary/validation-set)
- [predict_proba()](https://www.ask-codey.com/glossary/predict-proba)
- [Recall](https://www.ask-codey.com/glossary/recall-ai)

## Related

- [Probabilities and the 0.5 Line](https://www.ask-codey.com/lessons/ml-predicted-probabilities)
- [Precision and Recall](https://www.ask-codey.com/lessons/ml-precision-and-recall)
- [When the Answer Is Rare](https://www.ask-codey.com/lessons/ml-imbalanced-classes)
- [Exercise: Fraud Check Scorecard](https://www.ask-codey.com/lessons/exercise-ml-fraud-check-scorecard)

Source: https://www.ask-codey.com/lessons/ml-thresholds-and-tradeoffs
