# Exercise: Fraud Check Scorecard

Write the confusion matrix's four counts, accuracy, precision, recall, the F1 score and all three scores at a chosen threshold in plain Python for Northside Credit Union's fraud check, and let pytest compare each one with scikit-learn's own function, edge cases included.

Journeyman · AI Basics · Python · Updated November 19, 2026

By Gregor Dzierzon

Northside Credit Union's fraud team wants a scorecard for its fraud check that anyone on the team can read: what it caught, what it missed, how many honest members it bothered, and how all of that changes when the line moves. You'll write the six functions behind that scorecard in plain Python, the way *Accuracy*, *The Confusion Matrix*, *Precision and Recall* and *The F1 Score and the Classification Report* worked them out by hand, and the threshold step from *Moving the Threshold*. Tests compare each one with scikit-learn's own function on the same values, so you'll know your counting matches the library's, including the awkward cases where there's nothing to divide by.

## Get the exercise code

Open this lesson's GitHub repository, [github.com/ask-codey/exercise-ml-fraud-check-scorecard](https://github.com/ask-codey/exercise-ml-fraud-check-scorecard), also linked from the **Exercise** box beside the lesson. Clone it into your `python-course` folder. In VS Code's terminal:

```bash
cd ~/python-course
git clone https://github.com/ask-codey/exercise-ml-fraud-check-scorecard.git
```

Git makes a folder, `exercise-ml-fraud-check-scorecard`, with the exercise code inside. Open that folder in VS Code (**File > Open Folder**), open its terminal with **Terminal > New Terminal**, and make the exercise's virtual environment. On Windows (PowerShell):

```powershell
python -m venv .venv
.venv\Scripts\Activate.ps1
pip install -r requirements.txt
```

On a Mac (zsh):

```bash
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
```

The prompt starts with `(.venv)` once the virtual environment is on. The exercise's README says what each command does, and what to do if PowerShell won't run the second one.

The folder holds:

- `transactions.csv`: the 5,000 card payments, our own made-up data, the same file the chapter's lessons use. 100 of them are fraud.
- `seed_data.py`: the script that made the CSV, with the habits it planted in honest payments and in fraud. You don't need to run it: the committed CSV is what keeps everyone's numbers the same.
- `fraud_scorecard.py`: the six functions, for you to finish.
- `explore.ipynb`: a notebook for looking at the payments, drawing the fraud check's confusion matrix and precision-recall curve, and trying your functions beside scikit-learn's. Choose the `.venv` kernel with **Select Kernel** (top right) before you run it. The tests don't read it.
- `report.py`: trains the fraud check on 3,000 payments, keeps 1,000 for choosing a threshold and 1,000 for the final check, as *Moving the Threshold* did, and scores the test payments with your six functions. Ready to use.
- `tests/`: the tests, which call each function and check what comes back, and `conftest.py`, which gives a test a fresh copy of the payments. You don't need to change them.
- `requirements.txt` and `pytest.ini`: the packages the exercise needs, at exact versions, and where pytest finds the tests and your code.

## What to do

Every function in `fraud_scorecard.py` is already declared with its name and parameters, and for now it stops with a `NotImplementedError` that says *Not built yet.* The docstring under each one says what it should do, with examples:

```python
def precision(actual, predicted):
    """Return the precision: of the payments guessed fraud, the share that were fraud, tp / (tp + fp).

    If nothing was guessed fraud (tp + fp is 0), return 0.0.

    precision([1, 0, 0, 1, 0, 1, 0, 0], [1, 0, 1, 0, 0, 1, 1, 0]) -> 0.5   (2 / 4)
    precision([1, 0, 1], [0, 0, 0])                         -> 0.0   (nothing flagged)
    """
    raise NotImplementedError("Not built yet.")
```

In each function, replace the `raise NotImplementedError("Not built yet.")` line with your own code, in plain Python: no scikit-learn, NumPy or pandas inside the functions, and no rounding. In every list, 1 means fraud and 0 means honest. The examples use the same eight payments, `actual = [1, 0, 0, 1, 0, 1, 0, 0]` guessed `[1, 0, 1, 0, 0, 1, 1, 0]`.

1. **`confusion_counts(actual, predicted)`** returns a dictionary with the four counts: `{"tp": 2, "fp": 2, "fn": 1, "tn": 3}` for the example.
2. **`accuracy(actual, predicted)`** returns the share of guesses that match: `0.625` for the example (5 of 8).
3. **`precision(actual, predicted)`** returns tp / (tp + fp): `0.5` for the example. If nothing was guessed fraud, it returns `0.0`.
4. **`recall(actual, predicted)`** returns tp / (tp + fn): 2 / 3 for the example. If there was no fraud to find, it returns `0.0`.
5. **`f1(actual, predicted)`** returns 2 × precision × recall / (precision + recall): 4 / 7 for the example. If precision and recall are both 0, it returns `0.0`.
6. **`scores_at(chances, actual, threshold)`** turns each chance into a guess, 1 when the chance is at or above `threshold` and 0 otherwise, and returns a dictionary with that guess list's `"precision"`, `"recall"` and `"f1"`. `scores_at([0.9, 0.6, 0.35, 0.3, 0.2, 0.05], [1, 0, 1, 0, 1, 0], 0.3)` returns a precision of `0.5`, a recall of 2 / 3 and an F1 of 4 / 7.

Every function also works on a pandas column and a NumPy array, as `report.py` gives them: a `for` loop over `zip` and `len` work the same on all three.

## Check your work

Run the tests from the exercise folder, with the virtual environment on:

```bash
python -m pytest
```

At first all 21 tests fail. That's expected: each function you finish turns its tests green. A failing test's message says what it called, what it expected and what came back, for example *precision([1, 0, 1], [0, 0, 0]) should return 0.0, but it returned None.* Several tests compare your answers with scikit-learn's `confusion_matrix`, `accuracy_score`, `precision_score`, `recall_score` and `f1_score` on twelve more payments, to 9 decimal places, and one compares all four scores with scikit-learn's on the 1,000 test payments. The edge cases are tested too: nothing flagged, no fraud to find, and a chance exactly on the threshold. No test checks how good the fraud check is: they check your counting and your arithmetic.

To see your functions at work on the whole file, run `python report.py`.

## Hints

- **Count first.** Write `confusion_counts` before anything else; `precision`, `recall` and `f1` can each start by calling it, as *The Confusion Matrix*'s Do It counted the four cells.
- **Check the bottom number.** Before each division, ask whether it can be 0, as the Do It in *Precision and Recall* did, and return `0.0` when it is.
- **Reuse your own work.** `f1` can call your `precision` and `recall`, and `scores_at` can build a list of guesses and call all three.
- **At or above.** A chance exactly on the threshold is flagged: `chance >= threshold`, the rule `precision_recall_curve` uses.

## When you're done

Every test is green: pytest's last line says `21 passed`. And `python report.py` prints:

```text
1) payments: 5000
2) training payments: 3000
3) validation payments: 1000
4) test payments: 1000
5) counts at 0.5: {'tp': 8, 'fp': 4, 'fn': 12, 'tn': 976}
6) accuracy at 0.5: 0.984
7) precision at 0.5: 0.667
8) recall at 0.5: 0.4
9) F1 at 0.5: 0.5
10) counts at 0.05: {'tp': 17, 'fp': 32, 'fn': 3, 'tn': 948}
11) scores at 0.05: {'precision': 0.347, 'recall': 0.85, 'f1': 0.493}
```

The same numbers the chapter's lessons got from scikit-learn, now from your own code. Compare lines 9 and 11: the F1 score barely moved, from 0.5 to 0.493, while the fraud caught went from 8 to 17. What does that say about choosing a threshold by F1 when a missed fraud costs ten times a false alarm? In the notebook, look at the 3 frauds still missed at 0.05: what makes them look honest?

## Submit your exercise

The exercise is how you finish this lesson. When your work is done, put it on GitHub in a repo of your own, then submit that repo's address here. *Hand In Your Work with Git* walks through each step.

1. On GitHub, click **+**, then **New repository**. Name it (for example `exercise-ml-fraud-check-scorecard`), choose **Public**, and click **Create repository**. Leave out the README: your folder already has one.
2. In VS Code's terminal, in your exercise folder, send your work to your new repo. Point your copy at your own repo, then commit and push:

   ```bash
   git remote set-url origin https://github.com/your-name/exercise-ml-fraud-check-scorecard.git
   git add .
   git commit -m "My solution"
   git push -u origin main
   ```

3. Paste your repo's address, such as `https://github.com/your-name/exercise-ml-fraud-check-scorecard`, into **Submit your exercise** on this page. The site checks that the repo is public and holds your commits, and then the lesson is complete.

Changed something after you submitted? Push it, and submit again: your latest submission is the one that counts.

## So what

Every score on a classifier's report card is a few lines of counting you can now write and check yourself, which is what lets you explain a fraud check's numbers to the people who pay for its mistakes.

## Key terms

- [Confusion Matrix](https://www.ask-codey.com/glossary/confusion-matrix)
- [Precision](https://www.ask-codey.com/glossary/precision)
- [Recall](https://www.ask-codey.com/glossary/recall-ai)
- [F1 Score](https://www.ask-codey.com/glossary/f1-score)
- [Decision Threshold](https://www.ask-codey.com/glossary/decision-threshold)
- [pytest](https://www.ask-codey.com/glossary/pytest)

## Related

- [Accuracy](https://www.ask-codey.com/lessons/ml-accuracy)
- [The Confusion Matrix](https://www.ask-codey.com/lessons/ml-confusion-matrix)
- [Precision and Recall](https://www.ask-codey.com/lessons/ml-precision-and-recall)
- [The F1 Score and the Classification Report](https://www.ask-codey.com/lessons/ml-f1-score)
- [Moving the Threshold](https://www.ask-codey.com/lessons/ml-thresholds-and-tradeoffs)

Source: https://www.ask-codey.com/lessons/exercise-ml-fraud-check-scorecard
