Exercise: Fraud Check Scorecard
Write the confusion matrix's four counts, accuracy, precision, recall, the F1 score and all three scores at a chosen threshold in plain Python for Northside Credit Union's fraud check, and let pytest compare each one with scikit-learn's own function, edge cases included.
Journeyman · AI Basics · Python · Updated
Northside Credit Union's fraud team wants a scorecard for its fraud check that anyone on the team can read: what it caught, what it missed, how many honest members it bothered, and how all of that changes when the line moves. You'll write the six functions behind that scorecard in plain Python, the way Accuracy, The Confusion Matrix, Precision and Recall and The F1 Score and the Classification Report worked them out by hand, and the threshold step from Moving the Threshold. Tests compare each one with scikit-learn's own function on the same values, so you'll know your counting matches the library's, including the awkward cases where there's nothing to divide by.
Get the exercise code
Open this lesson's GitHub repository, github.com/ask-codey/exercise-ml-fraud-check-scorecard, also linked from the Exercise box beside the lesson. Clone it into your python-course folder. In VS Code's terminal:
cd ~/python-course
git clone https://github.com/ask-codey/exercise-ml-fraud-check-scorecard.git
Git makes a folder, exercise-ml-fraud-check-scorecard, with the exercise code inside. Open that folder in VS Code (File > Open Folder), open its terminal with Terminal > New Terminal, and make the exercise's virtual environment. On Windows (PowerShell):
python -m venv .venv
.venv\Scripts\Activate.ps1
pip install -r requirements.txt
On a Mac (zsh):
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
The prompt starts with (.venv) once the virtual environment is on. The exercise's README says what each command does, and what to do if PowerShell won't run the second one.
The folder holds:
transactions.csv: the 5,000 card payments, our own made-up data, the same file the chapter's lessons use. 100 of them are fraud.seed_data.py: the script that made the CSV, with the habits it planted in honest payments and in fraud. You don't need to run it: the committed CSV is what keeps everyone's numbers the same.fraud_scorecard.py: the six functions, for you to finish.explore.ipynb: a notebook for looking at the payments, drawing the fraud check's confusion matrix and precision-recall curve, and trying your functions beside scikit-learn's. Choose the.venvkernel with Select Kernel (top right) before you run it. The tests don't read it.report.py: trains the fraud check on 3,000 payments, keeps 1,000 for choosing a threshold and 1,000 for the final check, as Moving the Threshold did, and scores the test payments with your six functions. Ready to use.tests/: the tests, which call each function and check what comes back, andconftest.py, which gives a test a fresh copy of the payments. You don't need to change them.requirements.txtandpytest.ini: the packages the exercise needs, at exact versions, and where pytest finds the tests and your code.
What to do
Every function in fraud_scorecard.py is already declared with its name and parameters, and for now it stops with a NotImplementedError that says Not built yet. The docstring under each one says what it should do, with examples:
def precision(actual, predicted):
"""Return the precision: of the payments guessed fraud, the share that were fraud, tp / (tp + fp).
If nothing was guessed fraud (tp + fp is 0), return 0.0.
precision([1, 0, 0, 1, 0, 1, 0, 0], [1, 0, 1, 0, 0, 1, 1, 0]) -> 0.5 (2 / 4)
precision([1, 0, 1], [0, 0, 0]) -> 0.0 (nothing flagged)
"""
raise NotImplementedError("Not built yet.")
In each function, replace the raise NotImplementedError("Not built yet.") line with your own code, in plain Python: no scikit-learn, NumPy or pandas inside the functions, and no rounding. In every list, 1 means fraud and 0 means honest. The examples use the same eight payments, actual = [1, 0, 0, 1, 0, 1, 0, 0] guessed [1, 0, 1, 0, 0, 1, 1, 0].
confusion_counts(actual, predicted)returns a dictionary with the four counts:{"tp": 2, "fp": 2, "fn": 1, "tn": 3}for the example.accuracy(actual, predicted)returns the share of guesses that match:0.625for the example (5 of 8).precision(actual, predicted)returns tp / (tp + fp):0.5for the example. If nothing was guessed fraud, it returns0.0.recall(actual, predicted)returns tp / (tp + fn): 2 / 3 for the example. If there was no fraud to find, it returns0.0.f1(actual, predicted)returns 2 × precision × recall / (precision + recall): 4 / 7 for the example. If precision and recall are both 0, it returns0.0.scores_at(chances, actual, threshold)turns each chance into a guess, 1 when the chance is at or abovethresholdand 0 otherwise, and returns a dictionary with that guess list's"precision","recall"and"f1".scores_at([0.9, 0.6, 0.35, 0.3, 0.2, 0.05], [1, 0, 1, 0, 1, 0], 0.3)returns a precision of0.5, a recall of 2 / 3 and an F1 of 4 / 7.
Every function also works on a pandas column and a NumPy array, as report.py gives them: a for loop over zip and len work the same on all three.
Check your work
Run the tests from the exercise folder, with the virtual environment on:
python -m pytest
At first all 21 tests fail. That's expected: each function you finish turns its tests green. A failing test's message says what it called, what it expected and what came back, for example precision([1, 0, 1], [0, 0, 0]) should return 0.0, but it returned None. Several tests compare your answers with scikit-learn's confusion_matrix, accuracy_score, precision_score, recall_score and f1_score on twelve more payments, to 9 decimal places, and one compares all four scores with scikit-learn's on the 1,000 test payments. The edge cases are tested too: nothing flagged, no fraud to find, and a chance exactly on the threshold. No test checks how good the fraud check is: they check your counting and your arithmetic.
To see your functions at work on the whole file, run python report.py.
Hints
- Count first. Write
confusion_countsbefore anything else;precision,recallandf1can each start by calling it, as The Confusion Matrix's Do It counted the four cells. - Check the bottom number. Before each division, ask whether it can be 0, as the Do It in Precision and Recall did, and return
0.0when it is. - Reuse your own work.
f1can call yourprecisionandrecall, andscores_atcan build a list of guesses and call all three. - At or above. A chance exactly on the threshold is flagged:
chance >= threshold, the ruleprecision_recall_curveuses.
When you're done
Every test is green: pytest's last line says 21 passed. And python report.py prints:
1) payments: 5000
2) training payments: 3000
3) validation payments: 1000
4) test payments: 1000
5) counts at 0.5: {'tp': 8, 'fp': 4, 'fn': 12, 'tn': 976}
6) accuracy at 0.5: 0.984
7) precision at 0.5: 0.667
8) recall at 0.5: 0.4
9) F1 at 0.5: 0.5
10) counts at 0.05: {'tp': 17, 'fp': 32, 'fn': 3, 'tn': 948}
11) scores at 0.05: {'precision': 0.347, 'recall': 0.85, 'f1': 0.493}
The same numbers the chapter's lessons got from scikit-learn, now from your own code. Compare lines 9 and 11: the F1 score barely moved, from 0.5 to 0.493, while the fraud caught went from 8 to 17. What does that say about choosing a threshold by F1 when a missed fraud costs ten times a false alarm? In the notebook, look at the 3 frauds still missed at 0.05: what makes them look honest?
Submit your exercise
The exercise is how you finish this lesson. When your work is done, put it on GitHub in a repo of your own, then submit that repo's address here. Hand In Your Work with Git walks through each step.
On GitHub, click +, then New repository. Name it (for example
exercise-ml-fraud-check-scorecard), choose Public, and click Create repository. Leave out the README: your folder already has one.In VS Code's terminal, in your exercise folder, send your work to your new repo. Point your copy at your own repo, then commit and push:
git remote set-url origin https://github.com/your-name/exercise-ml-fraud-check-scorecard.git git add . git commit -m "My solution" git push -u origin mainPaste your repo's address, such as
https://github.com/your-name/exercise-ml-fraud-check-scorecard, into Submit your exercise on this page. The site checks that the repo is public and holds your commits, and then the lesson is complete.
Changed something after you submitted? Push it, and submit again: your latest submission is the one that counts.
So what
Every score on a classifier's report card is a few lines of counting you can now write and check yourself, which is what lets you explain a fraud check's numbers to the people who pay for its mistakes.