# Measuring a Classifier

Is 98% accuracy good? Not when only 2% of payments are fraud. Count a classifier's right and wrong answers in a confusion matrix, then ask the two questions that matter: of the payments it flagged, how many were fraud (precision), and of the fraud, how much did it catch (recall). You'll compute each by hand, then with scikit-learn, and move the threshold to trade one for the other. The running example is a credit union's fraud check.

Updated: November 19, 2026

## [Counting Right and Wrong](https://www.ask-codey.com/courses/machine-learning-python/topics/ml-counting-right-and-wrong)

1. [Accuracy](https://www.ask-codey.com/lessons/ml-accuracy) (Apprentice): Accuracy is the share of a classifier's guesses that are right, worked out by counting matches or with scikit-learn's accuracy_score, and it's a fair first number, but the same accuracy can hide very different mistakes, and a fraud check that never says fraud scores 98% on payments where 2% are fraud.
2. [The Confusion Matrix](https://www.ask-codey.com/lessons/ml-confusion-matrix) (Journeyman): A confusion matrix sorts a classifier's guesses into four counts, true positives, false positives, false negatives and true negatives, so you can see which mistakes it makes, and scikit-learn's confusion_matrix puts the actual class in rows and the guessed class in columns, in sorted label order.

## [Which Mistakes Matter](https://www.ask-codey.com/courses/machine-learning-python/topics/ml-which-mistakes-matter)

1. [Precision and Recall](https://www.ask-codey.com/lessons/ml-precision-and-recall) (Journeyman): Precision asks how many of the payments a fraud check flagged were really fraud, TP divided by TP plus FP, and recall asks how much of the fraud it caught, TP divided by TP plus FN, and which one matters more depends on what each kind of mistake costs, worked out by hand or with precision_score and recall_score.
2. [The F1 Score and the Classification Report](https://www.ask-codey.com/lessons/ml-f1-score) (Journeyman): The F1 score joins precision and recall into one number with their harmonic mean, which stays low when either one is low, scikit-learn gives it with f1_score, and classification_report prints precision, recall and F1 for every class with macro and weighted averages, which you read class by class, because one number can hide the class you care about.

## [Rare Events and Thresholds](https://www.ask-codey.com/courses/machine-learning-python/topics/ml-rare-events-and-thresholds)

1. [When the Answer Is Rare](https://www.ask-codey.com/lessons/ml-imbalanced-classes) (Journeyman): When the class you're looking for is rare, accuracy rewards ignoring it, so you score a DummyClassifier that always guesses the most common class as the baseline to beat, read recall and balanced accuracy (the average of each class's recall) instead, and know that class_weight='balanced' moves the model toward the rare class while only more examples of it teach the model more.
2. [Moving the Threshold](https://www.ask-codey.com/lessons/ml-thresholds-and-tradeoffs) (Freelancer): Lowering a classifier's threshold below 0.5 flags more payments, so recall goes up and precision usually goes down, precision_recall_curve and PrecisionRecallDisplay show the whole trade, and you choose the threshold from what each mistake costs, on validation data set aside for choosing, then score the test data once.

## [Exercise: Fraud Check Scorecard](https://www.ask-codey.com/courses/machine-learning-python/topics/exercise-ml-fraud-check-scorecard)

1. [Exercise: Fraud Check Scorecard](https://www.ask-codey.com/lessons/exercise-ml-fraud-check-scorecard) (Journeyman): Write the confusion matrix's four counts, accuracy, precision, recall, the F1 score and all three scores at a chosen threshold in plain Python for Northside Credit Union's fraud check, and let pytest compare each one with scikit-learn's own function, edge cases included.

## [Up Next: Making Models Better](https://www.ask-codey.com/courses/machine-learning-python/topics/ml-up-next-making-models-better)

1. [Up Next: Making Models Better, Honestly](https://www.ask-codey.com/lessons/ml-up-next-making-models-better) (Journeyman): Look back at what Module 3 taught, classifiers that guess a category and the confusion matrix, precision, recall and thresholds that measure them, try a short fraud check memo with all of it, and see what Module 4 adds, scores you can trust and the mistakes that make a model look better than it is.

Source: https://www.ask-codey.com/courses/machine-learning-python/chapters/ml-measuring-a-classifier
