Which Mistakes Matter

Which Mistakes Matter: Precision and Recall, The F1 Score and the Classification Report.

Updated

  1. Precision and Recall (Journeyman): Precision asks how many of the payments a fraud check flagged were really fraud, TP divided by TP plus FP, and recall asks how much of the fraud it caught, TP divided by TP plus FN, and which one matters more depends on what each kind of mistake costs, worked out by hand or with precision_score and recall_score.
  2. The F1 Score and the Classification Report (Journeyman): The F1 score joins precision and recall into one number with their harmonic mean, which stays low when either one is low, scikit-learn gives it with f1_score, and classification_report prints precision, recall and F1 for every class with macro and weighted averages, which you read class by class, because one number can hide the class you care about.