Rare Events and Thresholds

Rare Events and Thresholds: When the Answer Is Rare, Moving the Threshold.

Updated

  1. When the Answer Is Rare (Journeyman): When the class you're looking for is rare, accuracy rewards ignoring it, so you score a DummyClassifier that always guesses the most common class as the baseline to beat, read recall and balanced accuracy (the average of each class's recall) instead, and know that class_weight='balanced' moves the model toward the rare class while only more examples of it teach the model more.
  2. Moving the Threshold (Freelancer): Lowering a classifier's threshold below 0.5 flags more payments, so recall goes up and precision usually goes down, precision_recall_curve and PrecisionRecallDisplay show the whole trade, and you choose the threshold from what each mistake costs, on validation data set aside for choosing, then score the test data once.