Cross-Validation

Cross-Validation: Cross-Validation, Validation Sets: Keep the Test Set for Last, Splits for Time and Groups.

Updated

  1. Cross-Validation (Journeyman): Cross-validation splits the rows into k folds and tests the model on each fold in turn, learning from the others, so every row is tested exactly once; cross_val_score gives the k scores, you report their mean and their spread, and KFold with shuffle=True and a random_state makes the folds random and repeatable.
  2. Validation Sets: Keep the Test Set for Last (Journeyman): Every time you look at the test score while choosing a model or a setting, the choice learns a little about the test rows, so choose with a validation set or with cross-validation on the training part, and score the test set once, at the end, for the model you chose.
  3. Splits for Time and Groups (Freelancer): Shuffled folds assume every row is independent of the others, which isn't true for data in time order, where shuffling lets a model learn from the future, or for data with repeats, where the same customer lands on both sides; TimeSeriesSplit always tests on rows that come after the ones it learned from, and GroupKFold keeps each group on one side.