Tuning and Comparing Models Fairly
Every model has settings you choose before training: how deep a tree goes, how many neighbors vote. Search for good settings with cross-validation instead of guesswork, meet the random forest, a crowd of trees that is often hard to beat, and compare models on equal terms: the same folds, the same metric, a baseline, and one final look at the test set. The running example is an inn's bookings that get canceled.
Updated
Tuning
- Hyperparameters: Settings You Choose (Journeyman): A hyperparameter is a setting you choose before fit, such as a tree's max_depth, while fit learns the rest, such as the tree's questions; get_params and set_params read and change the settings, the defaults are only a starting point, and settings are chosen with cross-validation on the training rows, never with the test set.
- Searching for Good Settings (Freelancer): GridSearchCV tries every combination of the settings you list, scores each by cross-validation on the training rows, keeps the best in best_params_ and best_score_ with every result in cv_results_, and refits the best pipeline on all the training rows, ready for one final test score; a pipeline's settings are named step__setting, and RandomizedSearchCV tries a sample when the grid is too big.
A Stronger Model
- Random Forests (Journeyman): A random forest trains many decision trees, each on a bootstrap sample of the training rows and choosing among a random few features at each split, then averages their predicted probabilities, so it changes less with the rows it was given than one tree does; RandomForestClassifier(n_estimators=200, random_state=0) builds one, it's harder to read than a single tree, and gradient boosting is the next model to learn.
Fair Comparisons
- Comparing Models Fairly (Freelancer): A fair comparison scores every model on the same training rows with one seeded splitter, one metric and a baseline, puts each model's mean and spread side by side, treats a difference smaller than the spread as a tie that goes to the simpler model, and ends with one test-set score for the chosen model, reported as it is.
Exercise: Fair Model Showdown
- Exercise: Fair Model Showdown (Freelancer): Write four functions that build a tree's settings grid with step names, set up a grid search scored by accuracy on the folds you're given, compare models on the same folds with each one's mean and spread, and score the chosen model once on the test bookings, and let pytest check the grid, the shared folds and that the test rows are never learned from.
Up Next: Groups, Fairness and Explaining
- Up Next: Groups, Fairness and Explaining (Journeyman): Look back at what Module 4 taught, catching overfitting, cross-validation, pipelines, data leakage, tuning and fair comparisons, try a short honest write-up of the inn's model with all of it, and see what Module 5 adds, finding groups in data that has no labels, and explaining a model's results, limits and fairness to the people it affects.