Trees and Neighbors

Two more ways to classify, each easy to picture: a decision tree that plays twenty questions with your data, and nearest neighbors, which looks for the most similar examples it has seen and lets them vote. You'll read a tree's questions in plain words, measure distance between examples by hand, and see why the units of each feature matter. The running example is a fruit packing line.

Updated

Decision Trees

  1. Decision Trees (Journeyman): A decision tree sorts examples into classes with yes-or-no questions about their features, picking each question so the groups it makes are as unmixed as possible (the Gini impurity), with max_depth to limit how many questions it asks and no need to rescale the features first.
  2. Reading a Decision Tree (Journeyman): A decision tree's rules can be printed with export_text or drawn with plot_tree, one fruit's guess can be said as a sentence by following its path from the top question to a leaf, and the leaf's mix of training fruits is the probability the tree gives, but a deep tree's rules stop being readable and every rule describes the training data, not the world.

Nearest Neighbors

  1. Distance Between Examples (Apprentice): Two examples become points when each feature is a direction, the straight-line (Euclidean) distance between them comes from the Pythagorean theorem and math.dist works it out for any number of features, and a feature measured in big units, like grams, swamps one scored from 0 to 10 until the features are put on a common scale.
  2. Nearest Neighbors (Journeyman): A nearest neighbors classifier guesses a new example's class by finding the k training examples closest to it and letting them vote, with KNeighborsClassifier(n_neighbors=5) storing the training rows instead of learning a rule, a small k following every quirk, a large k blurring the classes together, and a tied vote, which in the lesson's run went to the class listed first.

Exercise: Fruit Sorter

  1. Exercise: Fruit Sorter (Journeyman): Write four functions for Three Rivers Fruit Packers, a straight-line distance and a nearest-neighbors search in plain Python, then a decision tree and a nearest neighbors classifier with the settings you choose, and let pytest check the distances, the neighbors' order and the models' settings.