Selecting, Filtering and Sorting

Answering a question starts with the right rows and columns. Using an outdoor gear store's catalog, pick rows and columns by label and position, filter with one condition and several, sort into top lists, add calculated columns such as price bands and sale prices, and meet NumPy, the engine that makes pandas fast.

Updated

Picking Rows and Columns

  1. Picking Rows and Columns: loc and iloc (Apprentice): Square brackets pick columns by name, set_index makes a column such as sku the row labels, loc picks rows and columns by those labels (a label slice includes its end), and iloc picks them by position, counting from 0 (a position slice stops before its end).

Filtering

  1. Filtering Rows with a Condition (Apprentice): A comparison such as products["price"] < 100 gives a True or False answer for every row, called a mask; putting the mask in square brackets, products[mask], keeps only the rows where it is True; and sum() and mean() on the mask count the matching rows and give their share of the table.
  2. Combining Conditions with &, | and ~ (Apprentice): Python's and, or and not can't combine whole columns of True and False, so pandas uses & (and), | (or) and ~ (not), with parentheses around each comparison; isin, between and str.contains replace long chains of conditions, and query writes a filter as text.

Sorting

  1. Sorting and Top Lists (Apprentice): sort_values puts a table's rows in order by one column or several, with ascending= saying which way for each; nlargest and nsmallest give the top or bottom rows by a column in one call; and reset_index(drop=True) numbers a top list 0, 1, 2 instead of keeping the old row labels.

Calculated Columns

  1. New Columns from Old: Math, np.where and cut (Journeyman): Math between columns makes a new column for every row at once, np.where picks one of two values for each row from a True or False column, pd.cut sorts numbers into labeled bands such as budget, mid and premium, and apply runs a function of your own on every value, more slowly than whole-column math.
  2. NumPy: The Engine Under pandas (Journeyman): NumPy's array holds many values of one type and does math on all of them at once, a vectorized operation that a Python list can't do; every pandas column keeps its values in an array (to_numpy hands it over), np.mean gives NaN when a value is missing where pandas skips it, and one type per column is what makes the math fast.

Exercise: Outdoor Gear Finder

  1. Exercise: Outdoor Gear Finder (Apprentice): Write five functions for an outdoor gear store's online gear finder, filtering its catalog by budget and by stock, listing the best-rated products fairly, adding price bands with pd.cut and sale prices with np.where, and let pytest check the table each one returns.