Tidying Text

Tidying Text: Tidying Text with .str, Making Categories Match: replace and map, Splitting and Extracting Text.

Updated

  1. Tidying Text with .str (Apprentice): Text that people type differs in spaces and capital letters, so the same name fails to match itself; the .str tools strip, lower and title tidy a whole column at once, and len, contains and startswith test every value so you can find the ones that look wrong.
  2. Making Categories Match: replace and map (Apprentice): value_counts shows every spelling a column holds, replace with a dictionary fixes the few misspellings you name and leaves every other value alone, map turns every spelling into one standard value and leaves missing whatever it doesn't know, which you count before moving on, and the category type stores a column with a fixed list of values.
  3. Splitting and Extracting Text (Journeyman): str.split with expand=True turns one text column into several, with n=1 to split at the first space only, and str.extract with the short regular expression (\d+), read 'one or more digits, kept', pulls a number out of text typed in many different ways, where a plain replace would need a call for every wording.