Missing Values, Types and Duplicates

Real data comes with gaps and mistakes. Using an animal shelter's intake log typed in by volunteers, find the missing values, decide whether to drop or fill each one honestly, turn text such as '12 lbs' and '$75' into numbers you can add up, and remove the rows that were entered twice.

Updated

Missing Values

  1. Finding Missing Values: NaN and isna (Apprentice): pandas marks a missing value as NaN, isna().sum() counts the missing values in every column, na_values in read_csv turns hidden blanks such as 'unknown' and '-' into NaN, and count and mean quietly leave missing values out.
  2. Dropping or Filling Missing Values (Apprentice): Once you know where a table's gaps are, dropna removes rows (all of them with any gap, or only those missing the columns you name in subset), fillna puts a value, a median or 'Unknown' in each gap, and a flag column records which values were filled so none is passed off as measured.

Column Types

  1. Column Types: dtypes and astype (Apprentice): Every pandas column has one dtype that decides what its values can do, dtypes lists them (int64, float64, str, bool, datetime64, category), astype changes a column's type, and astype stops on text such as '59 lbs' because it can't guess which part is the number.
  2. Turning Text into Numbers: to_numeric (Apprentice): Text such as '59 lbs' and '$75' becomes numbers in two steps, .str.replace to take off the units and symbols, then pd.to_numeric with errors='coerce' to convert what's left, followed by a list of the values that became missing so none is lost silently.

Duplicates

  1. Finding and Removing Duplicates (Apprentice): duplicated marks every row that repeats one before it, on every column or only on the columns named in subset, drop_duplicates removes those rows and keep says which copy stays, and choosing the columns is how a double entry is told apart from two real visits.

Exercise: Tidy the Shelter Intake Log

  1. Exercise: Tidy the Shelter Intake Log (Apprentice): Write five cleaning functions for an animal shelter's messy intake log, reading hidden blanks as missing, turning weights and fees into numbers, filling missing ages honestly with a flag, and dropping double entries, and let pytest check the table each one returns.