The Python and pandas You Need
A refresher of the Python and pandas this course uses, so you can start here even if it's been a while: lists, dictionaries and functions, then a pandas table loaded from a CSV, a first look at what's in it, the rows you pick from it, and the missing values real data always has. The exercise uses a farmers' market's weekly sales.
Updated
The Python You Need
- Lists: Keeping Things in Order (Novice): A list keeps many values in order under one name, so you can read any item by its index, counting from 0 at the front or from -1 at the back, add, insert and remove items, and check whether a value is there with in.
- Dictionaries: Looking Things Up by Name (Novice): A dictionary stores each value under a key you choose, so a program finds a value by its name instead of its position, and adds, changes and removes entries as the data changes.
- Parameters and Return Values (Novice): Parameters let a function work on whatever values each call hands it, and return hands a result back to the code that called it, which is different from printing it and gives None when a function returns nothing.
pandas Tables
- DataFrames: Rows, Columns and the Index (Apprentice): A DataFrame is pandas' table, and shape, columns and dtypes describe it, df['col'] picks one column as a Series, df[['a', 'b']] picks several as a smaller table, a new column can be made from two others in one line, and the index labels every row.
- Loading a CSV File (Apprentice): pd.read_csv loads a CSV file into a DataFrame, usecols and nrows load only the columns and rows you need, sep reads a file whose values are split by semicolons, and a FileNotFoundError means the path doesn't lead from the folder the notebook runs in to the file.
- A First Look: head, info and describe (Apprentice): head, tail and sample show a few of a DataFrame's rows, info lists every column's type and how many values are there (the first hint of missing values), and describe sums up each number column in eight numbers, from count and mean to the quartiles, with include='all' adding the text columns.
- Counting What's There: value_counts and unique (Apprentice): value_counts counts how many rows hold each value of a column, most common first, normalize=True turns the counts into shares of the whole, unique lists the different values in a column and nunique counts them.
Picking Rows and Filling Gaps
- Filtering Rows with a Condition (Apprentice): A comparison such as products["price"] < 100 gives a True or False answer for every row, called a mask; putting the mask in square brackets, products[mask], keeps only the rows where it is True; and sum() and mean() on the mask count the matching rows and give their share of the table.
- Finding Missing Values: NaN and isna (Apprentice): pandas marks a missing value as NaN, isna().sum() counts the missing values in every column, na_values in read_csv turns hidden blanks such as 'unknown' and '-' into NaN, and count and mean quietly leave missing values out.
- Dropping or Filling Missing Values (Apprentice): Once you know where a table's gaps are, dropna removes rows (all of them with any gap, or only those missing the columns you name in subset), fillna puts a value, a median or 'Unknown' in each gap, and a flag column records which values were filled so none is passed off as measured.
Exercise: Market Day Warm-Up
- Exercise: Market Day Warm-Up (Apprentice): Write five functions that load a farmers' market's 260 weekly sales from a CSV file, count the missing values in each column, fill the blank sales counts with the median on a copy and flag them, pick out one stall's rows, and work out each stall's share of the rows, and let pytest check what each one returns.