NumPy: The Engine Under pandas

NumPy's array holds many values of one type and does math on all of them at once, a vectorized operation that a Python list can't do; every pandas column keeps its values in an array (to_numpy hands it over), np.mean gives NaN when a value is missing where pandas skips it, and one type per column is what makes the math fast.

Journeyman · Python · Updated

By

After this lesson you can

  • Turn a column into a NumPy array with to_numpy, and do math on every value at once, a vectorized operation
  • Explain what np.nan does to np.mean, and why an array of one type is faster than a column of mixed values

Key takeaways

  • A Python list times 2 repeats the list. A NumPy array times 2 doubles every value.
  • Every pandas column keeps its values in a NumPy array, and to_numpy() hands you that array.
  • Math on a whole array at once, with no loop of your own, is called a vectorized operation.
  • np.mean gives nan when one value is np.nan, while pandas' mean skips missing values. np.nanmean skips them too.
  • An array holds one type, such as float64, packed side by side, which is why column math is fast. A column of mixed values becomes object and loses that speed.

Open almost any data science notebook, statistics class handout or machine learning tutorial, and import numpy as np is near the top. Error messages from pandas mention numpy.ndarray, and values print as np.float64(95.99). NumPy is the library pandas is built on, and knowing what it does explains why some pandas code is fast, why some is slow, and what those messages mean.

The example is Trailhead Outfitters' catalog, products.csv: 300 outdoor gear products with their product name, price, stock, weight_oz and more. The catalog is our own made-up data.

The problem: lists don't do math

Take three tent prices in a plain Python list and try to double them:

prices = [182.99, 231.99, 113.99]
print(prices * 2)
[182.99, 231.99, 113.99, 182.99, 231.99, 113.99]

A list times 2 is the list twice, not the prices doubled. To change every price you write a loop, or a list comprehension such as [price * 0.9 for price in prices], and Python works through the values one at a time.

An array: math on every value at once

NumPy (read "num pie", short for Numerical Python) gives Python a new container, the array: a row of values, all of one type, that does math on every value at once. np is its short name, the way pd is pandas':

import numpy as np

prices_array = np.array([182.99, 231.99, 113.99])
prices_array * 2
array([365.98, 463.98, 227.98])

A 10% discount is one line, with no loop:

prices_array * 0.9
array([164.691, 208.791, 102.591])

Doing the math on a whole array at once is called a vectorized operation (a vector is the math word for a row of numbers). It's the name your textbook uses, and it's what products["price"] * products["stock"] does in pandas.

Every column is an array

A pandas column keeps its values in a NumPy array, with the index and the column's name added around it. to_numpy() hands you the array itself:

import pandas as pd

products = pd.read_csv("products.csv")
price_values = products["price"].to_numpy()
print(type(price_values))
print("type of values:", price_values.dtype)
print("first four:", price_values[:4])
print("average price:", round(np.mean(price_values), 2))
<class 'numpy.ndarray'>
type of values: float64
first four: [182.99 231.99 113.99 176.99]
average price: 193.8

ndarray is NumPy's full name for an array (n-dimensional array). np.mean averages it, and gives the same $193.80 as products["price"].mean(). You'll reach for to_numpy() when a tool wants a plain array, as many machine learning libraries do.

np.nan, and why one type is fast

NumPy is where NaN, read "not a number", comes from: np.nan is the value pandas shows as NaN for something missing. NumPy and pandas treat it differently, though:

weights = np.array([103.5, np.nan, 65.3])
print("np.mean:", np.mean(weights))
print("pandas mean:", pd.Series(weights).mean())
print("np.nanmean:", np.nanmean(weights))
np.mean: nan
pandas mean: 84.4
np.nanmean: 84.4

NumPy's np.mean says "one value is unknown, so the average is unknown". pandas' mean() skips the missing value and averages the other two, and so does np.nanmean. Neither is wrong; know which one you're calling.

Why is an array fast? Because every value has the same type, NumPy stores them packed side by side, like eggs in a carton, and runs one compiled loop over them. A Python list is more like a box of mixed groceries: each value is a separate Python object that has to be unpacked and checked before every step. A pandas column gets the carton only when its values share a type:

mixed = pd.Series([12.5, "n/a", 3])
print("prices:", products["price"].dtype)
print("mixed:", mixed.dtype)
prices: float64
mixed: object

float64 is a carton of decimal numbers. object means "any Python object", and math on it falls back to the slow, one-at-a-time way, when it works at all. That's one more reason to turn text such as "n/a" into real missing values and numbers before you analyze, and why whole-column math beats apply (New Columns from Old: Math, np.where and cut).

Check yourself

2 questions to try on the lesson page.

So what

pandas is fast because each column is a NumPy array of one type that does its math all at once: keep your columns clean, use whole-column math, and you're using the engine the way it was built to run.

Key terms

Related