Column Types: dtypes and astype

Every pandas column has one dtype that decides what its values can do, dtypes lists them (int64, float64, str, bool, datetime64, category), astype changes a column's type, and astype stops on text such as '59 lbs' because it can't guess which part is the number.

Apprentice · Python · Updated

By

After this lesson you can

  • Read a DataFrame's dtypes and say what each common type holds, int64, float64, str, bool, datetime64 and category
  • Change a column's type with astype, and explain why astype stops on text such as '59 lbs'

Key takeaways

  • Every column has one dtype, the kind of value it holds, and dtypes lists them all. A number stored as text can't be added up; sum() glues the text together instead.
  • The common types are int64 (whole numbers), float64 (decimals, and the type that can hold NaN), str (text), bool (True or False), datetime64 (dates) and category (a short list of labels).
  • astype() changes a column's type, such as astype(int) for text that holds whole numbers or astype("category") for a few repeated labels.
  • astype stops on the first value it can't convert, such as '59 lbs', and a whole-number column with gaps can't become int64, because int64 has no NaN.

A spreadsheet will happily show you a column of numbers that it secretly treats as text, and the first sign is a total that comes out wrong. pandas is stricter: every column has one type, and that type decides what you can do with it. Knowing how to read a column's type, and change it, is how you catch the "numbers" that aren't numbers before they reach a report.

The example is Happy Tails Animal Shelter's intake log, intake.csv: 800 rows, one for each time an animal arrived, typed in by volunteers, with columns such as species, age_years and weight_lbs. It's loaded with unknown and - read as missing, as Finding Missing Values: NaN and isna shows.

Numbers that are really text

The shelter's vet wants the total weight of the animals in care, to plan food orders. Here are the first three weights:

import pandas as pd

intake = pd.read_csv("intake.csv", na_values=["unknown", "-"])
intake["weight_lbs"].head(3)
0       5.5
1    59 lbs
2        42
Name: weight_lbs, dtype: str

The last line says dtype: str: these are text. Adding text glues it together, and the largest text is the one that sorts last alphabetically:

print(intake["weight_lbs"].head(3).sum())
print(intake["weight_lbs"].max())
5.559 lbs42
about 20

No error, and two wrong answers. A volunteer typed 59 lbs with its unit, so pandas couldn't read the column as numbers, and every weight came in as text.

dtypes: what each column holds

A column's dtype (short for "data type") is the kind of value it holds. Every value in a column shares it. dtypes lists the whole table's:

intake.dtypes
animal_id           str
name                str
species             str
breed               str
age_years       float64
weight_lbs          str
intake_date         str
intake_type         str
adopted             str
adoption_fee        str
dtype: object

Only age_years is a number. These are the types you'll meet most:

dtype What it holds In the log
int64 Whole numbers, with no gaps allowed Counts, such as a number of rows
float64 Numbers with decimals, and NaN for missing age_years
str Text name, species, and for now weight_lbs
bool True or False What isna() gives back
datetime64 Dates and times intake_date, once it's parsed as a date
category A short list of labels, repeated species could be one

In pandas 3, a text column shows as str. Older tutorials and answers online show object for the same thing.

The 64 is how much room each value gets: 64 bits, the usual size on today's computers. And float64 has a job beyond decimals: it's the type with a place for NaN. That's why age_years is float64 even though volunteers typed whole years.

Changing a type with astype

astype() turns a column into another type. Here are the first ages, as text, the way a file can deliver them:

typed = pd.Series(["3", "2", "5"])
print(typed.sum())
print(typed.astype(int).sum())
325
10

As text, sum() glues "3", "2" and "5" into 325. After astype(int), they're numbers, and the sum is 10. A column with a few labels repeated over and over can become a category, which stores each label once and knows the full list:

intake["species"].astype("category").head(3)
0    cat
1    dog
2    dog
Name: species, dtype: category
Categories (3, str): ['cat', 'dog', 'rabbit']

When astype stops

astype changes every value or none. The ages are whole years, so astype(int) looks right, but some ages are missing:

intake["age_years"].astype(int)
IntCastingNaNError: Cannot convert non-finite values (NA or inf) to integer. Replace or remove non-finite values or cast to an integer type that supports these values (e.g. 'Int64')

int64 has no place for NaN, so the gaps have to be filled or dropped first (Dropping or Filling Missing Values), or the column stays float64, which is fine. Now the weights:

intake["weight_lbs"].astype(float)
ValueError: could not convert string to float: '59 lbs'

Read the error's last line: it names the first value it couldn't convert, '59 lbs'. astype won't guess that 59 is the number and lbs is a unit, and that's a good thing: a tool that guessed would get some values wrong without telling you. Taking the units off, and dealing with values like about 20, is the next lesson, Turning Text into Numbers: to_numeric.

Check yourself

2 questions to try on the lesson page.

So what

A column's dtype decides what its values can do, so dtypes is worth a look before any math: a total that glues text together is wrong without ever raising an error.

Key terms

Related