Missing Values in a Model

Most scikit-learn models stop with an error when a feature holds NaN, and dropping those rows throws away examples and leaves a new pet with a blank unscorable, so SimpleImputer fills each blank with a value learned from the training data (the median for numbers, the most frequent value for categories), and add_indicator=True keeps a column saying which values were missing.

Journeyman · AI Basics · Python · Updated

By

After this lesson you can

  • Fill in missing values with SimpleImputer, the median for numbers and the most frequent value for categories, learned from the training data
  • Choose between dropping rows, filling them in and adding a missing indicator, and explain what each one costs

Key takeaways

  • LogisticRegression, nearest neighbors and most other scikit-learn models refuse NaN. Decision trees and HistGradientBoosting models accept it.
  • Dropping rows with blanks loses examples, and a new pet with a blank still can't get a guess.
  • SimpleImputer fills each blank with a value it learned from the training data, the median for a number column and the most frequent value for a category column.
  • add_indicator=True adds a 0-or-1 column that says which values were missing, for when being missing is itself a clue.

Real data has gaps: a form left half filled in, a scale that was broken that week, a field added after the first records were made. Before a model can learn from a table, every blank needs a plan, and the plan has to work again on the day a new example arrives with a blank of its own. You'll make this choice for nearly every dataset you train on.

The example is Happy Tails Pet Adoption's pets.csv, from Exercise: Adoption Pipeline. 40 of its 800 pets have no recorded weight and 24 have no recorded color. Each code block is one notebook cell, run in order: the first one loads the data.

Most models refuse a blank

pandas reads a blank cell as NaN, read "not a number". Here's logistic regression asked to learn from the age, weight and photo columns:

import pandas as pd
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split

pets = pd.read_csv("pets.csv")
X = pets.drop(columns=["pet_id", "adopted_in_30_days"])
y = pets["adopted_in_30_days"]
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, stratify=y, random_state=42
)
numbers = ["age_years", "weight_lbs", "has_photo"]

model = LogisticRegression()
model.fit(X_train[numbers], y_train)
ValueError: Input X contains NaN.
LogisticRegression does not accept missing values encoded as NaN natively. For supervised learning, you might want to consider sklearn.ensemble.HistGradientBoostingClassifier and Regressor which accept missing values encoded as NaNs natively. Alternatively, it is possible to preprocess the data, for instance by using an imputer transformer in a pipeline or drop samples with missing values. See https://scikit-learn.org/stable/modules/impute.html You can find a list of all estimators that handle NaN values at the following page: https://scikit-learn.org/stable/modules/impute.html#estimators-that-handle-nan-values

The message names the two ways out: drop the rows, or fill the blanks in. It also names the exception: a few models accept NaN as it is, among them decision trees (DecisionTreeClassifier) and the HistGradientBoosting models, which learn which way to send a blank at each question. Nearest neighbors, logistic regression and linear regression don't.

Dropping rows has a cost

Dropping is the first idea, and the pandas refresher showed how. Here's what it costs:

print("Training pets with no weight:", X_train["weight_lbs"].isna().sum())
print("Test pets with no weight:", X_test["weight_lbs"].isna().sum())
print("Training pets left after dropna:", len(X_train[numbers].dropna()))
Training pets with no weight: 22
Test pets with no weight: 18
Training pets left after dropna: 578

Twenty-two training pets would be thrown away, along with everything else we know about them: their ages, photos and outcomes. Worse, the 18 test pets with no weight, and every future arrival whose scale reading is missing, still can't get a guess. Dropping can be fine when a few rows out of many are blank at random; it doesn't help a model that must answer for every pet.

Filling blanks with SimpleImputer

The other way is to fill each blank with a sensible stand-in. Your textbook may call this imputation. scikit-learn's SimpleImputer learns one stand-in per column with fit, from the training data, and fills blanks with it in transform:

from sklearn.impute import SimpleImputer

fill = SimpleImputer(strategy="median")
fill.fit(X_train[numbers])
print("Medians learned:", fill.statistics_)

filled = fill.transform(X_test[numbers])
blank = X_test["weight_lbs"].isna().to_numpy()
print("First test pet with no weight, filled in:", filled[blank][0])
Medians learned: [ 3.2  11.25  1.  ]
First test pet with no weight, filled in: [ 5.2  11.25  0.  ]

statistics_ holds each column's stand-in: the training pets' median age, weight and photo value. The median suits weight well, because a few 90-pound dogs drag the mean upward but barely move the middle value. The test pet kept its own age and photo, and its blank weight became the training median, 11.25 pounds. The default strategy is "mean".

A category column can't have a median, so it gets its most frequent value instead:

fill_color = SimpleImputer(strategy="most_frequent")
fill_color.fit(X_train[["color"]])
print("Most frequent color:", fill_color.statistics_)
Most frequent color: ['white']

"most_frequent" works on words as well as numbers; when two values tie, it picks the smallest. As with the scaler in Scaling Features, the stand-ins come from the training part only, so nothing about the test pets shapes them.

When being missing is a clue

A filled-in weight looks like any other weight, so the model can't tell it was a guess. Sometimes the blank itself means something: a pet whose weight was never recorded may be one that arrived on a busy day, or was never fully checked in. add_indicator=True keeps that information as an extra column, 1 where the value was missing and 0 where it wasn't:

flagged = SimpleImputer(strategy="median", add_indicator=True)
flagged.fit(X_train[numbers])
print(flagged.get_feature_names_out())
print(flagged.transform(X_test[numbers])[blank][0])
['age_years' 'weight_lbs' 'has_photo' 'missingindicator_weight_lbs']
[ 5.2  11.25  0.    1.  ]

Only weight gets an indicator: a column with no blanks in the training data gets none. Whether the indicator helps is something to test, not assume. In the Happy Tails data the blanks were planted at random, so here it has nothing to find.

So there are three choices, each with a cost. Dropping rows loses examples and leaves new rows with blanks unscored. Filling in keeps every row but makes up values that look real. Adding an indicator keeps the fact that a value was made up, at the price of one more column. Whichever you choose, the filling has to happen again, the same way, for every new pet: the next lesson, Pipelines: Every Step, Every Time, makes sure it does.

Check yourself

2 questions to try on the lesson page.

So what

A blank needs a plan that works on the training data and on every new example: fill it with a value learned from the training part, and keep a missing indicator when the blank itself might be a clue.

Sources

  • scikit-learn, Imputation of missing values, User Guide: datasets with blanks or NaNs are incompatible with scikit-learn estimators that assume all values are numerical; discarding rows comes at the price of losing data, so a better strategy is to impute the missing values; SimpleImputer supports strings with 'most_frequent'; MissingIndicator and add_indicator keep which values were missing, which can be informative; and the list of estimators that handle NaN values, which includes DecisionTreeClassifier and HistGradientBoostingClassifier.
  • scikit-learn 1.9.1, SimpleImputer, API reference: missing_values is np.nan by default; strategy is 'mean' by default, 'median' for numeric data, 'most_frequent' for strings or numbers, returning the smallest value when several tie; statistics_ holds the fill value for each feature; with add_indicator=True a feature with no missing values at fit time gets no indicator column.
  • scikit-learn, Decision Trees: Missing values support, User Guide: DecisionTreeClassifier and DecisionTreeRegressor have built-in support for missing values with splitter='best'.
  • pandas, Working with missing data: NaN as the missing-value marker for float columns, isna, and dropna.

Key terms

Related