Why Explain a Model

A model's guess lands on people who ask why, the person who gets the estimate, the team that keeps the model working and the manager who decides whether to use it, so you explain either the whole model (what it leans on across every example) or one guess (why this home got this number), and every explanation describes what the model learned from its data, not why the world works that way.

Journeyman · AI Basics · Python · Updated

By

After this lesson you can

  • Name who needs a model explained and what each one asks, and tell explaining the whole model from explaining one guess
  • Explain why a model's explanation describes what it learned from the data, not why the world works that way

Key takeaways

  • Three people need a model explained, each with a question of their own. The person who gets the guess asks why, the team asks whether it's working, and the manager asks whether to trust it.
  • Explaining the whole model says what it leans on across every example. Explaining one guess says why one example got its number.
  • An explanation describes the model, not the world. A model can lean on a column that has no effect of its own, because that column moves with one that does.
  • Check how far off a model is on test data before you explain it. A clear explanation of a poor model is still a poor model.

Greenway Home Energy, a utility in a made-up town, sends every customer an estimate of next month's energy bill. The owner of a 2,720-square-foot home built in 1962 opens a letter that says about $230, and picks up the phone: why so high? When a model's guess lands on a real person (a bill, a loan, a scholarship, a delivery time), someone will ask why, and "the computer said so" isn't an answer. This chapter is how you give a real one.

The example for the whole chapter is Greenway Home Energy: 700 homes in home_energy.csv, our own made-up data. It comes with Exercise: Explain the Energy Bill, beside the seed script that made it. Each home has its size (size_sqft), the year it was built (year_built), its insulation (poor, fair or good), its heating (gas, electric or a heat pump), how many people live there (occupants), whether it has a pool (has_pool, 1 or 0), and the label the models guess: monthly_bill, in dollars.

Three people, three questions

The same model gets asked about by different people, and each wants something different:

  • The homeowner gets the estimate. They ask: why is mine $230, and what could I change? They need plain words and dollars, not a chart of a model's insides.
  • The team that keeps the model running asks: is it leaning on what we'd expect? A model that leans hard on a customer number, or on a column that's only filled in after the bill arrives (What Data Leakage Is), is broken even when its score looks good. Explaining the model is how they catch it.
  • The manager who decides asks: can we send these estimates? How far off are they, and which homes shouldn't get one?

One model, three explanations. A good one starts from the question the person is asking.

The whole model or one guess

Those questions come in two kinds. "What does the model lean on, across all 700 homes?" is about the whole model: your textbook may call it a global explanation. "Why did this home get $230?" is about one guess: a local explanation.

Question About Where this chapter answers it
What did a straight-line model learn for each feature? The whole model Reading Coefficients and Trees
Which features does a big model lean on most? The whole model Permutation Importance
How does the guess change as one feature changes? The whole model, and one home's line How a Guess Changes with One Feature
What do I tell the homeowner? One guess, in words Explaining a Result to Someone Who Isn't Technical

How easily a person can follow why a model guesses what it does is called interpretability. A straight-line model or a small tree can be read directly; a forest of 200 trees can't, so it gets explained from the outside, by watching how its guesses change.

What an explanation can't tell you

The seed script behind home_energy.csv planted the bills: they go up with size, poor insulation, electric heat, more people and a pool. The year a home was built has no effect of its own. But newer homes in this town were built with better insulation, so the two columns move together.

Give a model the year built and leave out the insulation, and it leans on the year: newer homes, lower bills. Reading Coefficients and Trees shows a model doing exactly that. As a description of the model, it's true. As advice to a homeowner, it's wrong: the year on the deed doesn't heat a house, and in this data insulation is what made the difference. A model is built from what went together in its examples, and correlation is not causation.

So every explanation in this chapter says what the model learned from its data. It doesn't say why the world works that way. With made-up data you can check the model against the planted rule; real data comes with no such answer key, which is why the words matter.

Explain a model worth explaining

Before explaining a model, check how far off it is on test data, the examples it never learned from. scikit-learn's own guide gives the same warning before computing which features matter: first measure the model on held-out data. A clear explanation of a model that misses by $60 a month is a clear explanation of a bad model. Greenway's best model, in Permutation Importance, misses by about $11 on average, which is worth explaining.

And sometimes, after measuring and explaining, the honest answer is that the model shouldn't be used for some homes, or at all. That's a result too.

Check yourself

2 questions to try on the lesson page.

So what

People affected by a model deserve to know what drives it: decide whose question you're answering and whether it's about the whole model or one guess, check the model is worth explaining, and say what it learned, never what causes what.

Sources

  • scikit-learn, Permutation feature importance, User Guide: its warning that features found useless for a poor model could be very important for a good one, so a model's predictive power is evaluated on a held-out set (or with cross-validation) before importances are computed.
  • scikit-learn, Common pitfalls in the interpretation of coefficients of linear models, example: an interpretation made about a model may not generalize to the true process that made the data; linear models measure statistical association, and correlation doesn't always imply causation.

Key terms

Related