# When the World Changes

A model learns the world as it was in its training data, so when new examples stop looking like those rows (data drift), or the same inputs start leading to different answers (concept drift), its guesses slip; checking each column's range and mean on the new rows catches the first, and the error on new real answers catches both, which is when you retrain.

Journeyman · AI Basics · Python · Updated November 19, 2026

By Gregor Dzierzon

## After this lesson you can

- Check new rows against the training data, column by column, for values outside the training range and a change in the mean
- Tell a change in the inputs from a change in how the inputs lead to the answer, and decide when to retrain by watching the error on new answers

## Key takeaways

- A model only knows the data it was trained on. When new examples stop looking like those rows, its guesses can slip without any error message.
- A guess for a value outside the training range is a guess about something the model never saw. A line keeps going and a tree stays flat, and neither knows which is right.
- Checking each column's range and mean on new rows needs no answers, so it can run the day the rows arrive. It catches new inputs (data drift).
- When the same inputs start leading to different answers (concept drift), the columns look the same. Only the error on new real answers shows it.
- Retrain on recent data when the error grows, and keep measuring the error on new answers as they arrive.

A model is trained once, on the data of one moment, and then it's used for months. The world doesn't wait: a store opens in a new town, prices go up, a school changes its grading. Whenever a model you built keeps working after you hand it over, at a job or in a project someone still uses next year, someone has to ask whether today's data still looks like the data it learned from.

The example is Pine Ridge Delivery again, a year after its app started saying "your order arrives in about 31 minutes." The 500 orders the model learned from are in `deliveries.csv`, and 300 orders from this year are in `deliveries_new.csv`: our own made-up data, both from *Exercise: Model Check-Up*. Each code block is one notebook cell, run in order: the first loads the data, and the others use what it made.

## A year later, the guesses slip

Customers say the estimates run short. The first cell trains the model the way *Errors and the Mean Absolute Error* did, as a pipeline (*Pipelines: Every Step, Every Time*), and scores it twice: on last year's 100 test orders, and on this year's orders, now that their real minutes are known:

```python
import pandas as pd
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_absolute_error
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

deliveries = pd.read_csv("deliveries.csv")
new_orders = pd.read_csv("deliveries_new.csv")
features = ["distance_miles", "stops_before", "is_raining"]
train, test = train_test_split(deliveries, test_size=0.2, random_state=42)

model = make_pipeline(StandardScaler(), LinearRegression())
model.fit(train[features], train["minutes"])

print("MAE, last year's test orders:", round(mean_absolute_error(test["minutes"], model.predict(test[features])), 2))
print("MAE, this year's orders:", round(mean_absolute_error(new_orders["minutes"], model.predict(new_orders[features])), 2))
```

```text
MAE, last year's test orders: 3.63
MAE, this year's orders: 4.29
```

Nothing crashed and no warning appeared. The same model is about two-thirds of a minute worse on average, and only measuring showed it. Something about the orders changed.

## Has the new data moved?

A model learned from its **training data**: the 400 training orders. The first question is whether this year's orders look like them. For each feature, find the training rows' smallest and largest value, the **training range**, then count how many new values fall outside it, and compare the averages:

```python
for column in features:
    low, high = train[column].min(), train[column].max()
    outside = (new_orders[column] < low) | (new_orders[column] > high)
    print(column, "| range:", low, "to", high, "| outside:", outside.mean(),
          "| mean then:", round(train[column].mean(), 2), "now:", round(new_orders[column].mean(), 2))
```

```text
distance_miles | range: 0.5 to 9.0 | outside: 0.25 | mean then: 4.71 now: 6.59
stops_before | range: 0 to 3 | outside: 0.0 | mean then: 1.53 now: 1.48
is_raining | range: 0 to 1 | outside: 0.0 | mean then: 0.18 now: 0.2
```

`outside.mean()` is the share of `True` values: a quarter of this year's orders go farther than 9 miles, and the average distance grew by almost 2 miles.

![Two sets of bars comparing the share of orders in each distance band, last year's 400 training orders beside this year's 300: 0 to 3 miles 30% then 19%, 3 to 6 miles 37% then 26%, 6 to 9 miles 33% then 30%, 9 to 12 miles 0% then 16%, and 12 to 14 miles 0% then 9%, with a dashed line at 9 miles marking the farthest training order: a quarter of this year's orders lie beyond anything the model learned from](https://fsdtrainingmedia.blob.core.windows.net/public/images/2026/11/0bce4520bd3b4094bc9519361515f291.webp) Pine Ridge now delivers to new neighborhoods. When new examples stop looking like the training data, that's called **data drift**. This check needs no real minutes at all, so it can run the day the new orders arrive.

## Outside the training range

What does a model guess for 12 miles when it never saw more than 9? Compare the straight line with a regression tree trained on the same orders:

```python
from sklearn.tree import DecisionTreeRegressor

tree = DecisionTreeRegressor(max_depth=5, random_state=0)
tree.fit(train[features], train["minutes"])

trips = pd.DataFrame({"distance_miles": [8.0, 12.0, 14.0], "stops_before": [1, 1, 1], "is_raining": [0, 0, 0]})
print("line:", model.predict(trips).round(1))
print("tree:", tree.predict(trips).round(1))
```

```text
line: [35.4 47.5 53.6]
tree: [31.8 39.4 39.4]
```

The line keeps going, about 3 minutes a mile. The tree gives 12 and 14 miles the same 39.4 minutes, because its guesses are averages of training orders, as *Regression Trees* showed. Here the line happens to be right: the pattern planted in the data stays straight. But no model can know what happens where it has no examples. A guess outside the training range is a guess about something the model never saw.

## A change the columns don't show

The stops looked the same in the check: same range, nearly the same mean. Now look at what the model misses on this year's orders within 9 miles, by the number of stops the driver makes first:

```python
near = new_orders[new_orders["distance_miles"] <= 9.0]
leftover = near["minutes"] - model.predict(near[features])
for stops in range(4):
    print("Stops first:", stops, "average residual:", round(leftover[near["stops_before"] == stops].mean(), 1))
```

```text
Stops first: 0 average residual: -0.1
Stops first: 1 average residual: 3.0
Stops first: 2 average residual: 4.9
Stops first: 3 average residual: 6.4
```

The more stops, the later the order runs past its estimate: about 2 extra minutes a stop. A new routing rule made each stop take longer. The inputs didn't change; what they lead to did. That's called **concept drift**, and the range check can't see it: only the error on new real answers shows it. Books use these two words in slightly different ways, but the difference is the one to keep: new kinds of inputs, or the same inputs leading to new answers.

## Retrain, then keep watching

When the error grows, train again on recent data. Hold back some of this year's orders to test on, as always:

```python
new_train, new_test = train_test_split(new_orders, test_size=0.2, random_state=42)
retrained = make_pipeline(StandardScaler(), LinearRegression())
retrained.fit(new_train[features], new_train["minutes"])

print("Old model:", round(mean_absolute_error(new_test["minutes"], model.predict(new_test[features])), 2))
print("Retrained:", round(mean_absolute_error(new_test["minutes"], retrained.predict(new_test[features])), 2))
```

```text
Old model: 4.38
Retrained: 2.76
```

On the same 60 held-back orders, the retrained model is off by 2.76 minutes instead of 4.38. Retraining isn't a one-time fix: run the range check on each new batch of orders, and when their real minutes come in, measure the error again. Write down what "too much" means before you need it (for Pine Ridge, say, an MAE above 4.5 minutes for a week).

## Do it: find the new deliveries outside the training range

Pine Ridge Delivery wants a quick drift check for one column: what share of this week's orders fall outside the distances the model was trained on.

The inputs:

- `train_values`: one column's values in the training orders, such as each order's distance in miles.
- `new_values`: the same column's values in the new orders.

For example:

```python
train_values = [1.2, 3.5, 8.8, 0.6, 5.0]
new_values = [2.0, 9.5, 11.2, 4.4]
```

The training range runs from the smallest training value, 0.6, to the largest, 8.8. Of the four new values, 9.5 and 11.2 are above 8.8, and 2.0 and 4.4 are inside. That's 2 out of 4, so `share_outside(train_values, new_values)` returns `0.5`.

**Your job:** write `share_outside(train_values, new_values)`.

- It returns the number of new values below the smallest training value or above the largest, divided by the number of new values.
- A new value equal to the smallest or the largest training value is inside the range.
- Both lists always have at least one value.

Practiced live on the lesson page: write the Python right there and check it.

## Check yourself

2 questions to try on the lesson page.

## So what

A model's score is only true for data like its training data, so check new rows against the training ranges as they arrive, and keep measuring the error as real answers come in: that's how you notice the world changed before your users do.

## Sources

- scikit-learn 1.9.1, [Common pitfalls and recommended practices](https://scikit-learn.org/stable/common_pitfalls.html), *Inconsistent preprocessing* and *Data leakage*: the transformations used in training must be used the same way on test data and on data in a production system, and a model's real test is how it does on novel data, for example in production.
- scikit-learn, [Decision Trees](https://scikit-learn.org/stable/modules/tree.html), User Guide: a tree's predictions are piecewise constant approximations, so trees are not good at extrapolation.
- scikit-learn 1.9.1, [DecisionTreeRegressor](https://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeRegressor.html), API reference: `max_depth` and `random_state`.
- scikit-learn 1.9.1, [mean_absolute_error](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.mean_absolute_error.html), API reference: the mean absolute error of the true values against the predictions.

## Key terms

- [Data Drift](https://www.ask-codey.com/glossary/data-drift)
- [Concept Drift](https://www.ask-codey.com/glossary/concept-drift)
- [Training Data](https://www.ask-codey.com/glossary/training-data)
- [Mean Absolute Error](https://www.ask-codey.com/glossary/mean-absolute-error)

## Related

- [Errors and the Mean Absolute Error](https://www.ask-codey.com/lessons/ml-mean-absolute-error)
- [Regression Trees](https://www.ask-codey.com/lessons/ml-regression-trees)
- [Saving and Loading a Model](https://www.ask-codey.com/lessons/ml-saving-a-model)
- [Exercise: Model Check-Up](https://www.ask-codey.com/lessons/exercise-ml-model-check-up)

Source: https://www.ask-codey.com/lessons/ml-when-the-world-changes
