R Squared: Better Than Guessing the Average?

R squared compares a model's squared errors with those of always guessing the average of the values being scored, so 1 is perfect, 0 is no better than the average, and below 0 is worse, which can happen on test data; scikit-learn's r2_score computes it, and on its own it can't say whether the misses are small enough to use.

Journeyman · AI Basics · Python · Updated

By

After this lesson you can

  • Compute R squared by comparing a model's squared errors with those of always guessing the average
  • Explain what an R squared of 1, 0 or below 0 means, and what R squared alone can't tell you

Key takeaways

  • R squared is 1 minus the model's squared errors divided by the squared errors of always guessing the average of the values being scored.
  • An R squared of 1 is perfect, 0 is no better than always guessing the average, and below 0 is worse than that.
  • On test data, R squared can be negative, because the average it compares with is the test values' own average, which the model never saw.
  • r2_score(y_true, y_pred) needs the actual values first. Swapped, it gives a different, wrong number.
  • R squared depends on how spread out the values are, so it can't say on its own whether the misses are small enough to use.

Pine Ridge Delivery's model is off by about 3.6 minutes on average. Is that good? If every order took between 29 and 31 minutes, a model off by 3.6 would be worse than saying "30 minutes" to everyone. If orders take anywhere from 2 minutes to an hour, it's doing real work. A number on its own needs something to compare with, and the simplest thing to compare with is a guess that needs no model at all. You'll see that comparison as R squared in nearly every report on a model that predicts a number, so it pays to know exactly what it says, and what it doesn't.

The example is Pine Ridge Delivery's deliveries.csv, from Exercise: Delivery Time Errors. The scikit-learn code blocks are notebook cells, run in order: the first loads the data, and the others use what it made.

The guess that needs no model

If you knew nothing about an order (not its distance, its stops or the weather), your best single guess would be the average: the same number of minutes for everyone. Any model worth having should beat that. R squared, written R² and read "R squared", measures how much better a model's squared errors are than the squared errors of always guessing the average:

R² = 1 − (the model's squared errors, added up) ÷ (the squared errors of always guessing the average, added up)

Here it is by hand on four orders:

actual = [20, 30, 40, 50]
guessed = [22, 29, 37, 52]

average = sum(actual) / len(actual)
model_total = 0
average_total = 0
for a, g in zip(actual, guessed):
    model_total += (a - g) ** 2
    average_total += (a - average) ** 2

print("Average:", average)
print("Model's squared errors:", model_total)
print("Always the average:", average_total)
print("R squared:", 1 - model_total / average_total)
Average: 35.0
Model's squared errors: 18
Always the average: 500.0
R squared: 0.964

Always guessing 35 minutes misses by 15, 5, 5 and 15, whose squares add up to 500. The model's misses square to only 18, so R² = 1 − 18 ÷ 500 = 0.964. Your textbook calls R² the coefficient of determination, and may describe it as the share of the variation in the label that the model explains.

1, 0 and below 0

  • 1 means the model's squared errors are 0: every guess is exactly right.
  • 0 means the model does no better than always guessing the average.
  • Below 0 means it does worse than the average. There's no lowest value: a bad enough model can go as far below 0 as it likes.

Despite its name, R² isn't always a square of anything, which is why it can be negative.

r2_score in scikit-learn

r2_score(y_true, y_pred) takes the actual values first and the guesses second. Unlike the MAE, the order matters:

import pandas as pd
from sklearn.linear_model import LinearRegression
from sklearn.metrics import r2_score
from sklearn.model_selection import train_test_split

deliveries = pd.read_csv("deliveries.csv")
features = ["distance_miles", "stops_before", "is_raining"]
train, test = train_test_split(deliveries, test_size=0.2, random_state=42)

model = LinearRegression()
model.fit(train[features], train["minutes"])
guesses = model.predict(test[features])

print("R squared:", round(r2_score(test["minutes"], guesses), 2))
print("Swapped by mistake:", round(r2_score(guesses, test["minutes"]), 2))
R squared: 0.69
Swapped by mistake: 0.55

The model's R² on the 100 test orders is 0.69: its squared errors are about a third of the average guess's. Swapped, the average comes from the guesses instead of the real minutes, and the answer is a different, wrong number.

The average R² compares with is the average of the values being scored: the test orders' own average. That's a number the model never saw, which is how R² can drop below 0 on test data. Guess the training orders' average for every test order, the guess a model that looks at no features would make:

train_average = train["minutes"].mean()
same_guess = [train_average] * len(test)
print("Training average:", round(train_average, 2))
print("Test average:", round(test["minutes"].mean(), 2))
print("R squared:", round(r2_score(test["minutes"], same_guess), 2))
Training average: 28.42
Test average: 30.4
R squared: -0.04

The test orders happen to run about two minutes longer on average than the training orders, so guessing 28.42 for all of them does a little worse than the test set's own average would: an R² a little below 0.

What R squared can't tell you

R² compares the model with the average, so it depends on how spread out the values are. Here are the same four misses, 2, 1, 3 and 2 minutes, on orders that vary a lot and on orders that hardly vary:

from sklearn.metrics import mean_absolute_error

wide = [20, 30, 40, 50]
narrow = [33, 34, 36, 37]
print("Wide orders, MAE:", mean_absolute_error(wide, [22, 29, 37, 52]), "R squared:", r2_score(wide, [22, 29, 37, 52]))
print("Narrow orders, MAE:", mean_absolute_error(narrow, [35, 33, 33, 39]), "R squared:", r2_score(narrow, [35, 33, 33, 39]))
Wide orders, MAE: 2.0 R squared: 0.964
Narrow orders, MAE: 2.0 R squared: -0.8

The same 2-minute misses score 0.964 on one set and −0.8 on the other. So R² can't be compared across datasets whose values spread differently, and it can't say whether the misses are small enough to use. A 3.6-minute miss is fine for a pizza and useless for a train timetable, whatever the R². Report it beside the MAE or RMSE, which are in minutes, and decide with what a miss costs. And a high R² says the model's guesses follow the data, not that its features cause what it predicts.

Do it: score a model against the average

Pine Ridge Delivery wants an R² for any set of orders, worked out by hand, to check its reports.

The inputs:

  • actual: how many minutes each order really took. It always holds at least two different values.
  • predicted: the model's guesses for the same orders, in the same order.

For example, with four orders:

actual = [10, 20, 30, 40]
predicted = [12, 18, 33, 37]

The average of actual is 25.0. Always guessing 25 misses by 15, 5, 5 and 15, and their squares add up to 225 + 25 + 25 + 225 = 500, so average_guess_total(actual) returns 500.0. The model misses by −2, 2, −3 and 3, whose squares add up to 4 + 4 + 9 + 9 = 26. R² is 1 − 26 ÷ 500 = 0.948, so r_squared(actual, predicted) returns 0.948.

Your job: write two functions.

  • average_guess_total(actual) returns the squared errors of always guessing the average of actual, added up: each value minus the average, squared, all added together.
  • r_squared(actual, predicted) returns 1 minus (the model's squared errors added up, divided by average_guess_total(actual)), rounded to 3 decimal places with round(..., 3).

Practiced live on the lesson page: write the Python right there and check it.

Check yourself

2 questions to try on the lesson page.

So what

R squared answers one question, how much better the model is than always guessing the average, and answers it well; but whether the misses are small enough to use is a question for the MAE and RMSE, in the label's own units.

Sources

  • scikit-learn, Metrics and scoring: R² score, the coefficient of determination, User Guide: the best possible score is 1.0 and it can be negative, because a model can be arbitrarily worse; a constant model that always predicts the average of y, disregarding the features, gets 0.0; and because the variance it uses depends on the dataset, R² may not be meaningfully comparable across different datasets.
  • scikit-learn 1.9.1, r2_score, API reference: r2_score(y_true, y_pred), the coefficient of determination; it is not a symmetric function; it may be negative, since it need not be the square of a quantity R; and it isn't well-defined for fewer than two samples.

Key terms

Related