Logistic Regression

Logistic regression adds up a weighted score for each example, as a linear regression does, then bends it with an S-shaped curve into a probability between 0 and 1, which makes it a classifier despite its name, with C setting how simple scikit-learn keeps it and max_iter how long it may search.

Journeyman · AI Basics · Python · Updated

By

After this lesson you can

  • Explain how logistic regression turns a weighted score into a probability between 0 and 1, and read the sign of each coefficient
  • Train a LogisticRegression, set C and max_iter, and say what a smaller C does and what a ConvergenceWarning means

Key takeaways

  • A straight line fit to 0s and 1s can guess below 0 or above 1, which makes no sense as a chance.
  • Logistic regression adds up a weighted score, then an S-shaped curve (the logistic or sigmoid curve) turns any score into a probability between 0 and 1. A score of 0 is 0.5.
  • Despite "regression" in its name, it's a classifier. A positive coefficient pushes an email's score toward spam, a negative one away from it.
  • C sets how simple the model is kept, and a smaller C keeps the coefficients smaller. The penalty setting is deprecated, so this course uses C only.
  • A ConvergenceWarning means the search for the coefficients stopped at max_iter before it finished. Raise max_iter, or scale the features.

What Classification Is trained a spam filter as a black box. Before you trust a model with Inbox Guard's mail, or a bank's payments, or a library's reminder list, you should know what it does inside. Logistic regression is the classifier most teams try first, the one textbooks start with, and the one whose workings you can say in two sentences.

The example is Inbox Guard's emails.csv, from Exercise: Spam or Not. Each code block is one notebook cell, run in order: the first one loads and splits the data.

A straight line gives impossible answers

The first idea is the model you already know. is_spam is a number, 0 or 1, so fit a linear regression to it and read its guess as a chance of spam:

import pandas as pd
from sklearn.linear_model import LinearRegression, LogisticRegression
from sklearn.model_selection import train_test_split

emails = pd.read_csv("emails.csv")
FEATURES = ["num_links", "num_exclamations", "mentions_free", "sender_in_contacts", "subject_all_caps"]
X_train, X_test, y_train, y_test = train_test_split(
    emails[FEATURES], emails["is_spam"], test_size=0.2, random_state=42
)

two_emails = pd.DataFrame({
    "num_links": [0, 9],
    "num_exclamations": [0, 6],
    "mentions_free": [0, 1],
    "sender_in_contacts": [1, 0],
    "subject_all_caps": [0, 1],
})
line = LinearRegression()
line.fit(X_train, y_train)
print(line.predict(two_emails))
[-0.15089448  1.40732702]

The first email, from a contact with no links, gets −0.15: a chance of minus 15%. The second, with 9 links, 6 exclamation marks, "free" and an all-caps subject, gets 1.41: a 141% chance. A straight line keeps going in both directions, and a chance can't.

A score, bent into a probability

Logistic regression keeps the line's first step and fixes the second. It adds up a score for each email the way a linear regression adds up a guess: a starting amount plus a weight times each feature. Then it bends the score with an S-shaped curve that squeezes any number into the range 0 to 1.

An S-shaped curve with the score across the bottom from −6 to 6 and the chance of spam up the side from 0 to 1. The curve is nearly flat close to 0 on the left, rises steeply through 0.5 at a score of 0, and flattens close to 1 on the right. Two emails are marked on it: one from a contact with no links, at a score of −4.21 and a chance of 0.015, and one with 9 links, 6 exclamation marks, "free" and an all-caps subject, at a score of 5.60 and a chance of 0.996

A score of 0 becomes a chance of 0.5. A big positive score comes close to 1 without reaching it, and a big negative score comes close to 0. Your textbook calls the curve the logistic function or the sigmoid (read "SIG-moyd"), and the score the log-odds. The model's job in fit is to find the weights whose probabilities best match the 0s and 1s in the training emails.

That makes the name confusing: logistic regression works out a probability, a number, and then picks a class from it, so it's a classifier. scikit-learn's own guide says it plainly: despite its name, it's a linear model for classification.

The weights it learned

model = LogisticRegression()
model.fit(X_train, y_train)
for feature, coefficient in zip(FEATURES, model.coef_[0]):
    print(feature, round(coefficient, 2))
print("intercept", round(model.intercept_[0], 2))
num_links 0.35
num_exclamations 0.2
mentions_free 1.75
sender_in_contacts -2.54
subject_all_caps 1.22
intercept -1.67

coef_ holds one row of weights for a binary classifier, so coef_[0] is that row. Read the signs: a positive coefficient pushes the score toward spam (each link adds 0.35), and a negative one pushes it away (a sender in your contacts takes off 2.54). Each coefficient moves the score, not the chance directly: near a score of 0, a push changes the chance a lot; far out on the flat ends, hardly at all.

print("Scores:", model.decision_function(two_emails).round(2))
print("Chances of spam:", model.predict_proba(two_emails)[:, 1].round(3))
print("Guesses:", model.predict(two_emails))
Scores: [-4.21  5.6 ]
Chances of spam: [0.015 0.996]
Guesses: [0 1]

decision_function gives each email's score, and predict_proba the curve's answer: a 1.5% chance for the first email and 99.6% for the second, both between 0 and 1. Probabilities and the 0.5 Line reads them.

The model learned that emails from contacts were rarely spam in these emails. It found what went with what, not why; correlation is not causation.

C: how simple the model is kept

scikit-learn holds the weights back a little on purpose, so they don't chase every quirk of the training emails. That's regularization, and it's on by default. C sets how much: a smaller C holds the weights back more, which keeps the model simpler. The default is C=1.0:

for c in [0.01, 1, 100]:
    small = LogisticRegression(C=c)
    small.fit(X_train, y_train)
    print("C =", c, "contacts coefficient:", round(small.coef_[0][3], 2))
C = 0.01 contacts coefficient: -0.55
C = 1 contacts coefficient: -2.54
C = 100 contacts coefficient: -2.74

Older code sets the kind of holding back with penalty; that setting is deprecated since scikit-learn 1.8 and goes away in 1.10, so this course sets C only. Hyperparameters: Settings You Choose shows how to choose C without peeking at the test emails.

The convergence warning and max_iter

fit finds the weights by searching, step by step, and stops after max_iter steps, 100 by default. Add the word count, a number in the hundreds beside features from 0 to 9, and the search runs out of steps:

with_words = FEATURES + ["word_count"]
X_train_words, X_test_words, y_train, y_test = train_test_split(
    emails[with_words], emails["is_spam"], test_size=0.2, random_state=42
)
wordy = LogisticRegression()
wordy.fit(X_train_words, y_train)
print("Steps used:", wordy.n_iter_[0])
ConvergenceWarning: lbfgs failed to converge after 100 iteration(s) (status=1):
STOP: TOTAL NO. OF ITERATIONS REACHED LIMIT

Increase the number of iterations to improve the convergence (max_iter=100).
You might also want to scale the data as shown in:
    https://scikit-learn.org/stable/modules/preprocessing.html
Please also refer to the documentation for alternative solver options:
    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression
Steps used: 100

A warning, not an error: the model is fitted, but its weights may not be the best ones yet. Give the search more steps, and it finishes:

wordy = LogisticRegression(max_iter=1000)
wordy.fit(X_train_words, y_train)
print("Steps used:", wordy.n_iter_[0])
print("word_count coefficient:", round(wordy.coef_[0][5], 5))
Steps used: 122
word_count coefficient: -0.00024

The word count's weight is close to 0: the model found no pattern there, and the seed script planted none. The better fix is the one the warning names, putting the features on similar scales, which Scaling Features teaches.

Check yourself

2 questions to try on the lesson page.

So what

Logistic regression is a weighted score bent into a probability: read its coefficients' signs as pushes toward or away from the positive class, keep C in mind as how simple it's kept, and treat a ConvergenceWarning as unfinished work.

Sources

  • scikit-learn, Linear Models: Logistic regression, User Guide: despite its name, logistic regression is a linear model for classification; the probabilities are modeled with a logistic function; in the binary case predict_proba gives the positive class's probability as the logistic function of the weighted sum plus the intercept; regularization is applied by default, which is common in machine learning but not in statistics; the predicted probability is used as a classifier by applying a threshold (by default 0.5); lbfgs is the default solver.
  • scikit-learn 1.9.1, LogisticRegression, API reference: C (default 1.0) is the inverse of regularization strength, smaller values specify stronger regularization; max_iter (default 100) is the maximum number of iterations taken for the solvers to converge; penalty was deprecated in version 1.8 and will be removed in 1.10 (use l1_ratio and C instead); coef_, intercept_, n_iter_, and decision_function (in the binary case, the score for classes_[1], where above 0 means that class is predicted).

Key terms

Related