k-means with scikit-learn
scikit-learn's KMeans runs the whole k-means loop, with fit_predict giving each example a group number and cluster_centers_ holding the centers, but its distances weigh features by their units, so the features are scaled first in a pipeline, several starts are run (n_init) with a fixed random_state, and the result suits round groups of similar spread.
Journeyman · AI Basics · Python · Updated
After this lesson you can
- Cluster a table with a StandardScaler and KMeans pipeline, setting n_clusters, n_init and random_state, and read its labels and cluster_centers_
- Explain why features are scaled before k-means, what n_init and k-means++ starts do, and why k-means suits round groups of similar spread
Key takeaways
- KMeans(n_clusters=4, n_init=10, random_state=0) asks for 4 groups, runs k-means from 10 starts and keeps the tightest, and gives the same answer every run.
- fit_predict gives each row a group number, also kept in labels_. The numbers are only names, and cluster_centers_ holds each group's center.
- k-means measures distance, so a feature with big numbers outweighs one from 0 to 1. Scale first, with StandardScaler and KMeans in one pipeline.
- The n_init default is "auto", one start with k-means++. This course sets n_init=10 so the result never rests on one start.
- k-means suits round groups of similar spread. A long, stretched group or one much more spread out can be cut up or merged.
How k-means Works grouped six café members by hand. The Cozy Corner Café has 600 members with four numbers each, and that's a job for scikit-learn's KMeans, which runs the same loop. Two things the hand version never had to face come up here: features measured on very different scales, and the risk of a poor start. Both decide whether the groups you hand to the café's owner are real.
The example is the café's 600 members in cafe_members.csv, which comes with Exercise: Café Regulars. Each code block below is one notebook cell, and the cells run in order: the first one loads the data, and the others use what it made.
KMeans in a few lines
KMeans follows scikit-learn's usual pattern, with one difference: there's no y. fit_predict(X) runs k-means on the features and gives back each row's group number:
import pandas as pd
from sklearn.cluster import KMeans
members = pd.read_csv("cafe_members.csv")
features = ["visits_per_month", "avg_spend", "share_before_9am", "share_with_food"]
X = members[features]
kmeans = KMeans(n_clusters=4, n_init=10, random_state=0)
labels = kmeans.fit_predict(X)
print("First ten labels:", labels[:10])
print("Same as labels_:", (labels == kmeans.labels_).all())
print("Members in each group:", pd.Series(labels).value_counts().sort_index().tolist())
First ten labels: [3 3 3 1 1 3 2 1 0 0]
Same as labels_: True
Members in each group: [159, 151, 118, 172]
The settings, one sentence each:
n_clusters=4is k, the number of groups to find. Its default is 8, so always set it.random_state=0fixes the random starting centers, so every run, yours included, gives the same groups on the same package versions.n_init=10runs k-means from 10 different starts and keeps the tightest result. More on this below.
fit_predict also stores the group numbers in labels_ (read "labels underscore"). The numbers are names, not ranks: group 3 isn't "more" than group 1, and another setting can number the same groups differently. Each group's center is in cluster_centers_, one row per group, one column per feature:
print(pd.DataFrame(kmeans.cluster_centers_, columns=features).round(2))
visits_per_month avg_spend share_before_9am share_with_food
0 2.78 9.81 0.43 0.42
1 18.38 5.49 0.84 0.15
2 4.16 21.49 0.11 0.83
3 10.63 6.85 0.23 0.32
Group 3 is odd: about 11 visits a month, yet 23% of them before 9, between the early regulars of group 1 and the afternoon crowd. Something blurred it.
Scale first
k-means measures straight-line distance, so, as Distance Between Examples showed, a feature counts by the size of its numbers:
print(X.std().round(2))
visits_per_month 6.44
avg_spend 6.20
share_before_9am 0.34
share_with_food 0.27
dtype: float64
Visits and spend vary by about 6; the two shares by about 0.3. One extra visit counts as much in the distance as the difference between never coming before 9 and always coming before 9. The shares barely count, so k-means groups mostly by visits and spend.
The fix is Scaling Features: StandardScaler puts every feature on the same footing (each with a mean of 0 and a standard deviation of 1), and a pipeline, as in Pipelines: Every Step, Every Time, runs it before KMeans every time. The centers come out in scaled units, so inverse_transform turns them back into visits, dollars and shares:
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
model = make_pipeline(StandardScaler(), KMeans(n_clusters=4, n_init=10, random_state=0))
scaled_labels = model.fit_predict(X)
print("Members in each group:", pd.Series(scaled_labels).value_counts().sort_index().tolist())
centers = model.named_steps["standardscaler"].inverse_transform(model.named_steps["kmeans"].cluster_centers_)
print(pd.DataFrame(centers, columns=features).round(2))
Members in each group: [146, 122, 180, 152]
visits_per_month avg_spend share_before_9am share_with_food
0 2.62 9.85 0.47 0.40
1 4.16 21.19 0.10 0.84
2 17.46 5.50 0.85 0.15
3 9.89 7.18 0.10 0.35
Now the afternoon group (3) comes before 9 only 10% of the time. Count which raw group each member was in against their scaled group:
print(pd.crosstab(scaled_labels, labels, rownames=["scaled"], colnames=["raw"]))
raw 0 1 2 3
scaled
0 143 0 3 0
1 7 0 115 0
2 0 149 0 31
3 9 2 0 141
Most members land in the same group under a new number, but 52 don't. The biggest change: 31 morning regulars who visit a little less were lumped in with the afternoon crowd when the shares didn't count. There's no test set to protect here, so the pipeline learns its scaling from all 600 members, and model.predict can place a new member later, scaled with those same numbers.
Starts: n_init and k-means++
How k-means Works showed a poor start ending in a worse grouping. Here's the same thing at full size: one start each, with completely random starting centers, for six values of random_state. inertia_ is the total squared distance from each member to its center, the number that compared Start A and Start B (smaller is tighter):
X_scaled = model.named_steps["standardscaler"].transform(X)
for seed in range(6):
one_start = KMeans(n_clusters=4, init="random", n_init=1, random_state=seed).fit(X_scaled)
print("random_state", seed, "inertia", round(one_start.inertia_, 1))
random_state 0 inertia 298.8
random_state 1 inertia 474.0
random_state 2 inertia 298.8
random_state 3 inertia 476.7
random_state 4 inertia 477.0
random_state 5 inertia 298.8
Half the starts got stuck at about 475. scikit-learn guards against this in two ways. k-means++ (its default init) picks the starting centers one at a time, favoring points far from the centers already picked, so the starts spread out across the data instead of landing anywhere. And n_init runs k-means from several starts and keeps the one with the lowest inertia. Its default is "auto": since scikit-learn 1.4 that means one start when init is k-means++, and 10 with random starts. This course writes n_init=10 every time, so a result never rests on one start and the code says what it does.
Round groups
k-means judges a group by how close its members sit to one center, in every direction alike, so it favors compact, round groups. scikit-learn's guide says it assumes groups that are convex and the same in every direction (isotropic, in your textbook) with equal spread, and it responds poorly to long, stretched groups or odd shapes: it may cut one in half or join part of it to a neighbor. The café's planted groups are round blobs, which is why k-means suits them; on real data, check the groups make sense before trusting them.
Check yourself
2 questions to try on the lesson page.
So what
KMeans runs the loop for you, but only scaled features, several starts and a fixed seed make its groups ones you can trust and repeat.
Sources
- scikit-learn 1.9.1, KMeans, API reference:
n_clusters(default 8);init(default'k-means++', which picks initial centroids by sampling based on each point's contribution to the overall inertia, or'random');n_init(default'auto': the number of runs with different centroid seeds, the result being the best in inertia;'auto'means 10 runs withinit='random'and 1 withinit='k-means++'; the default changed to'auto'in version 1.4);random_state(an int makes the randomness deterministic);labels_,cluster_centers_andinertia_(the sum of squared distances of samples to their closest cluster center);fit_predictandpredict. - scikit-learn 1.9.1, Clustering: K-means, User Guide: k-means separates samples into groups of equal variance by minimizing the inertia; inertia assumes clusters are convex and isotropic and responds poorly to elongated clusters or manifolds with irregular shapes; k-means can converge to a local minimum that depends on the initialization, which k-means++ helps address.
- scikit-learn 1.9.1, Demonstration of k-means assumptions, example: k-means is more appropriate for clusters that are isotropic and normally distributed; anisotropic blobs and unequal variances are real limitations; more initializations (
n_init=10) help avoid a sub-optimal local minimum. - scikit-learn 1.9.1, StandardScaler, API reference: standardizes features by removing the mean and scaling to unit variance; a feature whose variance is orders of magnitude larger than the others' might dominate the objective function.
- scikit-learn, Glossary of Common Terms and API Elements: fit_predict, in clusterers, fits the model and returns the predictions on the training data, also stored in
labels_.