Describing Each Group

A clustering is only useful once each group is described, so you groupby the cluster label to get each group's means and size, compare them with the whole table's, name each group in plain words from what its numbers say, and state the limits, that other features or another k would give other groups and that a member isn't their group.

Journeyman · AI Basics · Python · Updated

By

After this lesson you can

  • Describe each cluster with groupby on its label, comparing each group's means and size with the whole table's
  • Name a group in plain words from its numbers, and state the honest limits of a clustering and why a member isn't their group

Key takeaways

  • groupby on the cluster label gives each group's mean for every feature, and size gives how many members each group has.
  • A group's numbers mean most beside the whole table's. A group that averages 17 visits stands out against an average of 9.
  • Name a group from what its numbers say, such as "frequent early visitors", never from what they don't, such as their job or the day of the week.
  • Groups depend on the features and the k you chose, a member near a border could go either way, and the groups describe the data without explaining why.

k-means hands back a group number for each of the café's 600 members, and the owner can't plan anything with "group 2". She needs to hear "about 180 members come 17 times a month, mostly before 9, and spend $5.50". Turning group numbers into a description someone can act on, with the limits stated, is the step that makes a clustering worth doing, whether the groups are customers, support tickets or anything else.

The example is the café's 600 members in cafe_members.csv, which comes with Exercise: Café Regulars, grouped with k = 4 as How Many Groups? chose. Each code block below is one notebook cell, and the cells run in order.

Each group's numbers

A description is a groupby on the group number, as Grouping with groupby shows: split the members by group, take each feature's mean, and add each group's size:

import pandas as pd
from sklearn.cluster import KMeans
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

members = pd.read_csv("cafe_members.csv")
features = ["visits_per_month", "avg_spend", "share_before_9am", "share_with_food"]
model = make_pipeline(StandardScaler(), KMeans(n_clusters=4, n_init=10, random_state=0))
members["group"] = model.fit_predict(members[features])

summary = members.groupby("group")[features].mean()
summary["members"] = members.groupby("group").size()
print(summary.round(2))
       visits_per_month  avg_spend  share_before_9am  share_with_food  members
group
0                  2.62       9.85              0.47             0.40      146
1                  4.16      21.19              0.10             0.84      122
2                 17.46       5.50              0.85             0.15      180
3                  9.89       7.18              0.10             0.35      152

The means are in the features' own units, visits, dollars and shares, even though k-means worked on scaled numbers: groupby reads the original columns. Size matters too: a group of 12 members isn't worth a menu of its own, however clear its numbers.

Compared with what?

A mean of 9.85 dollars means little alone. Put it beside the whole café's means:

print(members[features].mean().round(2))
visits_per_month     9.23
avg_spend           10.17
share_before_9am     0.41
share_with_food      0.40
dtype: float64

Now each group stands out by where it differs. Group 2 visits about twice as often as the average member (17.46 against 9.23) and comes before 9 twice as often (0.85 against 0.41). Group 1 spends about twice the average (21.19 against 10.17) and orders food with most visits. Group 0 visits rarely, at any hour; its spend and food share are close to everyone's. Group 3 visits about as often as average, but rarely before 9.

Naming a group honestly

A name helps people remember a group, as long as it says only what the numbers say:

Group Members Name Why
2 180 Frequent early visitors 17 visits a month, 85% before 9, $5.50 a visit, little food
3 152 Afternoon regulars about 10 visits, 10% before 9, $7.18 a visit
1 122 Big spenders who eat about 4 visits, $21.19 a visit, food 84% of the time
0 146 Occasional visitors under 3 visits a month, at any hour, a middling spend

Tempting names go further than the data: "commuters" claims a job, and "weekend brunchers" claims a day of the week, and the file holds neither. Our seed script happens to say who it planted (seed_data.py describes four hidden groups, and k-means found them closely), but real data comes with no such list, so name groups from the columns you have.

The honest limits

A clustering comes with limits, and the description should say them.

A member isn't their group. Member CC-1170 is in group 3, but look at their distances (in scaled units) to the four centers:

member = members[members["member_id"] == "CC-1170"]
print(member.to_string(index=False))
print("Distance to each center:", model.transform(member[features]).round(2))
member_id  visits_per_month  avg_spend  share_before_9am  share_with_food  group
  CC-1170                 7       9.54              0.26             0.55      3
Distance to each center: [[1.08 2.27 2.9  1.07]]

1.07 from group 3's center and 1.08 from group 0's: a coin toss. An offer for "afternoon regulars" may not fit this member at all. Every group has members near its edges.

Other features, other groups. Cluster on visits and spend alone, and compare:

two_features = ["visits_per_month", "avg_spend"]
other = make_pipeline(StandardScaler(), KMeans(n_clusters=4, n_init=10, random_state=0))
members["group_2"] = other.fit_predict(members[two_features])
print(pd.crosstab(members["group"], members["group_2"]))
print(members.groupby("group_2")["share_before_9am"].mean().round(2))
group_2    0    1    2    3
group
0          0    0    3  143
1          0    0  115    7
2         31  149    0    0
3        141    2    0    9
group_2
0    0.23
1    0.84
2    0.11
3    0.43
Name: share_before_9am, dtype: float64

Without the time of day, 31 frequent early visitors join the afternoon crowd, whose before-9 share rises to 0.23. Neither grouping is wrong; each answers a different question. Another k would change the groups too, as How Many Groups? showed.

Groups describe; they don't explain. Group 1 spends more and eats more, but the clustering doesn't say why, or whether a deal would change what anyone does.

Check yourself

2 questions to try on the lesson page.

So what

A group number becomes useful when it's a size, a few means beside the whole table's, an honest name, and the limits said out loud.

Sources

  • pandas, Group by: split-apply-combine, User Guide: groupby splits the data into groups, applies a function such as mean to each, and combines the results; size gives each group's number of rows.
  • scikit-learn 1.9.1, KMeans, API reference: transform turns X into a cluster-distance space, each row's distance to every cluster center; labels_ is the label of each point.
  • scikit-learn 1.9.1, Selecting the number of clusters with silhouette analysis on KMeans clustering, example: a silhouette value of 0 means a sample is on or very close to the boundary between two neighboring clusters.
  • scikit-learn 1.9.1, Demonstration of k-means assumptions, example: there is no uniquely defined true number of clusters; it has to be decided from data-based criteria and knowledge of the intended goal.

Key terms

Related