Beyond sum and mean

Grouping and Aggregations covered the basics: split rows into groups, then collapse each group down to one number with .sum().sum() or .agg().agg(). This page covers three things aggagg can’t do on its own:

• transformtransform — compute a per-group statistic, but keep the original number of rows (broadcast the group’s answer back onto every row in that group).
• filterfilter — keep or drop entire groups based on a condition, instead of individual rows.
• Fine control over aggagg: naming your output columns, and bucketing continuous values with qcutqcut before grouping.

transform: broadcast a group stat back onto every row

groupby(...).mean()groupby(...).mean() collapses rows down to one row per group. transform("mean")transform("mean") computes the same group means, but hands back a Series the same length as your original data — perfect for adding a “how does this row compare to its group?” column.

import pandas as pd
 
sales = pd.DataFrame({
    "city": ["pune", "pune", "delhi", "delhi", "delhi"],
    "amount": [100, 200, 150, 120, 300],
})
 
sales["city_avg"] = sales.groupby("city")["amount"].transform("mean")
print(sales)

import pandas as pd
 
sales = pd.DataFrame({
    "city": ["pune", "pune", "delhi", "delhi", "delhi"],
    "amount": [100, 200, 150, 120, 300],
})
 
sales["city_avg"] = sales.groupby("city")["amount"].transform("mean")
print(sales)

Every punepune row now shows 150.0150.0 (the average of 100 and 200); every delhidelhi row shows 190.0190.0. This is the “unwrapped” groupby McKinney describes — you can now do plain arithmetic like sales["amount"] - sales["city_avg"]sales["amount"] - sales["city_avg"] without a manual merge.

Filling missing values with the group mean

A common real use of transformtransform: instead of filling every missing value with one global average, fill it with that row’s own group’s average.

import numpy as np
 
data = pd.Series(
    [10.0, np.nan, 30.0, np.nan, 100.0, 200.0],
    index=["a1", "a2", "a3", "b1", "b2", "b3"],
)
group_key = ["a", "a", "a", "b", "b", "b"]
 
filled = data.groupby(group_key).transform(lambda x: x.fillna(x.mean()))
print(filled)

import numpy as np
 
data = pd.Series(
    [10.0, np.nan, 30.0, np.nan, 100.0, 200.0],
    index=["a1", "a2", "a3", "b1", "b2", "b3"],
)
group_key = ["a", "a", "a", "b", "b", "b"]
 
filled = data.groupby(group_key).transform(lambda x: x.fillna(x.mean()))
print(filled)

a2a2 was missing, so it gets group "a""a"’s mean of its other values ((10 + 30) / 2 = 20.0(10 + 30) / 2 = 20.0) — not the mean of the whole Series.

Group-weighted average with apply

Some calculations combine two columns within a group — like a weighted average, where each row contributes according to its own weight. That needs applyapply, since it operates on the whole group at once rather than one column transformed independently:

df = pd.DataFrame({
    "category": ["a", "a", "a", "b", "b"],
    "score": [80, 90, 70, 60, 95],
    "weight": [1, 2, 1, 3, 1],
})
 
def weighted_avg(group):
    return np.average(group["score"], weights=group["weight"])
 
result = df.groupby("category").apply(weighted_avg)
print(result)

df = pd.DataFrame({
    "category": ["a", "a", "a", "b", "b"],
    "score": [80, 90, 70, 60, 95],
    "weight": [1, 2, 1, 3, 1],
})
 
def weighted_avg(group):
    return np.average(group["score"], weights=group["weight"])
 
result = df.groupby("category").apply(weighted_avg)
print(result)

filter: keep or drop whole groups

transformtransform and applyapply change or summarize values within a group. filterfilter decides whether an entire group stays in the result at all — pass a function that returns TrueTrue/FalseFalse for a group, and only groups where it’s TrueTrue survive:

orders = pd.DataFrame({
    "key": ["a", "a", "b", "b", "b"],
    "value": [1, 2, 3, 4, 5],
})
 
kept = orders.groupby("key").filter(lambda g: g["value"].sum() > 8)
print(kept)
print(len(kept))

orders = pd.DataFrame({
    "key": ["a", "a", "b", "b", "b"],
    "value": [1, 2, 3, 4, 5],
})
 
kept = orders.groupby("key").filter(lambda g: g["value"].sum() > 8)
print(kept)
print(len(kept))

Group "a""a" sums to 33 (dropped); group "b""b" sums to 1212 (kept) — so only "b""b"’s rows remain, and the DataFrame keeps its original columns and row labels, unlike aggagg which would collapse each group to one row.

Named aggregation: control your output column names

Passing plain function names to aggagg on multiple columns gives you generic labels like "amount""amount" repeated under a hierarchy. Named aggregation — passing new_col=("source_col", "func")new_col=("source_col", "func") keyword arguments — lets you choose the output column names directly:

orders = pd.DataFrame({
    "city": ["pune", "pune", "delhi", "delhi"],
    "amount": [100, 200, 150, 300],
})
 
summary = orders.groupby("city").agg(
    total=("amount", "sum"),
    orders=("amount", "count"),
)
print(summary)

orders = pd.DataFrame({
    "city": ["pune", "pune", "delhi", "delhi"],
    "amount": [100, 200, 150, 300],
})
 
summary = orders.groupby("city").agg(
    total=("amount", "sum"),
    orders=("amount", "count"),
)
print(summary)

Bucket analysis: qcut + groupby

pandas.qcutpandas.qcut slices a continuous column into roughly equal-sized buckets by quantile. Feeding that result straight into groupbygroupby gives you quick “which bucket performs best?” answers:

scores = pd.DataFrame({
    "student": list("ABCDEFGH"),
    "score": [55, 60, 65, 70, 75, 80, 85, 90],
})
 
scores["bucket"] = pd.qcut(scores["score"], 4, labels=["Q1", "Q2", "Q3", "Q4"])
print(scores.groupby("bucket")["score"].mean())

scores = pd.DataFrame({
    "student": list("ABCDEFGH"),
    "score": [55, 60, 65, 70, 75, 80, 85, 90],
})
 
scores["bucket"] = pd.qcut(scores["score"], 4, labels=["Q1", "Q2", "Q3", "Q4"])
print(scores.groupby("bucket")["score"].mean())

Common pitfalls

• transformtransform must return either a single scalar (broadcast) or a Series the exact same length as the input group — anything else raises an error.
• filterfilter’s function must return one TrueTrue/FalseFalse per group, not per row — testing g["value"] > 8g["value"] > 8 (no .sum().sum()) would compare row-by-row and error out.
• Named aggregation only accepts new_name=("column", "func")new_name=("column", "func") tuples — you can’t mix it with the older {"col": ["sum", "mean"]}{"col": ["sum", "mean"]} dict style in the same call.

Visualize it

  flowchart LR
  G["groupby('key')"] --> A["agg()
one row per group"]
  G --> T["transform()
same row count, group stat broadcast back"]
  G --> F["filter()
whole groups kept or dropped"]

🧪 Try It Yourself

Exercise 1 – Broadcast a Group Mean With transform

Exercise 2 – Fill NaN With the Group Mean

Exercise 3 – Drop Whole Groups With filter

Next

With grouping mastered in depth, continue to Cross-Tabulation and Pivot Table Depth to summarize two or more categorical columns at once.