Cherry-picking

Regression to the mean

Pick the extremes and they improve without your help.

a.k.a. the regression fallacy · selecting on an extreme · the sophomore slump

Almost any measurement is a lasting tendency plus a bit of luck. Sort a group by that measurement and you sort partly on the luck — so the group you picked is more extreme than it really is, and its next reading drifts back toward the average all by itself. Chart the drift and it looks like a treatment working, a trend arriving, or a curse falling.

How to spot it

The group was chosen for being top or bottom of a first measurement.
“Most improved” and “biggest fallers” lists whose names never repeat.
An effect with no control group picked the same way as the treated one.
Run the grouping backwards: sort on the second measurement and look at the first. If it converges in that direction too, nothing is moving.

The fix

Choose the group on something other than the number you are about to compare it against — an earlier period, a random draw, a control selected identically. Show the whole distribution rather than the fate of the tails, and expect a slice picked at its extreme to move toward the middle whether or not anything happened to it.

In the gallery

Dark navy slide, masthead “The Stature Register · 205 families · 928 adult children”, headlined “The extremes are being bred back to the middle”. A subhead reads “Every mid-parent height class in the 1886 hereditary-stature survey, joined to the median height of its own adult children. One shared scale in inches, every class shown, all 928 children counted.” A slope chart has two vertical columns: mid-parent height on the left, the median height of their adult children on the right. Nine gold and blue lines run between them: 72.5 to 72.2 (19 children), 71.5 to 69.9 (43), 70.5 to 69.5 (68), 69.5 to 68.9 (183), 68.5 to 68.2 (219), 67.5 to 67.6 (211), 66.5 to 67.2 (78), 65.5 to 66.7 (66) and 64.5 to 65.8 (23). Gold lines mark families above the middle and blue ones families below; a dashed rule crosses the chart at the level of mediocrity, 68¼ inches, and every line leans toward it. Three stat cards down the right side read −1.6 in given back in one generation by the 71½-inch families, 8.0 → 6.4 in span of the nine height classes parents then children, and ⅔ of its parents’ distance from the middle is all a generation keeps. A gold note box adds “Run the two-thirds rule forward. A family sitting 4¼ inches above the middle today is 0.6 inches above it in five generations. The tall are not being replaced — they are being averaged away.” A footnote credits Francis Galton, “Regression towards Mediocrity in Hereditary Stature”, Journal of the Anthropological Institute 15 (1886), Table I, row medians exactly as published, all female heights multiplied by 1.08.

The convergence that runs just as well backwards

Regression to the mean