Cherry-picking

The self-selected sample

Ten thousand replies, every one of them from somebody who felt like replying.

a.k.a. voluntary response bias · self-selection bias · the opt-in poll · the write-in survey · convenience sample · nonresponse bias · the feedback form

Some samples are drawn and some samples walk in. When the respondents chose themselves — wrote in, clicked through, left a review, answered the pop-up — what you have is not a slice of the population but a slice of the people moved to reply, and what moves a person to reply is usually the very thing being measured. That makes the bias directional and unbounded, and it means more data cannot fix it: ten thousand volunteers is ten thousand draws from the wrong population. It is the near neighbour of survivorship bias without being the same thing — survivorship is about the cases that dropped out before you looked, self-selection about the ones that stepped forward.

How to spot it

Ask what a person had to do to appear in this dataset, then ask whether doing it correlates with the answer.
A big n offered as the credential. Sample size buys precision, which is a different thing from being right.
“Respondents”, “readers who wrote in”, “users who rated”, “those who completed the survey” — every one of those is a volunteer.
The response rate is missing. Not the sample size: the response rate — what share of the people approached actually answered.
A margin of error printed next to a figure nobody sampled at random. The formula assumes the draw; it cannot check that one happened.
A result far more positive or negative than any random sample of the same population. Feedback arrives from the delighted and the furious, rarely from the fine.

The fix

Draw the sample instead of letting it assemble itself: pick who to ask, then chase the ones who do not answer, because the non-responders are the sample. Where that is impossible — and for most product feedback it is — label the chart with the population it actually describes (“of the 3% who rated the app”), publish the response rate beside every figure, and never put an opt-in number on the same axis as a drawn one without saying which is which. Anchor it when you can: run a small random survey alongside the big self-selected one and report both, since a few hundred people drawn at random will beat ten thousand volunteers, and the gap between the two is the most useful number you will get all year. And treat the direction as knowable even when the size is not — ask who was moved to reply and you can usually say which way the figure leans before you have seen it.

In the gallery

Dark navy briefing slide, masthead “The Mailbag Desk · National Parenthood Survey · 1976”, headlined “Seven in ten parents would not do it again”. A subhead reads: More than 10,000 readers answered a single question — “If you had it to do over again, would you have children?” Every reply was counted. Nothing was weighted, adjusted or estimated. Under the label “Share of responding parents · by answer given”, two columns stand on a zero-based axis marked 0%, 25%, 50%, 75% and 100%: a tall gold column at 70% labelled NO, “would not have children if they had it to do over”, and a shorter blue column at 30% labelled YES, “would have children again”. A short white whisker sits on top of each column, annotated “± 0.9 pts”. A strip beneath the axis reads “Replies counted — more than 10,000 — seven times the sample of a national opinion poll”. Down the right, a card carries 70% in large white type over the words “of parents say they would not have children if they had it to do over again”. A second card, “The sample, in numbers”, lists replies received 10,000+, margin of error ± 0.9 pts, confidence 19 times in 20, replies discarded none. A gold-bordered note reads “The arithmetic leaves almost no room for doubt. At ten thousand responses the interval is under a single point,” with a small grey disclosure beneath it: “Respondents were readers who chose to write in.” Three ticked boxes run along the bottom: “Full axis — 0% to 100%, no truncation, no log scale, no second axis”, “Every reply counted — no weighting, no adjustment, no estimation, nothing set aside”, and “Uncertainty shown — the 95% interval is drawn on both bars; at this sample size it is thinner than the line”. A footnote credits Ann Landers’ syndicated column of 23 January 1976 and her account of the survey for Good Housekeeping, June 1976, notes that the margin of error is the textbook 95% interval for a simple random sample of n = 10,000 at p = 0.30 computed by us, and calls the slide a Mailbag Desk demonstration drawn by us.

The ±1-point poll that missed by 61

The self-selected sample