Cherry-picking

The size-biased count

A school of forty counts once. So does a school of four thousand.

a.k.a. the inspection paradox · length-biased sampling · size-biased sampling · the class-size paradox · the waiting-time paradox · the friendship paradox · unit of analysis · counting schools instead of students · sampling containers, not contents

Every count needs a unit, and a chart of containers gives each container one tally however much it holds. That is the right answer to some questions and the wrong answer to most of the questions people actually ask, because the reader is usually not a school, a class, a bus or a firm — the reader is inside one. Count the contents instead and each container is weighted by its own size, which drags the whole distribution to the right: the big ones, few enough to look like a tail, hold a share of the contents wildly out of proportion to their number. The two averages that result are not rivals and neither is wrong; they differ by exactly the variance divided by the mean, so they agree only when every unit is the same size and diverge fastest where the spread is widest. Nothing is excluded, no axis is bent and the arithmetic is right. The unit is simply not the one the sentence was about.

How to spot it

Ask what one tally is. A school, a class, a bus, a firm, a household, a hospital — then ask whether the claim is about those or about the people inside them.
A claim about typical experience made from a count of containers. “Most classes are small” and “most students are in small classes” are different sentences, and one chart cannot say both.
A long right tail of very large units. The tail is a sliver of the count and a slab of the population, and it is exactly the part a count of containers draws as negligible.
Two averages that ought to agree and don’t: the average class against the average student’s class, the average firm against the average employee’s firm. The gap is the spread, and it is often a factor of two.
Waiting time. If buses come every ten minutes on average, the average wait is more than five unless they run to the second, because a long gap is a bigger target to land in — which is the same arithmetic wearing a clock.
Surveys of institutions. One questionnaire per school, per practice, per company means the smallest respondent and the largest carry equal weight in every percentage the report prints.
Averages of averages. Averaging each store’s mean basket, each ward’s mean stay, each class’s mean score gives every unit one vote and is not the average customer, patient or pupil.
The friendship paradox and its relatives, where the sampling happens through the connections rather than the nodes: your friends have more friends than you do, on average, because a popular person appears on more lists.

The fix

Decide who the sentence is about, then count that. If the claim is about children, weight the bands by children; if it is about sites, weight them by sites — and say in the caption which unit the bars are, the way you would print a unit on an axis. Best of all is to draw both, because the pair is more informative than either and costs one extra series: the gap between the two distributions is itself the finding, and a reader who sees it will never mistake one for the other again. Print both means beside them, since the container mean and the contents-weighted mean are a matched pair and the distance between them is the spread in plain sight. Where only one can be shown, choose the unit the decision runs on: a staffing model and a maintenance budget really are about sites, and a class-size promise really is about children, so the same dataset honestly supports two different charts for two different meetings. Treat a long right tail as a warning that the two answers will be far apart, and be especially careful with an average of averages, which quietly picks the container as its unit while looking like an average of everything. And keep it apart from its neighbours, because the remedies differ: survivorship bias is about cases that left the dataset and self-selection about cases that walked in, while here nothing is missing at all — every unit is present in both counts, and the selection happens at the moment of counting, in the choice of what one tally represents. The nearest miss is the missing denominator, and the difference is worth holding on to: that is a count that should have been a rate, whereas here both figures are already rates and the whole argument is over what they are rates of. It sits under cherry-picking for the same reason, since choosing the unit is choosing the population the chart describes — which is a selection, even though it deletes nobody.

In the gallery

Dark navy briefing slide, masthead “Facilities briefing · US public schools · Enrollment size of school · School year 2021–22 · Census · not a sample”, headlined “Six in ten American public schools enroll fewer than 500 students”. A standfirst says this is a census rather than a survey — every public elementary and secondary school in the fifty states and the District of Columbia files to the federal Common Core of Data, 99,239 of them in 2021–22 — and that the publisher’s size distribution runs over the 95,839 that reported an enrollment above zero, the other 3,400 having reported none, those 95,839 schools holding 49,089,640 students between them. It adds that the axis starts at zero, every band is plotted at the publisher’s own boundaries, nothing is smoothed, indexed, annualized or rebased and there is no second axis, and that on that count the commonest school in the country holds 300 to 399 students, 61.5% of schools enroll fewer than 500, 32.0% fewer than 300 and very nearly one school in ten — 9.96% — fewer than 100, while only 9.0% reach 1,000 students and 0.34% reach 3,000. The main panel, labelled “Share of public schools, by enrollment size · 2021–22 · axis 0 to 16% · every school counted once”, is a column chart of thirteen blue bars with gridlines at 0, 4, 8, 12 and 16 percent, each bar labelled with its value: Under 100 at 10.0, 100–199 at 9.4, 200–299 at 12.6, 300–399 at 15.3 picked out in gold as the tallest, 400–499 at 14.2, 500–599 at 10.9, 600–699 at 7.7, 700–799 at 5.2, 800–999 at 5.7, 1,000–1,499 at 5.1, 1,500–1,999 at 2.0, 2,000–2,999 at 1.5 and 3,000+ at 0.3. The x-axis is captioned “Enrollment size of school · percent of the 95,839 reporting schools in each band”. A table headed “The distribution as drawn” lists every band with its exact percentage of schools and the school count that percentage implies: 9.96 and 9,546; 9.41 and 9,018; 12.62 and 12,095; 15.35 and 14,711; 14.15 and 13,561; 10.88 and 10,427; 7.71 and 7,389; 5.18 and 4,964; 5.73 and 5,492; 5.11 and 4,897; 2.02 and 1,936; 1.53 and 1,466; 0.34 and 326. A second table, “Small schools dominate the count”, gives cumulative shares — fewer than 100 at 9.96%, fewer than 200 at 19.37%, fewer than 300 at 31.99%, fewer than 500 at 61.49%, fewer than 800 at 85.26%, 1,000 or more at 9.00%, 2,000 or more at 1.87% and 3,000 or more at 0.34% — and notes beneath that the median school, the fiftieth of every hundred counted from the smallest, falls in the 400–499 band, and that mean enrollment per reporting school is 512.2 students. Down the right-hand side a card carries the figure “61.5%” over “Public schools enrolling fewer than 500 · 2021–22” and “Modal band 300–399 · mean 512.2 students”. A card headed “Read-out for the facilities board” says America is a nation of small schools, that the median school sits in the 400–499 band and the average school holds 512.2 students, so a site of roughly 500 is the country’s normal case rather than its small case. A “Planning note” card says a staffing model, a counselling ratio and a per-site technology package specified for a school of about 500 will fit the majority of American sites without modification, and that large-campus provision is the exception since fewer than one site in ten reaches 1,000 students. A card headed “Why a census, and what its base is” says these are not estimates, that every public school files to the Common Core of Data so there is no sample and no margin of error and the percentages are counts, and that the publisher computes them over the 95,839 schools reporting enrollment above zero while the 3,400 reporting none are outside the distribution. A “What we did not do” card lists a zero-based axis, every band plotted at the publisher’s own boundaries, nothing smoothed, indexed, annualized, projected or rebased, no second axis, no log scale and no break, and “one year, one country, one unit”. A “Basis of measurement” card names NCES, Digest of Education Statistics 2022, Table 216.40, with underlying data from the NCES Common Core of Data, 2021–22. Five ticked boxes run along the foot: “Zero-based axis”, “Every reporting school counted”, “Publisher’s own size bands”, “Census, not a sample” and “Base stated”. A footnote gives the source, notes that percentages are of the 95,839 schools out of 99,239 reporting enrollment above zero and sum to 99.99 through rounding, that the school counts in the table are those percentages applied to that base, and says the slide is a demonstration drawn by us in the manner of a facilities briefing, that the board, the read-out and the planning note are invented, and that every figure behind it is real.

The 61.5% of schools that hold 32.7% of the children

The size-biased count Uneven intervals