Five states fail the Benford screen, and chance alone fails two or three

Light grey data-review slide, kicker “Data-integrity review · Census county population file · Benford first-digit screen”, headlined “Five states’ county figures fail Benford’s law — hold them out until reviewed”. A subhead says honest counts should start with a 1 about 30% of the time and with a 9 under 5%, that every state’s county populations (Vintage 2024 estimates) went through a chi-square test against Benford’s law, that five states fail at p < 0.05, that Iowa’s counties start with a 1 nearly half the time and that Florida has no county starting with a 5. A badge reads “Demonstration · drawn by us from public data”. Five white panels, each with a red “FAIL” tag, show red bars for the share of a state’s counties starting with each digit 1 to 9 against a dashed black Benford curve on a 0–50% axis: Iowa, 99 counties, p = 0.0017, a 47% bar at digit 1 against Benford’s 30%; Nevada, 17 counties, p = 0.020, tall bars at 4 and 5 and none at 3; Colorado, 64 counties, p = 0.033, a bump at 5 and 7; Florida, 67 counties, p = 0.045, a gap at 5; Georgia, 159 counties, p = 0.047, a high bar at 2. A sixth panel explains how to read them. On the right a big red “5 states fail” says Iowa is worst, 47 of 99 counties starting with a 1 against 30 expected; a read-out says Benford’s law is the forensic accountant’s first screen, that fabricated figures rarely reproduce its curve, and that the five states cover 406 counties; a recommendation says to exclude Iowa, Nevada, Colorado, Florida and Georgia from the county model until their figures are reconciled. The footer says the test is a chi-square goodness-of-fit on first digits with p-values by simulation, that states that passed are not shown, that the source is the US Census Bureau’s Vintage 2024 county population totals, and that the slide is a demonstration drawn by us in which no review flagged these states. — Showing the misleading chart

Cream-background page headlined “Run Benford’s test fifty times and two or three states fail by luck”. A subhead says a test at p < 0.05 rejects one in twenty datasets whose digits truly follow Benford’s curve, so 50 states expect about 2.5 failures by chance alone, and that the file as a whole looks like Benford, all 3,143 counties pooled at p = 0.66. A grid of fifty small panels, ten across and five down, one per state and ordered by p, shows grey bars for the share of counties starting with each digit 1 to 9, on a common 0–50% scale, under a thin black Benford curve, each labelled with the state, its county count and its p-value. The first five, Iowa, Nevada, Colorado, Florida and Georgia, are shaded and drawn in ochre; the next, Nebraska at p = 0.062 and Indiana at 0.091, are unshaded, and the grid runs down to North Carolina at p = 0.988. Small states such as Delaware (3 counties), Hawaii and Rhode Island (5 each) have ragged bars that still pass. A side panel, “Deal the counties into pretend states”, says that shuffling all 3,143 counties into 50 random states of the same sizes and screening again flags 2.3 states on average across 2,000 deals, that 91% of deals flag at least one and 7.4% flag five or more, over a histogram of flagged states per deal peaking at 2 (26.6%), with the bar at 5 (4.9%) in ochre. A second panel says no state clears a Bonferroni correction for 50 tests, a bar of 0.001 against Iowa’s 0.0017, and that Iowa, the most extreme, has a mundane cause on view: 41 of its 99 counties hold 10,000 to 19,999 people, and since Benford’s law needs values spread evenly across orders of magnitude, a cluster of similar-sized rural counties puts its first digits where the cluster sits, on the 1s; a failed screen is a reason to look, not a finding. The footer gives the method, notes that the District of Columbia is excluded and Connecticut is counted as its nine planning regions, and credits the US Census Bureau. — Showing the misleading chart

Be the first to star this exhibit

A data-integrity slide we drew from the Census Bureau’s county population estimates: five states’ county figures fail a Benford’s-law test at p < 0.05, and the slide recommends holding them out. It ran the test fifty times, once per state, and showed only the failures. By chance alone, fifty states whose digits truly follow Benford’s curve would fail about 2.5 times, and the counties pooled fit that curve with p = 0.66.

US Census Bureau, Population Estimates Program — county population totals, Vintage 2024 (co-est2024-alldata.csv), July 1, 2024 estimates, 2025Added October 27, 2026

01The claim

Five states’ county population figures fail Benford’s law and should be held out until someone has checked them. A data-integrity slide of the kind an analyst might show before building on a public file, which we drew from real Census Bureau estimates: every state’s county populations go through a chi-square test of their first digits against Benford’s curve, and five fail at p < 0.05. Iowa is worst at p = 0.0017, with 47 of its 99 counties starting with a 1 where Benford’s law expects 30. Nevada has no county starting with a 3, Florida none starting with a 5. The recommendation is to exclude Iowa, Nevada, Colorado, Florida and Georgia from the model.

02The trick

Every figure on the slide is real and every p-value is right for its own state. What the slide leaves out is the number of tests: it ran the same test fifty times, once per state, and showed only the five that failed. At p < 0.05, fifty states whose digits truly follow Benford’s curve would still produce about 2.5 failures. We checked that directly: shuffling all 3,143 counties at random into fifty pretend states of the same sizes and screening again flags 2.3 states on average over 2,000 deals. Nine deals in ten flag at least one state, and 7.4% flag five or more. Benford’s name does the rest of the work. The law is famous as a fraud detector, so “fails Benford” reads as “looks made up”. The test only says the digits are unlikely if they follow Benford’s curve, and that assumption, that honest county populations follow it, is itself only roughly true. Small states make it look worse than it is. Nevada has 17 counties, so a single county moves a bar by six points, and a ragged histogram is what a small honest sample looks like. On that evidence the slide proposes dropping 406 counties from a model.

03The fix

Show all fifty. The redraw puts every state in one grid of small multiples on a common scale, ordered by p, the passes beside the failures. The five flagged states look much like Nebraska (p = 0.062) and Indiana (0.091), which passed. For context, the file as a whole looks like Benford: all 3,143 counties pooled give a chi-square of 5.9 on 8 degrees of freedom, p = 0.66, though a good pooled fit can’t clear any one state on its own. None of the five survives a correction for fifty tests. Bonferroni’s bar is 0.05/50 = 0.001, and Benjamini–Hochberg flags nothing either; with fifty tests, a smallest p as low as Iowa’s 0.0017 turns up about 8% of the time by chance. Iowa, the most extreme, also has a mundane cause on view, and it has nothing to do with tampering. 41 of its 99 counties hold between 10,000 and 19,999 people (median about 15,000). Benford’s law needs values spread evenly across orders of magnitude, so a cluster of similar-sized rural counties puts its first digits wherever the cluster sits, and Iowa’s sits on the 1s. A Benford screen across many groups is a way to choose where to look first. Say how many groups were screened, set the threshold for that count, show the passes with the failures, and check whether a group’s numbers are spread widely enough for Benford’s law to apply before reading a failure as anything more.