Framing & context

The missing base rate

Catches four in five. Wrong seven times in ten. Both are true.

a.k.a. base rate neglect · the false positive paradox · sensitivity without specificity · the half-shown confusion matrix · positive predictive value · detection rate without false alarms

Anything that looks for something — a test, an alarm, a screening rule, a model — produces four outcomes, not one: it fires and the thing is there, it fires and it isn’t, it stays quiet and the thing is there, it stays quiet and it isn’t. A detection rate is built from one column of that table, hits over hits plus misses, and it never counts a single false alarm. So it can be pushed to 100% by firing more often, which makes it a threshold setting rather than a score. How much that costs depends on a number the chart almost never carries: how rare the thing being looked for actually is. Hunt something rare and even a good detector’s alarms are mostly wrong, because there is so much more not-it to be wrong about.

How to spot it

One number where a detector has four outcomes: hits, misses, false alarms, correct all-clears.
Read the label. “Detection rate”, “sensitivity”, “accuracy”, “% caught”, “recall” — none of them counts a false alarm.
Ask how often the thing being detected actually happens. The rarer it is, the more of the alarms are wrong, however good the detector.
A hit rate with no volume beside it. How many times did it fire to earn that number?
Two metrics that never appear on the same page. A detection rate that improves in the same year a false-alarm rate worsens is one knob, not two results.

The fix

Show the whole table, or at least both sides of it: what share of the events it caught, and what share of its alarms were right. Neither number means much alone, because moving the threshold trades one for the other — only the pair says whether the detector got better or merely got louder. Put the base rate in the picture rather than the footnote: an icon array of a hundred alarms shows the hits against the false alarms instead of against nothing, and it takes one glance. Then answer the question the reader actually brought, which is rarely “what fraction does it catch” and almost always “when it goes off, how often is it right”. And when the honest answer is that most alarms are false, say so plainly and say why — for a rare event that is arithmetic, not negligence, and a reader who understands it will trust the next alarm more, not less.

In the gallery

Dark navy briefing slide, masthead “Warning Desk · Severe weather · United States, 1986–2011”, headlined “Four tornadoes in five now arrive with a warning”. A subhead reads: every tornado in the national record, twenty-six years of it, scored by one agency under one method — a tornado counts as detected if a warning covered it; in 1986, 33.9% did, and in 2011, 78.6% did; nothing sampled, nothing modelled, on an axis that runs from 0% to 100%. Under the label “Share of tornadoes covered by a warning · probability of detection · national record”, a gold line with a filled area beneath it rises across a zero-based axis marked 0%, 25%, 50%, 75% and 100%, with years 1986, 1990, 1995, 2000, 2005 and 2011 along the bottom. The line starts at a labelled 33.9% in 1986, dips to about 29% in 1987 and 1988, climbs through the mid-forties in the early nineties, jumps to 60% in 1995, reaches the mid-seventies by 2002 and then runs roughly flat, ending at a labelled 78.6% in 2011. A caption under the axis reads “share of all tornadoes in the national record, per year”. Three ticked boxes run along the bottom: “Every tornado in the record — the full national verification archive, 1986 to 2011, nothing sampled”, “One definition throughout — a tornado is detected if a warning covered it, before it began or while it was on the ground”, and “Zero-based axis — 0% to 100%, no truncation, no log scale, no second axis”. Down the right, a card carries 78.6% in large white type over the words “of tornadoes covered by a warning in 2011, against 33.9% in 1986”. A second card, “The record in three numbers”, gives 1986 the first year scored 33.9%, 2003 the high-water mark 79.1%, and 2011 the latest year shown 78.6%. A third, “Warned before it touched down”, gives 1986 at 26.2% and 2011 at 70.1%. A gold-bordered note at the bottom reads: “The detection rate has more than doubled in twenty-six years. Radar got better, and so did the people reading it.” Beneath it, in small grey type, sits the disclosure: “Detection rate only. Warning volume and false alarm ratio are published separately.” A footnote credits Brooks and Correia, “Long-Term Performance Metrics for National Weather Service Tornado Warnings”, Weather and Forecasting 33(6), 2018, Table 1, and calls the slide a Warning Desk demonstration drawn by us.

The detection rate that never counts a false alarm

The missing base rate The cherry-picked window