The detection rate that never counts a false alarm

The missing base rate The cherry-picked window

Dark navy briefing slide, masthead “Warning Desk · Severe weather · United States, 1986–2011”, headlined “Four tornadoes in five now arrive with a warning”. A subhead reads: every tornado in the national record, twenty-six years of it, scored by one agency under one method — a tornado counts as detected if a warning covered it; in 1986, 33.9% did, and in 2011, 78.6% did; nothing sampled, nothing modelled, on an axis that runs from 0% to 100%. Under the label “Share of tornadoes covered by a warning · probability of detection · national record”, a gold line with a filled area beneath it rises across a zero-based axis marked 0%, 25%, 50%, 75% and 100%, with years 1986, 1990, 1995, 2000, 2005 and 2011 along the bottom. The line starts at a labelled 33.9% in 1986, dips to about 29% in 1987 and 1988, climbs through the mid-forties in the early nineties, jumps to 60% in 1995, reaches the mid-seventies by 2002 and then runs roughly flat, ending at a labelled 78.6% in 2011. A caption under the axis reads “share of all tornadoes in the national record, per year”. Three ticked boxes run along the bottom: “Every tornado in the record — the full national verification archive, 1986 to 2011, nothing sampled”, “One definition throughout — a tornado is detected if a warning covered it, before it began or while it was on the ground”, and “Zero-based axis — 0% to 100%, no truncation, no log scale, no second axis”. Down the right, a card carries 78.6% in large white type over the words “of tornadoes covered by a warning in 2011, against 33.9% in 1986”. A second card, “The record in three numbers”, gives 1986 the first year scored 33.9%, 2003 the high-water mark 79.1%, and 2011 the latest year shown 78.6%. A third, “Warned before it touched down”, gives 1986 at 26.2% and 2011 at 70.1%. A gold-bordered note at the bottom reads: “The detection rate has more than doubled in twenty-six years. Radar got better, and so did the people reading it.” Beneath it, in small grey type, sits the disclosure: “Detection rate only. Warning volume and false alarm ratio are published separately.” A footnote credits Brooks and Correia, “Long-Term Performance Metrics for National Weather Service Tornado Warnings”, Weather and Forecasting 33(6), 2018, Table 1, and calls the slide a Warning Desk demonstration drawn by us. — Showing the misleading chart

The same record on a cream background, kicked “The same detection rate · with the other three cells put back” and headlined “79% of tornadoes caught. Seven warnings in ten with no tornado.” A subhead explains that both numbers describe the same warnings and neither is in dispute: a detection rate counts tornadoes — of all the tornadoes that happened, what share had a warning over them — and never counts a warning, while the false alarm ratio counts warnings, and the National Weather Service’s own performance target for that number is 71% or lower, a target it has met in 9 of the last 12 years; roughly 2,900 tornado warnings go out in a typical year against something like 1,200 to 1,800 tornadoes. On the left, an icon array headed “100 tornado warnings · at the official ceiling” shows a hundred dots in ten rows of ten, the first 29 in gold and the remaining 71 pale, with a legend reading “29 verified — a tornado followed” and “71 not verified — at the official ceiling of 71%”. Beside it, a table headed “Every warning decision has four outcomes” crosses “Warning issued” and “No warning” against “Tornado occurred” and “No tornado”, giving four cells: Hit, counted by both metrics; False alarm, counted by the false alarm ratio only; Miss, counted by the detection rate only; and Correct all-clear, counted by neither. The tornado-occurred column is shaded. A note beneath reads that the detection rate is built from the shaded column, hits over hits plus misses, that nothing in it counts a false alarm, so any detector can drive it to 100% by warning more often and pay for it in the column the chart does not show — which is why the two numbers move together: they are not two results, they are one threshold read from two sides. Below, a chart headed “The same detection rate, five years further on · 1986–2016” redraws the record with the years the first chart showed tinted and marked “the window the first chart showed”, and a dashed line at 2011. A gold line, “Covered by a warning, before or during the tornado”, peaks at a labelled 78.6% in 2011 and falls away to a labelled 64.2% in 2016; a blue line, “Warned in advance, before it touched down”, peaks near 70% and falls to a labelled 50.2%. A note beside them reads: both fall after 2011, when the threshold for issuing a warning was raised to bring false alarms down — one knob, two readings. Five cards run down the right. “The knob, not the score” explains that after the spring of 2011 forecasters raised the bar for issuing a warning to cut false alarms, and the detection rate went with it, 78.6% in 2011 to 64.2% in 2016 and the share warned in advance from 70.1% to 50.2%, with nothing about the radar having got worse. “The base rate does the rest” notes that warnings are issued on rotation, and that of 5,322 radar mesocyclone detections meeting the algorithm’s objective criteria, 26% were associated with a tornado — 15% of mid-altitude detections, more than 40% of low-altitude ones — so most rotating storms never produce one, and a forecaster who waited for certainty would warn on almost nothing. “Three numbers, published together” gives the agency’s three official targets, detection at or above 72%, lead time at or above 13 minutes and false alarms at or below 71%, reported side by side every year because any one alone is a setting rather than a score. “The question a reader actually has” contrasts “what share does it catch”, a fact about tornadoes, with “when it goes off, how often is it right”, a fact about warnings that needs the base rate to answer. “The check that catches it” says: find the detector’s two-by-two table and ask which cells the chart used, because one number is never a scorecard for something that can be wrong in two directions. A footnote credits Brooks and Correia (2018) for the detection rates, the Department of Commerce Office of Inspector General report OIG-25-026-I of July 2025 for the official definitions and targets, the NOAA tornado-warning indicator on the Commerce performance portal for warnings per year, Trapp, Stumpf and Manross (2005) for the mesocyclone figures and NOAA for tornado counts, notes that the 29/71 array is drawn at the official ceiling rather than a single year’s reported figure, and says both drawings are our own. — Showing the misleading chart

Be the first to star this exhibit

The share of American tornadoes that arrived with a warning over them climbed from 33.9% in 1986 to 78.6% in 2011 — every tornado in the national record, one agency, one method, a zero-based axis. The number is real. It is also one column of a four-cell table: nothing in a detection rate counts a warning that came to nothing, and roughly seven tornado warnings in ten do. The National Weather Service’s own performance target for that second number is 71% or lower.

Detection rates from Harold E. Brooks and James Correia Jr., “Long-Term Performance Metrics for National Weather Service Tornado Warnings”, Weather and Forecasting 33(6), 2018, Table 1; official metric definitions, targets and the record against them from the U.S. Department of Commerce Office of Inspector General, “Independent Evaluation of NOAA’s National Weather Service Tornado Forecasting and Warning Services”, OIG-25-026-I, July 2025; mesocyclone figures from Trapp, Stumpf and Manross, “A Reassessment of the Percentage of Tornadic Mesocyclones”, Weather and Forecasting 20(4), 2005, 2018Added August 17, 2026

01The claim

Four tornadoes in five now arrive with a warning. Every tornado in the national record, twenty-six years of it, scored by one agency under one method: a tornado counts as detected if a warning covered it, before it began or while it was on the ground. In 1986, 33.9% did. By 1995 it was 60%, by 2003 a high-water mark of 79.1%, and in 2011, 78.6%. The share warned before touchdown went from 26.2% to 70.1% over the same span. Nothing sampled, nothing modelled, on an axis that runs from 0% to 100% with no truncation and no second scale. The detection rate has more than doubled in twenty-six years — radar got better, and so did the people reading it.

02The trick

Every figure is right, the axis is honest, and the improvement is real. The catch is what a detection rate is made of. A warning decision has four outcomes, not two: the warning fires and a tornado follows (a hit), it fires and none does (a false alarm), it stays quiet and a tornado comes anyway (a miss), or it stays quiet and nothing happens (a correct all-clear). The detection rate is hits over hits plus misses — one column of that table. It counts tornadoes, never warnings, so no warning that came to nothing can move it by a single point. Which means it is not really a score at all: any detector can push it toward 100% by firing more often, and pay for that entirely in the column the chart does not show. Here the price is steep, and the reason is the base rate. Warnings are issued on rotation seen by radar, and rotation is far commoner than tornadoes — of 5,322 mesocyclone detections meeting the detection algorithm’s objective criteria, 26% were associated with a tornado, 15% of the mid-altitude ones and more than 40% of the low-altitude ones. About 2,900 tornado warnings go out in a typical year against something like 1,200 to 1,800 tornadoes, and roughly seven warnings in ten are not followed by a verified tornado. That second number is published too: the agency’s own performance target for the false alarm ratio is 71% or lower, alongside targets of 72% for detection and 13 minutes for lead time, and it has met the false-alarm target in 9 of the last 12 years. The chart’s other quiet choice is where it stops. The record does not end in 2011. After the spring of that year forecasters raised the bar for issuing a warning, precisely to bring false alarms down, and the detection line went with it: 78.6% in 2011 to 58.7% in 2014 and 64.2% in 2016, with the share warned in advance falling from 70.1% to 50.2%. Cropping at 2011 hides the one thing that proves the point. (This exhibit is our own demonstration, drawn in the style of an agency performance slide — the published record carries all three metrics together, which is exactly why the trick needs a chart to pull it off.)

03The fix

Put the other three cells back. Draw the two-by-two table and mark which cells each metric uses: the detection rate takes the tornado-occurred column, the false alarm ratio takes the warning-issued row, and one of them alone tells you where the threshold was set rather than how good the detector is. Then show the base rate as a picture instead of a footnote. A hundred dots for a hundred warnings — 29 gold for the ones a tornado followed, 71 pale for the ones it did not, drawn at the official ceiling — answers in one glance the question a detection rate cannot touch: when this thing goes off, how often is it right? Then run the line to the end of the record. Both series fall after 2011 and they fall together, which is the finding: the false alarm ratio and the detection rate are one knob read from two sides, and an agency that pushes one down will watch the other follow. None of this is a mark against the forecasters. A tornado warning is a decision made in minutes on ambiguous radar about a rare event, and when the thing you are hunting is rare, most of your alarms being wrong is arithmetic rather than negligence — which is why the National Weather Service publishes detection, lead time and false alarms side by side every year, and why a reader who understands the trade will trust the next warning more, not less. The habit transfers to any detector you meet, from a screening test to a fraud model: find its four outcomes, ask which ones the chart counted, and if only one number is on offer, ask how often it fired to earn it.