MisleadingCharts
Back to the gallery

The 54% fall in avalanche deaths that followed the record, as falls usually do

Showing the misleading chart

Be the first to star this exhibit

A donor report we drew for an invented safety charity credits its new pledge with cutting US avalanche deaths from a record 37 to 17 in one winter. The counts are real. Six of the seven US seasons with 33 or more deaths were followed by a quieter winter, five of them with no pledge in sight, and the five winters since the record average 21.6 deaths against 24.4 for the ten before it.

01The claim

The pledge cut avalanche deaths by more than half. A donor report like this one, which we drew for an invented backcountry safety charity from the Colorado Avalanche Information Center’s national count, says that after a record 37 people died in US avalanches in 2020–21 the charity launched a beacon-check pledge, and the next winter the toll fell to 17, down 54%, on a zero-based chart with every death counted.

02The trick

Every bar is CAIC’s count and the axis starts at zero. The problem is when the program started. It launched because 2020–21 was the worst season on record, and a season that tops seven decades of records has usually had more than its share of whatever makes a winter bad: a weak snowpack, the timing of storms, who happened to be on which slope. The next winter is dealt a new hand, so it is likely to come in lower whether or not anything changed. That is regression to the mean, and the chart gives the reader nothing to measure the drop against: it starts in 2016–17, so none of the earlier peak seasons, or what followed them, are on the page. The rest of the record shows how ordinary the drop is. The US has had seven seasons with 33 or more avalanche deaths, and six of them were followed by fewer. The six that came before the pledge went from 34.8 deaths to 25.5 on average the next winter, a 27% fall with no pledge to explain it. The biggest drop belongs to 2013–14, 35 followed by 11, a 69% fall that no pledge was there to take credit for. The bounce works the other way too: the five quietest seasons of the last 35 were all followed by more deaths, and the pledge’s own 17 was followed by 30.

03The fix

Put the claimed drop next to the drops that happened without it. The redraw shows all 76 seasons since 1950–51, marks the seven with 33 or more deaths and the winter after each, and sets them side by side in a slope chart. 2020–21’s fall to 17 is the second-largest of the seven, behind 2013–14’s fall to 11, and the other five peaks were followed by anything from a 6% rise to a 31% fall. Then compare stretches of seasons on either side, leaving out the record itself. The five seasons since the pledge (17, 30, 16, 22, 23) average 21.6 deaths; the ten before the record averaged 24.4. Seasons vary so much (a standard deviation of about 7 deaths) that two averages this short differ by about 4 deaths by chance alone, so a 2.8-death gap is noise, and the counts can’t tell a real effect from luck. The post-pledge average even includes 2021–22, the very season that was due to drop back; without it the four later seasons average 22.75. None of this says rescue training doesn’t work. These counts can’t show it either way: they say nothing about how many people went into avalanche terrain each winter or how unstable the snow was. A fair test would compare a death rate per backcountry visitor, or riders who took the training against similar riders who didn’t. Whenever a program starts right after a bad year, expect the next year to look better on its own, and judge the program against that.