Journalists and statistics

In recent days there has been much play over the fact that CNN, the Associated Press, NBC and other news organisations swallowed a statistically worthless report which said that underage drinkers drank 25% of all the alcohol consumed in the US. It turns out that the true figure is closer to 11%, and that the 25% figure, distributed by the National Center on Addiction and Substance Abuse at Columbia University, was the result of not adjusting for an oversampling of teenagers in the National Household Drug Survey. (The Center's retraction of its original report makes it seem just like a surly teenager being forced to apologise for something he doesn't think he ought to apologise for: "In its report CASA estimates that underage drinkers consume 25 percent of the alcohol sold in the United States... If the over sample is adjusted to reflect the population, the percentage is 11.4 percent... Nevertheless, CASA's estimate is that underage drinkers consume 25 percent of the alcohol consumed in the U.S.")

I don't blame the news outlets for buying the 25% figure. It came from a highly reputable university, and it's within the bounds of possibility: teenagers account for 20% of the population, and there's no doubt that 18-20 year-olds do a lot of drinking. If the story is well sourced, busy journalists can't be blamed for not checking the statistical basis for every finding. Michael Kinsley says that the whole story demonstrates America's "national innumeracy"; I think it doesn't, but that other stories do.

The real problem is not the innumeracy of the nation in general, but rather of journalists specifically. Journalists are generally the sort of people who never really liked mathematics, stopped studying it as soon as their schools allowed them to, and like to rely on experts whenever numbers find their way into their stories. It's the old division between the arts and the sciences: journalists are writers, they shouldn't be expected to understand numbers.

Fact is, the experts upon whom journalists typically rely often don't have a strong grasp on their own numbers. I used to write up the weekly league tables for Bridge News: the obsessed-over rankings of which banks had underwritten the most bond deals. Pretty much every week, after receiving the fax from the people who drew up the numbers, I would find some problem with them, phone them up, and eventually get a corrected version a couple of hours later. No one else, as far as I knew, approached the league tables with any sort of critical eye: they just ran whatever numbers they were given.

But never mind statistical snafus: some stories in highly-respected publications quite happily print numbers which can't possibly be true. Reporting that 25% of alcohol is consumed by teenagers is a mistake; reporting that 98% of alcohol is consumed by teenagers would be an obvious sign of complete innumeracy.

Let's look at David Broder, dean of the Washington press corps, writing last week:

Between 1980 and 2000, an analysis by the nonpartisan Alliance for Better Campaigns showed, the amount spent on political ads in major market TV outlets more than quadrupled, from less than $200 billion to almost $800 billion, even after adjusting for inflation.

This factoid was picked up by The Week, where I read it as part of its round-up of the best columns in the US. The news summary magazine did appreciate how startling the figure was: "In 2000," it says, "political candidates paid TV stations an astonishing $800 billion to reach voters."

Astonishing? Unbelievable, more like. $800 billion works out at $3,200 per US citizen, or getting on for $10,000 per voter. It's roughly 16 times the market capitalisation of the Walt Disney Company, and is certainly a large multiple of the value of all the TV networks in America combined.

Going back to the source, you won't be surprised to hear, we find that the real number is not "almost $800 billion" but rather $771 million. The Washington Post was out by more than three orders of magnitude.

Two mistakes were made here. Someone, somewhere, misread a "million" as a "billion": this is understandable. But then no one stopped to think about whether the new number made sense: not David Broder, not the Washington Post copy editors, not the editors at The Week. These are people who are incredibly anal when it comes to errors of language, but they all seem to have a huge blind spot whenever a number enters a paragraph.

I think this is one of the reasons behind the press's failure to pull back the curtain on Enron. Journalists don't like numbers: they're nearly always happy to repeat whatever they're told, no matter how implausible it is. And as Broder shows us, they haven't learned their lesson.

Posted by Felix at 0:20 EST