In recent days there has been much play
over the fact that CNN, the Associated Press, NBC and other news organisations
swallowed a statistically worthless report which said that underage
drinkers drank 25% of all the alcohol consumed in the US. It turns out
that the true figure is closer to 11%, and that the 25% figure, distributed
by the National Center on Addiction and Substance Abuse at Columbia
University, was the result of not adjusting for an oversampling of teenagers
in the National Household Drug Survey. (The Center’s retraction
of its original report makes it seem just like a surly teenager being
forced to apologise for something he doesn’t think he ought to apologise
for: "In its report CASA estimates that underage drinkers consume
25 percent of the alcohol sold in the United States… If the over sample
is adjusted to reflect the population, the percentage is 11.4 percent…
Nevertheless, CASA’s estimate is that underage drinkers consume 25 percent
of the alcohol consumed in the U.S.")
I don’t blame the news outlets for buying the 25% figure. It came from
a highly reputable university, and it’s within the bounds of possibility:
teenagers account for 20% of the population, and there’s no doubt that
18-20 year-olds do a lot of drinking. If the story is well sourced,
busy journalists can’t be blamed for not checking the statistical basis
for every finding. Michael Kinsley says
that the whole story demonstrates America’s "national innumeracy";
I think it doesn’t, but that other stories do.
The real problem is not the innumeracy of the nation in general, but
rather of journalists specifically. Journalists are generally the sort
of people who never really liked mathematics, stopped studying it as
soon as their schools allowed them to, and like to rely on experts whenever
numbers find their way into their stories. It’s the old division between
the arts and the sciences: journalists are writers, they shouldn’t be
expected to understand numbers.
Fact is, the experts upon whom journalists typically rely often don’t
have a strong grasp on their own numbers. I used to write up the weekly
league tables for Bridge News: the obsessed-over rankings of which banks
had underwritten the most bond deals. Pretty much every week, after
receiving the fax from the people who drew up the numbers, I would find
some problem with them, phone them up, and eventually get a corrected
version a couple of hours later. No one else, as far as I knew, approached
the league tables with any sort of critical eye: they just ran whatever
numbers they were given.
But never mind statistical snafus: some stories in highly-respected
publications quite happily print numbers which can’t possibly be true.
Reporting that 25% of alcohol is consumed by teenagers is a mistake;
reporting that 98% of alcohol is consumed by teenagers would be an obvious
sign of complete innumeracy.
Let’s look at David Broder, dean of the Washington press corps, writing
Between 1980 and 2000, an analysis by the nonpartisan
Alliance for Better Campaigns showed, the amount spent on political
ads in major market TV outlets more than quadrupled, from less than
$200 billion to almost $800 billion, even after adjusting for inflation.
This factoid was picked up by The Week, where I read it as part of
its round-up of the best columns in the US. The news summary magazine
did appreciate how startling the figure was: "In 2000," it
says, "political candidates paid TV stations an astonishing $800
billion to reach voters."
Astonishing? Unbelievable, more like. $800 billion works out at $3,200
per US citizen, or getting on for $10,000 per voter. It’s roughly 16
times the market capitalisation of the Walt Disney Company, and is certainly
a large multiple of the value of all the TV networks in America combined.
Going back to the
source, you won’t be surprised to hear, we find that the real number
is not "almost $800 billion" but rather $771 million. The
Washington Post was out by more than three orders of magnitude.
Two mistakes were made here. Someone, somewhere, misread a "million"
as a "billion": this is understandable. But then no one stopped
to think about whether the new number made sense: not David Broder,
not the Washington Post copy editors, not the editors at The Week. These
are people who are incredibly anal when it comes to errors of language,
but they all seem to have a huge blind spot whenever a number enters
I think this is one of the reasons behind the press’s failure to pull
back the curtain on Enron. Journalists don’t like numbers: they’re nearly
always happy to repeat whatever they’re told, no matter how implausible
it is. And as Broder shows us, they haven’t learned their lesson.