Statistics on Statistics: Internet reporting wins again

by phil on Thursday Oct 23, 2008 4:34 PM

Statistics are horribly mishandled by the mainstream media. On the one hand you have networks cherry-picking polls that are the most sensational, for example the Zogby polls that show the presidential race neck-and-neck, while as an aggregation of polls shows Obama above by 9%. And then on the other hand, you have the cynics on the talk shows, who stop conversations by citing Mark Twain's, "There are three kinds of lies: lies, damned lies, and statistics."

The media needs to treat statistics not cavalierly and not cynically, but with a measured interest. There is one site that gets the attitude just right. If you follow the way the guys at FiveThirtyEight think, you will see a shining example of a proper handling of statistics. I point to this article where they criticize a IBD/TIPP poll by pointing out that it gives John McCain a 74% to 22% advantage over Obama among the age group 18-24. Nate Silver, one of the writers for 538, points out that "Using a binomial distribution, the odds are 54,604,929,633-to-1 against. That is, about 55 billion to one. So, there is an 0.000000002% chance that IBD/TIPP just got really unlucky."

What 538 does is aggregate polls, but it's not out of a blind love of statistics, but out of a measured skepticism toward them. If, for example, a poll comes in that summarily cites a close race between McCain and Obama, and yet in the footnotes shows that McCain is beating Obama among blacks, then the poll is thrown out. That's just common sense.

I think what we're seeing, with the Internet, is the greater meaning that can be elicited from aggregation. For example, why should one media outlet give a film review by their one critic. What if that critic simply had indigestion when he or she saw the movie? A Rotten Tomatoes score is much more meaningful than one silly review that the church lady at your local newspaper wrote.

It's becoming sad to me to see these CBS and ABC polls, because the 538 aggregate polling is miles ahead of these data points in terms of accuracy. If I were CBS or ABC, I'd outsource the polling or give it a different name, and instead report the more interesting, aggregate information.