David M: Beware Polls

A CBS News poll published May 24 states,

Overall, 49 percent of registered voters now say they would vote for Kerry, 41 percent for Bush.

The poll, which contained other bad news for Bush as well, saw a lot of media play.

But Captain's Quarters says the poll overweights Dems and is hence biased.

Just to put the polling in context, take a look at the Rasmussen poll, which shows a continual dead heat between Bush and Kerry, even on a head-to-head basis, using a sample of 3500 likely voters, with the last data reported 5/29. Quinnipiac has Bush ahead by one point in a three-way race, 43-42-6, as of 5/26, using a sample of 1160 registered voters. While you're there, take a look at the trends; Bush's favorability rating has stabilized at 40% over the past two months, while Kerry's has dropped four points and is below Bush's number.

The CBS poll, on the other hand, uses only 1113 registered voters, broken down in an unusual manner: 346 Republicans, 390 Democrats, and 377 independents. Since when are there that many more Democrats than Republicans? The poll then shows its "weighting" (although it doesn't explain what it means), and the numbers get even worse: 330-R, 401-D, 381-I. According to the University of Pennsylvania in 2003, Republicans accounted for 32.5% of the registered electorate, while Democrats accounted for 33.7%. In a sample of 1113 voters, you would then expect to see 361-R, 375-D, 376-I. The result of CBS's sample is to throw off representation for Republicans by 8.6%, while bolstering Democrats by 7% and independents by 1.3%, using CBS' weighting.

The outcome, therefore, should hardly surprise anyone -- overreported Democrats are surely likely to support John Kerry, while the underreported Republicans depres Bush's ratings. It could hardly escape anyone's notice that CBS' result differed dramatically with other polling taken at the same time, using larger samples and better vetting, such as Rasmussen's 3,500 likely voters -- or could it?

Only RealClearPolitics remarked on the CBS sampling problems, noting that it represented a solid trend at the Tiffany Network:

If you go back, as I did this morning, and look at job approval numbers from the same group of pollsters for the first five and a half months of 2004, you can see the consistency of the CBS/NYT bias more clearly.

In every instance except one this year (and a very iffy one at that), CBS/NYT produced the worst job approval number of any of the three polls during a comparative time period.
And then Tom Bevan gives you a handy table demonstrating the phenomenon. All other news sources, however, gave CBS numbers straight, without any mention of its rather anemic sample -- you'd think a broadcast network could afford to poll larger -- its unreliable sample type, and its poor representation of the electorate. The lesson is that in order to analyze the polling numbers, you have to actually do some work and check the samples.