Polls, Polls, Polls
- 12.20.2007 - 5:53 PMAnother day, another set of polls. One says John McCain is now tied with Mitt Romney for the lead in New Hampshire. But that does not comport with the findings of other polls, which have Romney ahead by seven or ten or twelve points. Barack Obama has closed the gap with Hillary Clinton nationally and trails her by seven — or is twenty points back. And then there is Iowa, where Hillary is ahead, Obama is ahead, or Edwards is ahead. Huckabee might be ahead by fifteen or by two.
Results this varied are what has caused most people to begin to rely on an average of all polls being taken. The Poll of Polls most frequently cited is the one at realclearpolitics.com, which popularized it. Mark Blumenthal of pollster.com explains why a poll average works with an analogy to darts. Let’s say you’re playing darts and you don’t know where the bullseye is. You can figure it out by looking at the pattern of the holes created by other dart-throwers, which in an effort to reach the bullseye will actually create a picture of it in absentia.
The problem with that analogy is that there isn’t just one bullseye in a primary poll. There are five or six. Each candidate a pollster asks about is a bullseye. And with all these other possible bullseyes, the pattern of the holes around each one of them is not going to be anywhere near as distinct. It stands to reason that if you ask 500 people about a choice between A or B, you’re going to get a large number for A and a large number for B, and that one of the two will be larger than the other. If you ask 500 people about a choice between A, B, C, D, E, F or G, you’re not going to get big numbers for any one of them but relatively small numbers for all of them. And the pattern created by each choice — corresponding to a single dart — is more like an impression than a solid pattern.
Add to this uncertainty the fact that 14 percent of Americans now only use cellphones. Pollsters haven’t figured out how to factor in cellphones, so that’s 14 percent of the potential electorate missing from their sample to begin with. Add further the fact that many people — no one has a number, but it is significant — now hang up on people they don’t know or don’t answer the phone when their Caller ID offers an unknown phone number, and you have another segment of the population that is offline.
Now consider Iowa and New Hampshire. These are states whose residents are being bombarded daily by phone calls from campaign volunteers, campaign staffers, and recorded messages from candidates. As Richelieu, a campaign guru who posts on the Weekly Standard’s Campaign blog, puts it:
Polling right now in Iowa and New Hampshire is a technical nightmare. Every three minutes the average voter’s phone rings with somebody coaxing them to trudge out into the snow and attend an Edward’s meeting, go to a coffee with one of Romney’s sons, or sign up for a Huckabee prayer circle. Not to mention the endless pre-recorded “robo-call” phone messages from various crank interest groups grinding their axe on some issue. With your phone ringing two dozen times a day with a political call, it is not easy for the 35 different media and private pollsters each trying to get a sample done each night. Voters don’t answer the phone or refuse to play along when they do answer. Which means response rates go way down, samples tilt away from a statistically reliable random frame of the population, and results go bad.
And now for the most important part: Turnout in the Iowa caucuses is expected to be somewhere around…this is serious…five percent. That means five percent of the state’s universe of Republicans will attend a Republican caucus meeting, and five percent of the state’s Democrats will attend a Democratic caucus meeting. According to Blumenthal of pollster.com, “The historical high for turnout in the Iowa Caucuses was 5.5% of adults for the Democrats in 2004 and 5.3% of adults for the Republicans in 1988.”
Now here’s what this means. For a poll to achieve a measurable degree of scientific accuracy, a pollster “would need to screen out nine out of ten otherwise willing adults in order to interview a combined population of Democratic and Republican caucusgoers strictly comparable in size to past caucus turnouts.” Because no pollster can afford to do such a thing — to reach thousands of people and then discard the results from 90 percent of the phone calls — each polling firm has to come up with its own theory of how best to locate and identify likely voters in sufficient numbers. That’s why, Blumenthal says, the results of each poll vary so wildly.
So. People who are polled are offered six options. People use cell phones exclusively. They know they’re getting political calls and don’t answer the phone if they’re at home. And only five percent of voters in each party actually turn out in Iowa.
So. Still confident there’s a Huckabee “surge”? Or that Romney has ended the Huckabee surge? Or that Obama is gaining on Hillary? Or that Edwards can’t win? If you are, I would like to sell you this.
| »Back to Contentions | »Back to Commentary |
December 20th, 2007 at 6:39 PM
I see your point about the problems with state by state polling. However, what’s really impressive is that in 2004, the RCP average accurately predicted which state would go to each presidential candidate except for Wisconsin. (Zogby was on WTOP that morning saying how the undercounted youth vote was going to carry Kerry into the White House.)
December 21st, 2007 at 7:42 AM
“…in 2004, the RCP average accurately predicted which state would go to each presidential candidate except for Wisconsin.”
2004 is ancient history. The dinosaurs were still running around at that time. I suspect that things have changed dramatically since then.
December 21st, 2007 at 7:52 AM
Hey, come on, John, that’s a very old picture of the Brooklyn Bridge. I might consider buying it from you if you link to a prettier color pic from flickr.com. Seriously, your post is right on the money. The RCP averages will likely work better when we get down to a two-person race in the general, but in a caucus state like Iowa, where only 5% of the registered voters show up and the process is arcane and filled with haggling, it’s still anyone’s guess who will win.
December 21st, 2007 at 8:39 AM
One thing that is certain is that Giuliani’s support is sinking everywhere (13% nationally in the latest Rasmussen poll) - exactly as an awful lot of conservative pundits predicted, once the race really got going, and an expectation that you derided, Mr. Podhoretz. What say you now?
December 21st, 2007 at 9:59 AM
Nice article. I especially got a kick out of the Weekly Standard quote of inviting people out to coffee with the Romney’s even though none of them drink coffee. But back to the point. For too long I have felt that polls are not just an attempt to gauge the current standings of the candidates, but rather a very clear attempt to influence the actual race itself. Who decided to include Gingrinch and Thompson in early polls when they were stil very ambivalent about entering the race, why was Gore continually included in polls even though he repeatedly said he had no interest in running. Why is it that the very respectable Rasmussen poll will sometimes leave a candidate out, for example, their latest head to head poll in Missouri includes guilani, and Huckabee, but leaves out Romney, McCain, Thompson. It compares the GOP candidates to Clinton and Obama, but leaves out Edwards. Polls have the very interesting quality of appearing to be impartial while yet at times being clearly structured to advance certain agendas or to influence opinion…Another example, clearly, electability has been one of the major concerns of members of either party…but while we see many head to head matchup of Obama and Clinton, there are far fewer polls showing numbers on Edwards electability. In some of the ones done for him he has performed better than Clinton or Obama. Doesn’t the democratci base deserve to see the numbers of each of their major candidates? same thing on the GOP side. Guiliani gets most of the head to head match ups, Romney and Huckabee are getting more and more…we see far less of Thompson and McCain. If electability is such a major issue, then why aren;t the GOPers treated to polls showing them the electability chances of each fo their five major candidates? Some in the GOP fear an electoral nightmare if Huckabee is nominated….why isn’t there any polling on that?
Being a numbers guy, I have always found the polls fascinating and great discussion starters. But not only do I agree with John on his bullseye example, I also feel that polls are not the “objective” reviewer than many take them to be.
December 21st, 2007 at 11:00 AM
Phil - forget about Rasmussen’s numbers most of all …. he relies on robo-calls which are the most notoriously unreliable and most volatile of all … Rasmussen has been an outlier all year when it came to his numbers for Rudy, since he especially targets extreme conservatives in most of his polling and analysis. Per the RCP average, BTW, Rudy still leads nationally. And in the state-by state polling, Rudy still leads in most of the large urban states, where the lion’s share of convention delegates come from, by large, double digit margins.
Most of Rudy’s erosion in the past month went to Huck who is everybody’s favorite target now, so Huck’s numbers will likely go down. Also, note that the NYT finally today came out with a story that the so-called “scandal” on the billing for Rudy’s trips to the Hamptons in 1999 and 2000 was all bogus - that 100% of the billings for the trips and security was billed to the Mayor’s budget and not the obscure agencies that Politico (a heavily-Dem-laden hack site) claimed that the charges went to. End of scandal, pffffft!
With Rudy laying back and letting Huck and Romney sling mud at each other, and with McCain now coming on strong in NH, I would not lay any bets on Romney (who is still mired in near single digits nationally, and in most of the big states), and the bloom is already coming off the Huck rose.
With Rudy pulling back in IA and NH and concentrating on FL and the Feb 5 Super Duper Tuesday states, you can liken his current campaign to Rope-a-dope … no way can anybody count Rudy out before the results are in on February 5.
December 21st, 2007 at 11:35 AM
Duane, Rasmussen’s ‘robo-calls’ nailed it in 2004. He was withing 1/10 of a point, the closest any pollster came to calling the presidential race. I think Rasmussen appears to be an outlier because he is one of the only pollsters that is not looking for the headline (all polls associated with news networks) and despite being Republican, his polls are not politically driven. As a Republican sometimes I don’t like looking at his numbers as they don’t usually bode too well for us but in the long run they are the most realistic.
December 21st, 2007 at 12:20 PM
most polls are mathematically a waste of time when attempting to divine a predicted winner … just because someone can conduct a poll doesn’t mean its valid, useful or even interesting data …
The only poll that is valid (remember 2004) is the one that samples 100% of actual voters, i.e. the actual election …
The real “race” starts the morning of the election. Today we are not in a race therefore we can’t have someone winning or losing. We are still on the practice range. Perception is not reality no matter how hard you squeeze your eyes shut and wish it.
December 21st, 2007 at 12:24 PM
On polling cell phones. My wife works for a ‘polling firm’ and one of their current clients decided that there were probably some important (to the client) differences between those who were totally cell phone as opposed to those who still had a landline (but might well have cells, too). She (the client has a lead researcher who is 70+, anyway — an academic) who wanted to ‘piggyback’ a section strictly for cell-only users (just an additional 400 or so interviews) and thought it should be included in the cost of the current contract). My wife and others said it was all but impossible to do and certainly couldn’t be included in the current cost due to the excessive resource use. Well, the client is a *really* good client, so a deal was arrived at.
The researcher would be responsible for finding appropriate phone numbers of potential cell only users and would pay a very low but flat charge for each ‘connection’ made (i.e. for each call answered by someone, *not* for each completed survey). I presume she used grad students to find the numbers (bad move, trust me). Anyway, what my wife actually got was a list of those ‘blocks’ of 10,000 numbers ‘reserved’ for cell phones. What her call center got was approximately 90% ‘no response’ (probably not a ‘live’ number), about a 5% answer rate, about a 95% hangup/refusal rate, and fewer than 10 useful responses in a week of calls. Oh, the researcher got a pretty big darn bill (at which point the researcher said, “nevermind.”)
Now. boot that up for political polls. Gigantic selection bias even if you get a few.
December 21st, 2007 at 1:04 PM
The only poll that is valid (remember 2004) is the one that samples 100% of actual voters, i.e. the actual election …
Precisely. I’ve been a poll skeptic for quite some time now, for two reasons:
1) I’ve never been asked, and neither has anyone I know (count me among those who mostly use a cellphone and don’t pick up for unfamiliar numbers).
2) I’m one of the people who believes that most people act as individuals and not as members of some specific group. Just because four or five 37-year-old suburban women with three kids who don’t work outside the home think that Political Party X would do a better job with the economy than Political Party Y doesn’t mean that all people sharing that demographic feel the same way, and I sure don’t think that anyone else who shares my demographic automatically speaks for me, nor should they be allowed to do so.
As far as I’m concerned, polls (save for the one that involves ballot boxes) are about as scientific as one’s horoscope, and should be given the same weight as the latter in these discussions.