Comments on: Polls, Polls, Polls

By: Frank Castiglione

Frank Castiglione — Sat, 22 Dec 2007 02:24:51 +0000

Not only do polls tend to be wildly inaccurate, they can be downright dangerous. Many people are unduly influenced by polls. Some people vote for candidates they wouldn’t ordinarily vote for, some stay home on election day and some believe the results of polls are generally credible without bothering to check who conducted a poll, why it was conducted, and how it was conducted.

The individuals and organizations that commission polls are well aware of this. It may be the main reason polls are conducted so often. It’s definitely the reason poll results are inaccurate so often.

I’m very confident about this.

By: Park Slope Pubby

Park Slope Pubby — Fri, 21 Dec 2007 18:51:10 +0000

I have been called by various pollsters over the years, usually market research. It is unbearably irritating and I really truly believe that 95% of people hang up immediately or in the middle of the poll. who would want to take the unpaid time for this nonsense?

By: Duane T.

Duane T. — Fri, 21 Dec 2007 18:05:13 +0000

Robin - The robo daily tracking polls of Rassmussen varied by as much as 6-10 points from day to day on the Republican primary polling earlier this year … any poll that volatile is by definition - from a statistical analysis perspective - unrealiable. Also, you cannot compare the 2004 election results to primary polling being done now for several reasons.

For one, 2004 was a general election with an either/or choice - Kerry or Bush - and not a wide open primary with 8 or 9 candidates running and no individual polling more than a small plurality of respondents. Name recognition and intra-party factional squabbling in a 2-man Presidential final are not factors as they are in a crowded primary.

Secondly, the percentage of people using cell phones exclusively has gone up a great deal since 2004, as John Podhoretz points out.

Thirdly, there was relatively little volatility in the final polls for the general election in 2004 - nothing like we’ve had in Rasmussen and even a few other polls this primary season. The numbers of all the pollsters in late October/early November 2004 changed relatively little in the leadup to the final election … and most of the major pollsters got it about right, to within their respective margins of error. It’s immaterial, from a statistical standpoint if any given poll was within 0.1% or 3% of actual voting, if the margin of error was 3% or 4%, as were the margins of error in most of the final polls in 2004.

By: Kev

Kev — Fri, 21 Dec 2007 18:04:52 +0000

The only poll that is valid (remember 2004) is the one that samples 100% of actual voters, i.e. the actual election …

Precisely. I’ve been a poll skeptic for quite some time now, for two reasons:

1) I’ve never been asked, and neither has anyone I know (count me among those who mostly use a cellphone and don’t pick up for unfamiliar numbers).

2) I’m one of the people who believes that most people act as individuals and not as members of some specific group. Just because four or five 37-year-old suburban women with three kids who don’t work outside the home think that Political Party X would do a better job with the economy than Political Party Y doesn’t mean that all people sharing that demographic feel the same way, and I sure don’t think that anyone else who shares my demographic automatically speaks for me, nor should they be allowed to do so.

As far as I’m concerned, polls (save for the one that involves ballot boxes) are about as scientific as one’s horoscope, and should be given the same weight as the latter in these discussions.

By: JorgXMcKie

JorgXMcKie — Fri, 21 Dec 2007 17:24:52 +0000

On polling cell phones. My wife works for a ‘polling firm’ and one of their current clients decided that there were probably some important (to the client) differences between those who were totally cell phone as opposed to those who still had a landline (but might well have cells, too). She (the client has a lead researcher who is 70+, anyway — an academic) who wanted to ‘piggyback’ a section strictly for cell-only users (just an additional 400 or so interviews) and thought it should be included in the cost of the current contract). My wife and others said it was all but impossible to do and certainly couldn’t be included in the current cost due to the excessive resource use. Well, the client is a *really* good client, so a deal was arrived at.

The researcher would be responsible for finding appropriate phone numbers of potential cell only users and would pay a very low but flat charge for each ‘connection’ made (i.e. for each call answered by someone, *not* for each completed survey). I presume she used grad students to find the numbers (bad move, trust me). Anyway, what my wife actually got was a list of those ‘blocks’ of 10,000 numbers ‘reserved’ for cell phones. What her call center got was approximately 90% ‘no response’ (probably not a ‘live’ number), about a 5% answer rate, about a 95% hangup/refusal rate, and fewer than 10 useful responses in a week of calls. Oh, the researcher got a pretty big darn bill (at which point the researcher said, “nevermind.”)

Now. boot that up for political polls. Gigantic selection bias even if you get a few.

By: Jeff

Jeff — Fri, 21 Dec 2007 17:20:19 +0000

most polls are mathematically a waste of time when attempting to divine a predicted winner … just because someone can conduct a poll doesn’t mean its valid, useful or even interesting data …
The only poll that is valid (remember 2004) is the one that samples 100% of actual voters, i.e. the actual election …
The real “race” starts the morning of the election. Today we are not in a race therefore we can’t have someone winning or losing. We are still on the practice range. Perception is not reality no matter how hard you squeeze your eyes shut and wish it.

By: Robin

Robin — Fri, 21 Dec 2007 16:35:31 +0000

Duane, Rasmussen’s ‘robo-calls’ nailed it in 2004. He was withing 1/10 of a point, the closest any pollster came to calling the presidential race. I think Rasmussen appears to be an outlier because he is one of the only pollsters that is not looking for the headline (all polls associated with news networks) and despite being Republican, his polls are not politically driven. As a Republican sometimes I don’t like looking at his numbers as they don’t usually bode too well for us but in the long run they are the most realistic.

By: Duane T.

Duane T. — Fri, 21 Dec 2007 16:00:31 +0000

Phil - forget about Rasmussen’s numbers most of all …. he relies on robo-calls which are the most notoriously unreliable and most volatile of all … Rasmussen has been an outlier all year when it came to his numbers for Rudy, since he especially targets extreme conservatives in most of his polling and analysis. Per the RCP average, BTW, Rudy still leads nationally. And in the state-by state polling, Rudy still leads in most of the large urban states, where the lion’s share of convention delegates come from, by large, double digit margins.

Most of Rudy’s erosion in the past month went to Huck who is everybody’s favorite target now, so Huck’s numbers will likely go down. Also, note that the NYT finally today came out with a story that the so-called “scandal” on the billing for Rudy’s trips to the Hamptons in 1999 and 2000 was all bogus - that 100% of the billings for the trips and security was billed to the Mayor’s budget and not the obscure agencies that Politico (a heavily-Dem-laden hack site) claimed that the charges went to. End of scandal, pffffft!

With Rudy laying back and letting Huck and Romney sling mud at each other, and with McCain now coming on strong in NH, I would not lay any bets on Romney (who is still mired in near single digits nationally, and in most of the big states), and the bloom is already coming off the Huck rose.

With Rudy pulling back in IA and NH and concentrating on FL and the Feb 5 Super Duper Tuesday states, you can liken his current campaign to Rope-a-dope … no way can anybody count Rudy out before the results are in on February 5.

By: RussC

RussC — Fri, 21 Dec 2007 14:59:48 +0000

Nice article. I especially got a kick out of the Weekly Standard quote of inviting people out to coffee with the Romney’s even though none of them drink coffee. But back to the point. For too long I have felt that polls are not just an attempt to gauge the current standings of the candidates, but rather a very clear attempt to influence the actual race itself. Who decided to include Gingrinch and Thompson in early polls when they were stil very ambivalent about entering the race, why was Gore continually included in polls even though he repeatedly said he had no interest in running. Why is it that the very respectable Rasmussen poll will sometimes leave a candidate out, for example, their latest head to head poll in Missouri includes guilani, and Huckabee, but leaves out Romney, McCain, Thompson. It compares the GOP candidates to Clinton and Obama, but leaves out Edwards. Polls have the very interesting quality of appearing to be impartial while yet at times being clearly structured to advance certain agendas or to influence opinion…Another example, clearly, electability has been one of the major concerns of members of either party…but while we see many head to head matchup of Obama and Clinton, there are far fewer polls showing numbers on Edwards electability. In some of the ones done for him he has performed better than Clinton or Obama. Doesn’t the democratci base deserve to see the numbers of each of their major candidates? same thing on the GOP side. Guiliani gets most of the head to head match ups, Romney and Huckabee are getting more and more…we see far less of Thompson and McCain. If electability is such a major issue, then why aren;t the GOPers treated to polls showing them the electability chances of each fo their five major candidates? Some in the GOP fear an electoral nightmare if Huckabee is nominated….why isn’t there any polling on that?

Being a numbers guy, I have always found the polls fascinating and great discussion starters. But not only do I agree with John on his bullseye example, I also feel that polls are not the “objective” reviewer than many take them to be.

By: Phil

Phil — Fri, 21 Dec 2007 13:39:08 +0000

One thing that is certain is that Giuliani’s support is sinking everywhere (13% nationally in the latest Rasmussen poll) - exactly as an awful lot of conservative pundits predicted, once the race really got going, and an expectation that you derided, Mr. Podhoretz. What say you now?

By: Jon S.

Jon S. — Fri, 21 Dec 2007 12:52:58 +0000

Hey, come on, John, that’s a very old picture of the Brooklyn Bridge. I might consider buying it from you if you link to a prettier color pic from flickr.com. Seriously, your post is right on the money. The RCP averages will likely work better when we get down to a two-person race in the general, but in a caucus state like Iowa, where only 5% of the registered voters show up and the process is arcane and filled with haggling, it’s still anyone’s guess who will win.

By: David Thomson

David Thomson — Fri, 21 Dec 2007 12:42:44 +0000

“…in 2004, the RCP average accurately predicted which state would go to each presidential candidate except for Wisconsin.”

2004 is ancient history. The dinosaurs were still running around at that time. I suspect that things have changed dramatically since then.

By: soccer dad

soccer dad — Thu, 20 Dec 2007 23:39:10 +0000

I see your point about the problems with state by state polling. However, what’s really impressive is that in 2004, the RCP average accurately predicted which state would go to each presidential candidate except for Wisconsin. (Zogby was on WTOP that morning saying how the undercounted youth vote was going to carry Kerry into the White House.)