The 2012 election is the first presidential contest in the age of Twitter. It’s also the one that may be remembered as the one in which analysis of poll data became the subject of mass discussion as opposed to the domain of a few political science and statistical freaks. The New York Times’s Nate Silver is as responsible for this as anyone, but the trend is fed by the proliferation of national polls whose results are as varied as their methodologies. Silver has become something of a lightening rod in this election as his forecast which, has continually favored President Obama’s prospects, is now coming in for almost as much scrutiny as the policies of the man he’d like to see re-elected. As someone who has occasionally criticized Silver’s conclusions, I think the focus on him is unfortunate. Silver is a brilliant stat man who whose work attempts to bring the unsparing realism and devotion to accuracy and understanding that is the hallmark of sabermetrics — the study of baseball statistics that derives from the acronym for the Society of American Baseball Research — to political writing. That, like some baseball writers, he cannot always rise above his prejudices, is unfortunate but does not mean his work isn’t worthwhile. Silver is always a good read and even if he seems to have an agenda, I always learn from his posts.
Nevertheless, given the importance that Democrats are placing on his “Five Thirty Eight Forecast,” it was only a matter of time before Silver was given a thorough takedown and Josh Jordan of National Review has done it in a must read analysis. In “Nate Silver’s Flawed Model,” Jordan details how Silver’s partisan leanings have influenced his judgment about how much weight to give to various polls. As Jordan points out, Silver tends to assess the reliability of certain polls based on his feelings about whether they are right, which is to say sufficiently pro-Obama. While I don’t think Silver’s purpose is deception, his bias has created a model that seems designed to produce one result even if it contradicts what many see as a pro-Romney trend. As such, he’s become the geekiest yet perhaps also the most important cheerleader in the country these days as liberals look to his blog for comfort in trying times. But Silver isn’t the only one making mistakes out there.
Trying to make sense of the various national polls would defeat a lesser mind than Silver’s and that’s the position the rest of us are placed in by the contradictory results being produced. But you don’t have to be as sharp as the Times blogger to understand that if you poll a Democrat-leaning sample, you get a Democrat-leaning result.
Thus, amid the flood of polls showing progress for Romney, there are some showing Obama holding his own. But almost all of them report that their poll sample is made up of respondents who appear to be disproportionately identified as Democrats. For example, that was the case with last week’s Washington Post/ABC News poll that showed the president with a three-point lead. It’s sample showed nine percent more Democrats as opposed to Republicans, a result that seems at variance with reality as well as outstripping the numbers that were produced by President Obama’s 2008 near-landslide.
This week, the same poll produced a slightly less favorable result for the president giving him only a one-point lead. While some might interpret this as part of a pro-Romney trend or more evidence that the president failed to get a bounce out of last week’s debate, the reason for the different result is found in the last question in the survey: Do you think of yourself as a Republican or a Democrat? In one week, the poll’s party identification sample changed from plus nine for the Democrats to only plus five.
Now, as Silver has often rightly pointed out, party identification is not set in stone and can vary. But do we really believe the country’s view of the parties, as opposed to the candidates, can change that much in one week. A review of the samples in this poll in the past year shows similar fluctuations.
Understanding that the sample you choose more or less dictates your poll results is the sort of unoriginal insight that puts even stat amateurs pretty much on an even playing field with smart guys like Silver. Put simply, the polls that show Obama winning only make sense if you believe the Democrats’ turnout will far eclipse that of the Republicans and at least match the “hope and change” fervor of 2008. That’s certainly possible but is not particularly likely. Which is why the healthy skepticism being shown toward outlier polls that favor Obama is not merely conservatives letting their wishes be father to their thoughts.
In the two weeks that remain before the election, we’ll have a lot more polls to dig through but that basic truth about samples should never be far from our thoughts. We don’t know what the results in the only poll that counts — the one at the ballot box — will be but until we get there, we’re stuck trying to, as Silver would say, filter out the “noise” that is obscuring the truth about public opinion. Unfortunately, some of that noise comes in the form of analysis that is as heavily influenced by partisan feelings as that shown by any hometown baseball scribe.