A new poll by the Public Religion Research Institute, which found the majority of Catholics support the Obama administration’s mandate for employers to provide health care plans covering free contraceptives, has been getting a ton of media attention the past two days.
But don’t buy into it so fast. A peek into the poll’s methodology raises enough red flags to invite serious questions about its conclusion. The leadership board of the organization that conducted the poll also includes several of President Obama’s current and former religious advisers (Rabbi David Saperstein sits on his religious advisory board, and Lisa Sowle Cahill advised his campaign in 2008).
Here are three problems that immediately jump out:
1.) No breakdown of the number of Catholic respondents.
The poll was taken from a random sample of 1,009 American adults aged 18 or older. But it doesn’t include the percentage of the sample that’s Catholic, making it impossible to figure out how meaningful the data from this group is.
2.) Strange sample weighting.
In a poll on religious opinions, you’d expect the pollster to weigh the sample to account for accurate representation of religious affiliation. But the sample in this poll was only weighted to the following five parameters: age, sex, geographic region, education and telephone usage. The last two – education and telephone usage – seem to be far less consequential to the poll than religion, and it’s hard to see why they’d be included when religious affiliation was not. Occasionally, weighting can be used to manipulate polling data.
3.) An odd disclaimer.
Polls typically include a margin of error. In this case, the margin is +/- 3.5 percentage points at the 95 percent level of confidence. But they don’t usually include disclaimers like this: “In addition to sampling error, surveys may also be subject to error or bias due to question wording, context and order effects.”
Those are all controllable errors that professional pollsters are expected to avoid.