How Cellphones Complicate Polling

Cellcom

Smart Phones

5s

Apple

Mobiles

Phones

How Cellphones Complicate Polling

With this election, math again messed with the magic* in a media stalwart. Television pundits, usually with the authority left over from past political victories, turned out to be inferior seers compared to fast-moving analysts armed with a raft of polling data. The Times’s own Nate Silver appears to be the biggest winner of all.

But other math, abetted by technology, could mean trouble down the line for our prognosticating overlords. Traditional polling is getting more expensive and less reliable. The emerging online alternatives are promising, but they have problems of their own. Problems with the polls may also mean problems for the people who read them. (Nate Silver made a comparison of polling accuracy last week.)

The fundamental difficulty has to do with changes in phone technology and human habits. Much of the polling data you see comes from phone calls. Caller identification has made it easier to ignore calls from polling outfits. Cellphones have caller ID, and people are likely to be using them from any number of places  where they don’t want to be disturbed.

In May, the Pew Research Center published a report that said that the number of households responding to phone polls had fallen to 9 percent today from 36 percent in 1997. If this trend continues, at some point response rates will be too low to show good representation.

Even if pollers do get through and persuade people to cooperate with an in-depth poll, taking these kinds of surveys to an increasingly mobile population is more expensive. A 1996 federal law states that calls to cellphones must be hand-dialed, not generated by computer. That increases the time required for getting the answers.

A study published last spring looked at an effort by the Department of Housing and Urban Development to survey rents. It found that the cost of obtaining one completed survey ranged from $77.18 for a call to a landline phone to $277.19 for a call to a cellphone.

While it is not clear that this study was a perfect match for the costs of a political poll, it is clear that calling the mobile population is expensive. That makes follow-up and in-depth polls, which are more valuable, less attractive.

“The ultimate question is, how representative are you of the population?” says Michael McDonald, a professor of statistics at George Mason University who studies polling. “I tend to trust organizations that go the extra mile, with personal interviews, calls and multiple callbacks. Fast polls are a strategy if you want to make news, but they aren’t as good.”

One alternative is to rely more on Internet-based surveys, something the pollers at Rasmussen Reports and other outfits already do. Professor McDonald says using Internet data, however, “trades one set of biases for another. We don’t have full Internet coverage, and not everyone uses computers.”

Still, as more people get online, the Internet-based polls get much better. SurveyMonkey, which sells tools for many kinds of collective voting, carried out over several months an online presidential poll that had 96 percent accuracy, compared with the actual results of the vote.

“We looked at nine battleground states over 11 weeks,” said Philip Garland, vice president of methodology at SurveyMonkey. “On the day before Election Day alone, 60,000 people took the survey.”

The cost per person was negligible, he said, and the results may be more illuminating. “We got twice as many ‘don’t knows’ compared with phone or personal surveys,” says Mr. Garland. “When people are asked questions by a person, they feel like they should make a choice.” Still, like other pollers, the online service was surprised at the turnout by Latino and African-American voters, indicating that the survey didn’t perfectly capture the national population.

SurveyMonkey, which didn’t make money from this poll, plans to continue the work for the 2014 midterm elections and will make its data available to the public. “We expect to get a lot of interest from political organizations,” says Mr. Garland.

Just in case you thought this election thing was over.

*Note: A saltier version of the phrase “messed with the magic” was supposedly uttered by an old-media big shot when he first toured Google and learned how its algorithms could make advertising both cheaper and more efficient.