Polls and Cell Phones

Are pre-election opinion polls accurate?

If you know anything about polling, you've probably heard about this notorious polling mistake from 1936:

One of the most accurate straw polls for a while was the Literary Digest poll. It gained a wide following for its straw polls of presidential elections in the early 20th Century. The Digest's straw polling methods were based on the then-current belief that the bigger the sample, the more accurate the results.

During the last year (1936) the Digest polled, over 10 million questionnaires were mailed out, and over two million people responded. Compared to the roughly 1,500 or fewer people interviewed in a modern poll, two million respondents seem almost incalculable.

It was, however, with the 1936 poll that the Literary Digest's techniques became a symbol of flawed polling, and the Digest thereafter would have a special place in polling research infamy. Based on a sample of two million, the Digest confidently predicted that Pennsylvania native Republican Alf Landon, now a Kansan, would overwhelm Democrat Franklin Roosevelt.

FDR got 63% of the vote and won 46 of 48 states!

The question of poll accuracy is again alive in the 2004 election season.

Columnist Jimmy Breslin makes the most controversial claim. The polls are bunk, he says, because pollsters call people to unearth their opinions and a substantial number of people use cells phones that are never called:

Anybody who believes these national political polls are giving you facts is a gullible fool.

Any editors of newspapers or television news shows who use poll results as a story are beyond gullible. On behalf of the public they profess to serve, they are indolent salesmen of falsehoods....

There are almost 169 million cell phones being used in America today - 168,900,019 as of Sept. 15, according to the cell phone institute in Washington.

There is no way to poll cell phone users, so it isn't done.

Why don't pollsters call cell phone users?

Well, there are technical hurdles, which you probably already considered:

A technological factor is that users of wired telephones screen out unwanted calls from pollsters and survey researchers with caller-ID and computer programs that filter out calls from any unrecognized phone number.

There are also legal barriers unique to cell phones:

For a decade, Federal Communications Commission regulations have restricted pollsters from using modern dialing equipment to call cell phones. And even if they dial by hand, another rule prohibits them from phoning anyone who would have to pay for the call.

“That’s just about everyone with a cell phone,” said Elyse Gammer, an executive with the Marketing Research Association in Connecticut. “They pay for airtime, and if somehow they don’t, how are you going to know?”

A violation makes the caller vulnerable to a lawsuit with a penalty of at least $500.

To avoid a barrage of lawsuits, pollsters have long used lists of phone numbers that exclude cell phone exchanges.

Pollsters are exempt from the "do not call" laws, but still cannot call people who pay for incoming calls.

The impact of these limits are potentially huge. It is possible that failure to call cell phones skews the polling sample and thus distorts the meaning of the results. In a voter pool that seems quite evenly divided, the implications are obvious.

Columnist Robert Landauer of Portland's Oregonian newspaper claims that there may be huge demographic effects that will distort the poll samples:

Young people are the demographic group least likely to have their own wired telephones and most likely to communicate via cell phone. "Just over 18 million young people (18-30) voted in 2000, and five states were decided by less than 8,000 votes," said a March 11 report in Wired News.

Breslin quotes pollster John Zogby to demonstrate the enormity of the problem faced by pollsters:

"The people who are using telephone surveys are in denial," Zogby was saying. "It is similar to the '30s, when they first started polling by telephones and there were people who laughed at that and said you couldn't trust them because not everybody had a home phone. Now they try not to mention cell phones. They don't look or listen. They go ahead with a method that is old and wrong."

I've read that pollsters this year are asking to speak to the youngest members (over age 18) of the households they call. This seems like an obvious attempt to compensate for the "cell phone problem."

Does it work?

Landauer quoted another polling expert in his column

"At the moment there is no ready answer on how to deal with cell phones and how to get people into surveys and how to design surveys to accurately represent the population," says John Tarnai, director of the Social & Economic Sciences Research Center at Washington State University.

This story came to a similar conclusion, and noted the difficulty of even finding out if the survey methods are correct:

Relatively few people have opted to go entirely wireless, and little research has been conducted into whether that group differs significantly from the wired community. Pollsters note the Catch-22 of wanting to profile wireless-only customers but being unable to poll them.

Zogby is using direct interviews and is apparently trying to figure out ways to employ email to conduct polling, which might also prove controversial.

Anyway, I'm going to blog about Zogby's latest results in my next post.