up

The Threesome Battle: Surveys That Elected the Wrong President

The Threesome Battle: Surveys That Elected the Wrong President

SURVEYS THAT CHANGED THE WORLD



What if you woke up one morning to discover that all survey predictions have really been guesswork all along? If you’re anything like us, you would be completely shocked. Utterly devastated. Entirely broken.

Well, something like that happened in 1948. Sort of.
Luckily we weren't there to witness it.


How the Business of Opinion Poll Surveys Got Into Trouble


It was still in the age of newspapers, before the common availability of television and the birth of Internet. It was also election year in the USA, the third since the 1936 election where George Gallup, founder of the Gallup polling firm, had introduced the world to survey sampling. As detailed in our previous article in this series, Gallup had correctly predicted the victory of Franklin Roosevelt using a sample of only 50,000 voters, beating one of the greatest media companies of that time, the Literary Digest, which had used a sample of 10 million people. For years after that – and through two presidential elections – public opinion surveys sampling only a few thousand people had become common in the media, and it had proved pretty accurate.

So, in 1948, as President Harry Truman faced Thomas Dewey, the Republican challenger for the White House, the three major pollsters of the day confidently ran their opinion poll surveys and predicted the election outcome. George Gallup’s firm was the top dog, with their polls featuring in over 200 newspaper through his syndicate. Archibald Crossley was running a polling service as part of his market researcher firm, while Elmo Roper polled for Fortune, a highly regarded business magazine.

The three firms used the quota sampling technique in what was then a proven scientific way of survey sampling. They carefully identified the different categories of the country’s population and incorporated that ratio-mix in their survey sampling. For instance, if the country’s census showed that 50% of the population was female and 60% of the people lived in rural areas, the sample was designed to include 50% women and 60% rural residents.

Then all the firms deployed teams of interviewers to interview a cross section of people matching the set quota for each segment of the population. Gallup, for example, interviewed about 3,250 people. He interviewed face-to-face rather than using a questionnaire to minimize the non-response bias, which means people not answering your survey.


The Results That Made Pollsters Blush With Embarrassment


As expected, the three firms came up with matching predictions, save for differences of a few percentage points.

Predictions Harry S. Truman Thomas E. Dewey Others
George Gallup 44% 50% 6%
Archibald Crossley 45% 50% 5%
Elmo Roper 38% 53% 9%

The Dewey team was in 7th heaven, while Truman’s people trembled with anxiety. Everyone else, having long leant not to second-guess pollsters, prepared for a Dewey presidency. One newspaper, the Chicago Tribune, reasonably figured the game was up for Truman and, on election night, decided not to waste time waiting for the final results and took their paper to the printer with the seemingly obvious headline Dewey Beats Truman.

Harry S. Truman Thomas E. Dewey Others
Election Results 50% 45% 5%

Then the last ballot was counted. The result was surprising. The United States of America had a laugh. Truman won the election with 50% of the vote against Dewey’s 45%. The three pollsters, for years America’s untouchable know-it-alls, had gotten it all terribly wrong. The results were actually more like a carbon copy of Gallup’s and Crossley’s predictions, only then with the wrong candidate winning. Roper’s poll predictions looked even worse.

It was embarrassing. People across America laughed when Truman, all grin and bravado, posed with a copy of the Chicago Tribune with the headline trumpeting his electoral loss and quipped “Ain’t the way I heard it!”


How the Pollsters Got It So Wrong


So what happened? Three things went wrong.

First was a quota-sampling problem. To be fair to the three polling firms, nobody had figured it out before then, but, as it turned out, quota sampling had a weakness. It essentially forced a survey sample to cover all segments of a population that were thought to have a different perspective or interest that in turn was thought to affect their choice or preference. But the key word here is ‘thought’ – population segments that are thought to determine differences in perspective, which in turn are thought to affect choice or voting preference. Put differently, the quota technique relies too much on the personal or professional judgment of the person designing the survey sample, and leaves big questions unanswered. Like, how is one to determine what certain segments of the population actually think or feel differently on a particular topic, and how is one to know if some important segment has been left out, or if some unnecessary segment has been thrown into the mix?

Second was that all three polling firms ended their polling two weeks before Election Day. Therefore, they did not capture any last minute swings in voter preferences.

Third, the polling firms did not put much premium on the undecided voters, and therefore miscalculated their margins of error.


The Impact of the 1948 Polling Debacle


The 1948 polling embarrassment threatened to destroy the credibility of the survey business. To the general public, it looked like polls had been predictions all along, merely based on a science that was suspiciously close to guesswork.
But, as fate would have it, the great failure turned out to be a blessing in disguise. Until that point, surveys of public opinion had been a preserve of the media. But after this debacle, social scientist waded back into the field.

This is how it happened. The Social Science Research Council, which brought together universities across the United States, formed a committee of experts to review the polling and to provide solutions. The committee’s report was then discussed in a major conference, bringing the media and social scientists together.

It was this committee and the subsequent conference that indentified the problems – including the weakness of quota sampling – and recommended ways to fix them. Among the solutions quickly taken on board by the industry were measures to always factor in undecided survey respondents, and to always take polls until the last possible moment. Most importantly, the quota sampling technique was relegated behind a more scientific survey sampling method still used today: probability sampling.

In the years since, probability sampling has been horned to such a level that never again have surveys of any kind failed so dismally across the board. In fact, there are now no less than four distinct types of recognized probability sampling methods, each with their own known limitations and advantages, and each with a whole body of scientific study to back it up. There is simple random sampling, systematic random sampling, stratified sampling, and cluster sampling. For more details on each, please see our article Survey Myths Debunked: the More People Surveyed the Better the Results.

And so, in a twist of fate, the failed public opinion surveys of 1948 are in fact some of the greatest surveys ever conducted. Because in their dismal failure, they created a window of opportunity that helped evolve the science of surveys to what it is today. In retrospect, it was a darn good thing that Old Truman had such a laugh at George Gallup and his contemporaries.



Do you know of an important survey that you would like us to cover? Feel free to send us your suggestions!

Photo credit: Frank Cancellare (thanks, Frank)


Sources

DeTurck, D. (2014) Case Study 2: The 1948 Presidential Election,
Available at: http://www.math.upenn.edu/~deturck/m170/wk4/lecture/case2.html, Last accessed 28th Jan 2014

Meier, Norman C., & Harold W. Saunders (1949) The Polls and Public Opinion, The Iowa Conference on Attitude and Opinion Research Sponsored by the State University of Iowa, New York: Holt

Mosteller, F., Doob, L.W. & Others (1949) The Pre-election polls of 1948; report to the Committee on Analysis of Pre-election Polls and Forecasts, New York: Social Science Research Council

Peters, G. (2014) Election Year Presidential Preferences: Gallup Poll Accuracy Record: 1936-2012, Available at: http://www.presidency.ucsb.edu/data/preferences.php, Last accessed 28th Jan 2014

Jonah Njonge
Jan 28, 2014
blog comments powered by Disqus