Pollsters Admit – Don’t Believe the Numbers

02-12-09

Have to say, this article in the Winnipeg Free Press is delightful, and if you hear champagne corks popping it is likely from SEO, Social Media and branding consultants partying under ‘Told You So‘ banners.

Metrics and Measurment

Metrics and Measurment

No doubt a well designed and implemented poll can provide data – actionable on many fronts, but the truth is far from the expected reality. Too often political surveys are commissioned at the lowest cost, and only to create a headline for a media group. In the worst case, a survey is used to justify a policy or decision – the polling company provided a narrow range of questions and criteria, among other parameters, which result in an expected outcome.

There’s broad consensus among pollsters that proliferating political polls suffer from a combination of methodological problems, commercial pressures and an unhealthy relationship with the media. Winnipeg Free Press

The ‘unhealthy relationship’ ties to the commissioning – the company, political party, media partner or government that creates the contract. This is especially true of municipalities that use polling companies as part of their ‘so-called’ engagement policy. Questions, and criteria, can be set at the beginning of a survey – ignoring facts, highlighting an ‘ideal’ – resulting in numbers supporting a policy, that is often not the majority preference come actual vote.

Polling companies are in survival mode, in the same declining influence as newspapers. There are plenty of great, long term, ethical pollsters but they face a fractured, dis-enfranchised pool. In the age (not so long ago) when newspapers, TV, radio and land-line phone connection reigned supreme, polling was accurate. Simple enough, after a major radio address by a sitting oligarch, or newly hatched contender, to phone a bunch of people and get a score.

The contemporary situation is far different. Media influence is widely disconnected between the traditional TV, radio and print mediums vs blogs, Twitter and Facebook – yet long and short tail connected. A confusion the internet and social media has created, making it impossible for any poll to make sense of data. Either they, the pollsters, miss a ‘trend’ or ‘meme’, or frankly, are unable to connect with the players who eventually will create the ultimate result. As mobile phones are often un-listed, and polling companies can’t survey social media accounts without biased results, general surveys (especially political) are so inaccurate they should never be published.

So why do pollsters continue to trumpet their imperfect data to the media?
Money. Or more precisely, the lack of it.
When Gregg started polling in the 1970s, there were only a handful of public opinion research companies. Polls were expensive so media outlets bought them judiciously.
Now, Gregg laments almost anyone can profess to be a pollster, with little or no methodological training. There is so much competition that political polls are given free to the media, in hopes the attendant publicity will boost business.
Turcotte says political polls for the media are “not research anymore” so much as marketing and promotional tools. Because they’re not paid, pollsters don’t put much care into the quality of the product, often throwing a couple of questions about party preference into the middle of an omnibus survey on other subjects which could taint results.
And there’s no way to hold pollsters accountable for producing shoddy results since, until there’s an actual election, there’s no way to gauge their accuracy.
“I believe the quality overall has been driven to unacceptably low levels by the fact that there’s this competitive auction to the bottom, with most of this stuff being paid for by insufficient or no resources by the media,” concurs Graves.

Winnipeg Free Press

Media knowledge is not an indication of action. That is true today as 20 years ago. Yet, it is interesting that Facebook ‘likes’ for candidates in the 2010 US midterm elections actually prove more accurate than mainstream traditional polling in actual votes.

Politicians like to say the only poll that really counts is on election day. But Marzolini worries the plethora of shoddy media polls, an annoyance between elections, can be self-fulfilling during a campaign.
With little or no time between polls and the media fixated on the flood of horse race numbers, he says voters don’t get a chance to reflect on platforms or leaders or their campaign messages. Hence, “the only movement in the polls is in fact motivated by the previous polls.”
Poll have always had some influence on elections, helping to drive strategic voting. But Marzolini fears the sheer volume now is creating a situation in which the media — and, by extension, voters, — “just want to get the score for the game; they don’t want to watch the game.”

In the upcoming Canadian election cycles, federal, provincial and municipal – it will be of interest to everyone who market predictions. My guess – a tie between traditional polling and social media scores.

[ad#Google Adsense]

Share

Discussion2 Comments

Leave a Reply


*