Should we trust surveys and opinion polls?

Jose Ramon G. Albert
Should we trust surveys and opinion polls?
Let politicians be more analytical about poll results, lest they unwisely run for posts and lose

With the election for 2016 expected to heat up, voting preferences have already been tracked by poll outfits even more than a year before the elections.

Last December, Rappler carried the results of a survey by the Social Weather Stations (SWS) that asked respondents to name three people who should succeed PNoy. The results indicated that Vice President Jojo Binay had nearly two out of every five respondents (37%) naming him, while both Senator Grace Poe (21%) and Interior Secretary Manuel Roxas II (19%) have one in five respondents naming them. 

A November poll conducted by Pulse Asia provided Binay with 26%, and Poe with 18%, while Roxas only landed sixth (with 6 %) trailing Senator Miriam Defensor Santiago (12%), former President and current Manila Mayor Joseph Estrada (10%) and Senator Francis “Chiz” Escudero (7%). Unlike the SWS survey, however, the Pulse Asia survey asked respondents who they would vote for if elections were held today.

Last month, the results of Pulse Asia’s survey suggested that if elections were held then, Binay would garner 29%, followed by Poe (14%), Davao City Mayor Rodrigo “Rody” Duterte (12%) and Erap (also at 12%). Others, including Roxas were landing single digit voter preferences.

Should these survey results be trusted if the results of surveys of these two “reputable” organizations do not seem to jibe? The short answer to this question is yes. 

There is a lot of uncertainty about elections as some people may make up their minds on who they will vote for only on election day or thereabouts. Some people may also be fickle-minded and may base their final decision on a random choice. Within the context of uncertainty (and assuming that elections are honest and clean), polls, if done well, can yield reliable assessments of voter preferences. 

The main challenge is understanding the results of polls.  

The discrepancies in results of polls done by SWS and Pulse Asia can be readily explained as a result of differences in question wording. In addition, while it may be a puzzle why the ratings of Binay in the Pulse Asia survey in November 2014 to March 2014 “increased,” the changes, however, are actually merely within the survey’s “margin of error” so that there may have been no change in voting preference for Binay. 

As of now, there is no one yet who has proclaimed interest in running for president except VP Binay, so he may have the upper hand. If Senator Poe decides to run for the highest office and others (such as Duterte and Erap) do not (and these candidates express support for one candidate), then the voting preferences may change. Let us remember that the current president only became an aspirant in 2009 after his mother passed on, and the landscape for the 2010 elections changed after that.     

Can’t we find faster ways of tracking public opinion these days on voting preferences, say through SMS polls, Facebook posts, email surveys? 

 

How about big data?

While undoubtedly new technologies have allowed us to collect Big Data, i.e., large volumes of data with velocity and variety, we should understand the nature of data to transform them into meaningful information. 

Nate Silver (the statistician who predicted accurately the results of the last presidential election) aptly warns in his book, The Signal and the Noise, that “If the quantity of information is increasing by 2.5 quintillion bytes per day, the amount of useful information almost certainly isn’t. Most of it is just noise, and the noise is increasing faster than the signal.”  

While polls on voting preferences through SMS messages and Facebook posts can always be conducted, their results are filled with too much noise as there is currently no way to determine the representativeness of respondents to such surveys.  

For instance, the results of an SMS poll that I participated in (see Figure 1), suggested a rather incredible result: Duterte garnering much, much, much more votes than every presidential wannabe, including Binay. Even if Binay’s share of votes are dwindling because of corruption charges, SMS polls do not have complete coverage of voters: not all voters have cellphones (especially among the poor), despite the increase in mobile phone usage over the years. 

In addition, a mere “random selection” of mobile phone numbers will in no way assure us of its representativeness of the voting population (even if everyone had a cell phone) as there will be nonresponses that have to be accounted for. 

Figure 1. Unscientific SMS polls.

Within the context of uncertainty, traditional face-to-face polls based on selecting respondents randomly remain the most reliable method of assessing public opinion, especially in developing countries like the Philippines.

However, not all opinion polls can be trusted. One must be cautious about looking at results of polls, especially those conducted by fly-by-night pollsters with dubious credentials, and those who resort to unprofessional practices. Media should be responsible enough to examine the track record of organizations that conduct polls.  

For example, when comparing the pre-election surveys of SWS and Pulse Asia with the final election results, we find that the polling yielded fairly reasonable estimates of voting preferences on election day (see Table 1).   

Table 1. Results of Pre-Election and Election Day Surveys of SWS and Pulse Asia on Voting Preferences for Presidential Candidates: 1992-2010

1992
Presidential candidates Comelec SWS Pulse Asia
May 10 Apr 26 – May 4  
Fidel Ramos 23.6 26.8  
Miriam Santiago 19.7 25.0  
Eduardo Cojuangco Jr. 18.2 16.2  
Ramon Mitra Jr. 14.6 14.1  
Imelda Marcos 10.3 6.5  
Other candidates 13.6 11.4  
# of respondents   1200  
1998
Presidential candidates Comelec SWS Pulse Asia
May 10 May 2-4 Apr 26-29
Joseph Estrada 39.9 33.0 37.0 
Jose De Venecia 15.9 15.0 31.0 
Raul Roco 13.8 11.0 11.0
Lito Osmeña 12.4 11.0 7.0
Alfredo Lim 8.7 10.0 5.0 
Other candidates 9.3 20.0 9.0
# of respondents   1500  
2004
Presidential candidates Comelec SWS Pulse Asia
May 10 May 1-4 April 23-25
Gloria Arroyo 40.0 37.0 37.0 
Fernando Poe Jr. 36.5 30.0 31.0 
Panfilo Lacson 10.9 11.0 11.0
Raul Roco 6.5 6.0 7.0
Eddie Villanueva 6.2 4.0 5.0 
Other candidates 0.0 12.0 9.0
# of respondents   2000  1800
2010
Presidential candidates Comelec SWS Pulse Asia
May 10 May 2-4 April 26-29
Noynoy Aquino 42.1 42.0 39.0
Joseph Estrada 26.3 20.0 20.0
Manny Villar 15.4 19.0 20.0
Gilbert Teodoro 11.3 9.0 7.0
Eddie Villanueva 3.1 3.0 3.0
Other candidates 1.8 7.0 11.0
# of respondents   2400 1800

Sources: SWS, Pulse Asia and Comelec

The SWS and Pulse Asia thus have a track record of good forecasting performance of election results, with estimates of voter preference for the top five candidates in pre-election surveys generally within 3 percentage points of the election day results.

Highest errors (of over 5 percentage points) were made in all presidential elections for the second highest ranking candidate, except for 1998 where the highest error was for the winning candidate (Estrada). Average forecast errors tend to be lower with increasing sample sizes. In addition, it should be noted that these organizations use scientific methods based on chance to select their respondents to ensure representativeness of their respondents.  These two organizations have also fully disclosed their sampling methodologies.

In 2004, the SWS also conducted with ABS-CBN a day of the election household survey. For 2010, SWS conducted an exit poll with TV5, while Pulse Asia conducted an exit poll with ABS-CBN. My predecessor at the now defunct NSCB, Romy Virola, provided an assessment of the 2010 exit polls in his Sexy Statistics article, stating that “the two exit polls did very well in predicting the percentages of votes cast for the winning  presidential and vice-presidential candidates, missing by not more than 2 percentage points.”  

When sample surveys and polls involve a random selection of respondents, as in the case of polls of SWS and Pulse Asia, their results carry a margin of error and a level of confidence. For example, a voter preference poll of 1,600 respondents has a “margin of error” of about plus or minus three points for a confidence level of 95%.

This means that if the survey were to be replicated with the same protocol of randomly interviewing respondents, we would expect that in 95 out of 100 surveys, the poll results will be within three percentage points of what we would expect if we had instead surveyed everyone.

Why not survey everyone?

So, why not survey everyone instead, and why should we “trust” polls if they are only interviewing a mere 1600 respondents, when there are actually millions of Filipinos who will be voting on election day?  

Conducting a full census of voters can be quite costly. In addition, sampling theory, developed a century ago, has shown that one does not need to conduct a census to obtain information, i.e. conducting a sample survey will do just as well. Even hospitals extract only blood samples from patients to subject the blood to medical tests (rather than extracting all the blood of the patient to determine whether or not the patient is in a clean bill of health).

Sample surveys, if carefully designed with chance methods for selecting respondents, should be able to estimate the true proportion of people who would vote for some candidate, up to a margin of error. Of course, there is always the possibility of uncertainty, i.e. getting “by chance,” too many who favor the candidate, or the other extreme, too many who do not favor the candidate. In normal situations, however, we should be able to yield fairly reliable estimates of voter preference. 

Even sample surveys conducted by the Philippine Statistics Authority, from household surveys to establishment surveys, to agricultural surveys (that may involve households and establishments) are also using chance methods to select their survey respondents. 

The statistics generated from these PSA surveys, e.g., poverty incidence, average employment size, and agricultural production, also carry margins of error, as well as measurement errors arising from non-responses and coverage issues. But these sample surveys were designed to yield estimates of the characteristics of the entire population with some degree of confidence. In the case of polls, these sample surveys are meant to track measures of public opinion, especially of voter preference.

If polls have the same protocol for respondent selection, we should also examine the trend across polls instead of looking merely at one set of figures, especially statistics that we like best. That is, when there are several polls with the same sampling methodology, and if many of them say the same story but one looks very, very different, while it may be tempting to focus on the poll that has a different story, it is very likely that the majority of polls show the more accurate picture – note the term likely, as there is no certainty with statistics.

There has been some suggestion in the literature that some voters may be influenced by results of these public opinion polls. The trends in surveys conducted by SWS and Pulse Asia on voter preferences certainly have been far from constant even in the last election, with then Senator Manny Villar going on a downward slump for the presidency (see Figure 2), and then Mayor Binay on an upward trajectory for the vice presidency suggesting people’s voting preferences can change, all the way up to election day itself (see Figure 3). The camp of the Veep candidate Mar Roxas may have made the mistake then of being too comfortable that they had the lead on the vice-presidency and not examined trends regarding their closest rival, until election day results started to trickle in, and by then, it was too late.

Figure 2.  Voter Preferences for the Presidency in 2010 according to SWS and Pulse Asia

(a) SWS

 (b) Pulse Asia 

  

Figure 3.  Voter Preferences for the Vice Presidency in 2010 according to SWS and Pulse Asia

 (a) SWS

 (b) Pulse Asia

Effect on individual choices

New information provided to likely voters, including information about voting preferences from surveys, are viewed by some as having an effect on individual choices.

A classic paper in the 1950s by Herb Simon discussed the notion of a bandwagon effect (when voters shift their preference to the leading candidate, which can make forecasts of surveys rather self-fulfilling), or its opposite, the underdog effect (resulting from sympathy for a candidate that may be perceived as losing).

Studies and evidence though on the underdog effect is scant compared to the bandwagon effect, which is believed to arise because of social acceptance (when voters feel more accepted if they share the majority opinion), social learning (when voters believe there is information about the quality of the candidates from the mass support granted to the candidate). In addition, voters may also want to resolve cognitive dissonance (i.e. holding two or more contradictory beliefs) by supporting the candidate who will most likely win anyways.

Since survey results can possibly influence voter preferences, some countries have regulated opinion polls, particularly restricting their conduct or publication of their results close to the election day.

A 2012 global survey on election poll policies and practices of 85 countries, however, suggests that more countries (45) still had absolutely no restrictions on pre-election polls than those (38) that had some restrictions. 

Among the 38 countries that have an embargo on publishing poll results before an election, a large majority of them reported that the main enforcers are government agencies or election administration offices (87%) followed by independent agencies (5%) and broadcast/press regulatory agencies (3%).

The Philippines, based on a Supreme Court May 5 2001 ruling, has been on the side of the majority of countries allowing pre-election polls to be unrestricted.  Supreme Court rulings though have been overturned, so there is no certainty that this practice will continue in the country. 

Even if pre-election surveys may swing votes on one side or another, every piece of information on candidates, on their trustworthiness or their corrupt leanings, on their platforms of government, and on their popularity, should be given to the voting population.

Let voters decide based on every bit of information they can gather, including survey results, but let media be more discerning about the kinds of survey results they disseminate to the public for not all polls are accurate.

But most of all, let politicians be more analytical about poll results, lest they unwisely run for posts and lose. Better that they consolidate their forces with like minds and make ultimate sacrifices on their ambitions, else they may just be giving away the elections to the person they do not want to win. – Rappler.com

 

Dr. Jose Ramon “Toots” Albert is a professional statistician who has written on poverty measurement, education statistics, agricultural statistics, climate change, macro-prudential monitoring, survey design, data mining, and statistical analysis of missing data. He is a Senior Research Fellow of the government’s think tank Philippine Institute for Development Studies, and the president of the country’s professional society of data producers, users and analysts, the Philippine Statistical Association, Inc. for 2014-2015. He also teaches at De La Salle University, Holy Angel University and at the Asian Institute of Management.  

Add a comment

Sort by

There are no comments yet. Add your comment to start the conversation.