“The Emergence of Political Polling Through Kalye Surveys” is a study presented by the authors in a #FactsFirstPH research briefing held on March 25, 2022. The full copy of the research is reposted with permission from the authors.

Second of 3 parts

Read the other parts of this series:
Part 1: The emergence of political polling through ‘kalye surveys’
Part 3: The ethics of ‘kalye surveys’

For surveys, the demand for scientific rigor is anchored on its positivist roots, which predisposes it to view reality objectively to avoid bias, to ask valid and reliable questions to generate valid and reliable answers, to draw a representative sample to enable making inferences about the population, and to count incidences to make sense of observations.

Prescription 1. View reality objectively to avoid bias

In order to view reality objectively, surveys utilize a standardized questionnaire, which contains a standard set of thoroughly-designed questions and, if applicable, a standard set of meticulously-crafted response options, that is then administered to an adequately-sized and randomly selected set of respondents that represent a target population. This safeguards the survey from subjective biases that may arise in questionnaire construction and administration as well as in respondent selection. Unbiased questionnaire construction and administration and unbiased respondent selection allow for impartial data collection and documentation.

Our review of kalye survey videos revealed the survey enumerator’s tendency to conduct the survey in an unstandardized manner across all respondents – e.g., unsystematic respondent selection, inconsistent phrasing of questions and response options, arbitrary asking of follow-up questions, non-uniform method of recording responses. These incidents tend to introduce bias which compromises the objectivity of the survey.

It is rather common across kalye surveys for the survey enumerator to inconsistently phrase the main question. For instance, one enumerator would ask, “Survey lang, Sir, sino presidente ‘nyo ngayong darating na halalan?,”  then would shift to another variation when asking another respondent, “Sino sa anim ang napupusuan ‘nyong iboto ngayong darating na halalan?,” or any other variations of these. We also observed an enumerator inconsistently phrase the follow-up question using these variations: “Dito, Sir, wala kang tiwala sa iba?;” “Naniniwala ba kayo na siya [Candidate X]  ang ina/ama ng bayan?;” “Kay [Candidate X], ina/ama ng bayan, ayaw ‘nyo po?.”

There are also occasions when a survey enumerator would affirm the expressed vote choice of the respondent; for example, a respondent said,  “Hindi siya mangungurakot,” to which the enumerator replied,  “…talagang mababago ang Pilipinas.” It is also common for some enumerators to offer additional comments, “Kay [Candidate X], ma’am, magbibigay daw ng bahay.”

When a respondent commented, “Hindi ko bet ang babae,” the enumerator replied, “Ah, so gusto mo dapat yung matapang.” Some enumerators or even some film crew would feed possible answers to the respondent. In one instance, as the respondent was forming her answer to the follow-up question, which asked why she did not choose a particular candidate, the film crew supplied, “Mag-aral daw muna kasi,” which was then echoed by the respondent,  “Mag-aral muna.”

We also noted occurrences of hyperpartisanship. We caught survey enumerators who hyped up the answer of respondents when it favored a particular candidate. For example, one enumerator, in an attempt to prod the respondents to repeat their vote choice in a louder and more emphatic manner, would repeat the question, “Sino ulit iboboto mo?” In another instance, an enumerator would allow the group of respondents to chant and cheer for the favored candidate’s name.

In one video, we caught a survey enumerator arguing against the vote choice of a respondent. This video was edited to also incorporate video clips of the favored candidate’s supporters that appear to echo and support the enumerator’s arguments.

In another occasion, the survey enumerator disclosed his presidential bet when the respondent asked him about it,  “[Candidate X] po ako.”

Thus, survey literacy training can help explain the importance of objectivity, sharpen the skills necessary for designing and implementing objective surveys, and introduce samples of codified protocols that outline how to professionally conduct surveys.

Moreover, since content creators function as enumerators, literacy training can also help them realize how they can affect respondents’ answers, which consequently introduces bias, and guide them on how such can be avoided when surveys are professionally administered. For instance, enumerators must avoid behaviors such as reacting to respondents’ answers, giving side comments, and offering interpretation to respondents while the survey is ongoing.

Prescription 2. Ask valid and reliable questions to generate valid and reliable answers

In order to ask valid and reliable questions, survey instrument development needs to follow the deductive approach, which is a scientifically-guided transformation of concepts into variables and variables into measures. This process produces measurement validity and reliability, wherein a questionnaire contains a set of well-calibrated questions and response options that is capable of sufficiently measuring the variables that the survey intends to measure at a certain point in time.

A survey questionnaire with measurement validity and reliability has been pretested to confirm its capability of being standardly administered across the survey’s adequately-sized and randomly selected set of respondents. Since a survey questionnaire also functions as a communication device, a questionnaire with measurement validity and reliability has been checked for its capability to effectively and efficiently articulate the set of instructions, questions, and response options to its target respondents. Hence, the scientific process produces a survey questionnaire that is not haphazardly designed. By extension, the scientific process also calls for a systematic administration of the survey so that the set of survey questions and response options is not carelessly asked. Asking valid and reliable questions and presenting valid and reliable response options facilitate the generation of valid and reliable answers. 

Our review of kalye survey videos bared that the main question is essentially phrased to draw presidential vote choice in a language that is accessible to the respondents. However, some enumerators asked this question in an open-ended manner, some aided respondents with a list of candidate options, while some offered options only after being prodded for it by the respondents. As earlier pointed out, there are enumerators who were inconsistent in their question and response option phrasing. There are also enumerators who arbitrarily asked some respondents to explain their vote choice. The capricious pattern of asking questions,  providing response options, and adding follow-up questions runs contrary to the systematic prescription of the scientific process, which clearly risks the validity and reliability of survey results.

Survey literacy training should then instruct kalye survey content creators on how to properly phrase vote choice questions and response options. Content creators can follow the lead of mainstream polling groups when doing so. They may even consult peer-reviewed studies to gather possibilities for question and response option phrasing. These options, which have been tried and tested, are publicly accessible and may be used in rolling out kalye surveys.

The value of careful and standard administration of questions across the entire pool of respondents must also be emphasized to content creators. The exact phrasing of questions and response options should be read like a script across the respondent pool, without deviation. The standardized manner of survey administration helps safeguard the survey from any subjective biases that may emanate from inconsistent modes of questioning and response option administration.

Prescription 3. Draw a representative sample to be able to make inferences about the population

A survey’s sampling design should generate a representative sample, which is basically an adequately-sized and randomly selected set of respondents from identified segments of the target population. Having a representative sample is a requirement to be able to test whether survey results can be used to make inferences about the larger population where the sample came from. From such testing, called inferential testing, decisions can be made whether results computed from the representative sample are within an acceptable confidence level and tolerable margin of error, and thus, can be used to describe the characteristics of the target population as well. Essentially, a survey has achieved external validity when results obtained from its representative sample can be used to make inferences about the target population.

It is evident in the kalye survey videos that respondents are selected through convenience sampling technique as survey enumerators intercept prospective respondents as they go about their daily lives – e.g., on the streets, in public markets, around food stalls, in a public transport terminal. This is a non-random manner of selecting respondents, which has repercussions on the result’s potential for generalizability.

There are even instances where some intercepted respondents are not residents of the video’s claimed target locale population. We captured an enumerator saying, “Okay lang po kahit hindi kayo taga dito, kung meron kayong bet na presidente sa susunod na eleksyon.”

As such, kalye surveys should be careful when framing the results of their survey so as not to misinform the viewers that their results speak for a group beyond the sample of respondents.

Can surveys with respondents drawn from non-random sampling techniques be considered scientific? While the gold standard prescribes random sampling, social science research does allow for non-random techniques. However, data from non-random samples must only be used to describe the pool of respondents and not the larger population. In fact, we observed that there are some kalye survey videos that did not attempt to generalize their findings, which is the ideal and recommended track to take, given that the members of their sample were non-randomly drawn. One enumerator even provided a disclaimer that the results of the survey cannot be expected to exactly predict election results, “Unfair naman sa mga kandidato kung sabihin kong sure na yan, [Candidate X] na yan.”

We also noted one occasion when the survey enumerator allowed a respondent to take over survey enumeration. This respondent took the whiteboard from the enumerator and proceeded to lead the enumerator to a location where she perceived to intercept potential respondents who will favor a particular candidate. The respondent remarked: “Kaya kita sinamahan, maraming maka [Candidate X] dito… sa labas puro [Candidate Y].”  The enumerator and the crew then followed to film the respondent. This episode showed the purposive manner of selecting respondents, which introduced sampling bias, contributing even more to the non-representativeness of the sample.

Can surveys with respondents selected through purposive sampling be considered scientific? It must be noted that purposive sampling is associated with qualitative forms of data gathering, such as ethnographic interviews, unlike surveys which fall under the domain of quantitative research.

While external validity can no longer be expected from kalye surveys that followed non-random sampling techniques, there will always be an expectation that surveys will follow the protocols for objectivity and adhere to the principles of measurement validity and reliability.  Doing so will enable counting of recurrence, even if such accounting can only describe the members of the sample.

Then again, can surveys be designed and conducted just for fun, or as a survey enumerator would say, “Katuwaan lang”? There are certainly pop-surveys that abound. However, when the subject of surveys involves important matters of public interest, then its design, administration, analysis, and dissemination must be founded on much care and responsibility.

Prescription 4. Count incidences to make sense of observations

To make sense of survey results, responses generated from an adequately-sized and randomly selected pool of respondents are counted. Given the random nature of respondent selection, statistical techniques are applied to assess whether the observed patterns from the adequately-sized and randomly drawn sample can be generalized to the target population.

Our review of kalye survey videos uncovered different styles of counting frequency of responses – e.g., some videos feature on-the-spot tallying of responses, some include reporting of frequency distribution towards the end of the video, while some did not perform any form of aggregation at all.  As some videos were edited or spliced, we are uncertain if all the responses have indeed been accounted for in the videos.

Kalye surveys should be careful when framing the results of their survey so as not to misinform the viewers that their results speak for a group beyond the sample of respondents.

Analysis-wise, given the non-random nature of respondent selection, the data acquired from kalye surveys cannot be subjected to statistical techniques that can assess whether the voting patterns observed in the sample can be generalized to the target population. Then again, our review of kalye survey videos revealed the tendency of the title of the videos to frame the surveys as if they were representative of the voting population of the survey locale (e.g., “Nagsalita na ang [Location]”).

These videos also tend to adopt a sensationalized manner of phrasing the video title: “Landslide sa [Location];” “Kalye survey [part X] titindig ang balahibo mo sa survey na ito sino ang presidente ng [Location];” “Kalye survey [part X] grabe napuno ang listahan sino kaya ang sinisigaw ng taga [Location].” We also observed at least one video with a title that claimed something, e.g., “May tumabla kay [Candidate X],” even though the results presented in the video did not support this claim.

Apart from the title, the thumbnail description of the video sometimes also contains generalizations: “Landslide sa [Location];” “[Candidate X] Number 1 sa [Location].” There is also an instance when the survey enumerator’s opening spiel hinted at a representative survey: “Aalamin natin sino ba ang sinisigaw ng mga taga [Location] para maging presidente sa susunod na halalan.” This video is accompanied by a description that promotes its supposed representativeness: “Ating pakinggan ang tinig ng ating mga kababayan mula sa [Location] kung sino ang gusto nilang maging susunod na Presidente ng Pilipinas.”

We also noted several occasions when the survey enumerator offered generalizations of the result while the survey was still ongoing: “[Candidate X] talaga malakas pati sa [Location];”

“Landslide si [Candidate X]!;” “Lamang po si [Candidate X] sa [Location];” “Taumbayan nagsasalita, si [Candidate X] malakas, hindi po si [Candidate Y].”

Thus, survey literacy training can guide content creators on the value of properly reporting response patterns; otherwise, the reporting only leads to viewer misinformation. Survey literacy training should underscore to content creators that, even if results cannot be generalized to the population, all responses should be accounted for when reporting results, including the undecided and the non-response.

Moreover, emphasis must be placed on explaining to the content creators the analytical impact of the non-random manner by which they typically select their respondents. Fundamentally, a non-random sampling scheme can only deliver results that describe the non-random sample and not the larger population. Hence, if the survey can only manage gathering data through non-random sampling, then the framing of the video title and the presentation of results should not mislead the viewers that the results represent the larger population.

In the third and last part of this series, we will be presenting instances where kalye surveys tend to deviate from the ethical standard of practice. We will also be offering two additional prescriptions that address these deviations in order to design, administer, analyze, and disseminate ethically-grounded surveys. – Rappler.com

Ma. Rosel S. San Pascual is an associate professor at the Department of Communication Research of the University of the Philippines Diliman College of Mass Communication and a member of the Philippine Media Monitoring Laboratory. She primarily teaches quantitative research methods, measurement and sampling, and quantitative data analysis in both undergraduate and graduate levels. She also actively engages in academic research on communication and new media, online discourse, media effects, mass media, ICT and development, political communication, and transnational migration and family communication.

Anjenica U. Durana is an MA in Communication student at the Department of Communication Research of the University of the Philippines Diliman College of Mass Communication. She previously received her Bachelor of Science, Major in Psychology degree from Ateneo de Manila University, and has been a licensed psychometrician since 2017.

Myrnelle A. Cinco is currently aiding the Department of Communication Research of the University of the Philippines Diliman College of Mass Communication in a number of research projects. She is also a research assistant for an international nonprofit organization working on human rights. She has a background in data analytics, data science, and research. She has a bachelor’s degree in communication research from the University of the Philippines Diliman.