MANILA, Philippines – A new study found that kalye surveys or on-the-spot street interviews, which continue to gain traction on YouTube, deviate from the gold standard of conducting surveys. They introduce bias, lead to viewer misinformation, and create groupthink, among others. 

The study, entitled “The Emergence of Political Polling Through Kalye Surveys,” was conducted by associate professor Ma. Rosel San Pascual, Anjenica Durana, and Myrnelle Cinco of the University of the Philippines Diliman Department of Communication Research.

Based on the tracking of the Philippine Media Monitoring (PMM) Laboratory since May 2021, kalye surveys started emerging as election-related content on YouTube around September 2021, and gained traction in terms of content uploads in October, when candidates filed their certificates of candidacy.

The PMM Laboratory had recorded over 350 election-related kalye survey videos on YouTube by January 2022. Using quarterly intensity sampling for their study, the researchers reviewed 43 videos from 10 YouTube channels with the most-watched kalye survey videos when election-related kalye surveys began surfacing as YouTube content.

The researchers outlined some of the ways in which kalye surveys deviate from the gold standard of conducting surveys. 

“The gold standard in surveys adheres to the principles of scientific and ethical practice across the entire phase of research…. When a survey fails to follow the gold standard of practice, survey results could lead to misinformation,” San Pascual said during a #FactsFirstPH briefing on this study on Friday, March 25.

While established public opinion research organizations are bound to a code of professional and ethical practice, kalye surveys are posited as an alternative to mainstream political polls. They try to project authenticity as respondents are filmed while responding, in contrast to mainstream public opinion surveys where participants are provided anonymity, the researchers said.

Since some say kalye surveys are “katuwaan lang (just for fun),” should they still be expected to follow the same gold standard? San Pascual said, “Our team believes so, especially if the intended outputs of the survey is to serve the purpose of public information and vote choice guidance.”

Methods that introduce bias

Researchers observed that kalye survey enumerators tend to conduct the survey in an unstandardized manner across all respondents, which includes practices like unsystematic respondent selection, inconsistent phrasing of questions and response options, arbitrary asking of follow-up questions, and non-uniform method of recording responses.

“These incidents tend to introduce bias, which compromises the objectivity of the survey,” their study said.

One enumerator they observed would ask seemingly pointed follow-up questions, such as, “Naniniwala ba kayo na siya (Candidate X) ang ina/ama ng bayan (Do you believe that [Candidate X] is the mother/father of the nation?)?” and “Kay [Candidate Y] ina/ama ng bayan, ayaw ‘nyo po (How about [Candidate Y], don’t you like him/her as mother/father of the nation)?” 

At times, they would also affirm the answer of the respondent, offer additional comments, or even feed possible answers to the respondent. One video even showed a survey enumerator arguing against a respondent’s choice. The researchers also noted instances of hyper-partisanship, wherein survey enumerators hyped up respondents’ answers when they supported a certain candidate.

‘Non-representativeness’ of samples

The researchers also found that while the main question asked in kalye survey videos is phrased using language accessible to respondents, some enumerators ask it in ways that go against the systematic prescription of the scientific process, risking the validity and reliability of the survey results.

Further, respondents for such surveys are selected through convenience sampling technique, meaning they are intercepted as they go about their daily lives in public spaces. This non-random manner of respondent selection “has repercussions on the result’s potential for generalizability,” the researchers said.

Some surveys even accept respondents who aren’t from the video’s claimed target locale population, with one enumerator saying, “Okay lang po kahit hindi kayo tagarito, kung meron kayong bet na presidente sa susunod na eleksyon (It’s okay if you’re not from here, if you have a preferred presidential candidate in the upcoming election).”

Sampling bias was observed in some cases, like one respondent leading the enumerator to an area she perceived to come across potential respondents who would favor a certain candidate. This further contributes to the non-representativeness of the sample.

Misinformation

The study also found that kalye survey content creators frame their videos in ways that may mislead and misinform viewers.

“While kalye surveys are generally packaged as an alternative to mainstream political polling, they are not excused from the standards expected from social science research methods given the agenda of their inquiry, the scope of their dissemination, and their potential to create actual voting effects. And while kalye surveys may be packaged as ‘for fun,’ they are prone to cause misinformation if handled carelessly. Any survey, if improperly done, can generate misinformation,” it said.

The researchers observed different styles of tallying responses across kalye survey videos. These include on-the-spot tallying of responses, reporting of frequency distribution towards the end of the video, while some did not do any sort of aggregation at all. Some were edited or spliced, thus casting doubt on whether or not all responses have been accounted for.

While the non-random respondent selection means data collected cannot be subjected to statistical techniques that will reveal voting patterns that are generalizable to the target population, video titles tend to frame surveys “as if they were representative of the voting population of the survey locale.” 

Some are also sensationalized, with some videos claiming a certain candidate wins in a “landslide” at a certain location. At least one video framed the title with a claim that was unsupported by the results presented. The thumbnail description in some videos also contains generalizations.

“Thus, survey literacy training can guide content creators on the value of properly reporting response patterns; otherwise, the reporting only leads to viewer misinformation,” the study said.

It added: “If the survey can only manage gathering data through non-random sampling, then the framing of the video title and the presentation of results should not mislead the viewers that the results represent the larger population.”

Groupthink

Most enumerators, the researchers said, would show a board that functions as their device for tallying responses for each candidate. Because of this, every succeeding respondent would see the latest tally of votes.

“The practice of using the board as a show card and a tally board does not only run contrary to the data privacy clause that is core to survey data collection, it also potentially creates a bandwagon effect (when respondents follow the vote of the majority) or underdog effect (when respondents opt to choose the minority) on succeeding respondents, which ultimately affects the validity and reliability of the survey results,” the study said.

They also pointed out that the non-private manner of conducting such surveys may undermine the independence of the respondents when they relay their preference. There are instances when other people hear and comment on a respondents’ answer.

“This manner of survey administration can create groupthink effect among the respondents, and some respondents may express a choice that appears to be socially desirable among the other respondents in the group. We noted occasions when some respondents changed their vote choice after they heard or saw the answer of another respondent. As such, the collected data may reflect voting preference that some respondents did not independently make,” they said.

Lacking informed consent

The researchers also raised concerns over violations of the ethical principles of informed consent, confidentiality, and data privacy.

They observed that enumerators would approach respondents and get them to participate only by announcing that they are conducting a kalye survey and asking the question on presidential vote choice, thus foregoing the process of obtaining informed consent from respondents.

“It is also quite concerning to observe how the reviewed kalye surveys appear to assume that the respondents consented to have their participation recorded on video, and worse, be later uploaded in YouTube. Ethical practice emphasizes that survey data collection should be treated as a private and confidential exercise,” their study said.

One respondent they noted said, “Naka-camera pala…. Baka pagalitan ako ah (I didn’t know it would be on camera…. I might get into trouble).” Another, upon finding out that the video would be uploaded on YouTube, said, “Baka mamaya ma-shoot-to-kill ako ah (I might get shot later on).”

There were also incidents when children’s faces were captured and weren’t blurred in the uploaded video. The inside space of some houses were also captured without apparent prior informed consent, the researchers said.

Given their findings, the researchers provided the following prescriptions for kalye surveys:

• View reality objectively to avoid bias.
• Ask valid and reliable questions to generate valid and reliable answers.
• Draw a representative sample to be able to make inferences about the population.
• Count incidences to make sense of observations.
• Obtain informed consent.
• Honor respondents’ voluntary participation and voluntary non-participation.

The researchers advocate for survey literacy not only for content creators but also for members of the public, as they are the ultimate consumers of survey reports. “Survey literacy should empower the agency of the public so that they can demand accountability among survey groups that deliberately misuse surveys for misinformation,” they said.

“While our article will surface incidents where kalye surveys tend to deviate from the gold standard of practice, we nonetheless optimistically believe that the gold standard can be learned and incorporated in any survey practice to validly and reliably contribute to sense-making.” – Rappler.com