This is a #PHVote newsletter sent to subscribers on May 15, 2019.

I’m Gemma Mendoza, Rappler research head and leader of the Rappler team that processed election results from the Pope Pius Catholic Center.

Monday, May 13, was the third elections when Rappler delivered live election results from the Transparency and Media servers of the Commission on Elections (Comelec).  

After months of follow-ups, we got included in the resolution which outlined rules for accreditation and listed the initial groups that got accredited.  

We had our election results microsite up and tested for various scenarios. We submitted our laptops and USBs to the Comelec’s IT division. We got ourselves a small office space at the Pope Pius Catholic Center just above the server room.  

Our team was ready for what we expected to be a literal relay process: with two persons inside the server room, runners were ready to bring the USB containing results files to our room upstairs, where we had a computer that did the results processing and uploading.  

This might sound strange in this age of cloud computing, but the law forbids any direct connection from the internet or any 3rd-party computers to the election servers.

Laptops submitted to the Comelec go through the digital equivalent of sanitization: all files inside, including the operating systems, are wiped out and replaced with a software that can only do these: reformat connected USBs, receive the files, check the file hash, and copy the results files to the USBs.  

Even USBs we used to copy the files from those sanitized laptops go through the same process. 

We knew it was going to be a long night. We have trained ourselves to expect anything. Experience from monitoring past elections told us to expect anything. We had a notebook documenting observations, for future reference.  

But we did not expect that it would go down the way it did last Monday night.  

For 7 hours after the first file was released to our laptops in the server room, the system failed to cough up results files. For the first couple of hours, no explanation was given other than the file being too big. 

Finally, at past 9 pm, the Comelec project team leads came in to explain the situation. They showed us what looked like a dashboard that was being refreshed every few minutes. It listed down all the regions and included columns showing voters who actually voted and the percentage of results received for each region. 

By that time, over 90% of precinct-level results for Metro Manila were already in. 

At past 1 am – roughly 8 hours after the precincts closed – we got our second and third files. Those files, however, already contained over 90% of expected precinct level results. In effect, tapos na ang boxing.

Any person who has worked on digital systems knows that things could always go wrong when they go live. But then again this is why people managing important, public-interest-imbued systems should be ready for anything.  

Systems need to be stress-tested. This is even more critical for a massive system like the Philippine Automated Election System, where any simple glitch could be amplified and used in electoral protests. (READ: Election data quash Marcos’ cheating pattern claim) 

There are things we know of already, as correctly pointed out by one Facebook user: we know how much data will be generated, we know from data analysis the pace at which this data will be received by the system. These should be used in stress-test scenarios ahead of elections.   

We hope the Comelec learns from this. To their credit, they appeared to have considered lessons learned from previous elections. (READ: Was transmission of 2019 PH election results better or worse?)

They have to do better. The country needs them to do better. 

The integrity of our elections depends on this. – Rappler.com 

Here’s a quick guide to Rappler’s election pages:
LIVE UPDATES
RESULTS OF LOCAL RACES
STORIES, VIDEOS, ANALYSIS, DATA