[OPINION] The science and art of pandemic modeling in the Philippines

In recent days, much has been said and written in the media about the pandemic models and forecasts of the Octa Research team, of which I am a grateful member. I am responsible for Octa’s modeling of hospital occupancy and health care utilization (HCUR) during this pandemic, so I thought that it may be helpful to explain to Filipinos what and why I do what I do, especially since I am doing this for them. 

I begin with an important quote attributed to the British statistician, George E.P. Box. He once said: “All models are wrong, but some are useful.” This is the underlying truth that grounds all mathematical modeling. Modelers start off their work knowing that their models will never be able to capture the complexities of the real world. I know that my models cannot completely explain the COVID-19 pandemic in the Philippines. The pandemic is ever changing and ever evolving, not only because the virus is ever changing and evolving, but also because our strategies to manage the pandemic are also changing and evolving. However, models can be good enough to be useful. That is my goal.

To determine if a model is useful, we have to test its predictions. We have to test its ability to make forecasts. Take the simple spreadsheet model for hospital occupancy and HCUR that I developed earlier this year to anticipate the March surge in the NCR, which was driven by the Alpha and Beta variants of SARS-CoV-2. Building on pandemic projections calculated by my colleague, Professor Guido David, this spreadsheet Excel model simulated hospital occupancy every day in the NCR using data for hospitalization and death from COVID-19 in the Philippines that I had obtained from the DOH, from the Quezon City CESU, and from published scientific reports. 

During a March 19, 2021, press briefing hosted by Octa, I presented forecasts that suggested that our hospitals in the NCR would exceed critical capacity during Holy Week. I displayed a series of potential future scenarios for a range of reproduction numbers in the NCR that ranged from 1.5 to 1.9. Regardless of the actual and true reproduction number in the NCR at that time, however, the forecasts were clear that the hospitals of the NCR would be overrun by Easter and we called the national government to undertake drastic measures to prevent this from happening. 

We now know what did happen: no drastic measures were imposed for another 10 days or so. At that point, we had wasted those 10 days, and we had effectively lost control of the pandemic. Tragically, because of that delay, our kababayans in Metro Manila had to endure weeks of hard lockdown and long waiting lists for admissions to our hospitals.

Our critics have argued that our use of the reproduction number, also known as R, is flawed because of the two-week delay in the DOH data. One of our most vociferous interlocutors has said: “Even a student of clinical epidemiology will tell you that that is not valid because cases that happened within the last two weeks are subject to error, because some get tested later or earlier. There are backlogs, so if you are making projections based on data that is incomplete or erroneous, you are going to come out with erroneous projections.”

Yes, there are shortcomings and limitations to the DOH data, but to account for them, we consider a range of projections that take into account the possible values of the true reproduction number at a given moment in time. This is what scientists do to account for uncertainty. Nonetheless, regardless of the weaknesses in the DOH data, our HCUR model consistently predicted that our hospitals in the NCR would be overrun regardless of the exact, true value of R, whether it was 1.5 or 1.9 or anything in between. In the end, history proved that our projections were not erroneous. Sadly, Filipinos suffered and died because the projections were ignored. Remember: “All models are wrong, but some are useful.”

We presented HCUR projections again during an April 3, 2021 press briefing also hosted by Octa. At that time, the NCR had just been returned to its second ECQ and many people were wondering about how long the hospital crisis would last. My spreadsheet model forecasted that hospital occupancy in the NCR would peak on April 15, 2021, and that the HCUR would not fall below the critical 70% threshold before the end of April. 

We now know what did happen: according to DOH data, hospital occupancy peaked in the NCR on April 15, 2021, with 8,019 cases. Hospital occupancy fell below the 70% critical threshold around May 6, 2021. Our simple spreadsheet model for HCUR had proven to be useful. 

I will be the first to admit that my models for hospital occupancy in the Philippines are relatively crude. However, they are crude for a reason: last year, I had developed a more elaborate and sophisticated model for the pandemic in the NCR based on an MIT prototype that had incorporated machine learning. It was a really cool, cutting-edge model that was peer-reviewed and described in the Philippine Journal of Science. But in the end, the model struggled with the shortcomings, messiness, and inherent flaws of the DOH data. That experience taught me that in the Philippines, simpler models that are useful are better than sophisticated models that are not. Deciding which approach to take in light of the problems associated with the available DOH data is the “artistic” part of pandemic modeling. 

As I write this, the NCR has returned to its third ECQ of the pandemic. Last week, on July 28, 2021, my Octa colleagues and I called for a circuit breaker lockdown because my hospital occupancy forecasts, this time based on the experience of the Delta surges in Thailand, Malaysia, and Vietnam, all indicated that our hospitals would be overrun by the middle to end of August. I used the data from our ASEAN numbers because I could not rely on the DOH data for the prevalence of the Delta variant in the NCR. 

Presenting simulations using the experience from three ASEAN countries with different vaccination rates, different lockdown strategies, and different prevalence for Delta, would give us a range of possible forecasts for the Delta surge in the NCR. Once again, despite their differences, all three simulations converged into one conclusion: without drastic measures, the pandemic would inevitably lead to overwhelmed hospitals, crowded ERs, and crushed healthcare workers by the end of the month.

I am grateful that the forecasts of the modelers working with the national government have corroborated our findings and their released forecasts also predict that our hospitals will be overrun within weeks. But this present-day experience has left me with lingering questions: If the government modelers today can foresee the catastrophic effect of this Delta surge on our hospital system weeks before it happens, what did they see before the Alpha/Beta surge in March? Why did they not raise the alarm then as they are doing now? 

Instead, why did they believe that the pandemic was under control? Why did one of our critics tweet on February 28, 2021, at what was already the beginning of the Alpha/Beta surge in the NCR, that we had contained the pandemic: ““Even WHO says so. You naysayers won’t be content with what’s in front of you. That’s your problem not ours. We save your butt. You’re welcome.” Was this “low risk” scenario a forecast of his self-proclaimed world-class pandemic model? If so, it was erroneous. It was not useful. And sadly, Filipinos died because of it.  

Clearly, all models are wrong, but some are deadly. – Rappler.com

Reverend Fr. Nicanor Austriaco, OP is Visiting Professor of Biological Sciences at the University of Santo Tomas, and an Octa Research Fellow.

Voices features opinions from readers of all backgrounds, persuasions, and ages; analyses from advocacy leaders and subject matter experts; and reflections and editorials from Rappler staff. 

You may submit pieces for review to opinion@rappler.com.