Why can't models get Covid-19 predictions right? Should we rely on them?

The models did work to a certain extent. The purpose of the models was not to do long-term forecasting but to inform public opinion and guide policy decisions

In May last year, as the pandemic was raging, the Department of Science and Technology (DST) constituted the Indian National Supermodel Committee to track the pandemic's evolution and recommend future policy interventions.
 
India hit its peak in September 2020, and the National Supermodel Committee report, released in October 2020, confirmed that India had indeed hit its peak of infections in September. The report further said that if all protocols were followed, the pandemic would be controlled by February 2021, and there would be minimal active symptomatic infections.
 
A few months earlier, a more rudimentary model had erroneously projected the end of the pandemic by May.

 

A Tata Institute of Fundamental Research (TIFR) study in November 2020 predicted a decline in infections in Mumbai by January 2021, saying that the epidemic would be under control by then.
 
It is not just models in India; modelling efforts from across the world predicted at one point or another that the pandemic would end soon or infections would fall to negligible levels.
 
The only saving grace for modellers and forecasters as countries have dealt with waves of infections has been the Latin phrase "ceteris paribus", meaning all other things being equal.
 
While model projections consider the current conditions to predict future outcomes, they have to rely on certain assumptions to make the models easily understandable and decipherable. So, when the National Supermodel Study projected that the cases would end in February or the TIFR study iterated that the virus would largely be under control in Mumbai by January, they had certain presuppositions in mind — people would follow social distancing norms, and there would be less slippage in Covid-appropriate behaviour or that there would be a calibrated reopening of the economy.

 
"Model outputs are a result of the assumptions made by the modeller. Early on for India, modellers did not sufficiently consider the scenario that when part of the country opens up further around February, people's behaviour would have changed. Lulled into complacency due to declining cases, people would assume that the disease had resided, and would lower their social distancing, mask compliance behaviour by January this year," opines Sandeep Juneja, head of the TIFR study.
 
The Supermodel, in the worst outcome, had stated that relaxation in protective measures could lead to a rise in Covid-19 infections again, and the country could record nearly 2.6 million infections a month.

 
Today, India is well past that mark. The country has averaged 233,091 a day cases in the past seven days and at this pace it would record 7 million infections in a month.
 
Moreover, while nearly all models did predict a second wave, none envisaged that the wave would be much worse than the first one. The National Supermodel Study had predicted an average of 87,000 cases daily — much lower than the peak of 93,000 cases the country had averaged for a week in September.



"All of them (models) have to be fitted to data and our understanding of disease transmission in a particular context. Second waves have been larger than the first in some contexts, and variants add another layer of complexity here. The most difficult variable in these models is human behaviour. Typically, people respond to increasing prevalence by taking precautions, but when testing levels are low, there is a false sense of security from vaccines or there is public messaging that Covid-19 is not harmful. That comes in the way of this feedback," says Ramanan Laxminarayan, founder and director of the Center for Disease Dynamics, Economics & Policy (CDDEP), Washington, DC.

 
To understand what did work and what didn't, it is important to decode how models work.
 
Modelling a virus
 
The most common and widely used epidemiological models are ‘suspected, infected and recovered’ (SIR), or ‘suspected, exposed, infected and recovered’ (SEIR) models. Using the SIR model, as the infected population and the recovered population rise, the susceptible population decreases, and researchers can predict the disease's final outcome and duration. The SEIR model adds another dimension to this by adding 'exposed' parameter, which covers people who have been exposed to the virus but are not infected. This adds more certainty to modelling outcomes. The initial pandemic models relied on these two approaches to determine the shape of the curve. Usually, the trajectory of daily infections in these models follow a bell-shaped curve. However, SARS Cov-2 has behaved a bit differently. There is another dimension to Covid-19 – the asymptomatic people, which the models mentioned above do not cover. So, many researchers constructed a ‘suspected, asymptomatic, infected and recovered’ (SAIR) model, which accounts for asymptomatic people.

 
The National Supermodel Committee has relied on yet another variation of the SAIR model devised during the pandemic. Manindra Agrawal, Madhuri Kanitkar and Mathukumalli Vidyasagar developed a new modelling technique called ‘susceptible, undetected, tested (positive), and removed approach (SUTRA), which takes into account the gaps in testing and those who are infected but are undetected; it also takes into consideration new variables like reach.
 
The other approach that has become common for limited geographical spaces is the agent-based model. While there are only a limited number of parameters in SAIR or SEIR modelling, agent-based models can cover a whole set of parameters and determine the impact of each on the spread of the pandemic. In the case of India, the Tata Institute of Fundamental Research followed this approach for determining the trajectory of Covid-19 infections in Bengaluru and Mumbai. The advantage of these models over the SAIR models is that they allow researchers to experiment with various scenarios. So, the TIFR study played with assumptions regarding the phased reopening of offices, schools and trains to determine the rate of fatalities and hospitalisation in the cities. The TIFR model for Mumbai had estimated that the transmission coefficient for schools and homes would be higher than the transmission coefficient at the workplace or in the community. In its initial phase, the model did not determine the impact of local trains, which are the city's lifeline, but in later studies, it estimated a limited spike with locals operating on a limited capacity.

So, is one model better than the other?
 
The agent-based models consider a variety of parameters and hence are more detailed than SAIR or other modelling techniques.
 
"Our projections were quite detailed and accurate till January. This was because ABS allows more detailed modelling of interventions and people's behaviour as well as behaviour change (reduction in compliance, increase in mobility, etc). Another advantage is that one can do more detailed what-if analysis using ABS," says Juneja.
 
But SAIR models have their own advantages.
 
Manindra Agarwal, professor, IIT Kanpur and one of the SUTRA technique co-creators, differs in opinion. "Agent-based models have a very large number of parameters, most of which need to be guessed. Such models do not work well on large regions due to significant heterogeneity. Our model, SUTRA, is designed to have very few parameters so that their value can be learnt from data. This makes SUTRA suited for large regions (states, countries, etc)," he points out.

 
Ramanan Laxminarayan says that there is a utility to both. "We prefer agent-based models when the data to back these models are available. That is not always the case, and I'm not sure it makes sense to depend on just one category of models."
 
Did it all go wrong?
 
"All models are based on assumptions and are only as good as the assumptions. Further, pandemics are fuelled by human behaviour, which is unpredictable and hard to model," says Shahid Jameel, director, Trivedi School of Biosciences, Ashoka University. "Most models did not take into account the generation of viral variants that would be more infectious and also escape prior immunity to various degrees," he adds.

 
The models did work to a certain extent. The purpose of the models was not to do long-term forecasting but inform public opinion and guide policy decisions.
 
"Many epidemiologists had warned the population to not let their guard down precisely because new variants could show up, reinfections could take off. In future, we need to heed these warnings more. We also need to be adequately sceptical of longer-term model forecasts because many unknown-unknowns can surface," says Juneja.
 
The TIFR model was used extensively by the government till January to predict trajectories.
 
However, some of the considerations did go out of the picture.

 
"Modellers also didn't know how to account for new variants that may surface and that they may be more infectious or more virulent. This is an understandable oversight," Juneja highlights.
 
However, K Srinath Reddy, president Public Health Foundation of India (PHFI), holds a different opinion. While he believes that the new variants and laxity in following social distancing norms and Covid-appropriate behaviour have also contributed to rising infections, he says that faulty assumptions have also played a huge role in not getting correct forecasts.
 
"Many people thought herd immunity would come. I didn't believe that and argued against it. Many people took it for granted and they started going into theories of how much of herd immunity might have been contributed by India-specific genetic factors and so on and so forth. So, I think curves were fitted with their perception," he says.

 
The role of public transport
 
The TIFR study on Mumbai elucidated three scenarios in case local trains were allowed to operate. In all three scenarios, the study projected a marginal rise in cases on restarting the locals. The high-contact-risk scenario was to have the highest jump in cases leading to a smaller second wave.
 
"Our model did take restarting locals into consideration. If you see our October report, we projected that the opening of local trains and the economy would lead to a relatively small (compared to the observed) second wave. Our modelling was under the assumption that people's behaviour would not change. We were very clear in the report that laxity in people's behaviour would lead to higher infection numbers than projected. In our paper, we outline how we model infection spread in the trains. This appeared to be a reasonable methodology, although it's hard to validate it to data (given that there may be many other new reasons for the increase in infections after February)," Juneja says.

 
However, given that data on contact tracing are limited, it cannot be said for sure that the trains did become superspreaders.
 
Another study last year by Laxminarayan had reached a similar conclusion as the TIFR study.
 
"Last November, we published our first paper from the largest Covid-19 contact-tracing study in the world, and we also found that the likelihood of transmitting Covid-19 when travelling with someone was much higher than any other sort of interaction. But decisions to run trains are meant to balance economic considerations against disease risks. Trains can facilitate superspreading, but it takes a very good contact tracing operation to pick that up. I'm not sure that anyone has systematically looked at secondary attack rates in the context of train travel or their role in superspreading."

 
"The longer the journey, the higher the risk. Local trains in Mumbai may have been only one reason. More infectious variants and most people not wearing masks compounded the problem," Jameel concurs.
 
But there is a contrarian view that Agrawal presents. Although he agrees that public transport leads to a faster spread, the second wave, he says, "is primarily driven by people who do not use public transport."
 
"The ones who do seem to have become immune already during the first wave. That is why one does not see trains causing superspreading events (same logic appears to hold for election rallies as well)," he points out.

 
Silver lining
 
Models may have failed to correctly predict the second wave or factor in various assumptions that came with new variants and people not following Covid-appropriate behavior. But that does not mean an end for modelling.
 
"When we model, our aim is to address somewhat immediate scenarios. It is difficult, somewhat unrealistic, and not entirely desirable to play out all possible future eventualities. Some of these what-ifs may distract from more urgent actions that policymakers need to focus on," Juneja says.
 
Besides, rising cases have led to a race in modelling again, with forecasters predicting when the second wave would subside.

 
"We have been predicting the trajectories of various states and some cities since last month," says Agrawal. He has been posting his findings on his Twitter handle and predicts that the second wave would be over by the end of this month or first week of May for most states.
 
Given that most states and Union Territories are going into a lockdown and governments have become strict again, cases are bound to fall. But will there be a third wave? More importantly, will the modellers get it right this time?