Decoding randomised controlled trials

Poverty is a complex, multi-dimensional issue. Is it amenable to solutions that deploy RCT?

In his 2017 book, Experimental Conversations: Perspectives on Randomised Trials in Development Economics, Timothy Ogden, an expert on financial inclusion, was looking for credible assessments on “randomised controlled trials” (RCT) to know “how we learn about the world, what evidence is and means, and how policy should and shouldn’t be formed” from the distinguished researchers who interact with RCT in every imaginable way.

Examples of RCT in real-life clinical trials can be traced back to the 16th century. Documented evidence of RCT in psychology and education in the late 19th century is available, and rigorous statistical formulation of RCT was done in the 1920s and 1930s. Later, in the second half of the 20th century, RCT became an essential tool — “rational therapeutics” — for clinical trials.

RCT has become an integral part of clinical trials due to the nature of the scientific question associated with the experiments. Suppose a new treatment/intervention is under experimentation for treating some unknown fever, and 80 of the 100 patients recover as a result of that treatment; can we safely conclude that the recovery rate of the treatment is 80 per cent? It is possible that a considerable proportion of patients recover even without any treatment.

Thus, to look at the true effect of the treatment, one needs an estimate of the recovery rate without “treatment”, and this may become available through enrolling some more patients into the study and not giving them any treatment. Suppose 60 out of 100 such “controlled” patients recover, yielding an estimated recovery rate of 60 per cent. The estimated “treatment difference” is thus 20 per cent, which is an indicator of the advantage of using “treatment” over “control”.

Now, the first 100 patients could be given the treatment, and the remaining 100 control, or vice versa. Any such prior knowledge of treatment assignment might induce “selection bias”. To circumvent that, one needs to employ “randomisation” — a random mechanism like tossing a coin or drawing a random number from a computer — to allocate patients in the treatment under experimentation and the control group. The procedure will then become a RCT.

It has been almost an ideological war concerning RCT in development economics for the last two decades or so. One group, called “randomistas”, considers RCT the holy grail of development economics, while the other, led by 2015 economics Nobel Prize winner Angus Deaton, has expressed reservations about RCT in terms of both philosophy and effectiveness. The future of development economics depends on who’s going to win this war.

Interestingly, about three years back, Esther Duflo had commented while batting in favour of RCT: “I think it’s been completely won in that I think it’s just happening...I think it is now understood to be one of the tools.” With this year’s Nobel Prize for economics having been won by three proponents of RCT — Michael Kremer, who is generally given credit for launching the RCT movement in development economics, and Abhijit Banerjee and Duflo — has the war over RCT been now won by the randomistas?

The “randomista” trio who won the Nobel economics prize: Abhijit Banerjee, Esther Duflo, and Michael Kremer

What if randomisation is not done in economic experiments? Certainly, selection bias would prevail. But, will that be very serious, especially if there is apprehension of imperfect randomisation in many experiments? Moreover, randomisation facilitates “blinding”, or masking of the identity of treatments from investigators, participants, and assessors in clinical trials, and thus reduces bias. It is impossible to ensure blinding in economic experiments, and proponents of RCT know that well. 

Also, advanced randomisation techniques like “adaptive randomisation” use accumulated data within the experiment to fix several features, such as the allocation pattern, test statistics and monitoring time. These are also very difficult to employ in social experiments due to their very nature. However, randomisation helps in using probability theory and statistical techniques for making inferences and finding standard errors and their estimates.

Interestingly, statisticians are usually an integral part of the clinical trials team, and its framework, design, randomisation, implementation, and data analyses are generally statistically rigorous, correct and of the desirable quality. The food and drug administration of the country concerned acts as a watchdog in such clinical trials. These are billion-dollar experiments, having a trillion-dollar market, for the benefit of billions of people worldwide.

Hundreds of thousands of clinical trials have been documented so far (by contrast, the number of RCT in development economics is less than a few thousand), mostly owing to the business interests of the pharmaceutical giants. Yet nobody ever thought that a Nobel Prize in medicine could be awarded for conducting such statistically accurate and precise life-saving experiments through RCT, having tremendous business potential. It is a century-old technique in statistics. By contrast, it is a daunting task to conduct RCT in different aspects of poverty all over the world.

The scientific question behind clinical trials is often unidimensional — the effectiveness of a new drug for a disease. By contrast, philosophically, economists like Deaton and many others believe that poverty is a very big and complex issue, with many inter-related components and dimensions. Every single event might have numerous different kinds of important (often long-term) economic and social impetus.

Now, in order to facilitate the use of RCT, poverty has been sliced and diced into numerous small parts by the randomistas. Thus, a complicated multivariate problem has been transformed into many univariate ones by ignoring the complexity defined by the inherent associations of these small parts. Are these univariate bits of “evidence” enough to solve the jigsaw puzzle of poverty? However, the randomistas never claim that they are out to solve the poverty problem completely; rather, they are interested to find evidence to eradicate some of those smaller parts.

The debate on RCT, and whether the future of economics is in good hands or in danger, will continue, with the randomistas in a more comfortable place since the announcement of this year’s Nobel for economics.    

The writer is professor, Indian Statistical Institute, Kolkata