No method in the madness

A new critique of the V-Dem democracy rankings highlights where the problem lies

India is surely a “flawed” democracy. We need not have any doubts on that score, for almost no democracy is perfect. However, the Narendra Modi government has been unduly upset that many “independent” think-tanks, from Freedom House to the Economist Intelligence Unit, to the Swedish V-Dem (Varieties of Democracy), have been consistently downgrading India’s rankings on the democracy indices for some years now.

Two points are worth making upfront. First, the government should not dismiss such reports merely because they are adverse, even if they are problematic. And, if we think the methodology is flawed, we should point out the flaws and engage with the rating agencies to correct them before dismissing them as motivated.

A problem arises when reasoned critiques are dismissed out-of-hand by these ranking agencies purely because the objections are coming from officials close to the government. V-Dem’s Director Staffan Lindberg in a recent interview asserted that his organisation’s rating of India was “fair”. He pooh-poohed allegations that the experts used by V-Dem to rank India at a lowly 93 in 2022 may somehow be biased, especially when two critics of the rankings, Sanjeev Sanyal and Akanksha Arora, mentioned in passing that Lesotho, which faced a coup in 2014, was ranked much higher than India. Mr Lindberg hinted that there may be something racist about the comparison with Lesotho. (You can read the full text of his interview at, https://bit.ly/41tvmD8). He claimed V-Dem used “high math” to arrive at its rankings, hence could not be faulted.

Mr Lindberg disclosed that the experts used to code three of the five subjective parameters — all anonymous, and possibly for good reason — were “academics who are experts in their field and have published scientific articles on elections in India or the judiciary in India. They have verified scientific knowledge about this specific area for this specific country.”

This is a bit rich, for any conclusion about what is happening in a complex polity like India cannot be pure science, but applied behavioural science, where biases can creep in inadvertently. You don’t need to be consciously biased in order to downgrade or upgrade a country.

Illustration: Ajay Mohanty

V-Dem’s balloon is about to be punctured by a research paper by Salvatore Babones, director of the Sydney-based Indian Century Roundtable, which debunks the methods used to rank India on democracy. You can read the full report, Inside The V-Dem Rankings (https://bit.ly/3Z8ReSF). Professor Babones is currently researching a book on Indian democracy.

Professor Babones notes that the V-Dem rankings are based on five sub-indices, namely, 1) elected officials, 2) universal suffrage, 3) clean elections, 4) freedom of association, and 5) freedom of expression. The first two criteria are seriously flawed and the last three leave the doors wide open for biases. For example, the head elected officials and universal suffrage are scored on the basis of an analysis of the constitutions of the countries being ranked. In short, if the constitution of a country claims it “elects” its officials, it gets a perfect score. Among those countries that got a perfect score on this criterion are Cuba, Libya, Russia, Syria and Vietnam. Some 131 countries got this perfect score. India falls short of a perfect score because till 2021 two Lok Sabha MPs were “selected” (the reference is to two representatives of the Anglo-Indian community) and not elected. This law was repealed in 2021, but still becomes the reason for V-Dem to give India a less than perfect score under this head in 2022!

The head “universal suffrage” gives 174 countries a perfect score, and these include China, Iran, North Korea, Myanmar and Venezuela. On these two counts, the world’s largest autocracies get the same scores as the world’s largest and highly-rated democracies. Apparently, a country that allows all to vote, but only for one official party, is as democratic as a country that allows anyone to vote for any party freely.

While the choice of the first two heads is seriously problematic, even from a theoretical perspective, Professor Babones is scathing when noting how the remaining three parameters, clean elections, freedom of association, and freedom of expression, are ranked purely with the help of scientific “experts”. He underlines the problem simply: “Political science is not a science, and political scientists’ expert evaluations of democracy indicators are not akin to chemists’ measurements of the temperature of a fluid.” Thus, there may be nothing precise or objective about a knowledgeable expert’s views on Indian democracy. It may be subject to individual political, ideological or personal biases. A judge, for example, may be an expert in law, but we know from experience that the same crime, when decided by different judges, can result in widely varying sentences and punishments. Professor Babones quotes from a 2009 overview of expert interview methodologies to emphasise this point: “First and foremost, the realisation that the naïve image of the expert as a source of objective information…has long become problematic.”

A perfect example of the subjectivity of experts comes from the sharp decline in India’s score on “clean elections”, which is essentially a rating on how fair and neutral the Election Commission is. The India rating on this score has fallen from 3.6 to 2.3 (out of a possible 4) since 2014, which is a fall from Swiss levels of fairness to that of Zambia’s. And this despite the fact that the ruling party has lost as many elections as it has won since 2014.

Professor Babones’s conclusion is simple: “The problems with V-Dem do not derive from its statistical methodology, but from its idiosyncratic choice of indicators, inexplicable scaling decisions, and vulnerability to expert biases. In short, the problem isn’t with the models, it’s in the data. Or, as the computer scientists say, ‘garbage-in, garbage out’.”

The two lessons India should take out from Professor Babones’s research are simple. One, it should engage with V-Dem to modify its methodology so that it is truly reflective of trends in Indian democracy, never mind the rankings we finally end up with. And two, it must evolve its own standards on how democracy must be ranked in a diverse and multi-religious, multi-ethnic, multi-lingual and pluralistic idea called India. Our objective should be to improve in our own rankings, and not try to figure out how to game the V-Dem system.

In the world of plural democracies, and there are very few of them around, India is often in a minority of one, a case study all by itself. It can probably —with some ifs and buts — be compared with truly ethnically diverse polities like the US or maybe Great Britain, but even these countries are not as poor as India. Also, in a country where the average district is larger than or equal to more than 100 countries, and where the largest state can be No 5 in the world by population, a simple set of universally applied parameters will not give the right picture. We should always keep this in mind.

The writer is the editorial director of Swarajya