Statistical theory says if we randomly pick a small number of households and obtain a truthful set of responses from them, the aggregate of those responses will be representative of the whole population. P C Mahalanobis, who was as close to god as statisticians will ever come, designed a phenomenal system to help decision makers access information critical to good policy. To man this system he built institutions, the best in the world, that researched and taught statistics. The National Sample Survey Organisation (NSSO) started conducting surveys covering tens of thousands of respondents. India, however, needed more respondents than surveyed in most other countries since greater details were required on states, demographic segments, occupational groups and so on. As time went by, sample sizes increased, eventually crossing a hundred thousand households. Questionnaires also increased in size and spread. Though the system was not perfect, it was coming out with estimates that appeared to make a lot of sense.
For many years much of the data never got used; only aggregates were used for policy purposes. The millions of cuts and cross-cuts that could enable a better understanding of the highly heterogeneous Indian population were rarely attempted, since there was no time for the limited number of people who had easy access to this data to do so. But the machinery continued to generate such data. Finally, in the nineties, NSSO started to make it very easy for outside researchers to access data — one of the best things that NSSO did. Its data were out in the public domain, with hundreds of researchers using them for a range of purposes. Few problems were found that were not known already. The kind of respect that NSSO earned was rare for the government departments. But that was not to last. Slowly and steadily, strange numbers, which made little sense, started coming into the public domain.
Today, that trickle has turned into a tide. Neither has the system collapsed, nor have good people disappeared, nor are the data useless. But the trend is quite clear: the numbers and estimates that the data machinery generates are becoming increasingly questionable.
The problem is not only with NSSO but also with government data in general. Typically, the problem has to do with inconsistency, under-reporting, low frequency, obsolescence and so on. Today everyone is questioning the data on the index of industrial production, inflation, employment, expenditure, income and poverty. Where does the problem lie? Are all data fudged? Clearly not. Are they deliberately biased in a certain direction? Again, no. Are they totally useless? No, or else there would have been no debate.
The problem is lack of innovation which, in turn, is a function of personnel and motivation. These are a result of poor administrative structure. Times are changing, technologies are changing and, more importantly, the economic structure is changing. We need to continuously update data collection methods and change the estimation processes, which convert raw data into usable estimates. Also, we need to have very good cross-checking mechanisms. However, many people are talking about expanding coverage (in estimating inflation numbers for example) and sample sizes (NSSO), increasing the in-house staff in government organisations (Central Statistical Organisation), outsourcing more, and bringing in laws to force people to respond. None of these is the solution, since the problem lies somewhere else. In fact, each of these will worsen the quality of available estimates.
The system is not functioning as well as we would like it to because we have stopped innovating. The systems, monitoring mechanisms, institutional processes, technologies, questionnaires and estimation processes that were being used many decades back are still in use today. Meanwhile, respondents today have less time, lower respect for the government or its agents and a higher opportunity cost of time than in the time of Mahalanobis. Moreover, the economy and demography are changing far more rapidly year-on-year and quarter-on -quarter. The quality of information that households and businesses provide is worsening — anyone in the market research domain will tell you this.
So why do we continue with the past systems? Decision makers today have little understanding of data collection and estimation. Some may have collected data at some point, but rarely have they been involved directly in the day-to-day overseeing of data collection. Even public debate is mis-informed and backed by bad interpretation of data. As a result, an understanding of the nuances is lacking in the debate.
More From This Section
There is an important story of how the best statisticians and economists at some point in the past moved away from the nitty-gritty of data into the intellectually (perhaps) more satisfying area of mathematical economics or policy formulation. Increasingly, this is based on principles and preferences rather than empirics. Data collection was left to junior staff or assistants, who were rarely mentored to reach high levels. And today, among all the senior people in the government or outside, you would rarely find anyone who has spent his life on data collection and estimation. It will take 20 years to build such a team and system once again. We should have started in 1991. Nevertheless, it would be good to start now.
Quick fixes such as rapidly increasing staff, sample sizes, coverage and frequency will only pressure a system that is admittedly working sub-optimally, but is working nevertheless. Overloading is not the solution. The pressure will worsen the quality of the output and will hardly improve either data or estimates. We need to figure out new ways to sample, query, estimate, cross-check and correct in an ongoing manner. This cannot be done top-down. Innovative organisations need to provide some freedom, delegation, flexibility and a risk-taking and forgiving environment to people who are hired for this purpose. All these need to be combined with answerability. This has been done within the government in times when we were much poorly resourced. Let’s start now, take time and do it properly.
This is the first of a three-part series. Part-II will appear on August 13