Ethics in statistical reporting

The headline in one of our leading newspapers recently blared: “Global Index: Over 8 million people live in modern slavery in India”. If you read the story, you find referenced a report by a global NGO, Walk Free Foundation. The Foundation has been producing reports on modern slavery for some years now; its work has attracted greater attention of late, in part because the ILO recently co-published its Global Slavery Index. Another factor increasing interest in the Foundation’s work is the Sustainable Development Goals. Goal 8 on “Decent Work and Economic Growth” has a target stating “Take immediate and effective measures to eradicate forced labour, end modern slavery and human trafficking, and secure the prohibition and elimination of the worst forms of child labour, including recruitment and use of child soldiers, and by 2025 end child labour in all its forms.” This clubbing of modern slavery with the other problems has had the effect of seriously muddying the waters. Goal 8’s reference to modern slavery, along with the association of the ILO in the report of the Foundation, leads to the unwarranted conclusion that such reports provide an authentic assessment of the problem, when in fact they do not. 

Why do I say this? Because there is a fundamental flaw underlying this emotive issue: At present, there is no agreed concept of modern slavery. The ILO does have conventions on forced labour and child labour, but modern slavery is as yet just a phrase, for which there is no agreed definition and hence, crucially, no methodology for estimation. The NGO, as a private entity, is within its rights to formulate and advocate for a definition, but until that definition is globally accepted, it remains a private opinion. The NGO’s methodology note defining modern slavery is essentially an omnibus term covering forced labour, debt bondage, forced marriage, slavery and slavery-like practices, and human trafficking. They themselves acknowledge this has no legal basis. 

As a consequence, the different components have different standards in measurability and quantifiability. For example, as mentioned above, while forced labour and child labour have been recognised and defined in the ILO conventions, there is a problem with forced marriage, i.e. marriage without consent.  It is well recognised that there are significant difficulties in measuring this, especially in regions where there is an extensive prevalence of child and arranged marriages. The Foundation commissioned Gallup World Poll to conduct a global perception survey of 71,000 individuals in 48 countries. Only 437 individuals reported marriage without consent! The small size of the sample and the low incidence rate makes their estimates, analysis and conclusions suspect. A study of their methodology raises many other issues as well (which can be discussed elsewhere), but what is important is the overall effort to somehow inflate the magnitude of the problem.  

If a measure of forced marriage is desired, it is first necessary to have a robust definition that is sensitive to, and accommodates, cultural differences. The challenges posed by child and arranged marriages need to be resolved in the approach to the definition. Only then can a measurement approach that is robust to reporting biases be developed. The ILO’s work on child labour is a good example of how this can be done in a manner which is conducive to solving the problem. 

Recognising the fact that UN agencies carry a higher measure of credibility in the public mind, and therefore, need to ensure higher standards in their statistical work, the Committee for Coordination of Statistical Activities, an apex group of international multilateral agencies, issued in 2005, and later reaffirmed in 2014, a set of principles governing international statistical activities.  These principles assert that statistical work should be impartial, and be based on concepts, sources, methods and procedures that meet professional scientific standards and are transparent to the users. This would require that all statistical estimates be endorsed by professional statisticians.  Indeed, the ILO over the years has set very high standards for statistical work in many areas of labour and employment policy.

Applying these principles to issues like modern slavery,  we need to first look at the debate on the problem of defining slavery in modern times, understand the complexity involved in the exercise of evolving a definition. Brushing under the carpet the need for a robust and transparent definition, and rushing straight to measurement, will result in spurious conclusions, and may also compromise our efforts to solve critical problems like forced & child labour. 

One of the objectives of the Foundation conducting the survey on modern slavery is to argue for trade restrictions on countries which have a high incidence. Not surprisingly many are in the developing world and are increasingly competing with producers in the developed world. Associating the need to provide decent work to thinly disguised efforts to promote protectionism is not only undesirable but likely to be counter-productive. The association of the ILO gives these estimates a highly unjustified degree of credibility, which in turn gives it greater legitimacy and media coverage. One would argue that for the ILO to give its imprimatur to definitions put out by private bodies and estimates based on these definitions, is questionable and in violation of statistical ethics. What is not clear is why the ILO agreed to be part of such a protectionist agenda.  

The writer is former Chief Statistician of India