The problem with opinion polls

As I write this column, we have just ended one more of our periodic public pageants of expressing opinions, i.e. the elections in Karnataka. The elections have bought out the usual panoply of opinion polls, perception surveys, and various data analytics, seeking to describe the mood of the people. And as always, there is a wide spectrum of results to choose from. As you read this column, you would have come to know which of these many efforts were on the target, and how many had missed the bus.

The use of opinion polls is not limited to time-bound self-verifying exercises at election times. They are now being used for analysing progress in public policy. Every other day, in newspapers, television and on social media some poll or the other is used to give a report on the state of our views. More recently, the indicator list to monitor and report progress on the Sustainable Development Goals (SDG) by the UN Statistical Commission includes many items that rely on perception surveys or opinion polls as the primary data set. To illustrate, for Target 16.2: “End abuse, exploitation, trafficking and all forms of violence against and torture of children,” the Indicator 16.2.1 is “Proportion of children aged 1–17 years who experienced any physical punishment and/or psychological aggression by caregivers in the past month”. The measurement of this indicator is through a sample survey, where respondents give answers to the questions, based on their understanding of the concepts being probed. 

Sample Surveys, in general, are subject to several errors. These errors arise for a variety of reasons, including issues of coverage, instrument design, mode of canvassing etc. Even when they seek to measure objective phenomena, e.g. “Expenditure Profile of Households”, the results are highly sensitive to the sampling design, the instrument used, and the respondents selected. Thus, for instance, the National Sample Survey has found that an individual’s assessment of his/her household expenditure is significantly impacted by factors such as the time horizon chosen, the degree of detail with which he/she is probed, length of interview, gender of the respondent (and even the interviewer), and so on. These errors have been well documented, and have, in many cases, been quantified and calibrated through careful experiments.

In contrast, perception or opinion polls seek to measure or quantify what is, at its heart, a nebulous and imprecise personal opinion. For instance, for SDG goal (16.6) “Develop effective, accountable and transparent institutions at all levels”, the proposed measure (16.6.2) “Proportion of population satisfied with their last experience of public services” requires a perception survey. Conceptually speaking “satisfaction with public service” is dependent on the prior level of service, so we could have a situation where a small increase in an earlier unavailable service yields a high level of satisfaction, compared to a situation where a small decline in a high level of service generates a lot of dissatisfaction! These problems are further compounded when the questionnaire design allows the investigator opportunities (deliberate or inadvertent) to frame issues in a manner which can produce manipulated results. The problem is that such data, based on people’s responses, as with political opinions for election analysis, is overly subjective and imprecise, and the resultant statistics are prone to misuse.

These characteristics of opinion polls can create different problems. Perception surveys, whether done by governments or by non-governmental agencies, face persistent issues of credibility. The use of these in SDG monitoring becomes particularly problematic when different social objectives are all competing for scarce budgetary resources. In the electoral domains, these concerns about credibility and manipulability of opinion polls have led many countries to ban or restrict their use in certain parts of the election process. This then leads to the problem of how they should be used in social policy discussions.

It should be noted that in spite of these problems, opinion polling is a form of free speech. Their value to the researcher who uses them to canvass public opinion in support of a particular policy position is indubitable. The credibility of any such use depends on the transparency of the survey process, documentation of the design, concepts and definitions, schedules of enquiry and availability of disaggregated data to analysts. Thus, in academic uses of such data, it becomes necessary that peer review processes mandate such disclosures. It should be pointed out that at present, adherence to these practices, in India, is minimal. 

The use of subjective assessments by governments and multilateral agencies needs to be subjected to higher standards. Their suggested use in high visibility social objectives such as the SDGs makes the need to develop standardised protocols critical. This would entail in the first instance a dialogue between government, academia and civil society. Till that happens, it is probably a good idea to take the results from such exercises with a large helping of salt.