A defeat or victory for pollsters?

In contrast, there is a perception that pollsters did a smart job for the just-concluded Delhi Assembly election - perhaps because the opinion and the exit polls mostly predicted a victory for AAP.

Psephologists around the world have produced several disastrous poll predictions in the recent past. For example, in the 2015 UK election, the average in the final stage of opinion polls resulted in 279 out of 650 seats, which is 42.9 per cent, to the Conservative Party, and 269 seats, or 41.4 per cent, to the Labour Party. The actual outcome of the election was 316 seats, or 48.9 per cent, to the Conservatives and 232 seats, roughly 35.7 per cent, for the Labour. 

Again, the 2009 Lok Sabha election is regarded as an example of a monumental blunder of exit polls — the average of the exit polls in this election results in 196 seats (36 per cent), whereas the actual result was 262 (48 per cent) for the United Progressive Alliance (UPA). 

In contrast, there is a perception that pollsters did a smart job for the just-concluded Delhi Assembly election — perhaps because the opinion and the exit polls mostly predicted a victory for the Aam Aadmi Party (AAP). However, the average opinion polls yielded 55 seats (which is less than 80 per cent of the total seats), and the exit polls gave an average of 53 seats (which is 75 per cent) to AAP. We know that AAP bagged 62 seats in the actual poll, which is nearly 90 per cent of the total seats. The allowable “margin of error” in poll prediction is only 3 per cent, as considered by leading psephologists and international organisations such as FiveThirtyEight, Pew Research and YouGov. Let’s try to understand the underlying interpretation. 

An opinion poll or an exit poll is basically a survey based on a small proportion of the population. And the prediction from the survey is not likely to match exactly the true result that could be obtained by interviewing all members of the population. The margin of sampling error quantifies how close one may expect a survey result to fall relative to the population value. A “margin of error” of ±3 percentage points at a 95 per cent confidence level indicates that if the survey is conducted 100 times, one might expect the result to be within 3 percentage points either side of the true population value 95 of those times, on an average. 

Thus, with a ±3 per cent margin of error, for example, a 60 per cent prediction indicates a range of 57 to 63 per cent is actually predicted — be that for voting or seat percentage. Clearly, a ±3 per cent prediction in an election having only 70 seats indicates that the prediction should be as precise as within ±2 seats. However, the pollsters even provided a prediction with a range of ±5 or ±7 seats (which is a range of ±10 per cent) for Delhi election, and they are now celebrating their success! These predictions were very loose by any standard. Since the overall trend was highly skewed favouring AAP in Delhi, such predictions don’t appear to be bad-looking. 

Yet, imagine if pollsters predict a 32 (±6) seats for any party in an election having 70 seats with only two contesting parties, and both 38 seats (which is victory) and 26 seats (a defeat) in the final outcome are cheered as correct predictions. Statistically, this is what happened in Delhi. And, what’s more, if a ±15 per cent margin of error is allowed, the prediction has a wide range covering 30 percentage points, which is absurd. Yet, people are branding it a success! Certainly, the margin of error can be reduced by increasing the sample size. Calculation of the number of seats from the voting percentages is a more complicated process. However, pollsters need to calculate for the seats also, separately for each election, if they aspire to predict the number of seats by different parties in our first-past-the-post system. And, if a 20 or 30 percentage points of error range is projected even for an effectively two-party contest like in Delhi, the predictions will be more off the mark in a multi-party contest, and in states with more economic, religious, and social heterogeneity. Let’s consider a simple illustrative example. Suppose, a 55 per cent support favouring a party is estimated with ±10 per cent margin of error using a sample size of 95. Then, a ±5 per cent margin of error could be achieved by increasing the sample size to 380. And, the margin of error could be made to ±3 per cent by a sample size of  1,056. Simple statistical theory can be employed for such sample size calculations, and these are needed a-priori to achieve a ±3 per cent projected margin of error considering the heterogeneity of the concerned population appropriately — be it for a pan-Indian vote, or the states. 

This is a problem not limited to Indian pollsters as highlighted in an article published in the Journal of the American Statistical Association in 2018, written by Professor Andrew Gelman of Columbia University jointly with Houshmand Shirani Mehr and Sharad Goel — both from Stanford University — and David Rothschild of Microsoft Research. They compared 4,221 late-campaign polls including 608 state-level, presidential, Senate and governor’s races of the US between 1998 and 2014, to find the historical margin of error to be plus or minus 6 to 7 percentage points. This means that the error range was 12 to 14 points for most poll predictions in the context of the US — not the typically reported 6 per cent. Pollsters in the US, too, are criticised for their not-so-accurate estimation. The same standard should be applied for their Indian counterparts.

The writer is a professor of statistics at the Indian Statistical Institute, Kolkata