Cutting through the noise

Book review of Noise: A Flaw in Human Judgment

A study of 1.5 million cases found that when judges are passing sentences on days following a loss by the local city’s football team, they tend to be tougher than on days following a win. The study was consistent with a steady stream of anecdotal reports beginning in the 1970s that showed sentencing decisions for the same crime varied dramatically — indeed scandalously — for individual judges and also depending on which judge drew a particular case.

A study at an oncology centre found that the diagnostic accuracy of melanomas was only 64 per cent, meaning that doctors misdiagnosed melanomas in one of every three lesions.

When a large insurance company, concerned about quality control, asked its underwriters to come up with estimates for the same group of sample cases, their suggested premiums varied by an eye-popping median of 55 per cent, meaning that one adjuster might have set a premium at $9,500 while a colleague set it at $16,700.

If employers rely on only one job interview to pick a candidate from among a similarly qualified group, the chances that this candidate will indeed perform better than the others are about 56 to 61 per cent. That’s “somewhat better than flipping a coin, for sure but hardly a fail-safe way to make important decisions,” Daniel Kahneman, Olivier Sibony and Cass R Sunstein explain in this tour de force of scholarship and clear writing.

These inconsistencies are all about noise, which the authors define as “unwanted variability in judgments.”

Sometimes we treasure variability — in artistic tastes, political views or picking friends. But in many situations, we seek consistency: Medicine, criminal justice, child custody decisions, economic forecasts, hiring, college admissions, fingerprint analysis or business choices about whether to greenlight a movie or consummate a merger.

Despite its prominence in so many realms of human judgment, the authors note that “noise is rarely recognized,” let alone counteracted. We are living in a moment of rampant polarisation and distrust in the fundamental institutions that underpin civil society. Eradicating the noise that leads to random, unfair decisions will help us regain trust in one another.

Despite the authors’ intimidating academic credentials, they take pains to explain their various categories of noise, the experiments and formulas that they introduce, as well as their conclusions and solutions.

Some decision hygiene is relatively easy. “Occasion noise” — the problem of a judge handing out stiffer sentences depending on whether a favourite sports team won or lost — can, like bias, be recognised during a “noise audit” and presumably dealt with. “System noise,” in which insurance adjusters, doctors, project planners or business strategists assess the same facts with that unfortunate variability, requires a more energetic decision hygiene. However, as the authors point out, the steps of decision hygiene “can be tedious. Their benefits are not directly visible; you might never know what problem they prevented from occurring.”

Noise: A Flaw in Human Judgment

Author: Daniel Kahneman, Olivier Sibony and Cass R Sunstein

Publisher: Harper Collins

Price: Rs 699; Pages: 454

A compelling example of “decomposing” a decision involves a case study of a corporate merger. Rather than the bankers and executive team giving the company’s board the usual pro or con presentation, the CEO first tasked various senior executives to come up with their assessments on seven aspects of the mer­ger, ranging from talent of the team to be acquired to the possible financial benefits. Importantly, there were separate teams working on each aspect, so that their judgment was not coloured by posit­ive or negative noise emanating from another verdict, falling into the trap of what the authors call “excessive coherence.”

The authors are sensitive to the costs of noise reduction, a point they illustrate in part with the story of the company that tangled itself up in an annual employee review process that included an overly complicated feedback questionnaire. Forty-six ratings on eleven dimensions for each rater and person being rated is just too much.

Beyond bureaucracy and cost, there’s a loss of dignity when people are treated like numbers instead of individuals. Think of Jack Welch, the former CEO of General Electric, who made it a set practice to fire a percentage of his lowest performers each year, even if many were still performing well. In other situations, the opposite approach can create probl­ems: Rating everyone individually with no compari­sons, such as the standards that allow over 98 per cent of the federal civil servant work force to be judged “fully successful.”

The trick is finding the right balance, not looking for perfect fairness or accuracy, which will always be illusory. A digital body scan examined might be an efficient way to check for melanoma, but I’d rather trust the terrific doctor who checks me every few months. Then again, I wouldn’t mind if he checked his conclusion against the algorithm.

Noise is about how our most important institutions can make decisions that are more accurate and credible. That its prescriptions will not achieve perfect fairness and credibility, while creating pitfalls of their own, is no reason to turn away from this welcome handbook for making life’s lottery a lot more coherent.

The reviewer is co-CEO of NewsGuard, which rates the reliability and trustworthiness of news websites

©2021 The New York Times News Service