The power of collective intelligence

Company chief executives might hate this fascinatingly well-crafted book. For, it raises one fundamental question: why do companies rely so much on the judgment of one person""namely the CEO?

Despite genuflections to decentralisation, it is no secret that most fail to tap the insights of their employees, suppliers, and others by using well-designed decision markets for everything""from forecasting demand to deciding which products to create. One reason: such an approach might limit CEOs' ability to justify their huge compensation packages!

A Delhi-based chief executive told me the other day that this is "literature in a hurry written by a journalist eager to meet his deadline."

But before dismissing the book, CEOs would do well to remember that its author is no ideologue. Financial journalist James Surowiecki doesn't shy from evidence that counters his theory.

Still, he musters ample proof that the payoff from heeding collective intelligence (the wisdom of crowds) is greater than many of us imagine.

In The Wisdom of Crowds, the New Yorker staff writer provocatively argues that, in many circumstances, the group collectively reaches better decisions""and solves problems more efficiently""than the smartest man or woman alone.

Consider the examples given by Surowiecki in support of his arguments for exploiting the power of a large number of independent minds in solving big problems.

Linux, the open-source operating system created by Finnish programmer Linus Torvalds in 1991 but effectively owned by no one, is now the major rival to Microsoft's Windows.

Independent computer programmers from around the world contribute to improving the operating system, and solving the problems that intrigue them, although Torvalds and his peers keep a tight rein on what changes are acceptable.

When a problem arises with the way Linux works, it only gets fixed if someone, on his own, offers a good solution. There are no bosses ordering people around, no organisational charts dictating people's responsibilities. Instead, people work on what they are interested in and ignore the rest.

This seems like""in fact, it is""a rather haphazard way to solve the problems. After all, if thousands of programmers are spending their time trying to come up with a solution that only a few of them are going to find, that's many hours wasted that could be spent doing something else.

But as Surowiecki has shown so far at least, Linux's seeming wastefulness is a kind of strength, which has made it the single most important challenger to Microsoft.

Or, take the case of the disappearance of USS Scorpion, a submarine on its way to Newport News. The US Navy had a general idea where the submarine sank, but it was an area 20 miles wide and many thousands of feet deep.

Naval officer John Craven hit upon a solution. He gathered a group of diverse experts and asked for their best guesses on why the submarine ran into trouble, its speed as it fell to the ocean floor, the slant of its descent, and so on.

Craven took all the speculations, ran them through a sophisticated mathematical formula, and ended up with the team's overall guesstimate.

The Navy found the ship 220 yards from where Craven's group had predicted it would be, yet not one individual had picked that spot. "The final estimate was a genuinely collective judgment that the group as a whole had made," says Surowiecki. "It was also a genuinely brilliant judgment."

The capital markets provide the classic examples of Surowiecki's thesis: Even if they are sometimes prone to bouts of enthusiasm or depression, they are an amazing social and economic institution for communicating all kinds of data and knowledge through price changes. The more pervasive the financial markets, the more investors will find and fund profitable ideas and, at the same time, flee from failed management strategies.

Of course, the crowd can go spectacularly wrong, as some of the best parts of this book reveal. For instance, the stock market, with its decentralised structure, can go horribly wrong with speculation at times spiralling out of control.

The reasons are many: Investors sometimes herd, preferring the safety of the company of others to make independent decisions. They give too much credence to recent high-profile news while underestimating the importance of longer-lasting trends or less dramatic events, in the same way that people worry about being killed in a plane crash while not paying attention to their high cholesterol.

Surowiecki doesn't wish away these imperfections. He merely discusses an interesting theory, gives arguments both for and against it, and leaves the judgement to "the wisdom of crowds."

The book begins with an interesting anecdote about the popular TV game show "Who wants to be a Millionaire" (remember "Kaun Banega Crorepati?"). The show's gimmick was that if a contestant got stumped by a question, he could pursue three avenues of assistance.

First, he could have two of the four multiple-choice answers removed. Second, he could call a person whom, before the show, he had singled out as one of the smartest people he knew. And third, he could poll the studio audience, which would immediately cast its votes by computer.

Conventional wisdom says that the "smartest" people would be able to do much better than the crowd of people with nothing better to do on a weekday afternoon than sit in a TV studio.

The actual result, however, was an eye-opener: while the "experts" got the right answer 65 per cent of the time, the crowd picked the right answer 91 per cent of the time. Sometimes, "literature in a hurry" can indeed leave a long-lasting impression.

THE WISDOM OF CROWDS

James Surowiecki
Little Brown
Price: $16.99, Pages: 295