TCA Srinivasa-Raghavan: The flawed method of empiricism

OKONOMOS

The way economists test their theories is no good. Here is a better way

For a long time, economics was a theory-oriented discipline. You thought of something, wrote it up as a set of equations and bingo! you were famous. All the old codgers made their reputations this way. It was so easy that excess became inevitable.

Then came the revenge of the empiricists in the 1970s. As a result, now only data-based work sells. Theoreticians have been reduced to selling apples in their spare time to keep body and soul together.

But I am delighted to report that the mindlessness of empiricism, too, is now being questioned. In what is one of the best-ever papers* this column has described, Daniel S Hamermesh, professor at the University of Texas at Austin, asks the all-important question about the method of empiricists, namely, can what they do be replicated?

I have quoted him rather more extensively below because, for an economist, he has a wonderfully felicitous turn of phrase. Here is an example: "Economists treat replication the way teenagers treat chastity""as an ideal to be professed but not to be practised..."

And another: "Without these changes, the paeans to the virtues of replication are as likely to enhance the scientific soundness of empirical research as programmes urging abstinence to reduce teenagers' sexual activity."

Why is replicability important? Because economics claims to be a science. An experiment that can't be repeated is of no value in proper science. Despite this, says Hamermesh, economists don't pay much attention to replication.

He puts this reluctance down to the absence of the right incentives to do what any garden variety physicist or chemist or biologist would do as a matter of course. There are, he says, too "... few incentives... to produce replications of others' research, and similarly few to increase the believability of one's own research by testing ideas on multiple sets of data."

In other words, economists have got into a bad habit. They cry victory and run. So he proposes "a way of generating more scientific replication that will make empirical economic research more credible... "

To start with, he says, it is important to distinguish between three types of replicability. The first is the dictionary meaning of it, which a judge recently used in a dispute between two economists. It basically means doing exactly the same thing.

The second is statistical replicability. This consists of taking a different sample, but applying the identical model. This is not of much use to an economist, he says.

Hence, third, scientific replicability which he says is the most appropriate for economists. "...different sample, different population and perhaps similar, but not identical model""appears much more suited to our methods of research and, indeed comprises most of what economists view as replication."

He also says that "the best empirical economists are adept at writing down clever, novel models that are wonderfully consistent internally, intellectually beautiful and supported by the single set of data on which they are 'tested'." But this is clearly not acceptable procedure for developing a general theory.

"One might take the Alfred E Newman approach to scientific replication: 'What, me worry?' about studies based on just one set of data. Eventually an idea will be tested in the market by subsequent empirical studies that are usually published further down the food chain of journals."

So he suggests that it is necessary to add a second data set to enhance validity "more than proportionately." What needs to be done, therefore, is to test a theory or an idea on different data sets drawn from different economies. If, for some reason, that is not possible at least data from different points of time can be used.

In a sense, Professor Hamermesh is making a very simple point: One swallow does not make a summer. But to wait for more requires patience and the possibility that your theory may be wrong (which, by the way, is why most journalists prefer not to double check their story).

So the issue really boils down to the willingness of economists to be as patient as scientists are. I don't think they will be. There are too many of them about in a profession that is too burdened by patronage. Elbows out is the method, not proper scientific replication.

* Replication in Economics, NBER Working Paper No 13026 http://www.nber.org/papers/w13026