The financial crisis and wild gyrations in commodity prices have exposed deep conceptual flaws in the way academics and regulators think about commodity markets that will force a fundamental re-think.
In particular, they have demolished three key main planks on which the laissez-faire approach to regulation has rested: Fundamentals linked to physical production and consumption of commodities are the principal drivers of prices. Speculators or investors provide an important source of market liquidity, smoothing out temporary imbalances. Their activity may accelerate the adjustment, or even cause the market to overshoot. But speculative influences cannot force prices away from fundamentally determined equilibria for any sustained period.
Commodity markets are highly competitive, with many buyers and sellers, each accounting for a small share of the overall market, unable to influence the prices set to any great extent.
Regulators can draw a meaningful distinction between attempts to "manipulate" prices (which are illegal) and the impact of "dominant" positions (which are not).
Each of these assumptions has proved incorrect, forcing a fundamental reappraisal.
Fundamentals and speculation
Traditional theories about commodity markets assume there are two basic types of participant: producers and consumers using the market to hedge away existing price risks ("commercials") and investors or speculators using the market to take exposure to price risk ("non-commercials").
Prices are assumed to converge on an equilibrium based on supply-demand fundamentals, with commercial players acting as the main drivers of pricing behaviour and speculators playing a subsidiary role balancing the market and providing liquidity.
More From This Section
But as the scale of investment in commodity markets has grown, speculation has emerged a factor in its own right, a new "fundamental" alongside traditional physical supply and demand.
It raises the difficult question of what happens if the weight of investment money moves prices away from their fundamentally determined equilibrium value for a prolonged period?
Most analysts insist this is not possible. But the widely acknowledged housing bubble and mispricing of risk in credit markets have shown markets can deviate from fundamental valuations for years at a time. If credit and housing markets can misprice assets, there is no reason commodity markets should be any more "accurate".
Manipulation and dominance
The idea commodity markets are characterised by lots of small buyers and sellers unable to influence prices is clearly not true in practice:
One market participant was revealed to be running positions in the NYMEX crude oil contract amounting to more than 300 million barrels of oil in the first half of 2008 when the Commodity Futures Trading Commission (CFTC) corrected an error in its commitment of traders reports.
Investigations by the Energy and Commerce Committee of the US House of Representatives revealed NYMEX had issued 117 exemptions to the normal position limits in the NYMEX crude contract alone since 1991. Positions on this scale clearly have the potential to disturb prices.
Liquidity in thin, segmented markets
Most commentators assume global markets such as oil are too large to be manipulated successfully. But most commodity markets are in fact quite small. Even in the US government bond market, supposedly the largest and most liquid market in the world, a string of "settlement failures" has highlighted the limits of liquidity, prompting warnings from the US Treasury about the behaviour of participants with dominant positions.
If liquidity and large positions can become a problem even here, the risks in smaller markets such as crude oil let alone niche markets such as carbon are far greater.
Market segmentation compounds these risks. Most futures contracts are in fact for very specific grades of material delivered in specific locations, restricting them to only a very small fraction of worldwide production and consumption.
Markets are also segmented into individual contracts which are not (completely) fungible: the market for Jun 2009 natural gas is not (quite) the same thing as Jul 2009 natural gas. Given this segmentation, it makes no sense to talk about the "bond market" or the "WTI market". Individual market segments are much smaller and more vulnerable to distortions.
(4) More regulation, but how?
Excessive volatility undermines the function of price signals by making it harder for producers and consumers to differentiate real long-term signals from the mass of short-term movements ("noise"). It imposes real costs in terms of resource misallocation when producers and consumers get it wrong, and these costs are non-trivial.
The damage bubbles do to the economy and the planning-investment process by producers and consumers provides a strong intellectual justification for some form of regulatory intervention. But what?
If the concept of a unique supply-demand equilibrium based on fundamentals is abandoned in favour of one that allows other factors (including speculation) to play a role, does that make the regulator irrelevant (if there is no natural equilibrium, how can speculation be said to distort it)? Or does it mean the regulator should step in to curb "excessive" volatility and ensure prices bear some resemblance to the physical supply-demand balance?
And if regulators do have a role, how should it be enforced? Past practice in the United States has favoured position limits, what might be termed a "structural" solution. But practice in the United Kingdom has favoured a more "behavioural" approach, allowing large positions but imposing additional burdens on those running them to do so in a way that minimises market distortions. Neither approach has proved effective.
In practice, any new regulation is bound to be more intrusive and will be strongly opposed by many market participants. But before regulators pick that fight, they need to forge a new consensus around how these markets work in reality.
Regulators are under pressure to consider whether some of the smaller, less liquid markets need special oversight, especially where they are linked to larger ones and there is a risk that a dominant position in the smaller market can be used to manipulate prices in the larger one.
The recent report on commodity market regulation by the International Organisation of Securities Commissions (IOSCO) admitted positions in unregulated over-the-counter (OTC) markets could, in principle, influence prices on regulated public exchanges, and called for greater information and oversight on positions in these markets.
So far the focus has been on soliciting greater information about OTC trading on larger markets such as oil and natural gas. But a case could be made that regulators should also concentrate on smaller markets (such as carbon) many of which are traded OTC and where linkages to larger markets such as power may make them more important than their size suggests.