Signal and Noise: The Rating Agencies

In The Signal and the Noise, Nate Silver is convinced "...that the best way to view the financial crisis is as a failure of judgement--a catastrophic failure of prediction" (p. 20). The big three Nationally Recognized Statistical Rating Agencies (NSROs), Standard & Poor's, Moody's and the Fitch Group, gave their triple-A rating to mortgage-backed securities that turned out to be junk. The triple-A rating were taken as a "forecast" that the securities had very low risk of default. The ratings (forecasts) were based solely on models since CDOs (collateralized debt obligations) had little track record and limited markets.

This is not the first time that reliance on theoretical models produced catastrophic results. In the late 1990's Long-Term Capital Management (LTCM) collapsed and had to be bailed out by the US Federal Reserve. LTCM was a hedge fund that traded securities based on the Black-Sholes model.

In both cases, theoretical models made predictions that were wrong. In both cases, the culprit was that the models were typical, myopic academic models that did not take account of system effects. The models used probability distributions that assumed securities were independent. When the system collapsed, it took the narrowly conceived models and the associated securities along with it--well, maybe just the securities, the models are still being used.

So we have two extremes here: (1) markets are perfectly efficient (the Efficient Market Hypothesis) and produce the most accurate prices and price forecasts (in futures markets?) or (2) precise mathematical equations based on Econophysics are perfect and produce the most accurate security prices and forecasts. Each of the extremes has a contradiction: perfectly efficient markets are random walks which can't be predicted and precise mathematical pricing models using normal distributions only hold when there are a large number of observations, the precise conditions when a market should produce a better result (for more on this, see my post on markets, models and forecasting here).

Chapter 1 of The Signal and the Noise not only discusses the NSROs but brings up a long list of other issues raised by the Financial Crisis of 2007-2008:

• The NSROs seemed to not have recognized that the period before the Subprime Mortgage Crisis was a housing bubble even though they studied the possible effect of a bubble. Are there tools they could have used to identify the development of housing bubbles?
• Risk is something that can be assigned a specific probability like the probability of drawing one card out of a 52 card deck (1/52). Uncertainty is much more difficult to measure but may be more important for real-world outcomes.
• Financial leverage seems an important indicator of problems in the Financial system but doesn't seem to be monitored or controlled in any meaningful way.
• Policy makers (Larry Summers, in this case) seem to recognize that there are feedback loops in the economy, supply-and-demand being a negative feedback loop controlling prices and fear-and-greed being a positive feedback loop creating bubbles. Unfortunately, this is as far as system thinking seems to go and it begs the question of whether these positive and negative loops actually control the economy in the way economists seem to think.
• The U.S. Congress passed a fiscal stimulus in 2009. The White House promised (and Keynesian theory predicted) that the stimulus would reduce unemployment below 8%. It didn't. Unemployment reached 10.1% in early 2009 and didn't approach 8% until the end of 2011. Was the stimulus a failure or did it prevent unemployment from actually getting worse (I've blogged about the stimulus here)?
• Nate Silver suggests that over confidence in forecasts for housing prices, CDO ratings, financial system performance and unemployment might be the result not of bad models but of sampling problems. Forecasters typically use time series data from periods of "normal" economic growth. Events like the Great Recession were simply out-of-sample. 

To demonstrate the last possibility, Nate Silver offers the following graphic (page 46):

A false sense of confidence in forecasts comes from predictions that seem precise but are not accurate. It's as if you have a weapon that produces a tight pattern of shots that are always off target (the third target from the left above, the other three display some other possibilities). The graphic above is really a demonstration of the concepts reliability (tight pattern) and validity (on target). These are important statistical concepts but I think the forecasting problem is far worse and involves the models and how they are used. In the graphic above, we know we're at a rifle range and are shooting at targets. If precise mathematical models don't accurately describe the real underlying system, it's as if you had a great understanding of your weapon but no idea of where your were or what you were shooting at.

It will be interesting to see if Nate Silver reconciles all of this in future chapters of The Signal and the Noise which will be the topic of future posts. It will also be interesting to see if anyone from the NSRO's gets convicted of fraud.

Introduction: The Signal and the Noise

Nate Silver has written a book on forecasting, The Signal and the Noise: Why So Many Predictions Fail--But Some Don't (discussed in the video above). The book has become popular enough to be translated into other languages and a colleague in Italy has asked me what I think of the book prior to its publication there. My initial reaction was that the book seems to be asking the right questions but not being critical enough and not providing a wider range of answers.  You can get a flavor for this from my last post (here) where I jumped into the middle of The Signal and the Noise to talked about Climate Change and Stock Market forecasting. I agreed to give my colleague in Italy a more detailed review and, while I'm reading the chapters more carefully, I might as well serialize my reactions (there is a synopsis of the entire book in Wikipedia here).

The introductory chapter makes a number of points:

• Ever since the invention of the printing press in 1440, there has been an explosion of information. The digital computer has in some ways made matters worse, but also holds out the promise of applying computer power to making sense out of Big Data. Unfortunately, "...prediction in the period of Big Data is not going very well." (page 10, all remaining quotes are from the Introductory chapter)
• Some well-known failures: The Terrorist Attacks of September 11, 2001 (9/11), the Global Financial Crisis of 2007-2008, Earthquake prediction, and Biomedical research (Why Most Published Research Findings are False).
• In response to this information overload (Future Shock), humans are good at using their innate ability for "Finding patterns in random noise" whether they are there or not.
• As system complexity increases, systems start failing in spectacular ways (for example, the 2010 Stock Market Flash Crash) and regulation might not be enough.
• Under these conditions, prediction becomes more important but: (1) "...we can never make perfectly objective predictions", (2) "...many of our ideas have not or cannot be tested at all", and (3) "We are undoubtedly living with many delusions that we do not even realize."

In future chapters, the book will look at the Global Financial Crisis of 2007-2008, Climate Change, sports forecasting (particularly Silver's PECOTA system), Bayes' Theorem, economic forecasting, Terrorism and weather forecasting--examples of both success and failures.

One interesting question mentioned in the Introduction (and in the video) is whether the exact predictions of the hard sciences are better than predictions based on statistical models, models that contain random error terms. Evidently, original hurricane forecasting models were statistical but the newer models which evidently perform better are derived from first principles (scientific theory). This is also an issue which comes up in climate change forecasting. It raises a lot of interesting issues about the role of scientific knowledge, particularly in the social sciences, and how that knowledge is generated.

All these issues are lurking beneath every posting in this blog. Reviewing The Signal and the Noise will help bring the issues to the surface.