Signal and Noise: The Rating Agencies

In The Signal and the Noise, Nate Silver is convinced "...that the best way to view the financial crisis is as a failure of judgement--a catastrophic failure of prediction" (p. 20). The big three Nationally Recognized Statistical Rating Agencies (NSROs), Standard & Poor's, Moody's and the Fitch Group, gave their triple-A rating to mortgage-backed securities that turned out to be junk. The triple-A rating were taken as a "forecast" that the securities had very low risk of default. The ratings (forecasts) were based solely on models since CDOs (collateralized debt obligations) had little track record and limited markets.

This is not the first time that reliance on theoretical models produced catastrophic results. In the late 1990's Long-Term Capital Management (LTCM) collapsed and had to be bailed out by the US Federal Reserve. LTCM was a hedge fund that traded securities based on the Black-Sholes model.

In both cases, theoretical models made predictions that were wrong. In both cases, the culprit was that the models were typical, myopic academic models that did not take account of system effects. The models used probability distributions that assumed securities were independent. When the system collapsed, it took the narrowly conceived models and the associated securities along with it--well, maybe just the securities, the models are still being used.

So we have two extremes here: (1) markets are perfectly efficient (the Efficient Market Hypothesis) and produce the most accurate prices and price forecasts (in futures markets?) or (2) precise mathematical equations based on Econophysics are perfect and produce the most accurate security prices and forecasts. Each of the extremes has a contradiction: perfectly efficient markets are random walks which can't be predicted and precise mathematical pricing models using normal distributions only hold when there are a large number of observations, the precise conditions when a market should produce a better result (for more on this, see my post on markets, models and forecasting here).

Chapter 1 of The Signal and the Noise not only discusses the NSROs but brings up a long list of other issues raised by the Financial Crisis of 2007-2008:

• The NSROs seemed to not have recognized that the period before the Subprime Mortgage Crisis was a housing bubble even though they studied the possible effect of a bubble. Are there tools they could have used to identify the development of housing bubbles?
• Risk is something that can be assigned a specific probability like the probability of drawing one card out of a 52 card deck (1/52). Uncertainty is much more difficult to measure but may be more important for real-world outcomes.
• Financial leverage seems an important indicator of problems in the Financial system but doesn't seem to be monitored or controlled in any meaningful way.
• Policy makers (Larry Summers, in this case) seem to recognize that there are feedback loops in the economy, supply-and-demand being a negative feedback loop controlling prices and fear-and-greed being a positive feedback loop creating bubbles. Unfortunately, this is as far as system thinking seems to go and it begs the question of whether these positive and negative loops actually control the economy in the way economists seem to think.
• The U.S. Congress passed a fiscal stimulus in 2009. The White House promised (and Keynesian theory predicted) that the stimulus would reduce unemployment below 8%. It didn't. Unemployment reached 10.1% in early 2009 and didn't approach 8% until the end of 2011. Was the stimulus a failure or did it prevent unemployment from actually getting worse (I've blogged about the stimulus here)?
• Nate Silver suggests that over confidence in forecasts for housing prices, CDO ratings, financial system performance and unemployment might be the result not of bad models but of sampling problems. Forecasters typically use time series data from periods of "normal" economic growth. Events like the Great Recession were simply out-of-sample. 

To demonstrate the last possibility, Nate Silver offers the following graphic (page 46):

A false sense of confidence in forecasts comes from predictions that seem precise but are not accurate. It's as if you have a weapon that produces a tight pattern of shots that are always off target (the third target from the left above, the other three display some other possibilities). The graphic above is really a demonstration of the concepts reliability (tight pattern) and validity (on target). These are important statistical concepts but I think the forecasting problem is far worse and involves the models and how they are used. In the graphic above, we know we're at a rifle range and are shooting at targets. If precise mathematical models don't accurately describe the real underlying system, it's as if you had a great understanding of your weapon but no idea of where your were or what you were shooting at.

It will be interesting to see if Nate Silver reconciles all of this in future chapters of The Signal and the Noise which will be the topic of future posts. It will also be interesting to see if anyone from the NSRO's gets convicted of fraud.

Introduction: The Signal and the Noise

Nate Silver has written a book on forecasting, The Signal and the Noise: Why So Many Predictions Fail--But Some Don't (discussed in the video above). The book has become popular enough to be translated into other languages and a colleague in Italy has asked me what I think of the book prior to its publication there. My initial reaction was that the book seems to be asking the right questions but not being critical enough and not providing a wider range of answers.  You can get a flavor for this from my last post (here) where I jumped into the middle of The Signal and the Noise to talked about Climate Change and Stock Market forecasting. I agreed to give my colleague in Italy a more detailed review and, while I'm reading the chapters more carefully, I might as well serialize my reactions (there is a synopsis of the entire book in Wikipedia here).

The introductory chapter makes a number of points:

• Ever since the invention of the printing press in 1440, there has been an explosion of information. The digital computer has in some ways made matters worse, but also holds out the promise of applying computer power to making sense out of Big Data. Unfortunately, "...prediction in the period of Big Data is not going very well." (page 10, all remaining quotes are from the Introductory chapter)
• Some well-known failures: The Terrorist Attacks of September 11, 2001 (9/11), the Global Financial Crisis of 2007-2008, Earthquake prediction, and Biomedical research (Why Most Published Research Findings are False).
• In response to this information overload (Future Shock), humans are good at using their innate ability for "Finding patterns in random noise" whether they are there or not.
• As system complexity increases, systems start failing in spectacular ways (for example, the 2010 Stock Market Flash Crash) and regulation might not be enough.
• Under these conditions, prediction becomes more important but: (1) "...we can never make perfectly objective predictions", (2) "...many of our ideas have not or cannot be tested at all", and (3) "We are undoubtedly living with many delusions that we do not even realize."

In future chapters, the book will look at the Global Financial Crisis of 2007-2008, Climate Change, sports forecasting (particularly Silver's PECOTA system), Bayes' Theorem, economic forecasting, Terrorism and weather forecasting--examples of both success and failures.

One interesting question mentioned in the Introduction (and in the video) is whether the exact predictions of the hard sciences are better than predictions based on statistical models, models that contain random error terms. Evidently, original hurricane forecasting models were statistical but the newer models which evidently perform better are derived from first principles (scientific theory). This is also an issue which comes up in climate change forecasting. It raises a lot of interesting issues about the role of scientific knowledge, particularly in the social sciences, and how that knowledge is generated.

All these issues are lurking beneath every posting in this blog. Reviewing The Signal and the Noise will help bring the issues to the surface.

Calculating Forecasting Odds: Think Like a Stockbroker

In a Take Part article (here), Amy Luers, Director of Climate Change at the Skoll Global Threats Fund, takes on the issue of variability and forecasting. Leurs makes the argument that global temperature forecasting and stock forecasting have similar problems with statistical variability. Working with Leonard Sklar, Professor of Geology at San Francisco State University, Leurs developed the two graphs above. On the left is the Dow Jones Industrial Average (DJI, my forecast for the SP500 is here), on the right is Global Temperature (my forecast is here). The bottom left graph shows the odds of making money if you invested in any random year (1913-2010) and stayed in the stock market for up to 50 years. After about 35 years of staying invested, there is a 100% chance of making money. On the right is the same graph for global temperature. It takes a little longer, but after 45 years you have a 100% chance of finding a temperature increase. Notice that in both cases, if you look over only brief periods, the odds are closer to 50-50.

The graph above is from page 381 of Nate Silver's book The Signal and the Noise. In 2007, the notorious Scott Armstrong, Professor of Marketing at the Wharton Business School, made a $10,000 bet with Al Gore that Armstrong modestly called "The Global Warming Challenge." Armstrong and his colleague Kesten Green, an Australian Business School Lecturer,  have argued (here) that Global Temperature is too variable to forecast. The Armstrong-Gore bet was to be resolved monthly. Predicting "no-change," Armstrong argued from the data above that he had won the bet.

Given the Luers-Sklar data, we understand why Armstrong made this bet and we also understand why Al Gore never took the bet. Five years is far too short a time to determine whether temperature has increased or whether, for that matter, the stock market has made money. Having published "Principles of Forecasting, [...a book that...] should be considered canonical to anybody who is seriously interested in the field [...of forecasting]" (these are Silver's words), Prof. Armstrong should have known better.

The basic problem with Prof. Armstrong, his "Global Warming Challenge" and his forecasting principles (I have commented on the principles here) is summed up by a quote from Nelson Mandela "Where you stand depends on where you sit" and Prof. Armstrong sits in the business school and is paid to market business activity as having little impact on the environment. All the forecasting principles and outrageous bets in the world will not cover up this canonical conflict.

The Mystery of Why Portugal's Economy Has Performed So Poorly.

Matthew O'Brien recently wrote a piece for the Atlantic titled "The Mystery of Why Portugal Is Doomed". The reasons given in the piece are all essentially recitals of the conventional wisdom on economic development and, although each point might make sense in and of itself, the actual causes are simpler to understand and less complicated. From the graph above (real GDP in US$ from the World Data Bank) the economy in Portugal experienced an economic bubble after it joined the EU in 1999 and the bubble popped in 2007 as a result of the Subprime Mortgage Crisis. The bubble can be clearly seen in the dynamic attractor plot above (the dashed red line is the dynamic attractor path and the other dashed lines are the 98% bootstrap prediction intervals). Before going into more detail, let's review the conventional wisdom.

Matthew O'Brien lists the following problems with the economy of Portugal:

• Between 2000 and 2012, Portugal's economy wasn't growing fast enough. To prove this case, GDP per capita in Portugal from 2000-12 is compared to the USA 1929-41 (the Great Depression) and Japan 1992-2004 (the Lost Decade). The comparison supposedly supports the conclusion that "this wasn't the case of the bust erasing the boom, because there was no boom". But this is certainly a logical non-sequitur since these economies and time periods really have nothing to do with each other (more about the supposed "non-boom" and a-historical comparisons later).
• Portugal's Immature Financial Sector. After Portugal's entry into the EU, the financial sector "...misallocated the foreign capital that poured in to low productivity, non-tradeable sectors like wholesale and retail trade. In other words, it wasted money on things that never had a chance of paying off." This argument is rejected because German Banks in Portugal also made "bad bets". Bankers making bad bets sounds consistent with bubble behavior which seems inconsistent with the "non-boom".
• Portugal's Small Business Culture, Too Much Corruption and Regulation. Portugal and all of Southern Europe supposedly have too many small "mom-and-pop" businesses that stay small to fly under the radar of a corrupt government. The conventional wisdom: if small-and-medium-sized-enterprises (SMEs) play too big a role in the economy, the economy can't grow and take advantage of economies of scale. The article correctly rejects this explanation because it has been true for most of the late 20th century and hasn't changed much recently.
• Portugal Has To Fix All The Structural Problems.  Here the article trots out the laundry list of conventional criticism of Southern European countries: labor market inflexibility (they need to fire more workers), difficulty starting a business (too much red tape), and legal problems (inability to enforce contracts). "After all, Portugal's stagnation between 2000 and 2008 shows that adequate demand isn't sufficient in the face of these deep problems--but it is necessary. That's why Europe needs to stop insisting on punishment as the path the prosperity." What seems more interesting about the period after 2000 when looking at the graph above is why the bubble was able to stay inflated for so long?

In the end, the article concludes that Portugal's economic performance is a "puzzle". What creates the puzzle, I would argue, is illogical comparisons and arbitrary lines (explicitly or implicitly) drawn on graphs.

The starting point for all the consternation about Portugal is the counterfactual assertion that Portugal should have grown more rapidly. The counterfactual assertions can be conveniently summarized by drawing lines on graphs (the solid red lines with arrows in the graph above) projecting rapid future growth based on strong periods of (bubble driven?) economic performance.  Portugal should have taken-off after the mid-1990s (Dot-com bubble), or after 1999 (EU bubble, "I contend that the European Union itself is like a bubble." George Soros, June 2, 2012) or after 2007 (Subprime Mortgage Housing bubble). Something must have been preventing the take-off into sustained growth implied by the red-line counterfactuals and the usual culprits are trotted out to affirm the conclusion.

None of the red-line "take-off" extrapolations have anything to do with the capabilities of Portugal's economy but rather with the implications of neoclassical economic growth models. Unfortunately, Portugal's economy is not correctly described by the unrestrained exponential growth embedded in such models. Portugal's growth is slowing because the economy is moving toward a steady state. To understand these issues, which are very different from the conventional wisdom, will require a comparison of the neoclassical growth model to the PT20 state-space model of Portugal's economy (the model used to generate the dynamic attractor path in the plot above) and a more detailed discussion of macroeconomic analysis underlying the conventional wisdom, to be covered in future posts.

The Sequestration Experiment: Forecast, Counterfactual or Muddle?

On March 1, 2013 President Obama signed the executive order that put the 2013 Sequestration into effect. Budget Sequestration, first used in the  Gramm-Rudman-Hollings Deficit Reduction Act of 1985 (GRHDRA), places an automatic spending cap on the federal budget. If Congress exceeds the cap through spending authorizations (remember, Congress authorizes spending in the US Government) automatic cuts take place. The total size of the 2013 Sequestration is $85.4 Billion.

From Keynesian Economics 101 we all know that Y = I + C + (X-M) + G, that is, national income is composed of expenditures on investment (I), consumption (C), the balance of trade (Exports - Imports, X-M) and government expenditure (G). Everything else being equal, a cut in G will reduce Y. Therefore, the 2013 Sequestration should result in a recession, that is, a drop in overall national income.

In the graph above, produced by Macroeconomic Advisers (a nonpartisan organization that uses a complex macroeconomic model of the US economy to generate forecasts), the annual percentage change in real GDP (think National Income divided by the aggregate price level) with and without Sequestration is displayed. The blue line shows that Sequestration knocks a few percentage points off real GDP growth for all of 2013 but afterward there is no lasting impact compared to the gradual reductions in government spending assumed in the baseline (the red line).

Essentially the graph shows that Sequestration (really, enforced Austerity) inflicts unnecessary hardship on everyone affected by it from the Defense Department to poor children in Head Start. This is an important point but it misses the bigger picture. The Right Wing has been arguing ever since the Great Depression in the 1930s that Austerity (cutting government expenditure) is the correct response to Depressions and Recessions. Their argument is that since government gets its income from taxation, any reduction in taxation will put money in everyone's pockets to be used to pull the country out of Recession. If the government spends through borrowing, the spending is taking money away from investment (I) so there will be no effects on national income (Y). If government spends by printing money this will create inflation and have no impact on real GDP.

There are many arguments that can and have been made against this reasoning (you can read these arguments in economist Paul Krugman's blog on the NY Times here). Let's assume for the moment that  each position (Keynes vs. the Classics) is just a theory. Testing these two theories is not easy because it would require a macroeconomic experiment that no economist can run. I cannot simply cut aggregate government expenditure and see what happens. I can develop a model of the economy (such as the one developed by Macroeconomic Advisers), include government expenditure as an input variable, cut the level of this input variable holding everything else constant and see what happens. But, I am working with a model not the real economy and the model is subject to specification error.

In the $85.4 Billion 2013 Sequestration, however, we have a perfect natural experiment. Now we will finally get an answer to the major macroeconomic debate of the last 100 years. Unfortunately, we will not get this answer because political forces are working to muddle the experiment. The House GOP promises to limit the effects of Sequestration on Defense, Law Enforcement and Border Patrol. The old arguments about big government and excessive spending get put aside when arguably the most wasteful part of government, the Defensed Department, is facing cuts.

In what different Universe would we just run the experiment and settle the question once and for all? In the end, all that will remain is our ability to run counterfactual experiments with our models, experiments no one seems willing to accept.

Looking Back on 2011-2012 Fictions

This blog is based (somewhat) on a book by Nelson Goodman titled Fact, Fiction and Forecast (click the link to read a free pdf copy of the Fourth Edition). Goodman argued (summarized here) that Facts (things we observe from past history) and Fictions (counterfactuals) are more difficult to form into law-like statements that we can use to make Forecasts (predictions about the future) than we might think. My reaction to reading the initial edition in the 1980's was all these issue are related to building mathematical models. A mathematical-statistical model estimated from data (facts) can be used to generate counterfactuals by change some of the model parameters and also used to make forecasts by running the models into the future. The quality of a model will depend on its ability to do these things and, rather than arguing about formal aspects of the model, we need to get on with the enterprise and see how well models perform.

Surprisingly or not, in the social sciences this isn't really done with much enthusiasm. Models are estimated, journal articles are published or policy recommendations made, and the performance of the models is rarely critiqued over time. This state of affairs became a problem in the Economics profession during the Subprime Mortgage Crisis (for example, the Federal Reserve econometric models were "wildly inaccurate"). Why hadn't complex econometric models seen the crisis coming? And, if they had, why weren't alarms raised? For example, econometric models of mortgage default risk were found to be unstable and basically useless in predicting future mortgage defaults (here). And, Early Warning System (EWS) models, based on standard indicators, "...frequently do not provide much advance warning of currency and banking crises" (here). I discussed the forecasting problem in an early post (here).

Since I have been developing macro-societal statistical models since the late 1970's and have not really followed up much in my career on how well the models were performing, I thought now we be a good time to get on with it. I have about two years worth of experience looking at how well state-space time series models perform when estimated from historical data (facts), how well the same models can be used to generate counterfactuals (What if the US had increased levels of deficit spending? Would the crisis have been of shorter duration?) and how well the model forecasts have compared to those of other forecasters (particularly the Financial Forecasting Center which uses Artificial Intelligence models).

My first attempt at a counterfactual using the USL20 model was in September of 2011 (here). The Obama administration was essentially making the argument that without the bailout of the financial system, the US economy would have "gone off the cliff" after 2009 (when the Obama administration came into office). They way I constructed this counterfactual was to estimate the state space model up to 2008, the end of the Bush administration, and then run model forward as if the Bush administration had stayed in power and the Obama administration policies had never been enacted. In other words, forecast forward from 2008 to 2011 without knowledge of what actually happened. The economy (GDP in this case) should have gone off the cliff. It didn't. In fact, the economy actually performed a little better in the counterfactual world (dotted red line in the graphic above) than in the real world (solid black line).

Now, there were lots of reasons why the economy did not perform very well in the later part of 2011, the Sovereign Debt Crisis in Europe for one, in addition to the policies of the administration in power. But one thing the counterfactual did demonstrate was that the stimulus was not a "tremendous success" as was being argued by Time magazine (here).

There is a lot more counterfactual work that needs to be done surrounding fiscal policy and monetary policy in response to the US Subprime Mortgage Crisis. The counterfactual above suggests a pretty limited role for fiscal and monetary policy once a bubble pops (more work at different times in different countries needs to be done to demonstrate this point). If this assertion is confirmed, it still doesn't mean that policy measures might not have prevented the bubble from developing in the first place. The counterfactual would be to find some set of policy measures that would prevent the Housing and Stock Market Bubbles. Again, more work would have to be done at different time points and in different countries. And, we need better ways of identifying bubbles while they are developing (this is the topic of another blog here).