Looking Back on 2011-2012 Fictions

This blog is based (somewhat) on a book by Nelson Goodman titled Fact, Fiction and Forecast (click the link to read a free pdf copy of the Fourth Edition). Goodman argued (summarized here) that Facts (things we observe from past history) and Fictions (counterfactuals) are more difficult to form into law-like statements that we can use to make Forecasts (predictions about the future) than we might think. My reaction to reading the initial edition in the 1980's was all these issue are related to building mathematical models. A mathematical-statistical model estimated from data (facts) can be used to generate counterfactuals by change some of the model parameters and also used to make forecasts by running the models into the future. The quality of a model will depend on its ability to do these things and, rather than arguing about formal aspects of the model, we need to get on with the enterprise and see how well models perform.

Surprisingly or not, in the social sciences this isn't really done with much enthusiasm. Models are estimated, journal articles are published or policy recommendations made, and the performance of the models is rarely critiqued over time. This state of affairs became a problem in the Economics profession during the Subprime Mortgage Crisis (for example, the Federal Reserve econometric models were "wildly inaccurate"). Why hadn't complex econometric models seen the crisis coming? And, if they had, why weren't alarms raised? For example, econometric models of mortgage default risk were found to be unstable and basically useless in predicting future mortgage defaults (here). And, Early Warning System (EWS) models, based on standard indicators, "...frequently do not provide much advance warning of currency and banking crises" (here). I discussed the forecasting problem in an early post (here).

Since I have been developing macro-societal statistical models since the late 1970's and have not really followed up much in my career on how well the models were performing, I thought now we be a good time to get on with it. I have about two years worth of experience looking at how well state-space time series models perform when estimated from historical data (facts), how well the same models can be used to generate counterfactuals (What if the US had increased levels of deficit spending? Would the crisis have been of shorter duration?) and how well the model forecasts have compared to those of other forecasters (particularly the Financial Forecasting Center which uses Artificial Intelligence models).

My first attempt at a counterfactual using the USL20 model was in September of 2011 (here). The Obama administration was essentially making the argument that without the bailout of the financial system, the US economy would have "gone off the cliff" after 2009 (when the Obama administration came into office). They way I constructed this counterfactual was to estimate the state space model up to 2008, the end of the Bush administration, and then run model forward as if the Bush administration had stayed in power and the Obama administration policies had never been enacted. In other words, forecast forward from 2008 to 2011 without knowledge of what actually happened. The economy (GDP in this case) should have gone off the cliff. It didn't. In fact, the economy actually performed a little better in the counterfactual world (dotted red line in the graphic above) than in the real world (solid black line).

Now, there were lots of reasons why the economy did not perform very well in the later part of 2011, the Sovereign Debt Crisis in Europe for one, in addition to the policies of the administration in power. But one thing the counterfactual did demonstrate was that the stimulus was not a "tremendous success" as was being argued by Time magazine (here).

There is a lot more counterfactual work that needs to be done surrounding fiscal policy and monetary policy in response to the US Subprime Mortgage Crisis. The counterfactual above suggests a pretty limited role for fiscal and monetary policy once a bubble pops (more work at different times in different countries needs to be done to demonstrate this point). If this assertion is confirmed, it still doesn't mean that policy measures might not have prevented the bubble from developing in the first place. The counterfactual would be to find some set of policy measures that would prevent the Housing and Stock Market Bubbles. Again, more work would have to be done at different time points and in different countries. And, we need better ways of identifying bubbles while they are developing (this is the topic of another blog here).