Looking Back on 2011-2012 Fictions

This blog is based (somewhat) on a book by Nelson Goodman titled Fact, Fiction and Forecast (click the link to read a free pdf copy of the Fourth Edition). Goodman argued (summarized here) that Facts (things we observe from past history) and Fictions (counterfactuals) are more difficult to form into law-like statements that we can use to make Forecasts (predictions about the future) than we might think. My reaction to reading the initial edition in the 1980's was all these issue are related to building mathematical models. A mathematical-statistical model estimated from data (facts) can be used to generate counterfactuals by change some of the model parameters and also used to make forecasts by running the models into the future. The quality of a model will depend on its ability to do these things and, rather than arguing about formal aspects of the model, we need to get on with the enterprise and see how well models perform.

Surprisingly or not, in the social sciences this isn't really done with much enthusiasm. Models are estimated, journal articles are published or policy recommendations made, and the performance of the models is rarely critiqued over time. This state of affairs became a problem in the Economics profession during the Subprime Mortgage Crisis (for example, the Federal Reserve econometric models were "wildly inaccurate"). Why hadn't complex econometric models seen the crisis coming? And, if they had, why weren't alarms raised? For example, econometric models of mortgage default risk were found to be unstable and basically useless in predicting future mortgage defaults (here). And, Early Warning System (EWS) models, based on standard indicators, "...frequently do not provide much advance warning of currency and banking crises" (here). I discussed the forecasting problem in an early post (here).

Since I have been developing macro-societal statistical models since the late 1970's and have not really followed up much in my career on how well the models were performing, I thought now we be a good time to get on with it. I have about two years worth of experience looking at how well state-space time series models perform when estimated from historical data (facts), how well the same models can be used to generate counterfactuals (What if the US had increased levels of deficit spending? Would the crisis have been of shorter duration?) and how well the model forecasts have compared to those of other forecasters (particularly the Financial Forecasting Center which uses Artificial Intelligence models).

My first attempt at a counterfactual using the USL20 model was in September of 2011 (here). The Obama administration was essentially making the argument that without the bailout of the financial system, the US economy would have "gone off the cliff" after 2009 (when the Obama administration came into office). They way I constructed this counterfactual was to estimate the state space model up to 2008, the end of the Bush administration, and then run model forward as if the Bush administration had stayed in power and the Obama administration policies had never been enacted. In other words, forecast forward from 2008 to 2011 without knowledge of what actually happened. The economy (GDP in this case) should have gone off the cliff. It didn't. In fact, the economy actually performed a little better in the counterfactual world (dotted red line in the graphic above) than in the real world (solid black line).

Now, there were lots of reasons why the economy did not perform very well in the later part of 2011, the Sovereign Debt Crisis in Europe for one, in addition to the policies of the administration in power. But one thing the counterfactual did demonstrate was that the stimulus was not a "tremendous success" as was being argued by Time magazine (here).

There is a lot more counterfactual work that needs to be done surrounding fiscal policy and monetary policy in response to the US Subprime Mortgage Crisis. The counterfactual above suggests a pretty limited role for fiscal and monetary policy once a bubble pops (more work at different times in different countries needs to be done to demonstrate this point). If this assertion is confirmed, it still doesn't mean that policy measures might not have prevented the bubble from developing in the first place. The counterfactual would be to find some set of policy measures that would prevent the Housing and Stock Market Bubbles. Again, more work would have to be done at different time points and in different countries. And, we need better ways of identifying bubbles while they are developing (this is the topic of another blog here).

Fiction: The Obama GDP Counterfactual

Time magazine recently ran an article titled "The Counterfactual President: Obama Averted Disasters, but Getting Credit Is the Hard Part". The article explains how most of the accomplishments of the Obama administration are based on the counterfactual argument that things would have been worse without the policies enacted by the administration.

This particular quote from the article caught my attention since it implies a testable proposition:

The most extreme example, of course, was the $787 billion stimulus package that Obama signed during his first month in office, when the economy was shedding 700,000 jobs a month. The immediate goal was to avoid a depression, and in that sense it was a tremendous success, stopping the hemorrhaging and stabilizing the scariest economic situation since the Great Depression.

The idea that steps taken by the administration prevented the onset of the second Great Depression implies the following causal model.

The Obama administration, O, after it took office in 2009 not only solved the Subprime Mortgage Crisis, SC (which was negatively affecting the state of the economy, S), but also through the stimulus package improved the state of the economy. Since Gross Domestic product, GDP, is dependent on the state of the economy, improvements in the state of the economy should have improved GDP. To be clear, the model above supposedly explains the actual history we saw after the Obama administration came to office in January of 2009.

The counterfactual argument ("Things would have been worse without us") is described in the model above. The Subprime Mortgage Crisis would have continued to have a negative impact on the state of the economy (and thus GDP) to the present without the policy steps taken by the Obama Administration.

The null hypothesis for the two models above would be that the Subprime Mortgage crisis was over in 2008 and the economy was on the road to recovery in any event. The self-loop in the state of the economy, S, is meant to show that the US economy has its own internal feedback dynamic in response to external shocks such as the Subprime Mortgage Crisis, SC.

A complicated way to test the proposition would be to try to directly estimate the effects of the stimulus on the US economy and demonstrate that the effects were positive. This approach has been tried by the Congressional Budget Office (here), the council of Economic Advisors (here) and economists Alan Blinder and Mark Zandi (here). They have all concluded that the effects were positive.

A simpler way to test the counterfactual would be to estimate a state-space model of the US economy (the USL20 model) and use the state of the US economy to predict real GDP up to the last quarter of 2008, right before the start of the Obama administration. Then simulate the USL2008 model from the first quarter of 2009 until the present. In other words, run the model forward (counterfactually) as if the policies of the Obama administration had never been enacted. If the economy goes off the cliff, than there would be evidence for the counterfactual assertions of the Obama administration ("Things would have been worse without us"). If the economy recovers by itself, then there would be evidence for the null hypothesis ("The subprime Mortgage crisis ended in 2008").

The first question should be how good a job did the USL2008 model do of predicting real GDP (GDP2005 or GDP in 2005 dollars) for the late 20th century. The graphic above shows that the model did a very good job.
The USL2008 model can then be run forward to the present (BEA is currently publishing GDP data into the second quarter of 2011). The model indicates that the economy would not have fallen off a cliff and was in fact underperforming. However, this result cannot be used to argue that the Obama administration policies were inhibiting economic growth (the Republican argument), as the next graphic demonstrates.

There is the matter of probability and the associated prediction intervals for the simulation. As shown above, GDP is still within the 98% prediction interval for this model, but it is getting very close to the lower prediction interval (improbable poor performance) in the second quarter of 2011.

The counterfactual simulation, however, does show that the economy would have hit bottom in mid-2009 regardless of Obama administration policies. The poor performance from mid-2010 to the present could be the result of many other factors (e.g., poor performance in the world economy) other than or in addition to policies of the Obama administration. The simulation, however, does not demonstrate that the "stimulus was a tremendous success" as argued by the Time magazine article.