Introduction: The Signal and the Noise

Nate Silver has written a book on forecasting, The Signal and the Noise: Why So Many Predictions Fail--But Some Don't (discussed in the video above). The book has become popular enough to be translated into other languages and a colleague in Italy has asked me what I think of the book prior to its publication there. My initial reaction was that the book seems to be asking the right questions but not being critical enough and not providing a wider range of answers.  You can get a flavor for this from my last post (here) where I jumped into the middle of The Signal and the Noise to talked about Climate Change and Stock Market forecasting. I agreed to give my colleague in Italy a more detailed review and, while I'm reading the chapters more carefully, I might as well serialize my reactions (there is a synopsis of the entire book in Wikipedia here).

The introductory chapter makes a number of points:

• Ever since the invention of the printing press in 1440, there has been an explosion of information. The digital computer has in some ways made matters worse, but also holds out the promise of applying computer power to making sense out of Big Data. Unfortunately, "...prediction in the period of Big Data is not going very well." (page 10, all remaining quotes are from the Introductory chapter)
• Some well-known failures: The Terrorist Attacks of September 11, 2001 (9/11), the Global Financial Crisis of 2007-2008, Earthquake prediction, and Biomedical research (Why Most Published Research Findings are False).
• In response to this information overload (Future Shock), humans are good at using their innate ability for "Finding patterns in random noise" whether they are there or not.
• As system complexity increases, systems start failing in spectacular ways (for example, the 2010 Stock Market Flash Crash) and regulation might not be enough.
• Under these conditions, prediction becomes more important but: (1) "...we can never make perfectly objective predictions", (2) "...many of our ideas have not or cannot be tested at all", and (3) "We are undoubtedly living with many delusions that we do not even realize."

In future chapters, the book will look at the Global Financial Crisis of 2007-2008, Climate Change, sports forecasting (particularly Silver's PECOTA system), Bayes' Theorem, economic forecasting, Terrorism and weather forecasting--examples of both success and failures.

One interesting question mentioned in the Introduction (and in the video) is whether the exact predictions of the hard sciences are better than predictions based on statistical models, models that contain random error terms. Evidently, original hurricane forecasting models were statistical but the newer models which evidently perform better are derived from first principles (scientific theory). This is also an issue which comes up in climate change forecasting. It raises a lot of interesting issues about the role of scientific knowledge, particularly in the social sciences, and how that knowledge is generated.

All these issues are lurking beneath every posting in this blog. Reviewing The Signal and the Noise will help bring the issues to the surface.

Calculating Forecasting Odds: Think Like a Stockbroker

In a Take Part article (here), Amy Luers, Director of Climate Change at the Skoll Global Threats Fund, takes on the issue of variability and forecasting. Leurs makes the argument that global temperature forecasting and stock forecasting have similar problems with statistical variability. Working with Leonard Sklar, Professor of Geology at San Francisco State University, Leurs developed the two graphs above. On the left is the Dow Jones Industrial Average (DJI, my forecast for the SP500 is here), on the right is Global Temperature (my forecast is here). The bottom left graph shows the odds of making money if you invested in any random year (1913-2010) and stayed in the stock market for up to 50 years. After about 35 years of staying invested, there is a 100% chance of making money. On the right is the same graph for global temperature. It takes a little longer, but after 45 years you have a 100% chance of finding a temperature increase. Notice that in both cases, if you look over only brief periods, the odds are closer to 50-50.

The graph above is from page 381 of Nate Silver's book The Signal and the Noise. In 2007, the notorious Scott Armstrong, Professor of Marketing at the Wharton Business School, made a $10,000 bet with Al Gore that Armstrong modestly called "The Global Warming Challenge." Armstrong and his colleague Kesten Green, an Australian Business School Lecturer,  have argued (here) that Global Temperature is too variable to forecast. The Armstrong-Gore bet was to be resolved monthly. Predicting "no-change," Armstrong argued from the data above that he had won the bet.

Given the Luers-Sklar data, we understand why Armstrong made this bet and we also understand why Al Gore never took the bet. Five years is far too short a time to determine whether temperature has increased or whether, for that matter, the stock market has made money. Having published "Principles of Forecasting, [...a book that...] should be considered canonical to anybody who is seriously interested in the field [...of forecasting]" (these are Silver's words), Prof. Armstrong should have known better.

The basic problem with Prof. Armstrong, his "Global Warming Challenge" and his forecasting principles (I have commented on the principles here) is summed up by a quote from Nelson Mandela "Where you stand depends on where you sit" and Prof. Armstrong sits in the business school and is paid to market business activity as having little impact on the environment. All the forecasting principles and outrageous bets in the world will not cover up this canonical conflict.