Introduction: The Signal and the Noise

Nate Silver has written a book on forecasting, The Signal and the Noise: Why So Many Predictions Fail--But Some Don't (discussed in the video above). The book has become popular enough to be translated into other languages and a colleague in Italy has asked me what I think of the book prior to its publication there. My initial reaction was that the book seems to be asking the right questions but not being critical enough and not providing a wider range of answers.  You can get a flavor for this from my last post (here) where I jumped into the middle of The Signal and the Noise to talked about Climate Change and Stock Market forecasting. I agreed to give my colleague in Italy a more detailed review and, while I'm reading the chapters more carefully, I might as well serialize my reactions (there is a synopsis of the entire book in Wikipedia here).

The introductory chapter makes a number of points:

• Ever since the invention of the printing press in 1440, there has been an explosion of information. The digital computer has in some ways made matters worse, but also holds out the promise of applying computer power to making sense out of Big Data. Unfortunately, "...prediction in the period of Big Data is not going very well." (page 10, all remaining quotes are from the Introductory chapter)
• Some well-known failures: The Terrorist Attacks of September 11, 2001 (9/11), the Global Financial Crisis of 2007-2008, Earthquake prediction, and Biomedical research (Why Most Published Research Findings are False).
• In response to this information overload (Future Shock), humans are good at using their innate ability for "Finding patterns in random noise" whether they are there or not.
• As system complexity increases, systems start failing in spectacular ways (for example, the 2010 Stock Market Flash Crash) and regulation might not be enough.
• Under these conditions, prediction becomes more important but: (1) "...we can never make perfectly objective predictions", (2) "...many of our ideas have not or cannot be tested at all", and (3) "We are undoubtedly living with many delusions that we do not even realize."

In future chapters, the book will look at the Global Financial Crisis of 2007-2008, Climate Change, sports forecasting (particularly Silver's PECOTA system), Bayes' Theorem, economic forecasting, Terrorism and weather forecasting--examples of both success and failures.

One interesting question mentioned in the Introduction (and in the video) is whether the exact predictions of the hard sciences are better than predictions based on statistical models, models that contain random error terms. Evidently, original hurricane forecasting models were statistical but the newer models which evidently perform better are derived from first principles (scientific theory). This is also an issue which comes up in climate change forecasting. It raises a lot of interesting issues about the role of scientific knowledge, particularly in the social sciences, and how that knowledge is generated.

All these issues are lurking beneath every posting in this blog. Reviewing The Signal and the Noise will help bring the issues to the surface.