Are there business analytics failures - Failure to replicate the original effect?
Recently there were stories about
"The decline effect"
which seemed to plague some pharmaceutical
and social science studies. The effect - whatever it was - was strong in the original study, but each time the study was reproduced the effect was smaller - it "declined".
This received a lot of press in the popular media - see for example
- The Truth Wears Off, by Jonah Lehrer, New Yorker, Dec 13, 2010
- Cosmic Habituation, RadioLab Podcast, May 2011
See also an excellent analysis by Stanley Young from NISS Everything is Dangerous: A Controversy, June 2008, which shows many examples of bad analysis in epidemiology and medicine.
It seems that there should be similar examples in the business world, where
the original effect was strong, but appeared much less strong next time when someone tried to reproduce it.
One possible example is the
study by Johan Bollen et al, who claim that
collective mood states derived from large-scale Twitter feeds are correlated to the value of the Dow Jones Industrial Average (DJIA) over time. ... We find an accuracy of 87.6% in predicting the daily up and down changes in the closing values of the DJIA
Althouhgh this study was soundly criticized, a hedge fund was started using that strategy.
What happened next?
That Twitter-based hedge fund? It was "quietly liquidated" a month after launch.
Please comment below or email to editor1@kdnuggets.com and I will summarize, preserving confidentiality.
Gregory Piatetsky, Editor