Last week, researchers from the Universities of Indiana and Manchester published a paper in the online journal arXiv.org claiming that Twitter can be used to predict the stock market. It's a strange idea, though not without precedent. To understand the Twitter-Dow Jones link, first consider the relationship between ice cream and crime. If you were to compare ice cream sales and crime rates in a city like Brooklyn, you'd probably notice something odd: while one is up, so is the other. When people start buying more ice cream, it seems they also start committing more crimes.
Crime rates and ice cream sales move together because they are both influenced by warm weather. When it's hot, the days are longer, and more people are outside on the street, talking, walking, laughing, buying ice cream, and, inevitably, making trouble. The relationship between ice cream sales and crime rates is used in statistics classes to illustrate the old truism that correlation does not imply causation. Just because two variables move together, that doesn't mean they affect the other's behavior.
But the ice cream example also illustrates another point. We can look at ice cream sales to gain some insight into crime rates. They don't affect one another, but they nonetheless move together, and so we can use information we have about one system to learn about another less-understood one. Getting back to the stock market, it's notoriously difficult to predict. But perhaps there is another system that moves like the stock market—one that is easier to analyze. The paper from Indiana and Manchester argues, essentially, that Twitter might be to the stock market what ice cream is to crime rates.
[Read more]
