Sports fans, rejoice: You may have been right all along about hot streaks. They aren't a figment of your imagination.
Contradicting academic studies dating back 30 years, researchers at Stanford, Berkeley, and Harvard are now finding that a "hot hand" in basketball or baseball is not a statistical illusion. In fact, hot streaks can help predict a player's likelihood of getting another hit or sinking another basket.
Perhaps surprisingly, this has implications for a raging debate in finance about the rationality, or lack of it, in markets.
But first, about the sports.
Athletes, coaches, and sports fans overwhelmingly believe in the "hot hand," the idea that a player whose shooting percentage is higher than normal is likely to keep shooting better than normal — at least for a while. Indeed, coaches often rotate players in and out of play based on a sense of who has turned hot or cold.
Academic scholars thought they had debunked this idea years ago. Starting with a famous 1985 study of basketball shooting, experts have argued in dozens of papers that the hot hand is nothing more than statistical noise. Athletes may seem to go on hot streaks, goes the argument, but these are just random fluctuations without predictive value. People who believe in hot hands, the skeptics have argued, are seeing patterns that don't exist.
But in a major new study of baseball data, Jeffrey Zwiebel at Stanford Graduate School of Business and Brett Green at University of California's Haas School of Business at Berkeley find that hot hands are real and have predictive power.
Why would business professors jump into a sports debate? Because the "hot-hand fallacy" has become a staple in arguments by supporters of behavioral economics to argue that individuals can be irrational. For example, the behaviorists have argued that investors often get lured into bad decisions by seeing patterns that aren't real. According to the behaviorists, investors make a wide range of cognitive mistakes that range from overconfidence in their own abilities to a tendency to over-react to news.
That critique gained a lot of traction in the wake of the mortgage bust and the great financial crisis. A 2009 paper lays out the theory and potential financial applications of the hot-hand fallacy. A 2012 paper argues that the hot-hand fallacy explains why people pay for useless investment advice. And in a brand new paper, German scholars even found evidence that people who believe in the hot-hand fallacy are more at risk of long-term unemployment.
The hot-hand "fallacy" has its own roots at Stanford. Thomas Gilovich, a graduate student in the early 1980s, began comparing the widespread perception of hot streaks in basketball to the hard data. Gilovich, now at Cornell, led a study showing that the hot hand didn't really exist: The shooting records of the Philadelphia 76ers provided no predictive value of subsequent shots. A player might be hot one minute, but not the next. Fans and even sports professionals, they concluded, were making decisions based on myopic impressions.
Zwiebel and Green argue that the original finding failed to account for a key issue. In basketball, the opposing team quickly adjusts to a "hot" player, devoting extra coverage and forcing that player to attempt more difficult shots. As a result, it's inevitable that a hot player's shooting percentage will decline. It's not because the hot streak was an illusion, but rather that the hot player attracted more opposition.
Baseball is different, because pitchers and coaches have very limited ability to re-deploy resources against a hitter on a hot streak. Pitchers and coaches do play to a hitter's particular weaknesses, but they can't put more people in the hitter's way.
To test their ideas, Zwiebel and Green amassed data on 2 million Major League Baseball at-bats over 12 years. They looked at 10 categories of performance, from batting averages and home-run percentages to strike-out rates. For pitchers, they looked at data such as the average number of hits allowed. They also controlled for the fundamental ability of both pitchers and hitters, in order to isolate the actual "streakiness" of a player's performance.
The result: A player's most recent 25 times at bat was a significant predictor of how that player would do at his next time up — good enough to justify an adaptive reaction by coaches. When a player is "hot," the researchers calculated, his expected on-base percentage will be 25 to 30 points higher than it would be if he just has been "cold." Similarly, a player on a hot streak will be 30% more likely to hit a home run than if he has been on a cold streak.
"These results are important enough to rationalize common coaching decisions and fan perceptions with respect to hot players," Zwiebel and Green write in their paper.
Zwiebel says the earlier researchers were too quick to conclude that the belief in a hot hand was evidence of a cognitive or behavioral mistake. Most likely, what's really at work is not so much a mistake but an "equilibrium adjustment" around the hot-handed player — similar to the kinds of equilibrium adjustments that occur in finance and economics.
"The behavioral camp jumps too quickly to the conclusion that almost all sports fans and participants are under a dumb illusion that there are hot hands," Zwiebel says. "They have jumped to that conclusion because it fits their story that everyone is making cognitive mistakes and that these mistakes are extraordinarily pervasive."
As it happens, a team of Harvard researchers reached a similar conclusion in a new study of basketball. The Harvard team looked at data on 83,000 shots made during the National Basketball Association's 2012-2013 season. But instead of looking simply at completion rates, the researchers built a model for gauging the difficulty of shots. They found that players who had exceeded their expectations in recent shots were likely to face tighter opposition and take more difficult shots. Adjusting for the increased difficulty of the shots, the researchers found that hot players were likely to continue outperforming. The "hot-hand effect," they estimated, raised their chances of making a shot by 1.2% to 2.4%.
Zwiebel's larger point is that behaviorists use the supposed hot-hand fallacy to jump to unwarranted conclusions. They see cognitive mistakes everywhere they look. The problem, he argues, is that the misdiagnosis of a problem — such as the mortgage bust — can lead to the wrong remedies.
Investment bankers have been criticized for underestimating the risks of subprime mortgages, for being irrational in their optimism. But it's possible that their excessive risk-taking wasn't irrational as much as it was encouraged or even subsidized by regulatory policies, such as those that protected banks considered "too big to fail." The difference between those two diagnoses of the financial crisis leads to very different proposals for reform.
"One can portray the financial crisis as being triggered by a bunch of mistakes, or one can portray it as the consequence of risky decisions that were made based on the incentives and subsidies in the system," he says. "I see it as more of the latter."
Jeffrey Zwiebel is the James C. Van Horne Professor of Finance at Stanford GSB. Brett S, Green, who received his PhD in economics from Stanford GSB in 2009, is an assistant professor at U.C. Berkeley Haas School of Business.