We use a play-by-play dataset from the game show “Jeopardy!” to study the hot hand phenomenon, whereby people appear to exhibit “hot” states of elevated performance in domains with repeat trials. We first demonstrate that Jeopardy contestants exhibit strong belief in a hot hand effect as reflected in their wagering decisions during gameplay. In parallel, we find that a small, transient effect exists in contestants’ actual performances. We then quantify contestants’ “hot hand bias,” finding that they overestimate the true effect relative to a “rational” benchmark by up to an order of magnitude. We also find that more successful contestants, as well as those with more quantitative or analytical training, exhibit lower levels of bias. Our paper reconciles robust findings of belief in a hot hand with a growing consensus that a small effect often exists in reality, extends analysis of the phenomenon to a cognitive domain, and begins investigation of foundational mechanisms underlying these effects.