How Should We Think about Replicating Observational Studies? A Reply to Fowler and Montagnes

In their reply to our article, “Irrelevant Events and Voting Behavior,” Fowler and Montagnes reanalyze our replication study of college football’s effect on election outcomes. Although we agree with Fowler and Montagnes that the evidence supporting the irrelevant events hypothesis is weaker than earlier research suggested, they overstate this case. Philosophically, we disagree with Fowler and Montagnes’s preference for (1) running a plethora of tests rather than focusing on the most theoretically motivated tests and (2) privileging out-of-sample data over the full sample of available data. Empirically, we show that their claim that out-of-sample data weaken the original results depends on (1) an incorrect definition of out-of-sample years and (2) assigning two-thirds weight to the same hypothesis regarding heterogeneous effects. An amended version of Fowler and Montagnes’s analysis affirms our initial assessment: although the original finding was overstated, adding out-of-sample data strengthens it.