This article examines the causal effect of success and failure on future search behavior. Iteration based on frequent collection of market feedback has gained immense popularity as a method for innovation. Even as the literature has acknowledged this transformation, we have learned surprisingly little about how such feedback influences firms' search for innovation. This paper investigates firms' reaction to feedback using a unique data set consisting of 40,053 A/B tests (digital randomized controlled trials) conducted by 2,106 unique teams from 1,360 different firms. Using a sharp regression discontinuity around the 0.1 p-value (the default p-value for determining statistical significance in the particular software used to run the A/B tests), I compare experiments that had results just above or just below the 0.1 p-value. I demonstrate that failure causes teams to shrink their search, implementing fewer experiments and using fewer metrics to measure outcomes. Moreover, the difference between treatment and control is smaller in experiments following a failure, suggesting that the ideas implemented are relatively similar to the current version of their product.
Berk Can Deniz
PhD Program, Organizational Behavior
PhD Program Office
Graduate School of Business
655 Knight Way
Stanford, CA 94305
Berk Can Deniz
Job Market Paper
This paper studies how the adoption of experimentation as a selection method shapes the direction of innovation. The spread of A/B testing, or digital randomized experiments, has made experimentation one of the most common methods organizations use to evaluate and select internally generated ideas. The strength of experimental evidence can be a force for radical innovation. By providing seemingly irrefutable evidence, well-designed and well-executed experiments can disabuse people of their false beliefs and generate breakthrough discoveries. However, I argue that the adoption of experimentation can also result in incrementalism, whereby firms focus on minuscule yet reliable improvements. These divergent outcomes can be explained by the incentives driving the people who design and implement experiments. The incentives of managers in established firms may lead them to use experiments in a way that undermines the pursuit of novelty while encouraging the search for incremental improvements. I investigate the relationship between experimentation and in- novation in the context of US newspaper websites and their adoption of A/B testing. Using a historical archive of US newspaper websites and a novel computational method, I find that the adoption of A/B testing decreases the likelihood of radical change and makes websites more likely to change incrementally.
(R&R at Organization Science) This article examines the interaction between the variation and selection stages of organizational innovation. Most organizations rely on internal selection mechanisms, whereby creators submit their ideas to managers or experts within their firm for approval. Recent years have seen a growth in the uses of crowd-based selection mechanisms, whereby external audiences choose among ideas. While prior work has compared crowds and expert in terms of which kinds of ideas they select, we examine how these alternative selection mechanisms might influence the idea-generation process. We argue that internal selection generally lowers exploration by reducing the variation in ideas because creators with a clear conception of selection criteria constrain their search for ideas. We use two separate innovation tournaments to compare the two selection mechanisms on the variety of ideas generated. The findings are consistent with the claim that internal selection reduces variation. The results have implications for both the theory and practice of organizational innovation.
Last Updated 4 Feb 2021