Practitioner’s Guide: Designing Adaptive Experiments

Adaptive experiments present a unique opportunity to more rapidly learn which of many treatments work best, evaluate multiple hypotheses, and optimize for several objectives. For example, they can be used to pilot a large number of potential treatments when the researcher does not have strong hypotheses about what works and why; the data can then be used to narrow down a set of alternatives for further development, hypothesis testing, and evaluation. However, adaptive experiments may not be appropriate in all cases, as they create complexity and may require additional resources for implementation. This guide explains what adaptive experiments are, when they can be beneficial, and their limitations. It also offers insights into the questions to ask when considering running adaptive experiments on technology platforms.