Experimenting Under Stochastic Congestion

We study randomized experiments in a service system when stochastic congestion can arise from temporarily limited supply or excess demand. Such congestion gives rise to cross-unit interference between the waiting customers, and analytic strategies that do not account for this interference may be biased. In current practice, one of the most widely used ways to address stochastic congestion is to use switchback experiments that turn a target intervention on and off for the whole system in alternation. We find, however, that under a queueing model for stochastic congestion, the standard way of analyzing switchbacks is inefficient and that estimators that leverage the queueing model can be materially more accurate. Additionally, we show how the queueing model enables estimation of total policy gradients from unit-level randomized experiments, thus giving practitioners an alternative experimental approach they can use without needing to precommit to a fixed switchback length before data collection.