Experimental Design in Two-Sided Platforms: An Analysis of Bias

We develop an analytical framework to study experimental design in two-sided platforms. In the settings we consider, customers rent listings; rented listings are occupied for some amount of time, then become available. Platforms typically use two common designs to study interventions in such settings: customer-side randomization (CR), and listing-side randomization (LR), along with associated estimators. We develop a stochastic model and associated mean field limit to capture dynamics in such systems, and use our model to investigate how performance of these estimators is affected by interference effects between listings and between customers. Good experimental design depends on market balance: we show that in highly demand-constrained markets, CR is unbiased, while LR is biased; conversely, in highly supply-constrained markets, LR is unbiased, while CR is biased. We also study a design based on two-sided randomization (TSR) where both customers and listings are randomized to treatment and control, and show that appropriate choices of such designs can be unbiased in both extremes of market balance, and also yield low bias in intermediate regimes of market balance.