Return-aware platform experimentation: new directions for research
Аннотация
Platform companies run thousands of online randomized experiments (“A/B tests”) internally each year to test product innovations and decide which ones to roll out to their users. A systematic literature review finds that the vast majority of relevant research since 2000 has focused on questions around statistical inference within individual experiments, rather than on maximizing returns from many experiments. This paper proposes new research directions to bring the latter perspective to fruition. We anchor our proposals in the recent return-aware framework, which treats experimentation as a resource-allocation decision problem: given a finite user base and a set of possible ideas for innovations, how should a firm allocate experimentation effort and make rollout decisions to maximize expected returns? We outline two research directions not captured by the theoretical framework. The first concerns the statistical evidence and priors that enter decision-making, focusing on the use of non-experimental signals to pre-screen ideas for A/B testing and on how to account for ex ante unknown errors in A/B tests themselves. We discuss partial identification as a promising approach. The second concerns frictions that arise when organizations aim to implement return-maximizing decision rules, including congestion and diminished user experience from frequent product innovations, as well as misaligned incentives between decentralized product teams and the organization’s objectives. We call for research clarifying or resolving these frictions. Our aim is to inspire academic and industry research on how to align the practice of internal experimentation on platforms with intended business objectives, drawing upon concepts from statistics and economics.