Statistical challenges in online controlled experiments: A review of a/b testing methodology
The rise of internet-based services and products in the late 1990s brought about an
unprecedented opportunity for online businesses to engage in large scale data-driven …
unprecedented opportunity for online businesses to engage in large scale data-driven …
Efficient estimation for staggered rollout designs
J Roth, PHC Sant'Anna - Journal of Political Economy …, 2023 - journals.uchicago.edu
We study estimation of causal effects in staggered-rollout designs—that is, settings where
there is staggered treatment adoption and the timing of treatment is as good as randomly …
there is staggered treatment adoption and the timing of treatment is as good as randomly …
Balancing covariates in randomized experiments with the gram–schmidt walk design
The design of experiments involves a compromise between covariate balance and
robustness. This article provides a formalization of this tradeoff and describes an …
robustness. This article provides a formalization of this tradeoff and describes an …
Optimal experimental design for staggered rollouts
In this paper, we study the design and analysis of experiments conducted on a set of units
over multiple time periods in which the starting time of the treatment may vary by unit. The …
over multiple time periods in which the starting time of the treatment may vary by unit. The …
Adaptive neyman allocation
J Zhao - arXiv preprint arXiv:2309.08808, 2023 - arxiv.org
In experimental design, Neyman allocation refers to the practice of allocating subjects into
treated and control groups, potentially in unequal numbers proportional to their respective …
treated and control groups, potentially in unequal numbers proportional to their respective …
Seller-side experiments under interference induced by feedback loops in two-sided platforms
Z Zhu, Z Cai, L Zheng, N Si - arXiv preprint arXiv:2401.15811, 2024 - arxiv.org
Two-sided platforms are central to modern commerce and content sharing and often utilize
A/B testing for developing new features. While user-side experiments are common, seller …
A/B testing for developing new features. While user-side experiments are common, seller …
Estimating effects of long-term treatments
Estimating the effects of long-term treatments in A/B testing presents a significant challenge.
Such treatments--including updates to product functions, user interface designs, and …
Such treatments--including updates to product functions, user interface designs, and …
Tackling Interference Induced by Data Training Loops in A/B Tests: A Weighted Training Approach
N Si - arXiv preprint arXiv:2310.17496, 2023 - arxiv.org
In modern recommendation systems, the standard pipeline involves training machine
learning models on historical data to predict user behaviors and improve recommendations …
learning models on historical data to predict user behaviors and improve recommendations …
Data-driven switchback designs: Theoretical tradeoffs and empirical calibration
We study the design and analysis of experiments conducted on an aggregate unit over time,
and outcomes are measured on a sequence of events. The design problem is to partition the …
and outcomes are measured on a sequence of events. The design problem is to partition the …
[HTML][HTML] Population interference in panel experiments
The phenomenon of population interference, where a treatment assigned to one
experimental unit affects another experimental unit's outcome, has received considerable …
experimental unit affects another experimental unit's outcome, has received considerable …