Zhang Kelly W, Janson Lucas, Murphy Susan A
Department of Computer Science, Harvard University.
Departments of Statistics, Harvard University.
Adv Neural Inf Process Syst. 2021 Dec;34:7460-7471.
Bandit algorithms are increasingly used in real-world sequential decision-making problems. Associated with this is an increased desire to be able to use the resulting datasets to answer scientific questions like: Did one type of ad lead to more purchases? In which contexts is a mobile health intervention effective? However, classical statistical approaches fail to provide valid confidence intervals when used with data collected with bandit algorithms. Alternative methods have recently been developed for simple models (e.g., comparison of means). Yet there is a lack of general methods for conducting statistical inference using more complex models on data collected with (contextual) bandit algorithms; for example, current methods cannot be used for valid inference on parameters in a logistic regression model for a binary reward. In this work, we develop theory justifying the use of M-estimators-which includes estimators based on empirical risk minimization as well as maximum likelihood-on data collected with adaptive algorithms, including (contextual) bandit algorithms. Specifically, we show that M-estimators, modified with particular adaptive weights, can be used to construct asymptotically valid confidence regions for a variety of inferential targets.
强盗算法越来越多地应用于现实世界中的序贯决策问题。与此相关的是，人们越来越希望能够利用由此产生的数据集来回答科学问题，比如：一种广告类型是否能带来更多购买量？移动健康干预在哪些情况下有效？然而，经典统计方法在与通过强盗算法收集的数据一起使用时，无法提供有效的置信区间。最近已经为简单模型开发了替代方法（例如，均值比较）。然而，缺乏用于对使用（上下文）强盗算法收集的数据使用更复杂模型进行统计推断的通用方法；例如，当前方法不能用于对二元奖励的逻辑回归模型中的参数进行有效推断。在这项工作中，我们发展了理论，证明了M估计量（包括基于经验风险最小化的估计量以及最大似然估计量）在使用自适应算法（包括（上下文）强盗算法）收集的数据上的应用是合理的。具体来说，我们表明，用特定自适应权重修改后的M估计量可用于为各种推断目标构建渐近有效的置信区域。
