Department of Environmental Sciences, Informatics and Statistics, Ca' Foscari University of Venice, Via Torino 155, 30172, Venezia-Mestre, VE, Italy.
Department of Statistical Sciences, University of Bologna, Bologna, Italy.
Psychometrika. 2024 Jun;89(2):542-568. doi: 10.1007/s11336-024-09973-6. Epub 2024 Apr 25.
When analyzing data, researchers make some choices that are either arbitrary, based on subjective beliefs about the data-generating process, or for which equally justifiable alternative choices could have been made. This wide range of data-analytic choices can be abused and has been one of the underlying causes of the replication crisis in several fields. Recently, the introduction of multiverse analysis provides researchers with a method to evaluate the stability of the results across reasonable choices that could be made when analyzing data. Multiverse analysis is confined to a descriptive role, lacking a proper and comprehensive inferential procedure. Recently, specification curve analysis adds an inferential procedure to multiverse analysis, but this approach is limited to simple cases related to the linear model, and only allows researchers to infer whether at least one specification rejects the null hypothesis, but not which specifications should be selected. In this paper, we present a Post-selection Inference approach to Multiverse Analysis (PIMA) which is a flexible and general inferential approach that considers for all possible models, i.e., the multiverse of reasonable analyses. The approach allows for a wide range of data specifications (i.e., preprocessing) and any generalized linear model; it allows testing the null hypothesis that a given predictor is not associated with the outcome, by combining information from all reasonable models of multiverse analysis, and provides strong control of the family-wise error rate allowing researchers to claim that the null hypothesis can be rejected for any specification that shows a significant effect. The inferential proposal is based on a conditional resampling procedure. We formally prove that the Type I error rate is controlled, and compute the statistical power of the test through a simulation study. Finally, we apply the PIMA procedure to the analysis of a real dataset on the self-reported hesitancy for the COronaVIrus Disease 2019 (COVID-19) vaccine before and after the 2020 lockdown in Italy. We conclude with practical recommendations to be considered when implementing the proposed procedure.
在分析数据时，研究人员会做出一些选择，这些选择要么是任意的，基于对数据生成过程的主观信念，要么是可以做出同样合理的替代选择。这种广泛的数据分析选择可能会被滥用，并且是导致多个领域复制危机的根本原因之一。最近，多元宇宙分析的引入为研究人员提供了一种方法，可以评估在分析数据时可以做出的合理选择范围内结果的稳定性。多元宇宙分析仅限于描述性作用，缺乏适当和全面的推理程序。最近，规范曲线分析为多元宇宙分析添加了推理程序，但这种方法仅限于与线性模型相关的简单情况，并且只允许研究人员推断至少有一种规范是否拒绝零假设，但不能推断应该选择哪些规范。在本文中，我们提出了一种多元宇宙分析的后选择推断方法（PIMA），这是一种灵活和通用的推断方法，考虑了所有可能的模型，即合理分析的多元宇宙。该方法允许对各种数据规范（即预处理）和任何广义线性模型进行推断；它允许通过结合多元宇宙分析的所有合理模型的信息，检验给定预测因子与结果无关的零假设，并提供对家族错误率的强有力控制，允许研究人员声称可以拒绝任何显示显著效果的规范的零假设。该推断方法基于条件重抽样过程。我们正式证明了错误率是有控制的，并通过模拟研究计算了检验的统计功效。最后，我们将 PIMA 程序应用于意大利 2020 年封锁前后对冠状病毒病 2019（COVID-19）疫苗自我报告犹豫的真实数据集的分析。最后，我们提出了实施建议时需要考虑的实际建议。
