London School of Hygiene and Tropical Medicine, London International Development Centre, London, UK.
International Initiative for Impact Evaluation (3ie), c/o London International Development Centre, London, UK.
Eval Rev. 2023 Jun;47(3):563-593. doi: 10.1177/0193841X221116721. Epub 2022 Sep 1.
Non-randomized studies of intervention effects (NRS), also called quasi-experiments, provide useful decision support about development impacts. However, the assumptions underpinning them are usually untestable, their verification resting on empirical replication. The internal replication study aims to do this by comparing results from a causal benchmark study, usually a randomized controlled trial (RCT), with those from an NRS conducted at the same time in the sampled population. We aimed to determine the credibility and generalizability of findings in internal replication studies in development economics, through a systematic review and meta-analysis. We systematically searched for internal replication studies of RCTs conducted on socioeconomic interventions in low- and middle-income countries. We critically appraised the benchmark randomized studies, using an adapted tool. We extracted and statistically synthesized empirical measures of bias. We included 600 estimates of correspondence between NRS and benchmark RCTs. All internal replication studies were found to have at least "some concerns" about bias and some had high risk of bias. We found that study designs with selection on unobservables, in particular regression discontinuity, on average produced absolute standardized bias estimates that were approximately zero, that is, equivalent to the estimates produced by RCTs. But study conduct also mattered. For example, matching using pre-tests and nearest neighbor algorithms corresponded more closely to the benchmarks. The findings from this systematic review confirm that NRS can produce unbiased estimates. Authors of internal replication studies should publish pre-analysis protocols to enhance their credibility.
非随机干预效应研究（NRS），也称为准实验，为发展影响提供了有用的决策支持。然而，它们所依据的假设通常是无法检验的，其验证依赖于经验复制。内部复制研究旨在通过将因果基准研究（通常是随机对照试验（RCT））的结果与同时在抽样人群中进行的 NRS 的结果进行比较来实现这一点。我们旨在通过系统评价和荟萃分析来确定发展经济学中内部复制研究结果的可信度和可推广性。我们系统地搜索了在中低收入国家进行的社会经济干预 RCT 的内部复制研究。我们使用经过改编的工具对基准随机研究进行了批判性评估。我们提取并统计综合了偏差的经验衡量标准。我们包括了 600 项关于 NRS 与基准 RCT 对应关系的估计。所有内部复制研究都至少对偏差存在“一些关注”，有些研究存在高度偏差风险。我们发现，在不可观测因素（特别是回归不连续性）上存在选择的研究设计，平均产生的绝对标准化偏差估计值约为零，也就是说，与 RCT 产生的估计值相当。但研究设计也很重要。例如，使用预测试和最近邻算法进行匹配更符合基准。这项系统评价的结果证实，NRS 可以产生无偏估计。内部复制研究的作者应该公布预分析方案，以提高其可信度。
