Steiner Peter M, Wong Vivian C
1 Department of Educational Psychology, University of Wisconsin-Madison, Madison, WI, USA.
2 Curry School of Education, University of Virginia, Charlottesville, VA, USA.
Eval Rev. 2018 Apr;42(2):214-247. doi: 10.1177/0193841X18773807. Epub 2018 May 17.
In within-study comparison (WSC) designs, treatment effects from a nonexperimental design, such as an observational study or a regression-discontinuity design, are compared to results obtained from a well-designed randomized control trial with the same target population. The goal of the WSC is to assess whether nonexperimental and experimental designs yield the same results in field settings. A common analytic challenge with WSCs, however, is the choice of appropriate criteria for determining whether nonexperimental and experimental results replicate. This article examines different distance-based correspondence measures for assessing correspondence in experimental and nonexperimental estimates. Distance-based measures investigate whether the difference in estimates is small enough to claim equivalence of methods. We use a simulation study to examine the statistical properties of common correspondence measures and recommend a new and straightforward approach that combines traditional significance testing and equivalence testing in the same framework. The article concludes with practical advice on assessing and interpreting results in WSC contexts.
在研究内比较（WSC）设计中，将非实验设计（如观察性研究或回归断点设计）的治疗效果与针对相同目标人群的精心设计的随机对照试验所获得的结果进行比较。WSC的目标是评估非实验设计和实验设计在实际场景中是否产生相同的结果。然而，WSC常见的分析挑战是选择合适的标准来确定非实验结果和实验结果是否一致。本文研究了用于评估实验估计值和非实验估计值一致性的不同基于距离的对应性度量。基于距离的度量方法探究估计值之间的差异是否小到足以声称方法等效。我们通过模拟研究来检验常见对应性度量的统计特性，并推荐一种在同一框架中结合传统显著性检验和等效性检验的新的直接方法。本文最后给出了在WSC背景下评估和解释结果的实用建议。
