Department of Psychology, Northwestern University.
Trachtenberg School of Public Policy, George Washington University.
Psychol Methods. 2020 Dec;25(6):726-746. doi: 10.1037/met0000260. Epub 2020 Mar 12.
This study uses a within study comparison design (WSC) to conduct a novel test of how much causal bias results when researchers use a nonequivalent comparison group design type (NECGD) that combines: (a) a comparison group local to the treatment group; (b) a pretest measure of the study outcome; and (c) a rich set of 19 other multidimensional covariates. Most prior WSCs have dealt with the bias consequences of only 1 of these, revealing that each routinely reduces bias but does not necessarily eliminate it. Thus, a need exists to identify NECGDs that more robustly eliminate bias. This study is the first to examine how combining the 3 bias-control mechanisms above affects bias. The intervention we examine is a prekindergarten mathematics curriculum, for which a randomized control trial (RCT) produces a positive 1-year math effect. Final bias in the NECGD is assessed as the difference between its impact and that of the RCT when each design has the same intervention, outcome, and estimand. Over the many specifications we explore, NECGD bias is less than .10 standard deviations, indicating that minimal bias results when an NECGD combines all 3 design elements. The factorial design we use in this study also tests the bias associated with seven other NECGD types. Comparing the total pattern of results shows that the minimal bias when all 3 elements are combined is uniquely attributable to the locally chosen comparison group and not the availability of a pretest or other covariates. In actual research practice, it is impossible to predict in advance which design elements will affect bias by how much in any given application. So further research is needed to probe whether the simultaneous use of all three design elements achieves minimal bias dependably across diverse applications and not just in the preschool math context examined here. (PsycInfo Database Record (c) 2020 APA, all rights reserved).
本研究采用内部研究比较设计（WSC），对研究人员使用非等效对照组设计类型（NECGD）进行了一项新的测试，该设计类型结合了：（a）与治疗组所在地相近的对照组；（b）研究结果的预测试量；以及（c）一组丰富的 19 个其他多维协变量。大多数先前的 WSC 仅涉及这些因素中的一个因素所带来的偏差后果，结果表明，每种因素都能常规地减少偏差，但不一定能消除偏差。因此，需要确定更能有效消除偏差的 NECGD。本研究首次探讨了组合上述三种偏差控制机制如何影响偏差。我们研究的干预措施是学前数学课程，一项随机对照试验（RCT）产生了积极的 1 年数学效果。NECGD 的最终偏差是在每个设计都具有相同的干预、结果和估计量时，通过比较其影响与 RCT 的影响来评估的。在我们探索的许多规范中，NECGD 的偏差小于 0.10 个标准差，这表明当 NECGD 组合使用所有三个设计元素时，偏差很小。本研究中使用的析因设计还测试了七种其他 NECGD 类型相关的偏差。比较总结果模式表明，当所有三个元素组合时，最小偏差是由于选择了本地对照组，而不是预测试或其他协变量的可用性。在实际研究实践中，不可能提前预测在任何特定应用中，哪些设计元素会以何种程度影响偏差。因此，需要进一步研究以探究在不同应用中以及不仅仅在本研究中考察的学前数学背景下，同时使用所有三个设计元素是否可靠地实现最小偏差。（PsycInfo 数据库记录（c）2020 APA，保留所有权利）。
