MDRC, New York, NY, USA.
University of Michigan, Ann Arbor, MI, USA.
Eval Rev. 2023 Aug;47(4):701-726. doi: 10.1177/0193841X231159472. Epub 2023 Mar 4.
The Making Pre-K Count and High 5s studies represent a recent application of a phased two-stage, multi-level design, which was used to examine the effects of two aligned math programs implemented in early childhood settings. The purpose of this paper is to describe the challenges encountered in implementing this two-stage design and to describe approaches to resolving them. We then present a set of sensitivity analyses the study team used to examine the robustness of the findings. During the pre-K year, pre-K were randomly assigned either to receive an evidence-based early math curriculum and associated professional development (Making Pre-K Count) or to a pre-K-as-usual control condition. In the kindergarten year, who had been in Making Pre-K Count program classrooms in pre-K were then individually randomly assigned within schools to small-group supplemental math clubs that were designed to sustain the gains from the pre-K program, or to a business-as-usual kindergarten experience. Making Pre-K Count took place in 69 pre-K sites, comprising 173 classrooms across New York City. High 5s took place in the 24 sites that were part of the public school treatment arm of the Making Pre-K Count study and included 613 students. The study focuses on the effect of the Making Pre-K Count and High 5s programs on children's math skills at the end of kindergarten as measured by two instruments, the Research-Based Early Math Assessment-Kindergarten (REMA-K) and the Woodcock-Johnson Applied Problems test. The multi-armed design, while logistically and analytically challenging, balanced multiple considerations of power, the number of research questions that could be answered, and efficiency of resources. Robustness checks suggest that the design created groups that were both meaningfully and statistically equivalent. Decisions to use a phased multi-armed design should consider both its strengths and weaknesses. While the design allows for a more flexible, expansive research study, it also introduces complexities that need to be addressed both logistically and analytically.
《让学前教育计数和 High 5s 计划》代表了一种最近应用的分阶段两阶段、多层次设计，该设计用于检验在幼儿环境中实施的两个对齐的数学计划的效果。本文的目的是描述在实施这种两阶段设计时遇到的挑战，并描述解决这些挑战的方法。然后，我们提出了一组敏感性分析，研究小组使用这些分析来检查研究结果的稳健性。在学前一年，学前班儿童被随机分配接受基于证据的早期数学课程和相关的专业发展（让学前教育计数）或学前教育常规控制条件。在幼儿园一年，已经在学前班接受过让学前教育计数计划课程的儿童随后在学校内按个体被随机分配到旨在维持学前计划收益的小组补充数学俱乐部，或接受常规的幼儿园体验。让学前教育计数在纽约市的 69 个学前班地点进行，包括 173 个教室。High 5s 计划在作为让学前教育计数研究公立学校治疗组一部分的 24 个地点进行，包括 613 名学生。该研究侧重于让学前教育计数和 High 5s 计划对儿童在幼儿园结束时数学技能的影响，测量工具是两项，即基于研究的早期数学评估-幼儿园（REMA-K）和伍德科克-约翰逊应用问题测试。该多臂设计虽然在逻辑和分析上具有挑战性，但平衡了力量、可以回答的研究问题数量以及资源效率等多个考虑因素。稳健性检查表明，该设计创建的组在意义和统计学上都是等效的。决定使用分阶段多臂设计时应考虑其优缺点。虽然该设计允许更灵活、更广泛的研究，但它也引入了需要在逻辑和分析上解决的复杂性。
