Department of Psychological, Health & Learning Sciences, University of Houston, USA.
Department of Psychological, Health & Learning Sciences, University of Houston, USA.
J Sch Psychol. 2022 Jun;92:80-95. doi: 10.1016/j.jsp.2022.03.003. Epub 2022 Mar 28.
Although researchers have investigated technical adequacy and usability of written-expression curriculum-based measures (WE-CBM), the economic implications of different scoring approaches have largely been ignored. The absence of such knowledge can undermine the effective allocation of resources and lead to the adoption of suboptimal measures for the identification of students at risk for poor writing outcomes. Therefore, we used the Ingredients Method to compare implementation costs and cost-effectiveness of hand-calculated and automated scoring approaches. Data analyses were conducted on secondary data from a study that evaluated predictive validity and diagnostic accuracy of quantitative approaches for scoring WE-CBM samples. Findings showed that automated approaches offered more economic solutions than hand-calculated methods; for automated scores, the effects were stronger when the free writeAlizer R package was employed, whereas for hand-calculated scores, simpler WE-CBM metrics were less costly than more complex metrics. Sensitivity analyses confirmed the relative advantage of automated scores when the number of classrooms, students, and assessment occasions per school year increased; again, writeAlizer was less sensitive to the changes in the ingredients than the other approaches. Finally, the visualization of the cost-effectiveness ratio illustrated that writeAlizer offered the optimal balance between implementation costs and diagnostic accuracy, followed by complex hand-calculated metrics and a proprietary automated program. Implications for the use of hand-calculated and automated scores for the universal screening of written expression with elementary students are discussed.
虽然研究人员已经研究了书面表达课程基础测量（WE-CBM）的技术充分性和可用性，但不同评分方法的经济意义在很大程度上被忽视了。缺乏这些知识可能会破坏资源的有效分配，并导致为识别写作成绩不佳风险的学生采用次优的措施。因此，我们使用成分方法比较了手动计算和自动化评分方法的实施成本和成本效益。数据分析是基于一项评估 WE-CBM 样本定量评分方法的预测有效性和诊断准确性的研究中的二次数据进行的。研究结果表明，自动化方法比手动计算方法提供了更经济的解决方案；对于自动化评分，当使用免费的 writeAlizer R 包时，效果更强，而对于手动计算评分，更简单的 WE-CBM 指标比更复杂的指标成本更低。敏感性分析证实了自动化评分的相对优势，当每个学年的班级、学生和评估次数增加时；同样，writeAlizer 对成分变化的敏感性低于其他方法。最后，成本效益比的可视化表明，writeAlizer 在实施成本和诊断准确性之间提供了最佳平衡，其次是复杂的手动计算指标和专有的自动化程序。讨论了在小学对书面表达进行普遍筛查时使用手动计算和自动化评分的影响。
