Department of Psychology and Counseling, University of Taipei, Taiwan.
Br J Math Stat Psychol. 2020 Feb;73(1):109-141. doi: 10.1111/bmsp.12160. Epub 2019 Feb 22.
Methods of cognitive diagnostic computerized adaptive testing (CD-CAT) under higher-order cognitive diagnosis models have been developed to simultaneously provide estimates of the attribute mastery statuses of examinees for formative assessment and estimates of a latent continuous trait for overall summative evaluation. In a typical CD-CAT environment, examinees are often subject to a time limit, and the examinees' response times (RTs) for specific test items can be routinely recorded by custom-made programs. Because examinees are individually administered tailored sets of test items from the item pool, they may experience different levels of speededness during testing and different levels of risk of running out of time. In this study, RTs were considered during the item-selection procedure to control the test speededness and the RTs were treated as useful information for improving latent trait estimation in CD-CAT under the higher-order deterministic input, noisy 'and' gate (DINA) model. A modified posterior-weighted Kullback-Leibler (PWKL) method that maximizes the item information per time unit and a shadow-test method that assembles a provisional test subject to a specified time constraint were developed. Two simulation studies were conducted to assess the effects of the proposed methods on the quality of CD-CAT for fixed- and variable-length exams. The results show that, compared with the traditional PWKL method, the proposed methods preserve a lower risk of running out of time while ensuring satisfactory attribute estimation and providing more accurate estimates of the latent trait and speed parameters. Finally, several suggestions for future research are proposed.
方法认知诊断计算机自适应测试 (CD-CAT) 下的高阶认知诊断模型已经被开发出来，以同时提供被试的属性掌握状态的估计值，用于形成性评估和潜在连续特质的估计值，用于整体总结性评估。在典型的 CD-CAT 环境中，被试通常受到时间限制的限制，并且被试对特定测试项目的反应时间 (RT) 可以通过定制程序常规记录。由于被试从项目池中单独接受定制的测试项目集，因此他们在测试期间可能会经历不同程度的速度，并且面临时间不足的风险程度也不同。在这项研究中，在项目选择过程中考虑了 RT，以控制测试速度，并且将 RT 视为在高阶确定性输入、嘈杂“和”门 (DINA) 模型下进行 CD-CAT 中潜在特质估计的有用信息。开发了一种修改后的后验加权 Kullback-Leibler (PWKL) 方法，该方法最大化了每个时间单位的项目信息，以及一种影子测试方法，该方法根据指定的时间约束组装临时测试对象。进行了两项模拟研究，以评估所提出的方法对固定和可变长度考试的 CD-CAT 质量的影响。结果表明，与传统的 PWKL 方法相比，所提出的方法在确保满意的属性估计的同时，保留了较低的时间不足风险，并提供了更准确的潜在特质和速度参数估计。最后，提出了一些未来研究的建议。
