Department of Psychology, College of Social Science, Hallym University, Chuncheon, Korea.
Hallym Applied Psychology Institute, College of Social Science, Hallym University, Chuncheon, Korea.
J Educ Eval Health Prof. 2021;18:15. doi: 10.3352/jeehp.2021.18.15. Epub 2021 Jul 5.
Diagnostic classification models (DCMs) were developed to identify the mastery or non-mastery of the attributes required for solving test items, but their application has been limited to very low-level attributes, and the accuracy and consistency of high-level attributes using DCMs have rarely been reported compared with classical test theory (CTT) and item response theory models. This paper compared the accuracy of high-level attribute mastery between deterministic inputs, noisy “and” gate (DINA) and Rasch models, along with sub-scores based on CTT.
First, a simulation study explored the effects of attribute length (number of items per attribute) and the correlations among attributes with respect to the accuracy of mastery. Second, a real-data study examined model and item fit and investigated the consistency of mastery for each attribute among the 3 models using the 2017 Korean Medical Licensing Examination with 360 items.
Accuracy of mastery increased with a higher number of items measuring each attribute across all conditions. The DINA model was more accurate than the CTT and Rasch models for attributes with high correlations (>0.5) and few items. In the real-data analysis, the DINA and Rasch models generally showed better item fits and appropriate model fit. The consistency of mastery between the Rasch and DINA models ranged from 0.541 to 0.633 and the correlations of person attribute scores between the Rasch and DINA models ranged from 0.579 to 0.786.
Although all 3 models provide a mastery decision for each examinee, the individual mastery profile using the DINA model provides more accurate decisions for attributes with high correlations than the CTT and Rasch models. The DINA model can also be directly applied to tests with complex structures, unlike the CTT and Rasch models, and it provides different diagnostic information from the CTT and Rasch models.
诊断分类模型（DCM）旨在识别解决测试项目所需属性的掌握或未掌握情况，但它们的应用仅限于非常低层次的属性，与经典测试理论（CTT）和项目反应理论模型相比，使用 DCM 报告高级属性的准确性和一致性的情况很少。本文比较了确定性输入、噪声“与”门（DINA）和 Rasch 模型之间的高级属性掌握的准确性，以及基于 CTT 的子分数。
首先，一项模拟研究探讨了属性长度（每个属性的项目数）和属性之间的相关性对掌握准确性的影响。其次，一项真实数据研究检验了模型和项目拟合情况，并使用 2017 年韩国医师执照考试的 360 个项目，研究了 3 种模型中每个属性的掌握情况的一致性。
在所有条件下，随着用于测量每个属性的项目数量的增加，掌握的准确性也随之提高。在属性相关性较高（>0.5）和项目较少的情况下，DINA 模型比 CTT 和 Rasch 模型更准确。在真实数据分析中，DINA 和 Rasch 模型通常显示出更好的项目拟合和适当的模型拟合。Rasch 和 DINA 模型之间的掌握一致性范围为 0.541 至 0.633，Rasch 和 DINA 模型之间的人格属性得分的相关性范围为 0.579 至 0.786。
尽管所有 3 种模型都为每个考生提供了掌握决策，但 DINA 模型提供的高相关性属性的个人掌握情况比 CTT 和 Rasch 模型更准确。与 CTT 和 Rasch 模型不同，DINA 模型可直接应用于具有复杂结构的测试，并且提供了与 CTT 和 Rasch 模型不同的诊断信息。
