Srinivasan Malathi, McElvany Matthew, Shay Jane M, Shavelson Richard J, West Daniel C
University of California-Davis, Davis, California, USA.
Acad Med. 2008 Dec;83(12):1196-203. doi: 10.1097/ACM.0b013e31818c6e84.
To test the reliability of concept map assessment, which can be used to assess an individual's "knowledge structure," in a medical education setting.
In 2004, 52 senior residents (pediatrics and internal medicine) and fourth-year medical students at the University of California-Davis School of Medicine created separate concept maps about two different subject domains (asthma and diabetes) on two separate occasions each (four total maps). Maps were rated using four different scoring systems: structural (S; counting propositions), quality (Q; rating the quality of propositions), importance/quality (I/Q; rating importance and quality of propositions), and a hybrid system (H; combining elements of S with I/Q). The authors used generalizability theory to determine reliability.
Learners (universe score) contributed 40% to 44% to total score variation for the Q, I/Q, and H scoring systems, but only 10% for the S scoring system. There was a large learner-occasion-domain interaction effect (19%-23%). Subsequent analysis of each subject domain separately demonstrated a large learner-occasion interaction effect (31%-37%) and determined that administration on four to five occasions was necessary to achieve adequate reliability. Rater variation was uniformly low.
The Q, I/Q, and H scoring systems demonstrated similar reliability and were all more reliable than the S system. The findings suggest that training and practice are required to perform the assessment task, and, as administered in this study, four to five testing occasions are required to achieve adequate reliability. Further research should focus on whether alterations in the concept mapping task could allow it to be administered over fewer occasions while maintaining adequate reliability.
在医学教育环境中测试概念图评估的可靠性，该评估可用于评估个人的“知识结构”。
2004年，加利福尼亚大学戴维斯分校医学院的52名高级住院医师（儿科和内科）以及四年级医学生，分两次针对两个不同的主题领域（哮喘和糖尿病）分别创建概念图（共四张图）。使用四种不同的评分系统对概念图进行评分：结构评分（S；计算命题数量）、质量评分（Q；对命题质量进行评级）、重要性/质量评分（I/Q；对命题的重要性和质量进行评级）以及混合系统评分（H；将S的要素与I/Q相结合）。作者运用概化理论来确定可靠性。
对于Q、I/Q和H评分系统，学习者（全域分数）对总分变异的贡献为40%至44%，但对于S评分系统，这一贡献仅为10%。存在较大的学习者-测试场合-主题领域交互效应（19% - 23%）。随后分别对每个主题领域进行分析，显示出较大的学习者-测试场合交互效应（31% - 37%），并确定需要进行四到五次测试才能获得足够的可靠性。评分者变异一直较低。
Q、I/Q和H评分系统显示出相似的可靠性，且均比S系统更可靠。研究结果表明，执行评估任务需要培训和实践，并且在本研究的实施方式下，需要进行四到五次测试才能获得足够的可靠性。进一步的研究应聚焦于概念图任务的改变是否能够在保持足够可靠性的同时，减少测试次数。
