书签法和 Angoff 标准设定法在医学绩效测试中有效性的比较。
Comparison of the validity of bookmark and Angoff standard setting methods in medical performance tests.
机构信息
Department of Psychology, Faculty of Humanities, University of Zanjan, Zanjan, Iran.
出版信息
BMC Med Educ. 2021 Jan 2;21(1):1. doi: 10.1186/s12909-020-02436-3.
BACKGROUND
One of the main processes of determining the ability level at which a student should pass an assessment is standard setting. The current study aimed to compare the validity of Angoff and bookmark methods in standard-setting.
METHOD
190 individuals with an M.Sc. degree in laboratory science participated in the study. A test with 32 items, designed by a group of experts, was used to assess the laboratory skills of the participants. Moreover, two groups each containing 12 content specialists in laboratory sciences, voluntarily participated in the application of the Angoff and bookmark methods. To assess the process validity, a 5-item questionnaire was asked from two groups of panelists. To investigate the internal validity, the classification agreement was calculated using the kappa and Fleiss's Kappa coefficient. External validity was assessed by using five indices (correlation with criterion score, specificity, sensitivity, and positive and negative predictive values of correlation test with criterion score).
RESULTS
The results showed that the obtained cut-scores was 17.67 for Angoff and 18.8 for bookmark. The average total of items related to the quality of the execution process was 4.25 for the Angoff group and 4.79 for the bookmark group. Pass rates pass rates percentages for the Angoff and bookmark group were 55.78 and 41.36, respectively. Correlations of passing/failing, between employer ratings and test scores were 0.69 and 0.88 for Angoff and bookmark methods, respectively.
CONCLUSION
Based on the results, it can be concluded that the process and internal validities of the bookmark method were higher than the Angoff method. For evaluation of the external validity (concordance of the cut score with the criterion score), all five external validity indices supported the bookmark method.
背景
确定学生应通过评估的能力水平的主要过程之一是标准设定。本研究旨在比较 Angoff 和书签法在标准设定中的有效性。
方法
190 名具有实验室科学硕士学位的个人参加了这项研究。一项由一组专家设计的包含 32 个项目的测试用于评估参与者的实验室技能。此外，两组各包含 12 名实验室科学内容专家自愿参与了 Angoff 和书签法的应用。为了评估过程有效性，从两组专家中询问了一份包含 5 个项目的问卷。为了研究内部有效性，使用 kappa 和 Fleiss 的 Kappa 系数计算分类一致性。外部有效性通过使用五个指标（与标准分数的相关性、特异性、敏感性以及与标准分数的相关性测试的阳性和阴性预测值）进行评估。
结果
结果表明，Angoff 法获得的切割分数为 17.67，书签法为 18.8。与执行过程质量相关的项目平均总分分别为 Angoff 组的 4.25 和书签组的 4.79。Angoff 和书签组的通过率分别为 55.78%和 41.36%。雇主评分和测试分数之间的及格/不及格相关性分别为 Angoff 和书签方法的 0.69 和 0.88。
结论
根据结果可以得出结论，书签法的过程和内部有效性高于 Angoff 法。对于外部有效性（切割分数与标准分数的一致性）的评估，所有五个外部有效性指标都支持书签法。
Zotero 插件