Dwyer Tim, Wright Sarah, Kulasegaram Kulamakan Mahan, Theodoropoulos John, Chahal Jaskarndip, Wasserstein David, Ringsted Charlotte, Hodges Brian, Ogilvie-Harris Darrell
Women's College Hospital, 76 Grenville St, Toronto, M5S 1B1, Canada.
Mt Sinai Hospital, 600 University Avenue, Toronto, M5G 1X5, Canada.
BMC Med Educ. 2016 Jan 4;16:1. doi: 10.1186/s12909-015-0506-z.
The goal of the Objective Structured Clinical Examination (OSCE) in Competency-based Medical Education (CBME) is to establish a minimal level of competence. The purpose of this study was to 1) to determine the credibility and acceptability of the modified Angoff method of standard setting in the setting of CBME, using the Borderline Group (BG) method and the Borderline Regression (BLR) method as a reference standard; 2) to determine if it is feasible to set different standards for junior and senior residents, and 3) to determine the desired characteristics of the judges applying the modified Angoff method.
The results of a previous OSCE study (21 junior residents, 18 senior residents, and six fellows) were used. Three groups of judges performed the modified Angoff method for both junior and senior residents: 1) sports medicine surgeons, 2) non-sports medicine orthopedic surgeons, and 3) sports fellows. Judges defined a borderline resident as a resident performing at a level between competent and a novice at each station. For each checklist item, the judges answered yes or no for "will the borderline/advanced beginner examinee respond correctly to this item?" The pass mark was calculated by averaging the scores. This pass mark was compared to that created using both the BG and the BLR methods.
A paired t-test showed that all examiner groups expected senior residents to get significantly higher percentage of checklist items correct compared to junior residents (all stations p < 0.001). There were no significant differences due to judge type. For senior residents, there were no significant differences between the cut scores determined by the modified Angoff method and the BG/BLR method. For junior residents, the cut scores determined by the modified Angoff method were lower than the cut scores determined by the BG/BLR Method (all p < 0.01).
The results of this study show that the modified Angoff method is an acceptable method of setting different pass marks for senior and junior residents. The use of this method enables both senior and junior residents to sit the same OSCE, preferable in the regular assessment environment of CBME.
基于胜任力的医学教育（CBME）中客观结构化临床考试（OSCE）的目标是确定最低胜任水平。本研究的目的是：1）以边界组（BG）法和边界回归（BLR）法作为参考标准，确定CBME背景下改良安格夫标准设定方法的可信度和可接受性；2）确定为初级和高级住院医师设定不同标准是否可行；3）确定应用改良安格夫方法的评判者所需具备的特征。
采用先前一项OSCE研究的结果（21名初级住院医师、18名高级住院医师和6名研究员）。三组评判者对初级和高级住院医师均采用改良安格夫方法：1）运动医学外科医生；2）非运动医学骨科医生；3）运动医学研究员。评判者将处于边界水平的住院医师定义为在每个站点表现介于胜任和新手之间水平的住院医师。对于每个检查表项目，评判者回答“处于边界水平/高级初学者水平的考生会正确回答这个项目吗？”，答案为是或否。通过对分数求平均值来计算及格分数。将此及格分数与使用BG法和BLR法得出的及格分数进行比较。
配对t检验显示，与初级住院医师相比（所有站点p < 0.001），所有考官组预计高级住院医师在检查表项目上正确回答的百分比显著更高。评判者类型之间无显著差异。对于高级住院医师，改良安格夫方法确定的及格分数与BG/BLR方法确定的及格分数之间无显著差异。对于初级住院医师，改良安格夫方法确定的及格分数低于BG/BLR方法确定的及格分数（所有p < 0.01）。
本研究结果表明，改良安格夫方法是为高级和初级住院医师设定不同及格分数的可接受方法。使用该方法可使高级和初级住院医师参加相同的OSCE，最好是在CBME的常规评估环境中。
