Dimanche Ketsia, Klatt Edward C, Angle S Marshall
Medical Education Research, Mercer University School of Medicine, 1633 1st Ave, Columbus, 31901, GA, USA.
Mercer University School of Medicine, 1250 East 66th Street, Savannah, GA, 31404, USA.
BMC Med Educ. 2025 Mar 14;25(1):384. doi: 10.1186/s12909-025-06948-8.
Standard setting plays a critical role in determining student outcomes by defining the level of biomedical knowledge required to be considered competent. This is especially important for accurately classifying medical students as ready (or not) to progress through the pre-clinical curriculum. In multiple-choice medical knowledge exams, the Yes-No Angoff method may be used for setting passing scores. This method relies on faculty experts' judgments about whether students with borderline but adequate competence would answer each question correctly. Using a construct validity framework with the construct of academic success defined as the ability of a student to progress without obstacles, we examined the predictive validity of the passing standards set by this method.
We analyzed academic success for four pre-clinical semesters across three student cohorts. First, we identified passing standards for pre-clinical courses using the Yes-No Angoff method. Then, we applied binary logistic regression and receiver-operator characteristic (ROC) analyses with area under the curve (AUC) to evaluate passing standards. For binary outcomes, we defined academic success in terms of unimpeded progress through the curriculum and students' first-attempt passage of the United States Medical Licensing Examination (USMLE) Step 1. Model predictors for ROC analyses included Yes-No Angoff passing standards, Medical College Admissions test scores, and grade point averages for math and science courses.
ROC analyses showed a low but acceptable area under the curve for a single semester in one cohort and excellent or outstanding AUCs for the remaining 11 semesters. Rates of overall classification accuracy for the Yes-No Angoff passing scores ranged between 89% and 96% for predicting academic success for all pre-clinical semesters across all cohorts.
The Yes-No Angoff method yielded passing standards that aided in accurately predicting academic success, providing predictive validity evidence for our school's passing standards in a pre-clinical medical curriculum.
标准设定在通过定义被认为具备能力所需的生物医学知识水平来决定学生成绩方面起着关键作用。这对于准确将医学生分类为是否准备好进入临床前课程尤为重要。在多项选择题医学知识考试中，是/否安戈夫方法可用于设定及格分数。该方法依赖于教师专家对能力处于临界但足够的学生是否能正确回答每个问题的判断。使用一个以学术成功为构念的结构效度框架，将学术成功定义为学生无障碍进步的能力，我们检验了该方法设定的及格标准的预测效度。
我们分析了三个学生队列四个临床前学期的学术成功情况。首先，我们使用是/否安戈夫方法确定临床前课程的及格标准。然后，我们应用二元逻辑回归和带有曲线下面积（AUC）的受试者操作特征（ROC）分析来评估及格标准。对于二元结果，我们根据课程的顺利进展和学生首次通过美国医学执照考试（USMLE）第一步来定义学术成功。ROC分析的模型预测变量包括是/否安戈夫及格标准、医学院入学考试成绩以及数学和科学课程的平均绩点。
ROC分析显示，一个队列中一个学期的曲线下面积较低但可接受，其余11个学期的AUC为优秀或出色。对于所有队列中所有临床前学期的学术成功预测，是/否安戈夫及格分数的总体分类准确率在89%至96%之间。
是/否安戈夫方法产生的及格标准有助于准确预测学术成功，为我们学校临床前医学课程的及格标准提供了预测效度证据。
