Young Nicholas T, Mills Mark, Matz Rebecca L, Bell Eric F, Hayward Caitlin
Department of Physics and Astronomy, University of Georgia, Athens, Georgia, United States of America.
Center for Academic Innovation, University of Michigan, Ann Arbor, Michigan, United States of America.
PLoS One. 2025 May 30;20(5):e0323813. doi: 10.1371/journal.pone.0323813. eCollection 2025.
High-stakes exams significantly impact introductory physics students' final grades and have been shown to be inequitable, often to the detriment of students identifying with groups historically marginalized in physics. Certain types of exam questions may contribute more than other types to the observed equity gaps.
The primary objective of this study was to determine whether complex multiple-choice (CMC) questions may be a potential cause of inequity.
We used four years of data from Problem Roulette, an online, not-for-credit exam preparation program, to address our objective. This data set included 951 Physics II (Electricity and Magnetism) questions, each of which we categorized as CMC or non-CMC. We then compared student performance on each question type and created a multi-level logistic regression model to control individual student and question differences.
Students performed 7.9 percentage points worse on CMC questions than they did on non-CMC questions. We find minimal additional performance differences based on student performance in the course. The results from mixed-effects models suggest that CMC questions may be contributing to the observed equity gaps, especially for male and female students, though more evidence is needed.
We found CMC questions are more difficult for everyone. Future research should examine the source of this difficulty and whether that source is functionally related to learning and assessment. Our data does not support using CMC questions instead of non-CMC questions as a way to differentiate top-performing students from everyone else.
高风险考试对基础物理课程学生的期末成绩有重大影响，并且已被证明存在不公平现象，这往往对那些认同在物理学领域历史上处于边缘地位群体的学生不利。某些类型的考试题目对观察到的公平差距的影响可能比其他类型更大。
本研究的主要目的是确定复杂多项选择题（CMC）是否可能是不公平的一个潜在原因。
我们使用了来自“问题轮盘”（Problem Roulette）的四年数据，这是一个在线的非学分考试准备项目，以实现我们的目标。该数据集包含951道物理II（电与磁）题目，我们将每道题归类为CMC或非CMC。然后我们比较了学生在每种题型上的表现，并创建了一个多层次逻辑回归模型来控制个体学生和题目差异。
学生在CMC题目上的表现比在非CMC题目上差7.9个百分点。我们发现基于学生在课程中的表现，额外的表现差异极小。混合效应模型的结果表明，CMC题目可能导致了观察到的公平差距，尤其是对男女生而言，不过还需要更多证据。
我们发现CMC题目对每个人来说都更难。未来的研究应该探究这种难度的来源，以及该来源是否在功能上与学习和评估相关。我们的数据不支持使用CMC题目而非非CMC题目来区分成绩优异的学生和其他学生。
