Department of Assessment and Evaluation (AAE), Institute of Medical Education, University of Bern, Konsumstr 13, 3010, Bern, Switzerland.
Department of Education and Media, Institute of Medical Education, University of Bern, Bern, Switzerland.
Adv Health Sci Educ Theory Pract. 2018 Aug;23(3):455-463. doi: 10.1007/s10459-017-9805-y. Epub 2017 Nov 30.
Multiple true-false (MTF) items are a widely used supplement to the commonly used single-best answer (Type A) multiple choice format. However, an optimal scoring algorithm for MTF items has not yet been established, as existing studies yielded conflicting results. Therefore, this study analyzes two questions: What is the optimal scoring algorithm for MTF items regarding reliability, difficulty index and item discrimination? How do the psychometric characteristics of different scoring algorithms compare to those of Type A questions used in the same exams? We used data from 37 medical exams conducted in 2015 (998 MTF and 2163 Type A items overall). Using repeated measures analyses of variance (rANOVA), we compared reliability, difficulty and item discrimination of different scoring algorithms for MTF with four answer options and Type A. Scoring algorithms for MTF were dichotomous scoring (DS) and two partial credit scoring algorithms, PS where examinees receive half a point if more than half of true/false ratings were marked correctly and one point if all were marked correctly, and PS where examinees receive a quarter of a point for every correct true/false rating. The two partial scoring algorithms showed significantly higher reliabilities (α = 0.75; α = 0.75; α = 0.70, α = 0.72), which corresponds to fewer items needed for a reliability of 0.8 (n = 74; n = 75; n = 103, n = 87), and higher discrimination indices (r = 0.33; r = 0.33; r = 0.30; r = 0.28) than dichotomous scoring and Type A. Items scored with DS tend to be difficult (p = 0.50), whereas items scored with PS become easy (p = 0.82). PS and Type A cover the whole range, from easy to difficult items (p = 0.66; p = 0.73). Partial credit scoring leads to better psychometric results than dichotomous scoring. PS covers the range from easy to difficult items better than PS. Therefore, for scoring MTF, we suggest using PS.
多项真/假（MTF）项目是常用的单项最佳答案（A型）多项选择格式的广泛补充。然而，MTF 项目的最佳评分算法尚未建立，因为现有研究的结果相互矛盾。因此，本研究分析了两个问题：MTF 项目的最佳评分算法在可靠性、难度指数和项目区分度方面是什么？不同评分算法的心理测量特性与同一考试中使用的 A 型问题相比如何？我们使用了 2015 年进行的 37 次医学考试的数据（共有 998 个 MTF 和 2163 个 A 型项目）。使用重复测量方差分析（rANOVA），我们比较了四种答案选项的 MTF 和 A 型的不同评分算法的可靠性、难度和项目区分度。MTF 的评分算法为二分评分（DS）和两种部分计分评分算法，PS 中如果答对的真/假比例超过一半，考生得半分，如果全答对则得一分，PS 中考生每答对一个真/假比例得四分之一分。这两种部分计分算法的可靠性显著提高（α=0.75；α=0.75；α=0.70，α=0.72），这意味着需要更少的项目才能达到 0.8 的可靠性（n=74；n=75；n=103，n=87），且区分度更高（r=0.33；r=0.33；r=0.30；r=0.28）比二分评分和 A 型。用 DS 评分的项目往往较难（p=0.50），而用 PS 评分的项目则变得简单（p=0.82）。PS 和 A 型涵盖了从简单到困难的项目范围（p=0.66；p=0.73）。部分计分评分比二分评分产生更好的心理测量结果。PS 比 PS 更能覆盖从简单到困难的项目范围。因此，对于 MTF 的评分，我们建议使用 PS。
