Adelaide Medical School, The University of Adelaide, Adelaide, 5005, South Australia.
Department of Medical Education, Melbourne Medical School, The University of Melbourne, Melbourne, VIC, 3010, Australia.
BMC Med Educ. 2018 Apr 23;18(1):80. doi: 10.1186/s12909-018-1190-6.
Standard setting of assessment is critical in quality assurance of medical programs. The aims of this study were to identify and compare the impact of methods used to establish the passing standard by the 13 medical schools who participated in the 2014 Australian Medical Schools Assessment Collaboration (AMSAC).
A survey was conducted to identify the standard setting procedures used by participating schools. Schools standard setting data was collated for the 49 multiple choice items used for benchmarking by AMSAC in 2014. Analyses were conducted for nine schools by their method of standard setting and key characteristics of 28 panel members from four schools.
Substantial differences were identified between AMSAC schools that participated in the study, in both the standard setting methods and how particular techniques were implemented. The correlation between the item standard settings data by school ranged from - 0.116 to 0.632. A trend was identified for panel members to underestimate the difficulty level of hard items and overestimate the difficulty level of easy items for all methods. The median derived cut-score standard across schools was 55% for the 49 benchmarking questions. Although, no significant differences were found according to panel member standard setting experience or clinicians versus scientists, panel members with a high curriculum engagement generally had significantly lower expectations of borderline candidates (p = 0.044).
This study used a robust assessment framework to demonstrate that several standard setting techniques are used by Australian medical schools, which in some cases use different techniques for different stages of their program. The implementation of the most common method, the Modified Angoff standard setting approach was found to vary markedly. The method of standard setting used had an impact on the distribution of expected minimally competent student performance by item and overall, with the passing standard varying by up to 10%. This difference can be attributed to the method of standard setting because the ASMSAC items have been shown over time to have consistent performance levels reflecting similar cohort ability. There is a need for more consistency in the method of standard setting used by medical schools in Australia.
评估标准的制定对于医学项目的质量保证至关重要。本研究的目的是确定并比较参与 2014 年澳大利亚医学学校评估合作（AMSAC）的 13 所医学院校采用的方法对通过标准的影响。
进行了一项调查，以确定参与学校使用的标准制定程序。为 2014 年 AMSAC 用于基准测试的 49 个多项选择题收集了学校的标准设置数据。对九所学校按其标准设定方法和来自四所学校的 28 名小组成员的关键特征进行了分析。
在所研究的 AMSAC 学校中，标准设定方法以及特定技术的实施方式存在很大差异。学校之间的项目标准设定数据相关性在-0.116 到 0.632 之间。对于所有方法，小组成员都存在低估难题难度和高估简单题难度的趋势。9 所学校的中位数衍生分数标准为 49 个基准问题的 55%。尽管根据小组成员的标准设定经验或临床医生与科学家之间没有发现显著差异，但具有高课程参与度的小组成员通常对边缘候选人的期望明显较低（p=0.044）。
本研究使用了一个强大的评估框架，表明澳大利亚医学院校使用了几种标准设定技术，在某些情况下，针对其课程的不同阶段使用不同的技术。最常用的方法，即改良的 Angoff 标准设定方法的实施情况差异显著。标准设定方法的使用对每个项目和总体上的预期合格学生表现分布产生了影响，通过标准的差异可达 10%。这种差异可以归因于标准设定方法，因为随着时间的推移，ASMSAC 项目的表现水平一直保持一致，反映了相似的队列能力。澳大利亚医学院校在标准设定方法的使用上需要更加一致。
