Academic Centre for Medical Education, Division of Medical Education, University College London, Gower Street, London WC1E 6BT, UK.
BMC Med Educ. 2013 Jul 30;13:103. doi: 10.1186/1472-6920-13-103.
Bias of clinical examiners against some types of candidate, based on characteristics such as sex or ethnicity, would represent a threat to the validity of an examination, since sex or ethnicity are 'construct-irrelevant' characteristics. In this paper we report a novel method for assessing sex and ethnic bias in over 2000 examiners who had taken part in the PACES and nPACES (new PACES) examinations of the MRCP(UK).
PACES and nPACES are clinical skills examinations that have two examiners at each station who mark candidates independently. Differences between examiners cannot be due to differences in performance of a candidate because that is the same for the two examiners, and hence may result from bias or unreliability on the part of the examiners. By comparing each examiner against a 'basket' of all of their co-examiners, it is possible to identify examiners whose behaviour is anomalous. The method assessed hawkishness-doveishness, sex bias, ethnic bias and, as a control condition to assess the statistical method, 'even-number bias' (i.e. treating candidates with odd and even exam numbers differently). Significance levels were Bonferroni corrected because of the large number of examiners being considered.
The results of 26 diets of PACES and six diets of nPACES were examined statistically to assess the extent of hawkishness, as well as sex bias and ethnicity bias in individual examiners. The control (odd-number) condition suggested that about 5% of examiners were significant at an (uncorrected) 5% level, and that the method therefore worked as expected. As in a previous study (BMC Medical Education, 2006, 6:42), some examiners were hawkish or doveish relative to their peers. No examiners showed significant sex bias, and only a single examiner showed evidence consistent with ethnic bias. A re-analysis of the data considering only one examiner per station, as would be the case for many clinical examinations, showed that analysis with a single examiner runs a serious risk of false positive identifications probably due to differences in case-mix and content-specificity.
In examinations where there are two independent examiners at a station, our method can assess the extent of bias against candidates with particular characteristics. The method would be far less sensitive in examinations with only a single examiner per station as examiner variance would be confounded with candidate performance variance. The method however works well when there is more than one examiner at a station and in the case of the current MRCP(UK) clinical examination, nPACES, found possible sex bias in no examiners and possible ethnic bias in only one.
临床考官对某些类型的考生存在偏见，这种偏见基于性别或种族等特征，这将对考试的有效性构成威胁，因为性别或种族是“与构念无关”的特征。在本文中，我们报告了一种新方法，用于评估超过 2000 名考官在参加英国皇家内科医师学院（MRCP（UK））的 PACES 和 nPACES（新 PACES）考试中的性别和种族偏见。
PACES 和 nPACES 是临床技能考试，每个站点有两名考官独立打分。考官之间的差异不可能是由于考生表现的差异造成的，因为这对两名考官来说是相同的，因此可能是由于考官的偏见或不可靠性造成的。通过将每个考官与他们所有的共同考官的“篮子”进行比较，可以识别出行为异常的考官。该方法评估了严厉程度、性别偏见、种族偏见，以及作为评估统计方法的控制条件的“偶数偏见”（即对奇数和偶数考试编号的考生进行不同的处理）。由于考虑到大量的考官，因此使用了 Bonferroni 校正来确定显著性水平。
对 26 次 PACES 和 6 次 nPACES 的考试结果进行了统计学分析，以评估个别考官的严厉程度，以及性别偏见和种族偏见。控制（奇数）条件表明，大约 5%的考官在未经校正的 5%水平上显著，因此该方法按预期工作。与之前的一项研究（BMC Medical Education，2006，6:42）一样，一些考官相对于他们的同行较为严厉或温和。没有考官表现出显著的性别偏见，只有一位考官表现出与种族偏见一致的证据。仅考虑每个站点的一位考官（许多临床考试都是如此）对数据的重新分析表明，对单个考官的分析存在假阳性识别的严重风险，这可能是由于病例组合和内容特异性的差异造成的。
在每个站点有两名独立考官的考试中，我们的方法可以评估对具有特定特征的考生的偏见程度。在每个站点只有一名考官的考试中，该方法的敏感性要低得多，因为考官的方差会与考生表现的方差混淆。然而，当一个站点有多名考官时，该方法效果很好，而在当前的英国皇家内科医师学院临床考试 nPACES 中，没有发现考官存在性别偏见，只有一位考官可能存在种族偏见。
