Ryan J G, Mandel F S, Sama A, Ward M F
North Shore University Hospital, Department of Emergency Medicine, Manhasset, NY 11030, USA.
Acad Emerg Med. 1996 Dec;3(12):1124-30. doi: 10.1111/j.1553-2712.1996.tb03372.x.
To assess the reliability of faculty evaluations of non-emergency medicine (non-EM) residents during clinical ED rotations and to determine the effect that the "leniency" of grading by these evaluators had on the residents' final evaluations.
A prospective, observational study of the evaluation patterns of EM faculty was performed in an academic ED (50,000 visits yearly census). Each resident was evaluated on a daily basis by a board-certified or board-prepared emergency physician. The evaluation form rated 7 characteristics, but only the rating for overall clinical competence was used for data analysis. If an attending evaluated the same resident more than once, only the first evaluation was used to avoid bias from prior exposure. The scoring patterns of the evaluators, both individually and in groups, were analyzed using 1-way analysis of variance. Evaluator leniency was estimated using the mean evaluator score across all residents. Since each resident was evaluated by a different combination of evaluators, evaluator leniency for each resident was estimated from the mean leniency of the evaluators who specifically assessed that resident.
During the period of the study, 66 residents rotated through the ED, yielding a total of 401 evaluations. When the scoring patterns of individual evaluators were analyzed, a high degree of variability was found in the mean scores (range 5.23-8.09) and SDs (range 0.45-1.55) across evaluators; p = 0.0001. There was a moderate correlation between the mean overall competence score received by each resident and that resident's evaluators' leniency, r = 0.52; p = 0.0001.
There is significant variability in the scoring patterns of individual evaluators. The evaluators in this study showed large variations in both leniency (as measured by their mean score) and range restriction (as measured by their SD). The differences in evaluator scoring leniency have a moderate correlation with the overall score received by the resident.
评估急诊医学（EM）教员在临床急诊科轮转期间对非急诊医学（non-EM）住院医师评估的可靠性，并确定这些评估者评分“宽松”程度对住院医师最终评估结果的影响。
在一家学术性急诊科（年就诊量50000人次）对EM教员的评估模式进行了一项前瞻性观察研究。每位住院医师每天由一名获得委员会认证或具备委员会资质的急诊医师进行评估。评估表对7项特征进行评分，但数据分析仅使用总体临床能力评分。如果一名主治医师对同一名住院医师进行多次评估，则仅使用首次评估结果以避免先前接触造成的偏差。使用单因素方差分析对评估者个体及群体的评分模式进行分析。通过所有住院医师的评估者平均得分来估计评估者的宽松程度。由于每位住院医师由不同的评估者组合进行评估，因此根据专门评估该住院医师的评估者的平均宽松程度来估计每位住院医师的评估者宽松程度。
在研究期间，66名住院医师轮转至急诊科，共产生401份评估。分析个体评估者的评分模式时，发现评估者之间的平均得分（范围5.23 - 8.09）和标准差（范围0.45 - 1.55）存在高度变异性；p = 0.0001。每位住院医师获得的总体能力平均得分与其评估者的宽松程度之间存在中度相关性，r = 0.52；p = 0.0001。
个体评估者的评分模式存在显著变异性。本研究中的评估者在宽松程度（以其平均得分衡量）和范围限制（以其标准差衡量）方面均表现出较大差异。评估者评分宽松程度的差异与住院医师获得的总体评分存在中度相关性。
