Moonen-van Loon Joyce M W, Overeem Karlijn, Govaerts Marjan J B, Verhoeven Bas H, van der Vleuten Cees P M, Driessen Erik W
J.M.W. Moonen-van Loon is postdoctoral researcher, Department of Educational Development and Research, Maastricht University, Maastricht, The Netherlands. K. Overeem is postdoctoral researcher, Department of Educational Development and Research, Maastricht University, Maastricht, The Netherlands. M.J.B. Govaerts is assistant professor, Department of Educational Development and Research, Maastricht University, Maastricht, The Netherlands. B.H. Verhoeven is pediatric surgeon, Department of Surgery, Radboud University Medical Center, Nijmegen, and assistant professor, Department of Educational Development and Research, Maastricht University, Maastricht, The Netherlands. C.P.M. van der Vleuten is professor of education, Department of Educational Development and Research, Maastricht University, Maastricht, The Netherlands. E.W. Driessen is associate professor of education, Department of Educational Development and Research, Maastricht University, Maastricht, The Netherlands.
Acad Med. 2015 Aug;90(8):1093-9. doi: 10.1097/ACM.0000000000000763.
Residency programs around the world use multisource feedback (MSF) to evaluate learners' performance. Studies of the reliability of MSF show mixed results. This study aimed to identify the reliability of MSF as practiced across occasions with varying numbers of assessors from different professional groups (physicians and nonphysicians) and the effect on the reliability of the assessment for different competencies when completed by both groups.
The authors collected data from 2008 to 2012 from electronically completed MSF questionnaires. In total, 428 residents completed 586 MSF occasions, and 5,020 assessors provided feedback. The authors used generalizability theory to analyze the reliability of MSF for multiple occasions, different competencies, and varying numbers of assessors and assessor groups across multiple occasions.
A reliability coefficient of 0.800 can be achieved with two MSF occasions completed by at least 10 assessors per group or with three MSF occasions completed by 5 assessors per group. Nonphysicians' scores for the "Scholar" and "Health advocate" competencies and physicians' scores for the "Health advocate" competency had a negative effect on the composite reliability.
A feasible number of assessors per MSF occasion can reliably assess residents' performance. Scores from a single occasion should be interpreted cautiously. However, every occasion can provide valuable feedback for learning. This research confirms that the (unique) characteristics of different assessor groups should be considered when interpreting MSF results. Reliability seems to be influenced by the included assessor groups and competencies. These findings will enhance the utility of MSF during residency training.
世界各地的住院医师培训项目使用多源反馈（MSF）来评估学员的表现。关于MSF可靠性的研究结果不一。本研究旨在确定在不同场合下，由不同专业组（医生和非医生）的不同数量评估者实施的MSF的可靠性，以及两组评估者完成评估时对不同能力可靠性的影响。
作者收集了2008年至2012年通过电子方式完成的MSF问卷数据。共有428名住院医师完成了586次MSF评估，5020名评估者提供了反馈。作者使用概化理论分析MSF在多个场合、不同能力以及不同数量评估者和多场合评估者组中的可靠性。
每组至少10名评估者完成两次MSF评估，或每组5名评估者完成三次MSF评估时，可靠性系数可达0.800。非医生对“学者”和“健康倡导者”能力的评分以及医生对“健康倡导者”能力的评分对综合可靠性有负面影响。
每次MSF评估中可行数量的评估者能够可靠地评估住院医师的表现。单次评估的分数应谨慎解释。然而，每次评估都可为学习提供有价值的反馈。本研究证实，在解释MSF结果时应考虑不同评估者组的（独特）特征。可靠性似乎受所纳入评估者组和能力的影响。这些发现将提高MSF在住院医师培训期间的效用。
