Walsh Daniel P, Chen Michael J, Buhl Lauren K, Neves Sara E, Mitchell John D
Department of Anesthesiology, Beth Israel Deaconess Medical Center, Boston, Massachusetts, USA.
J Med Educ Curric Dev. 2022 Jun 2;9:23821205221093205. doi: 10.1177/23821205221093205. eCollection 2022 Jan-Dec.
High quality feedback on resident clinical performance is pivotal to growth and development. Therefore, a reliable means of assessing faculty feedback is necessary. A feedback assessment instrument would also allow for appropriate focus of interventions to improve faculty feedback. We piloted an assessment of the interrater reliability of a seven-item feedback rating instrument on faculty educators trained via a three-workshop frame-of-reference training regimen. The rating instrument's items assessed for the presence or absence of six feedback traits: actionable, behavior focused, detailed, negative feedback, professionalism / communication, and specific; as well as for overall utility of feedback with regard to devising a resident performance improvement plan on an ordinal scale from 1 to 5. Participants completed three cycles consisting of one-hour-long workshops where an instructor led a review of the feedback rating instrument on deidentified feedback comments, followed by participants independently rating a set of 20 deidentified feedback comments, and the study team reviewing the interrater reliability for each feedback rating category to guide future workshops. Comments came from four different anesthesia residency programs in the United States; each set of feedback comments was balanced with respect to utility scores to promote participants' ability to discriminate between high and low utility comments. On the third and final independent rating exercise, participants achieved moderate or greater interrater reliability on all seven rating categories of a feedback rating instrument using Gwet's agreement coefficient 1 for the six feedback traits and using intraclass correlation for utility score. This illustrates that when this instrument is utilized by trained, expert educators, reliable assessments of faculty-provided feedback can be made. This rating instrument, with further validity evidence, has the potential to help programs reliably assess both the quality and utility of their feedback, as well as the impact of any educational interventions designed to improve feedback.
关于住院医师临床能力表现的高质量反馈对于其成长和发展至关重要。因此，有必要采用一种可靠的方法来评估教师的反馈。一种反馈评估工具还能够使干预措施更有针对性，以改善教师的反馈。我们对一项包含七个条目的反馈评级工具进行了评估，该工具用于评估通过为期三个工作坊的参照框架培训方案接受培训的教师教育工作者的评分者间信度。该评级工具的条目评估了六种反馈特质的有无：可操作、以行为为导向、详细、负面反馈、专业素养/沟通以及具体；同时还评估了反馈在制定住院医师表现改进计划方面的总体效用，采用1至5的顺序量表。参与者完成了三个周期，每个周期包括为期一小时的工作坊，由一名教员带领对去识别化的反馈意见进行反馈评级工具的回顾，随后参与者独立对一组20条去识别化的反馈意见进行评级，研究团队则审查每个反馈评级类别的评分者间信度，以指导未来的工作坊。反馈意见来自美国四个不同的麻醉住院医师培训项目；每组反馈意见在效用得分方面保持平衡，以提高参与者区分高效用和低效用意见的能力。在第三次也是最后一次独立评级练习中，参与者使用格韦特一致性系数1对六种反馈特质以及使用组内相关系数对效用得分，在反馈评级工具的所有七个评级类别上都达到了中等或更高的评分者间信度。这表明，当经过培训的专家教育工作者使用该工具时，可以对教师提供的反馈进行可靠评估。这种评级工具，若有进一步的效度证据，有可能帮助各培训项目可靠地评估其反馈的质量和效用，以及任何旨在改善反馈的教育干预措施的效果。
