“但为什么呢？”：解释性反馈是手术表现高质量叙事评估的可靠标志。
"But Why?": Explanatory Feedback Is a Reliable Marker of High-Quality Narrative Assessment of Surgical Performance.
作者信息
Poeppelman Rachel Stork, Cho Junsang, Nachbor Kristine, Sekhar Tejas C, Pruett Jack, Baim Adam, Strul Sasha, Barsam Alex, Langworthy Benjamin, Waxman Evan L, Culican Susan M
出版信息
Acad Med. 2025 May 1;100(5):614-620. doi: 10.1097/ACM.0000000000005985. Epub 2025 Jan 31.
PURPOSE
This study examines the quality of short narrative comments collected using an online workplace-based assessment (WBA) tool.
METHOD
The quality of comments collected by a WBA tool at the UPMC Ophthalmology Residency Training Program was evaluated between July 2017-June 2020. A randomized rating exercise involving 10 meta-raters from 6 institutions was performed to evaluate the value of narratives from deidentified WBAs. The tool captured a single-item entrustment competency question with brief comments. Comments were evaluated using a Quality of Assessment of Learning (QuAL) score (range, 0-5; ≥3 considered high quality) and on whether the assessor provided a feedback rationale.
RESULTS
A total of 838 unique WBAs were collected from 15 attending evaluators. Comments were brief (median [interquartile range] length, 11 [7-17] words), yet 514 (61.3%) were rated as high quality (QuAL score ≥3). Of all 838 comments, 98 (11.7%) included a specific reason the evidence or suggestion was provided to the learner. Of these 98 comments, 94 (95.9%) met the high-quality feedback threshold. A higher QuAL score was associated with a higher postgraduate year (PGY) level (estimate [SE], 1.603 [0.428], P < .001 for PGY2 [reference]; 1.003 [0.389], P = .01 for PGY3; 1.079 [0.360], P = .003 for PGY4), suggesting more advanced learners receive higher-quality narrative comments. A correlation was found between a higher entrustment rating and a lower QuAL score (estimate [SE], -0.199 [0.053], P < .001). When the PGY level was controlled for, this association got stronger (estimate [SE], -0.310 [0.057], P < .001).
CONCLUSIONS
Analysis of WBA comments from attending physicians evaluated using the QuAL score demonstrated that most comments were high quality despite their brevity. Residents in later training years and with lower entrustment ratings received higher-quality comments. High-quality narrative assessments were longer and addressed rationale as part of the comment.
目的
本研究探讨使用基于工作场所的在线评估（WBA）工具收集的简短叙述性评语的质量。
方法
对2017年7月至2020年6月期间在匹兹堡大学医学中心眼科住院医师培训项目中通过WBA工具收集的评语质量进行评估。进行了一项随机评分活动，涉及来自6个机构的10名元评分者，以评估去识别化WBA中叙述的价值。该工具捕获了一个单项委托能力问题及简短评语。使用学习评估质量（QuAL）评分（范围为0 - 5；≥3被认为是高质量）以及评估者是否提供反馈理由来评估评语。
结果
共从15名主治评估者处收集到838份独特的WBA。评语简短（中位数[四分位间距]长度为11[7 - 17]个单词），但514份（61.3%）被评为高质量（QuAL评分≥3）。在所有838条评语中，98条（11.7%）包含向学习者提供证据或建议的具体原因。在这98条评语中，94条（95.9%）达到了高质量反馈阈值。较高的QuAL评分与更高的研究生年级（PGY）水平相关（估计值[标准误]，PGY2[参照组]为1.603[0.428]，P <.001；PGY3为1.003[0.389]，P =.01；PGY4为1.079[0.360]，P =.003），这表明更高级的学习者收到更高质量的叙述性评语。发现较高的委托评分与较低的QuAL评分之间存在相关性（估计值[标准误]， - 0.199[0.053]，P <.001）。当控制PGY水平时，这种关联更强（估计值[标准误]， - 0.310[0.057]，P <.001）。
结论
使用QuAL评分对主治医生的WBA评语进行分析表明，尽管评语简短，但大多数评语质量较高。处于后期培训阶段且委托评分较低的住院医师收到的评语质量更高。高质量的叙述性评估更长，并将理由作为评语的一部分。
Zotero 插件