Furey Michael J, Stemrich Raymond, Westfall-Snyder Jamaica, Gupta Tanvi, Rapp Megan, Hoffman Rebecca L
Department of Surgery, Geisinger Wyoming Valley Medical Center, Wilkes Barre, Pennsylvania.
Department of Surgery, Geisinger Medical Center, Danville, Pennsylvania.
J Surg Res. 2025 Apr;308:300-306. doi: 10.1016/j.jss.2025.01.029. Epub 2025 Mar 27.
Feedback is at the core of competency-based medical education. Learner perceptions of the evaluation process influence how feedback is utilized. Systems emphasize a fixed mindset, prioritizing evaluation over growth. Embracing growth mindset culture, the belief that ability is acquired through effort and human capabilities can be developed over time, will allow learners to gain greater benefits from feedback. Transitioning from fixed mindset language (FML) to growth mindset language (GML) will require faculty training. Artificial intelligence (AI) can assist faculty with incorporating GML concepts in written feedback. The aim of this study was to assess the ability of AI to assist in changing FML feedback statements into statements with GML.
A qualitative study was performed utilizing a sample of 83 summative and formative feedback statements provided to students (37) and residents (46) from surgery clerkship and national SIMPL-inguinal hernia evaluations. Of these 83 statements, a reviewer coded 41 statements as using GML and 42 using FML. Original statements identified as using FML were entered into the Google Chrome "Help me write" tool, a writing aid using Generative AI. The AI tool was prompted with the statement "rewrite using growth mindset language:," followed by an original FML statement. A dataset containing a combination of AI-altered and original statements, 99 statements in all, was provided to two additional blinded reviewers trained in GML concepts. Reviewers evaluated statements as predominantly GML or FML and commented on their perception of AI use in statements. Reviewer agreement was adjudicated by the original coder.
Of the 41 original GML statements, coders correctly identified 37 (90.2%) as using GML. Of the 26 original FML statements, coders correctly identified all 26 (100%) as using FML. Of the AI-modified FML to GML statements, coders correctly identified 17 of 18 (94.4%) as using GML. They correctly identified 56.3% as AI-modified and 44.8% as not AI-modified statements. They disagreed on AI use in 39.4% of statements. AI-assistance was unrecognized in 16 (8.1%) statements and mistaken for use in 47 (23.7%) statements.
AI was successful at modifying FML statements into feedback containing GML, and in a way that was not obviously AI-generated. This proof-of-concept study demonstrates that AI can be a helpful tool for faculty to increase the use of GML in written feedback. While AI cannot perfectly create GML feedback without initial input and understanding from faculty, it does serve as a promising educational aid. As the body of work on using GML in surgical education grows, the better AI can assist in the generation of quality feedback.
反馈是基于胜任力的医学教育的核心。学习者对评估过程的认知会影响反馈的利用方式。当前的系统强调固定型思维模式，将评估置于成长之上。而接受成长型思维模式文化，即相信能力是通过努力获得的，且人的能力会随着时间发展，将使学习者能从反馈中获得更大益处。从固定型思维模式语言（FML）转变为成长型思维模式语言（GML）需要对教师进行培训。人工智能（AI）可以帮助教师在书面反馈中融入GML概念。本研究的目的是评估AI将FML反馈语句转换为GML语句的能力。
进行了一项定性研究，使用了从外科实习和全国简化腹股沟疝评估中提供给学生（37名）和住院医师（46名）的83条总结性和形成性反馈语句样本。在这83条语句中，一名评审员将41条语句编码为使用GML，42条使用FML。被确定为使用FML的原始语句被输入到谷歌浏览器的“帮我写”工具中，这是一个使用生成式AI的写作辅助工具。AI工具被输入语句“使用成长型思维模式语言重写：”，后面跟着一条原始的FML语句。一个包含AI修改后的语句和原始语句组合的数据集，总共99条语句，被提供给另外两名接受过GML概念培训的盲评人员。评审员将语句评估为主要是GML或FML，并对他们对语句中AI使用的看法发表评论。评审员的一致性由原始编码人员裁定。
在41条原始GML语句中，编码人员正确识别出37条（90.2%）使用GML。在26条原始FML语句中，编码人员正确识别出所有26条（100%）使用FML。在AI将FML修改为GML的语句中，编码人员正确识别出18条中的17条（94.4%）使用GML。他们正确识别出56.3%为AI修改后的语句，44.8%为未修改的语句。他们在39.4%的语句中对AI的使用存在分歧。在16条（8.1%）语句中未识别出AI的帮助，在47条（23.7%）语句中被误认为使用了AI。
AI成功地将FML语句修改为包含GML的反馈，且方式并非明显由AI生成。这项概念验证研究表明，AI可以成为教师在书面反馈中增加GML使用的有用工具。虽然没有教师的初始输入和理解，AI无法完美地创建GML反馈，但它确实是一种有前景的教育辅助工具。随着在外科教育中使用GML的工作不断增加，AI能更好地协助生成高质量反馈。
