Nair Balakrishnan R, Moonen-van Loon Joyce M W, van Lierop Marion, Govaerts Marjan
University of Newcastle, Centre for Medical Professional Development, Newcastle, Australia.
School of Health Professions Education, Faculty of Health, Medicine and Life Sciences, Maastricht University, Maastricht, the Netherlands.
Adv Med Educ Pract. 2024 Jul 15;15:671-683. doi: 10.2147/AMEP.S465259. eCollection 2024.
Current assessment approaches increasingly use narratives to support learning, coaching and high-stakes decision-making. Interpretation of narratives, however, can be challenging and time-consuming, potentially resulting in suboptimal or inadequate use of assessment data. Support for learners, coaches as well as decision-makers in the use and interpretation of these narratives therefore seems essential.
We explored the utility of automated text analysis techniques to support interpretation of narrative assessment data, collected across 926 clinical assessments of 80 trainees, in an International Medical Graduates' licensing program in Australia. We employed topic modelling and sentiment analysis techniques to automatically identify predominant feedback themes as well as the sentiment polarity of feedback messages. We furthermore sought to examine the associations between feedback polarity, numerical performance scores, and overall judgments about task performance.
Topic modelling yielded three distinctive feedback themes: Medical Skills, Knowledge, and Communication & Professionalism. The volume of feedback varied across topics and clinical settings, but assessors used more words when providing feedback to trainees who did not meet competence standards. Although sentiment polarity and performance scores did not seem to correlate at the level of single assessments, findings showed a strong positive correlation between the average performance scores and the average algorithmically assigned sentiment polarity.
This study shows that use of automated text analysis techniques can pave the way for a more efficient, structured, and meaningful learning, coaching, and assessment experience for learners, coaches and decision-makers alike. When used appropriately, these techniques may facilitate more meaningful and in-depth conversations about assessment data, by supporting stakeholders in interpretation of large amounts of feedback. Future research is vital to fully unlock the potential of automated text analysis, to support meaningful integration into educational practices.
当前的评估方法越来越多地使用叙述来支持学习、指导和高风险决策。然而，对叙述的解读可能具有挑战性且耗时，这可能导致评估数据的使用次优或不充分。因此，在使用和解读这些叙述时为学习者、指导者以及决策者提供支持似乎至关重要。
我们探索了自动文本分析技术在支持叙述性评估数据解读方面的效用，这些数据来自澳大利亚一个国际医学毕业生执照项目中对80名学员的926次临床评估。我们采用主题建模和情感分析技术来自动识别主要的反馈主题以及反馈信息的情感极性。此外，我们试图研究反馈极性、数字绩效分数与对任务表现的总体判断之间的关联。
主题建模产生了三个不同的反馈主题：医学技能、知识以及沟通与职业素养。反馈的数量因主题和临床环境而异，但评估者在向未达能力标准的学员提供反馈时使用的词汇更多。虽然情感极性和绩效分数在单次评估层面似乎没有相关性，但研究结果表明平均绩效分数与算法分配的平均情感极性之间存在很强的正相关。
本研究表明，使用自动文本分析技术可以为学习者、指导者和决策者带来更高效、结构化且有意义的学习、指导和评估体验。适当地使用这些技术，通过支持利益相关者解读大量反馈，可能会促进关于评估数据的更有意义和深入的对话。未来的研究对于充分释放自动文本分析的潜力、支持其有意义地融入教育实践至关重要。
