Huisman Sophie M, Kraiss Jannis T, de Vos Jan Alexander
Department of Psychology, Health and Technology, Centre for eHealth and Wellbeing Research, University of Twente, Enschede, Netherlands.
Department of Research, GGZ Friesland Mental Healthcare Institution, Leeuwarden, Netherlands.
Front Psychiatry. 2024 Mar 13;15:1275236. doi: 10.3389/fpsyt.2024.1275236. eCollection 2024.
Clinicians collect session therapy notes within patient session records. Session records contain valuable information about patients' treatment progress. Sentiment analysis is a tool to extract emotional tones and states from text input and could be used to evaluate patients' sentiment during treatment over time. This preliminary study aims to investigate the validity of automated sentiment analysis on session patient records within an eating disorder (ED) treatment context against the performance of human raters.
A total of 460 patient session records from eight participants diagnosed with an ED were evaluated on their overall sentiment by an automated sentiment analysis and two human raters separately. The inter-rater agreement (IRR) between the automated analysis and human raters and IRR among the human raters was analyzed by calculating the intra-class correlation (ICC) under a continuous interpretation and weighted Cohen's kappa under a categorical interpretation. Furthermore, differences regarding positive and negative matches between the human raters and the automated analysis were examined in closer detail.
The ICC showed a moderate automated-human agreement (ICC = 0.55), and the weighted Cohen's kappa showed a fair automated-human (k = 0.29) and substantial human-human agreement (k = 0.68) for the evaluation of overall sentiment. Furthermore, the automated analysis lacked words specific to an ED context.
DISCUSSION/CONCLUSION: The automated sentiment analysis performed worse in discerning sentiment from session patient records compared to human raters and cannot be used within practice in its current state if the benchmark is considered adequate enough. Nevertheless, the automated sentiment analysis does show potential in extracting sentiment from session records. The automated analysis should be further developed by including context-specific ED words, and a more solid benchmark, such as patients' own mood, should be established to compare the performance of the automated analysis to.
临床医生在患者诊疗记录中收集每次诊疗的记录。诊疗记录包含有关患者治疗进展的宝贵信息。情感分析是一种从文本输入中提取情感基调与状态的工具，可用于评估患者在治疗过程中的情感变化。这项初步研究旨在针对人类评分者的表现，调查在饮食失调（ED）治疗背景下，对患者诊疗记录进行自动情感分析的有效性。
分别通过自动情感分析和两名人类评分者，对来自8名被诊断为饮食失调患者的460份患者诊疗记录的整体情感进行评估。通过在连续解释下计算组内相关系数（ICC）以及在分类解释下计算加权科恩kappa系数，分析自动分析与人类评分者之间的评分者间一致性（IRR）以及人类评分者之间的IRR。此外，还更详细地检查了人类评分者与自动分析在积极和消极匹配方面的差异。
ICC显示自动分析与人类评分者之间的一致性为中等（ICC = 0.55），加权科恩kappa系数显示在评估整体情感方面，自动分析与人类评分者之间的一致性为一般（k = 0.29），人类评分者之间的一致性为较高（k = 0.68）。此外，自动分析缺乏特定于饮食失调背景的词汇。
讨论/结论：与人类评分者相比，自动情感分析在从患者诊疗记录中辨别情感方面表现较差，如果认为现有基准足够充分，那么目前状态下的自动情感分析无法在实践中使用。然而，自动情感分析在从诊疗记录中提取情感方面确实显示出潜力。应通过纳入特定于饮食失调背景的词汇来进一步开发自动分析，并应建立一个更可靠的基准，例如患者自身的情绪，以便与自动分析的性能进行比较。
