Casey Arlene, Davidson Emma, Grover Claire, Tobin Richard, Grivas Andreas, Zhang Huayu, Schrempf Patrick, O'Neil Alison Q, Lee Liam, Walsh Michael, Pellie Freya, Ferguson Karen, Cvoro Vera, Wu Honghan, Whalley Heather, Mair Grant, Whiteley William, Alex Beatrice
Advanced Care Research Centre, Usher Institute, University of Edinburgh, Edinburgh, United Kingdom.
Centre for Clinical Brain Sciences, University of Edinburgh, Edinburgh, United Kingdom.
Front Digit Health. 2023 Sep 28;5:1184919. doi: 10.3389/fdgth.2023.1184919. eCollection 2023.
Natural language processing (NLP) has the potential to automate the reading of radiology reports, but there is a need to demonstrate that NLP methods are adaptable and reliable for use in real-world clinical applications.
We tested the F1 score, precision, and recall to compare NLP tools on a cohort from a study on delirium using images and radiology reports from NHS Fife and a population-based cohort (Generation Scotland) that spans multiple National Health Service health boards. We compared four off-the-shelf rule-based and neural NLP tools (namely, EdIE-R, ALARM+, ESPRESSO, and Sem-EHR) and reported on their performance for three cerebrovascular phenotypes, namely, ischaemic stroke, small vessel disease (SVD), and atrophy. Clinical experts from the EdIE-R team defined phenotypes using labelling techniques developed in the development of EdIE-R, in conjunction with an expert researcher who read underlying images.
EdIE-R obtained the highest F1 score in both cohorts for ischaemic stroke, ≥93%, followed by ALARM+, ≥87%. The F1 score of ESPRESSO was ≥74%, whilst that of Sem-EHR is ≥66%, although ESPRESSO had the highest precision in both cohorts, 90% and 98%. For F1 scores for SVD, EdIE-R scored ≥98% and ALARM+ ≥90%. ESPRESSO scored lowest with ≥77% and Sem-EHR ≥81%. In NHS Fife, F1 scores for atrophy by EdIE-R and ALARM+ were 99%, dropping in Generation Scotland to 96% for EdIE-R and 91% for ALARM+. Sem-EHR performed lowest for atrophy at 89% in NHS Fife and 73% in Generation Scotland. When comparing NLP tool output with brain image reads using F1 scores, ALARM+ scored 80%, outperforming EdIE-R at 66% in ischaemic stroke. For SVD, EdIE-R performed best, scoring 84%, with Sem-EHR 82%. For atrophy, EdIE-R and both ALARM+ versions were comparable at 80%.
The four NLP tools show varying F1 (and precision/recall) scores across all three phenotypes, although more apparent for ischaemic stroke. If NLP tools are to be used in clinical settings, this cannot be performed "out of the box." It is essential to understand the context of their development to assess whether they are suitable for the task at hand or whether further training, re-training, or modification is required to adapt tools to the target task.
自然语言处理（NLP）有潜力实现放射学报告阅读自动化，但有必要证明NLP方法在实际临床应用中具有适应性和可靠性。
我们测试了F1分数、精确率和召回率，以比较NLP工具在一项关于谵妄的研究队列中的表现，该队列使用了来自NHS法夫郡的图像和放射学报告以及一个涵盖多个国民保健服务健康委员会的基于人群的队列（苏格兰世代研究）。我们比较了四种现成的基于规则和神经网络的NLP工具（即EdIE-R、ALARM+、ESPRESSO和Sem-EHR），并报告了它们在三种脑血管表型（即缺血性中风、小血管疾病（SVD）和萎缩）方面的性能。EdIE-R团队的临床专家使用在EdIE-R开发过程中开发的标记技术，并结合一位阅读基础图像的专家研究人员来定义表型。
在两个队列中，EdIE-R在缺血性中风方面获得了最高的F1分数，≥93%，其次是ALARM+，≥87%。ESPRESSO的F1分数≥74%，而Sem-EHR的F1分数≥66%，尽管ESPRESSO在两个队列中的精确率最高，分别为90%和98%。对于SVD的F1分数，EdIE-R得分≥98%，ALARM+≥90%。ESPRESSO得分最低，≥77%，Sem-EHR≥81%。在NHS法夫郡，EdIE-R和ALARM+在萎缩方面的F1分数为99%，在苏格兰世代研究队列中，EdIE-R降至96%，ALARM+降至91%。Sem-EHR在萎缩方面表现最差，在NHS法夫郡为89%，在苏格兰世代研究队列中为73%。当使用F1分数将NLP工具输出与脑部图像解读进行比较时，在缺血性中风方面，ALARM+得分为80%，优于EdIE-R的66%。对于SVD，EdIE-R表现最佳，得分为84%，Sem-EHR为82%。对于萎缩，EdIE-R和两个版本的ALARM+相当，均为80%。
这四种NLP工具在所有三种表型上的F1（以及精确率/召回率）分数各不相同，尽管在缺血性中风方面更为明显。如果要在临床环境中使用NLP工具，不能“开箱即用”。必须了解其开发背景，以评估它们是否适合手头的任务，或者是否需要进一步训练、重新训练或修改，以使工具适应目标任务。
