Mottin Luc, Goldman Jean-Philippe, Jäggli Christoph, Achermann Rita, Gobeill Julien, Knafou Julien, Ehrsam Julien, Wicky Alexandre, Gérard Camille L, Schwenk Tanja, Charrier Mélinda, Tsantoulis Petros, Lovis Christian, Leichtle Alexander, Kiessling Michael K, Michielin Olivier, Pradervand Sylvain, Foufi Vasiliki, Ruch Patrick
HES-SO\HEG Genève, Information Sciences, Geneva, Switzerland.
SIB Text Mining Group, Swiss Institute of Bioinformatics, Geneva, Switzerland.
Front Digit Health. 2023 Jun 14;5:1195017. doi: 10.3389/fdgth.2023.1195017. eCollection 2023.
The objective of this study is the exploration of Artificial Intelligence and Natural Language Processing techniques to support the automatic assignment of the four Response Evaluation Criteria in Solid Tumors (RECIST) scales based on radiology reports. We also aim at evaluating how languages and institutional specificities of Swiss teaching hospitals are likely to affect the quality of the classification in French and German languages.
In our approach, 7 machine learning methods were evaluated to establish a strong baseline. Then, robust models were built, fine-tuned according to the language (French and German), and compared with the expert annotation.
The best strategies yield average F1-scores of 90% and 86% respectively for the 2-classes (Progressive/Non-progressive) and the 4-classes (Progressive Disease, Stable Disease, Partial Response, Complete Response) RECIST classification tasks.
These results are competitive with the manual labeling as measured by Matthew's correlation coefficient and Cohen's Kappa (79% and 76%). On this basis, we confirm the capacity of specific models to generalize on new unseen data and we assess the impact of using Pre-trained Language Models (PLMs) on the accuracy of the classifiers.
本研究的目的是探索人工智能和自然语言处理技术，以支持基于放射学报告自动分配实体瘤的四种疗效评价标准（RECIST）量表。我们还旨在评估瑞士教学医院的语言和机构特殊性如何可能影响法语和德语分类的质量。
在我们的方法中，评估了7种机器学习方法以建立一个强大的基线。然后，构建稳健的模型，根据语言（法语和德语）进行微调，并与专家注释进行比较。
对于两类（进展/非进展）和四类（疾病进展、疾病稳定、部分缓解、完全缓解）RECIST分类任务，最佳策略分别产生了90%和86%的平均F1分数。
根据马修斯相关系数和科恩卡方系数衡量，这些结果与人工标注具有竞争力（分别为79%和76%）。在此基础上，我们确认了特定模型对新的未见数据进行泛化的能力，并评估了使用预训练语言模型（PLM）对分类器准确性的影响。
