Rietberg Max Tigo, Nguyen Van Bach, Geerdink Jeroen, Vijlbrief Onno, Seifert Christin
Faculty of EEMCS, University of Twente, 7500 AE Enschede, The Netherlands.
Institute for Artificial Intelligence in Medicine, University of Duisburg-Essen, 45131 Essen, Germany.
Diagnostics (Basel). 2023 Mar 27;13(7):1251. doi: 10.3390/diagnostics13071251.
Understanding the diagnostic goal of medical reports is valuable information for understanding patient flows. This work focuses on extracting the reason for taking an MRI scan of Multiple Sclerosis (MS) patients using the attached free-form reports: Diagnosis, Progression or Monitoring. We investigate the performance of domain-dependent and general state-of-the-art language models and their alignment with domain expertise. To this end, eXplainable Artificial Intelligence (XAI) techniques are used to acquire insight into the inner workings of the model, which are verified on their trustworthiness. The verified XAI explanations are then compared with explanations from a domain expert, to indirectly determine the reliability of the model. BERTje, a Dutch Bidirectional Encoder Representations from Transformers (BERT) model, outperforms RobBERT and MedRoBERTa.nl in both accuracy and reliability. The latter model (MedRoBERTa.nl) is a domain-specific model, while BERTje is a generic model, showing that domain-specific models are not always superior. Our validation of BERTje in a small prospective study shows promising results for the potential uptake of the model in a practical setting.
了解医学报告的诊断目标对于理解患者流程是有价值的信息。这项工作专注于利用所附的自由格式报告提取对多发性硬化症（MS）患者进行磁共振成像（MRI）扫描的原因：诊断、病情进展或监测。我们研究了领域特定和通用的先进语言模型的性能及其与领域专业知识的一致性。为此，可解释人工智能（XAI）技术被用于深入了解模型的内部运作，并对其可信度进行验证。然后将经过验证的XAI解释与领域专家的解释进行比较，以间接确定模型的可靠性。荷兰的基于变换器的双向编码器表征（BERT）模型BERTje在准确性和可靠性方面均优于RobBERT和MedRoBERTa.nl。后一种模型（MedRoBERTa.nl）是特定领域模型，而BERTje是通用模型，这表明特定领域模型并不总是更优越。我们在一项小型前瞻性研究中对BERTje的验证显示，该模型在实际应用中具有潜在应用前景，结果令人鼓舞。
