Yasaka Koichiro, Nomura Takuto, Kamohara Jun, Hirakawa Hiroshi, Kubo Takatoshi, Kiryu Shigeru, Abe Osamu
Department of Radiology, The University of Tokyo Hospital, 7-3-1 Hongo, Bunkyo-Ku, Tokyo, 113-8655, Japan.
Department of Radiology, International University of Health and Welfare Narita Hospital, 852 Hatakeda, Narita, Chiba, 286-0124, Japan.
J Imaging Inform Med. 2024 Dec 13. doi: 10.1007/s10278-024-01370-w.
The aim of this study is to develop a fine-tuned large language model that classifies interventional radiology reports into technique categories and to compare its performance with readers. This retrospective study included 3198 patients (1758 males and 1440 females; age, 62.8 ± 16.8 years) who underwent interventional radiology from January 2018 to July 2024. Training, validation, and test datasets involved 2292, 250, and 656 patients, respectively. Input data involved texts in clinical indication, imaging diagnosis, and image-finding sections of interventional radiology reports. Manually classified technique categories (15 categories in total) were utilized as reference data. Fine-tuning of the Bidirectional Encoder Representations model was performed using training and validation datasets. This process was repeated 15 times due to the randomness of the learning process. The best-performed model, which showed the highest accuracy among 15 trials, was selected to further evaluate its performance in the independent test dataset. The report classification involved one radiologist (reader 1) and two radiology residents (readers 2 and 3). The accuracy and macrosensitivity (average of each category's sensitivity) of the best-performed model in the validation dataset were 0.996 and 0.994, respectively. For the test dataset, the accuracy/macrosensitivity were 0.988/0.980, 0.986/0.977, 0.989/0.979, and 0.988/0.980 in the best model, reader 1, reader 2, and reader 3, respectively. The model required 0.178 s required for classification per patient, which was 17.5-19.9 times faster than readers. In conclusion, fine-tuned large language model classified interventional radiology reports into technique categories with high accuracy similar to readers within a remarkably shorter time.
本研究的目的是开发一种经过微调的大语言模型，将介入放射学报告分类到技术类别中，并将其性能与阅片者进行比较。这项回顾性研究纳入了2018年1月至2024年7月期间接受介入放射学检查的3198例患者（男性1758例，女性1440例；年龄62.8±16.8岁）。训练、验证和测试数据集分别涉及2292例、250例和656例患者。输入数据包括介入放射学报告的临床指征、影像诊断和影像表现部分的文本。手动分类的技术类别（共15类）用作参考数据。使用训练和验证数据集对双向编码器表示模型进行微调。由于学习过程的随机性，此过程重复了15次。选择在15次试验中表现最佳、准确率最高的模型，以进一步评估其在独立测试数据集中的性能。报告分类由一名放射科医生（阅片者1）和两名放射科住院医师（阅片者2和3）进行。在验证数据集中，表现最佳的模型的准确率和宏敏感度（各类别敏感度的平均值）分别为0.996和0.994。对于测试数据集，最佳模型、阅片者1、阅片者2和阅片者3的准确率/宏敏感度分别为0.988/0.980、0.986/0.977、0.989/0.979和0.988/0.980。该模型对每位患者进行分类所需时间为0.178秒，比阅片者快17.5至19.9倍。总之，经过微调的大语言模型能够在显著更短的时间内，以与阅片者相似的高准确率将介入放射学报告分类到技术类别中。
