大语言模型在良性和恶性胆管狭窄鉴别诊断中的表现

Performance of large language models in the differential diagnosis of benign and malignant biliary stricture.

作者信息

Kang Chenxi, Li Jing, Yang Xintian, Ren Gui, Zhang Linhui, Wang Wei, Liu Xin, Wang Lei, Shang Guochen, Hong Jianglong, Wan Bingnian, Du Yu, Zeng Wei, Liu Yaling, Li Tongxin, Lou Lijun, Luo Hui, Liang Shuhui, Lv Yong, Pan Yanglin

机构信息

Xijing Hospital of Digestive Diseases, Air Force Medical University, Xi'an, China.

Department of Gastroenterology, People's Liberation Army Joint Logistics Support Force 940th Hospital, Lanzhou, Gansu, China.

出版信息

Front Oncol. 2025 Jul 3;15:1613818. doi: 10.3389/fonc.2025.1613818. eCollection 2025.

DOI:10.3389/fonc.2025.1613818

PMID:40678070

原文链接:https://pmc.ncbi.nlm.nih.gov/articles/PMC12267023/

Abstract

BACKGROUND

Distinguishing benign from malignant biliary strictures remains challenging. Large Language Models (LLMs) show promise in enhancing diagnostic accuracy. This study aimed to evaluate the performances of ten LLMs in the differential diagnosis of benign and malignant biliary strictures.

METHODS

Consecutive patients with biliary strictures undergoing endoscopic retrograde cholangiopancreatography (ERCP) at Xijing Hospital between January and December 2024 were retrospectively analyzed. Ten LLMs were systematically prompted with standardized clinical, laboratory, and imaging data. Performance was compared against tumor markers (CA19-9, CEA), a new multivariable clinical model, and ten independent pancreaticobiliary exoerienced physicians. Subgroup analyses assessed hilar (n=29) versus non-hilar strictures. Gold-standard diagnosis relied on histopathology and ≥3-month follow-up.

RESULTS

Among the 159 included patients (83 benign, 76 malignant), four LLMs (Kimi, Deepseek-R1, Claude-3.5S, Llama-3.1), the clinical model (AUC:0.83), and six physicians achieved >80% accuracy. Kimi demonstrated superior accuracy (87%), significantly outperforming 70% of physicians (7/10, p<0.01). Three other LLMs (Deepseek-R1:83%, Claude-3.5S:82%, Llama-3.1:81%) and the clinical model performed comparably to physicians (78-84%, p>0.05), collectively surpassing tumor markers (CA19-9 accuracy:66%, CEA:71%). Physicians demonstrated higher accuracy for hilar strictures (87% vs. 79% for non-hilar, p<0.001). LLMs showed similar performance across stricture locations (hilar:64-95%; non-hilar:62-88%, p>0.05). For hilar strictures, 7/10 physicians achieved significantly higher accuracy (87-90%) than 8/10 LLMs (64-84%, p<0.05).

CONCLUSIONS

Using clinical, lab, and imaging data, some LLMs achieved diagnostic accuracy comparable to or exceeding clinical models and experienced physicians for differentiating benign versus malignant strictures. However, for hilar strictures, LLM performance was inferior to over half of the physicians.

摘要

背景

鉴别良性与恶性胆管狭窄仍然具有挑战性。大语言模型（LLMs）在提高诊断准确性方面显示出前景。本研究旨在评估十个大语言模型在良性和恶性胆管狭窄鉴别诊断中的表现。

方法

对2024年1月至12月在西京医院接受内镜逆行胰胆管造影（ERCP）的连续性胆管狭窄患者进行回顾性分析。用标准化的临床、实验室和影像数据系统地提示十个大语言模型。将其表现与肿瘤标志物（CA19-9、癌胚抗原）、一种新的多变量临床模型以及十位独立的胰腺胆管领域经验丰富的医生进行比较。亚组分析评估肝门部（n=29）与非肝门部狭窄。金标准诊断依赖于组织病理学和≥3个月的随访。

结果

在159例纳入患者（83例良性，76例恶性）中，四个大语言模型（豆包、百川-R1、Claude-3.5S、Llama-3.1）、临床模型（AUC：0.83）和六位医生的诊断准确率>80%。豆包表现出卓越的准确率（87%），显著优于70%的医生（7/10，p<0.01）。其他三个大语言模型（百川-R1：83%、Claude-3.5S：82%、Llama-3.1：81%）和临床模型的表现与医生相当（78 - 84%，p>0.05），总体上超过肿瘤标志物（CA19-9准确率：66%，癌胚抗原：71%）。医生对肝门部狭窄的诊断准确率更高（87%对非肝门部的79%，p<0.001）。大语言模型在不同狭窄部位的表现相似（肝门部：64 - 95%；非肝门部：62 - 88%，p>0.05）。对于肝门部狭窄，7/10的医生达到了显著更高的准确率（87 - 90%），高于8/10的大语言模型（64 - 84%，p<0.05）。