Charité Comprehensive Cancer Center, Charité-Universitätsmedizin Berlin, Corporate Member of Freie Universität Berlin and Humboldt-Universität zu Berlin, Berlin, Germany.
Core Unit Bioinformatics, Berlin Institute of Health at Charité-Universitätsmedizin Berlin, Charitéplatz 1, Berlin, Germany.
JAMA Netw Open. 2023 Nov 1;6(11):e2343689. doi: 10.1001/jamanetworkopen.2023.43689.
Clinical interpretation of complex biomarkers for precision oncology currently requires manual investigations of previous studies and databases. Conversational large language models (LLMs) might be beneficial as automated tools for assisting clinical decision-making.
To assess performance and define their role using 4 recent LLMs as support tools for precision oncology.
DESIGN, SETTING, AND PARTICIPANTS: This diagnostic study examined 10 fictional cases of patients with advanced cancer with genetic alterations. Each case was submitted to 4 different LLMs (ChatGPT, Galactica, Perplexity, and BioMedLM) and 1 expert physician to identify personalized treatment options in 2023. Treatment options were masked and presented to a molecular tumor board (MTB), whose members rated the likelihood of a treatment option coming from an LLM on a scale from 0 to 10 (0, extremely unlikely; 10, extremely likely) and decided whether the treatment option was clinically useful.
Number of treatment options, precision, recall, F1 score of LLMs compared with human experts, recognizability, and usefulness of recommendations.
For 10 fictional cancer patients (4 with lung cancer, 6 with other; median [IQR] 3.5 [3.0-4.8] molecular alterations per patient), a median (IQR) number of 4.0 (4.0-4.0) compared with 3.0 (3.0-5.0), 7.5 (4.3-9.8), 11.5 (7.8-13.0), and 13.0 (11.3-21.5) treatment options each was identified by the human expert and 4 LLMs, respectively. When considering the expert as a criterion standard, LLM-proposed treatment options reached F1 scores of 0.04, 0.17, 0.14, and 0.19 across all patients combined. Combining treatment options from different LLMs allowed a precision of 0.29 and a recall of 0.29 for an F1 score of 0.29. LLM-generated treatment options were recognized as AI-generated with a median (IQR) 7.5 (5.3-9.0) points in contrast to 2.0 (1.0-3.0) points for manually annotated cases. A crucial reason for identifying AI-generated treatment options was insufficient accompanying evidence. For each patient, at least 1 LLM generated a treatment option that was considered helpful by MTB members. Two unique useful treatment options (including 1 unique treatment strategy) were identified only by LLM.
In this diagnostic study, treatment options of LLMs in precision oncology did not reach the quality and credibility of human experts; however, they generated helpful ideas that might have complemented established procedures. Considering technological progress, LLMs could play an increasingly important role in assisting with screening and selecting relevant biomedical literature to support evidence-based, personalized treatment decisions.
目前，临床解读精准肿瘤学的复杂生物标志物需要手动调查先前的研究和数据库。会话式大型语言模型 (LLM) 可能是一种有益的自动化工具，可以帮助临床决策。
使用最近的 4 个 LLM 作为精准肿瘤学的支持工具，评估性能并定义其作用。
设计、设置和参与者：本诊断研究检查了 10 名患有遗传改变的晚期癌症患者的 10 个虚构病例。每个病例都提交给了 4 个不同的 LLM（ChatGPT、Galactica、Perplexity 和 BiomedLM）和 1 名专家医生，以在 2023 年确定个性化的治疗方案。治疗方案被屏蔽并呈现给分子肿瘤委员会 (MTB)，其成员对 LLM 提出的治疗方案的可能性进行评分，范围从 0 到 10（0，极不可能；10，极有可能），并决定治疗方案是否具有临床意义。
治疗方案的数量、精度、召回率、LLM 与人类专家相比的 F1 评分、可识别性和推荐的有用性。
对于 10 名虚构的癌症患者（4 名肺癌，6 名其他；每名患者的中位数 [IQR] 有 3.5 [3.0-4.8] 个分子改变），人类专家和 4 个 LLM 分别确定了中位数（IQR）为 4.0 (4.0-4.0) 与 3.0 (3.0-5.0)、7.5 (4.3-9.8)、11.5 (7.8-13.0) 和 13.0 (11.3-21.5) 个治疗方案。当将专家作为标准进行考虑时，所有患者的 LLM 提出的治疗方案的 F1 评分分别为 0.04、0.17、0.14 和 0.19。结合来自不同 LLM 的治疗方案可以达到 0.29 的精度和 0.29 的召回率，F1 评分为 0.29。LLM 生成的治疗方案被识别为 AI 生成，中位数（IQR）为 7.5（5.3-9.0）分，而手动注释病例为 2.0（1.0-3.0）分。确定 AI 生成的治疗方案的一个关键原因是缺乏伴随的证据。对于每个患者，至少有 1 个 LLM 生成了 MTB 成员认为有用的治疗方案。只有 LLM 确定了 2 个独特有用的治疗方案（包括 1 个独特的治疗策略）。
在这项诊断研究中，精准肿瘤学中 LLM 的治疗方案没有达到人类专家的质量和可信度；然而，它们产生了有用的想法，可能补充了既定的程序。考虑到技术进步，LLM 可能在协助筛选和选择相关生物医学文献以支持基于证据的个性化治疗决策方面发挥越来越重要的作用。
Zotero 插件
