McWilliams School of Biomedical Informatics, The University of Texas Health Science Center at Houston, Houston, TX, United States of America.
Department of Artificial Intelligence and Informatics, Mayo Clinic, Jacksonville, FL, United States of America.
PLoS One. 2024 Mar 21;19(3):e0300919. doi: 10.1371/journal.pone.0300919. eCollection 2024.
Though Vaccines are instrumental in global health, mitigating infectious diseases and pandemic outbreaks, they can occasionally lead to adverse events (AEs). Recently, Large Language Models (LLMs) have shown promise in effectively identifying and cataloging AEs within clinical reports. Utilizing data from the Vaccine Adverse Event Reporting System (VAERS) from 1990 to 2016, this study particularly focuses on AEs to evaluate LLMs' capability for AE extraction. A variety of prevalent LLMs, including GPT-2, GPT-3 variants, GPT-4, and Llama2, were evaluated using Influenza vaccine as a use case. The fine-tuned GPT 3.5 model (AE-GPT) stood out with a 0.704 averaged micro F1 score for strict match and 0.816 for relaxed match. The encouraging performance of the AE-GPT underscores LLMs' potential in processing medical data, indicating a significant stride towards advanced AE detection, thus presumably generalizable to other AE extraction tasks.
虽然疫苗在全球卫生领域发挥了重要作用，能够减轻传染病和大流行病的爆发，但它们偶尔也会引发不良反应（AE）。最近，大型语言模型（LLM）在有效识别和分类临床报告中的不良反应方面显示出了巨大的潜力。本研究利用了 1990 年至 2016 年疫苗不良事件报告系统（VAERS）的数据，特别关注 AE，以评估 LLM 提取 AE 的能力。研究评估了多种流行的 LLM，包括 GPT-2、GPT-3 变体、GPT-4 和 Llama2，使用流感疫苗作为用例。微调后的 GPT 3.5 模型（AE-GPT）表现出色，在严格匹配时的平均微 F1 得分为 0.704，在宽松匹配时为 0.816。AE-GPT 的令人鼓舞的表现突出了 LLM 在处理医疗数据方面的潜力，表明在先进的 AE 检测方面取得了重大进展，因此可能可以推广到其他 AE 提取任务中。
