Fang Shichao, Holgate Ben, Shek Anthony, Winston Joel S, McWilliam Matthew, Viana Pedro F, Teo James T, Richardson Mark P
Department of Basic & Clinical Neuroscience, King's College London, London, UK.
King's College Hospital NHS Foundation Trust, London, UK.
Epilepsia. 2025 Jul 10. doi: 10.1111/epi.18475.
The emergence of large language models (LLMs) and the increasing prevalence of electronic health records (EHRs) present significant opportunities for advancing health care research and practice. However, research that compares and applies LLMs to extract key epilepsy-related information from unstructured medical free text is under-explored. This study fills this gap by comparing and applying different open-source LLMs and methods to extract epilepsy information from unstructured clinic letters, thereby optimizing EHRs as a resource for the benefit of epilepsy research. We also highlight some limitations of LLMs.
Employing a dataset of 280 annotated clinic letters from King's College Hospital, we explored the efficacy of open-source LLMs (Llama and Mistral series) for extracting key epilepsy-related information, including epilepsy type, seizure type, current anti-seizure medications (ASMs), and associated symptoms. The study used various extraction methods, including direct extraction, summarized extraction, and contextualized extraction, complemented by role-prompting and few-shot prompting techniques. Performance was evaluated against a gold standard dataset, and was also compared to advanced fine-tuned models and human annotations.
Llama 2 13b (a 13-billion-parameter LLM developed by Meta) demonstrated superior extraction capabilities across tasks by consistently outperforming other LLMs (F1 = .80 in epilepsy-type extraction, F1 = .76 in seizure-type extraction, and F1 = .90 in current ASMs extraction). Here, F1 score is a balanced metric indicating the model's accuracy in correctly identifying relevant information without excessive false positives. The study highlights the direct extraction showing consistent high performance. Comparative analysis showed that LLMs outperformed current approaches like MedCAT (Medical Concept Annotation Tool) in extracting epilepsy-related information (.2 higher in F1).
The results affirm the potential of LLMs in medical information extraction relating to epilepsy, offering insights into leveraging these models for detailed and accurate data extraction from unstructured texts. The study underscores the importance of method selection in optimizing extraction performance and suggests a promising avenue for enhancing medical research and patient care through advanced natural language processing technologies.
大语言模型（LLMs）的出现以及电子健康记录（EHRs）的日益普及为推进医疗保健研究和实践带来了重大机遇。然而，比较和应用大语言模型从未结构化的医学自由文本中提取关键癫痫相关信息的研究尚未得到充分探索。本研究通过比较和应用不同的开源大语言模型及方法，从非结构化的临床信件中提取癫痫信息，填补了这一空白，从而优化电子健康记录作为一种资源，以造福癫痫研究。我们还强调了大语言模型的一些局限性。
利用来自国王学院医院的280封带注释的临床信件数据集，我们探索了开源大语言模型（Llama和Mistral系列）提取关键癫痫相关信息的功效，包括癫痫类型、发作类型、当前的抗癫痫药物（ASMs）以及相关症状。该研究使用了各种提取方法，包括直接提取、汇总提取和情境化提取，并辅以角色提示和少样本提示技术。性能根据金标准数据集进行评估，并且还与先进的微调模型和人工注释进行了比较。
Llama 2 13b（Meta开发的一个拥有130亿参数的大语言模型）在各项任务中均表现出卓越的提取能力，始终优于其他大语言模型（癫痫类型提取的F1值为0.80，发作类型提取的F1值为0.76，当前抗癫痫药物提取的F1值为0.90）。在此，F1分数是一个平衡指标，表明模型在正确识别相关信息时不会出现过多误报的准确性。该研究突出了直接提取始终具有高性能。对比分析表明，在提取癫痫相关信息方面，大语言模型优于当前的方法，如MedCAT（医学概念注释工具）（F1值高0.2）。
研究结果证实了大语言模型在癫痫相关医学信息提取中的潜力，为利用这些模型从非结构化文本中进行详细准确的数据提取提供了见解。该研究强调了方法选择在优化提取性能中的重要性，并为通过先进的自然语言处理技术加强医学研究和患者护理提出了一条有前景的途径。
