Department of Radiology, Mayo Clinic, Phoenix, AZ.
Departments of Medicine and of Epidemiology & Population Health, Stanford University School of Medicine, Palo Alto, CA.
JCO Clin Cancer Inform. 2024 Aug;8:e2300258. doi: 10.1200/CCI.23.00258.
Patient-centered outcomes (PCOs) are pivotal in cancer treatment, as they directly reflect patients' quality of life. Although multiple studies suggest that factors affecting breast cancer-related morbidity and survival are influenced by treatment side effects and adherence to long-term treatment, such data are generally only available on a smaller scale or from a single center. The primary challenge with collecting these data is that the outcomes are captured as free text in clinical narratives written by clinicians.
Given the complexity of PCO documentation in these narratives, computerized methods are necessary to unlock the wealth of information buried in unstructured text notes that often document PCOs. Inspired by the success of large language models (LLMs), we examined the adaptability of three LLMs, GPT-2, BioGPT, and PMC-LLaMA, on PCO tasks across three institutions, Mayo Clinic, Emory University Hospital, and Stanford University. We developed an open-source framework for fine-tuning LLM that can directly extract the five different categories of PCO from the clinic notes.
We found that these LLMs without fine-tuning (zero-shot) struggle with challenging PCO extraction tasks, displaying almost random performance, even with some task-specific examples (few-shot learning). The performance of our fine-tuned, task-specific models is notably superior compared with their non-fine-tuned LLM models. Moreover, the fine-tuned GPT-2 model has demonstrated a significantly better performance than the other two larger LLMs.
Our discovery indicates that although LLMs serve as effective general-purpose models for tasks across various domains, they require fine-tuning when applied to the clinician domain. Our proposed approach has the potential to lead more efficient, adaptable models for PCO information extraction, reducing reliance on extensive computational resources while still delivering superior performance for specific tasks.
患者为中心的结局（PCOs）在癌症治疗中至关重要，因为它们直接反映了患者的生活质量。尽管多项研究表明，影响乳腺癌相关发病率和生存率的因素受到治疗副作用和长期治疗依从性的影响，但这些数据通常仅在较小规模或来自单个中心获得。收集这些数据的主要挑战是，结局是以临床医生撰写的临床叙述中的自由文本形式捕获的。
鉴于这些叙述中 PCO 文档的复杂性，需要计算机化方法来解锁隐藏在非结构化文本注释中的丰富信息，这些注释通常记录了 PCOs。受大型语言模型（LLMs）成功的启发，我们检查了 GPT-2、BioGPT 和 PMC-LLaMA 这三个 LLM 在梅奥诊所、埃默里大学医院和斯坦福大学三个机构的 PCO 任务上的适应性。我们开发了一个用于微调 LLM 的开源框架，该框架可以直接从诊所记录中提取五个不同类别的 PCO。
我们发现，这些未经微调的 LLM（零样本）在具有挑战性的 PCO 提取任务中表现不佳，即使提供了一些特定于任务的示例（少样本学习），性能也几乎随机。与非微调的 LLM 模型相比，我们专门针对任务进行微调的模型的性能明显更好。此外，微调后的 GPT-2 模型的性能明显优于其他两个更大的 LLM。
我们的发现表明，尽管 LLM 作为跨各种领域任务的有效通用模型，但在应用于临床医生领域时需要进行微调。我们提出的方法有可能为 PCO 信息提取带来更高效、适应性更强的模型，减少对大量计算资源的依赖，同时仍能为特定任务提供卓越的性能。
