Dao Nam, Quesada Luisa, Hassan Syed Moin, Campo Monica Iturrioz, Johnson Shelsey, Ghose Suchandra, San José Estépar Raúl, Waxman Aaron, Washko George, Rahaghi Farbod N
Division of Pulmonary and Critical Care, Brigham and Women's Hospital, Boston, MA, United States.
Division of Sleep Medicine, Brigham and Women's Hospital, Boston, MA, United States.
JAMIA Open. 2025 Sep 4;8(5):ooaf097. doi: 10.1093/jamiaopen/ooaf097. eCollection 2025 Oct.
Unstructured data, such as procedure notes, contain valuable medical information that is frequently underutilized due to the labor-intensive nature of data extraction. This study aims to develop a generative artificial intelligence (GenAI) pipeline using an open-source Large Language Model (LLM) with built-in guardrails and a retry mechanism to extract data from unstructured right heart catheterization (RHC) notes while minimizing errors, including hallucinations.
A total of 220 RHC notes were randomly selected for pipeline development and 200 for validation from the Pulmonary Vascular Disease Registry. The pipeline comprised three main components: the Engineered Preload Framework (EPF), which integrated schemas and instructions; the LLM module, enhanced by reasoning capabilities; and the validation and retry mechanism, which ensured data accuracy through iterative self-correction. A clinical expert manually extracted data from the validation cohort to establish the ground truth. Pipeline performance was evaluated using precision, recall, and F1 score. Additionally, the dataset was stratified into quartiles to assess the pipeline's ability to handle varying levels of data availability.
The pipeline achieved 99.0% precision, 85.0% recall, and a 91.5% F1 score, with an overall accuracy of 90% when evaluated at the note level. The most common error was missed values (5.2%), while hallucinations were the least frequent (<0.01%).
This study demonstrates the feasibility of a robust GenAI pipeline for automating structured data extraction from unstructured RHC procedure notes. The approach highlights the potential of LLMs in medical data mining, improving research efficiency and clinical applications.
诸如手术记录等非结构化数据包含有价值的医学信息，但由于数据提取工作强度大，这些信息常常未得到充分利用。本研究旨在开发一种生成式人工智能（GenAI）管道，使用具有内置防护机制和重试机制的开源大语言模型（LLM），从非结构化的右心导管检查（RHC）记录中提取数据，同时将包括幻觉在内的错误降至最低。
从肺血管疾病登记处随机选择220份RHC记录用于管道开发，200份用于验证。该管道由三个主要组件组成：工程预负荷框架（EPF），它整合了模式和指令；通过推理能力增强的LLM模块；以及验证和重试机制，通过迭代自我校正确保数据准确性。一名临床专家从验证队列中手动提取数据以确定真实情况。使用精确率、召回率和F1分数评估管道性能。此外，将数据集分层为四分位数，以评估管道处理不同数据可用性水平的能力。
该管道在记录层面评估时，精确率达到99.0%，召回率为85.0%，F1分数为91.5%，总体准确率为90%。最常见的错误是遗漏值（5.2%），而幻觉是最不常见的（<0.01%）。
本研究证明了一个强大的GenAI管道用于从非结构化RHC手术记录中自动提取结构化数据的可行性。该方法突出了大语言模型在医学数据挖掘中的潜力，提高了研究效率和临床应用。
