School of Computing and Information Technology, University of Wollongong, Wollongong, NSW 2522, Australia; School of Computer Science, Qassim University, Qassim 51452, Saudi Arabia.
School of Computing and Information Technology, University of Wollongong, Wollongong, NSW 2522, Australia.
J Biomed Inform. 2024 Aug;156:104662. doi: 10.1016/j.jbi.2024.104662. Epub 2024 Jun 14.
Malnutrition is a prevalent issue in aged care facilities (RACFs), leading to adverse health outcomes. The ability to efficiently extract key clinical information from a large volume of data in electronic health records (EHR) can improve understanding about the extent of the problem and developing effective interventions. This research aimed to test the efficacy of zero-shot prompt engineering applied to generative artificial intelligence (AI) models on their own and in combination with retrieval augmented generation (RAG), for the automating tasks of summarizing both structured and unstructured data in EHR and extracting important malnutrition information.
We utilized Llama 2 13B model with zero-shot prompting. The dataset comprises unstructured and structured EHRs related to malnutrition management in 40 Australian RACFs. We employed zero-shot learning to the model alone first, then combined it with RAG to accomplish two tasks: generate structured summaries about the nutritional status of a client and extract key information about malnutrition risk factors. We utilized 25 notes in the first task and 1,399 in the second task. We evaluated the model's output of each task manually against a gold standard dataset.
The evaluation outcomes indicated that zero-shot learning applied to generative AI model is highly effective in summarizing and extracting information about nutritional status of RACFs' clients. The generated summaries provided concise and accurate representation of the original data with an overall accuracy of 93.25%. The addition of RAG improved the summarization process, leading to a 6% increase and achieving an accuracy of 99.25%. The model also proved its capability in extracting risk factors with an accuracy of 90%. However, adding RAG did not further improve accuracy in this task. Overall, the model has shown a robust performance when information was explicitly stated in the notes; however, it could encounter hallucination limitations, particularly when details were not explicitly provided.
This study demonstrates the high performance and limitations of applying zero-shot learning to generative AI models to automatic generation of structured summarization of EHRs data and extracting key clinical information. The inclusion of the RAG approach improved the model performance and mitigated the hallucination problem.
营养不良是养老院（RACF）中普遍存在的问题，导致不良健康后果。能够从电子健康记录（EHR）中的大量数据中高效提取关键临床信息，可以提高对问题严重程度的认识，并开发有效的干预措施。本研究旨在测试零样本提示工程在生成人工智能（AI）模型中的功效，这些模型单独使用以及与检索增强生成（RAG）相结合，可用于自动化 EHR 中结构化和非结构化数据的摘要以及提取重要营养信息。
我们使用 Llama 2 13B 模型进行零样本提示。该数据集包含与澳大利亚 40 家 RACF 中营养不良管理相关的非结构化和结构化 EHR。我们首先对模型进行零样本学习，然后将其与 RAG 相结合，完成两项任务：生成关于客户营养状况的结构化摘要和提取营养风险因素的关键信息。我们在第一项任务中使用了 25 条笔记，在第二项任务中使用了 1399 条笔记。我们根据黄金标准数据集手动评估模型在每个任务中的输出。
评估结果表明，零样本学习应用于生成式 AI 模型，在总结和提取 RACF 客户营养状况信息方面非常有效。生成的摘要简洁准确地表示了原始数据，整体准确率为 93.25%。添加 RAG 改善了摘要过程，准确率提高了 6%，达到 99.25%。该模型在提取风险因素方面也表现出很高的准确率，达到 90%。然而，在这项任务中添加 RAG 并没有进一步提高准确率。总体而言，该模型在笔记中明确说明信息时表现出强大的性能；然而，它可能会遇到幻觉限制，尤其是在没有明确提供细节的情况下。
本研究证明了将零样本学习应用于生成式 AI 模型，以自动生成 EHR 数据的结构化摘要和提取关键临床信息的高效性和局限性。包括 RAG 方法提高了模型性能并减轻了幻觉问题。
