用于低资源医学问答的基于Reddit数据的双层检索增强生成框架：概念验证研究
Two-Layer Retrieval-Augmented Generation Framework for Low-Resource Medical Question Answering Using Reddit Data: Proof-of-Concept Study.
作者信息
Das Sudeshna, Ge Yao, Guo Yuting, Rajwal Swati, Hairston JaMor, Powell Jeanne, Walker Drew, Peddireddy Snigdha, Lakamana Sahithi, Bozkurt Selen, Reyna Matthew, Sameni Reza, Xiao Yunyu, Kim Sangmi, Chandler Rasheeta, Hernandez Natalie, Mowery Danielle, Wightman Rachel, Love Jennifer, Spadaro Anthony, Perrone Jeanmarie, Sarker Abeed
机构信息
Department of Biomedical Informatics, School of Medicine, Emory University, Atlanta, GA, United States.
Department of Computer Science and Informatics, Emory University, Atlanta, GA, United States.
出版信息
J Med Internet Res. 2025 Jan 6;27:e66220. doi: 10.2196/66220.
BACKGROUND
The increasing use of social media to share lived and living experiences of substance use presents a unique opportunity to obtain information on side effects, use patterns, and opinions on novel psychoactive substances. However, due to the large volume of data, obtaining useful insights through natural language processing technologies such as large language models is challenging.
OBJECTIVE
This paper aims to develop a retrieval-augmented generation (RAG) architecture for medical question answering pertaining to clinicians' queries on emerging issues associated with health-related topics, using user-generated medical information on social media.
METHODS
We proposed a two-layer RAG framework for query-focused answer generation and evaluated a proof of concept for the framework in the context of query-focused summary generation from social media forums, focusing on emerging drug-related information. Our modular framework generates individual summaries followed by an aggregated summary to answer medical queries from large amounts of user-generated social media data in an efficient manner. We compared the performance of a quantized large language model (Nous-Hermes-2-7B-DPO), deployable in low-resource settings, with GPT-4. For this proof-of-concept study, we used user-generated data from Reddit to answer clinicians' questions on the use of xylazine and ketamine.
RESULTS
Our framework achieves comparable median scores in terms of relevance, length, hallucination, coverage, and coherence when evaluated using GPT-4 and Nous-Hermes-2-7B-DPO, evaluated for 20 queries with 76 samples. There was no statistically significant difference between GPT-4 and Nous-Hermes-2-7B-DPO for coverage (Mann-Whitney U=733.0; n=37; n=39; P=.89 two-tailed), coherence (U=670.0; n=37; n=39; P=.49 two-tailed), relevance (U=662.0; n=37; n=39; P=.15 two-tailed), length (U=672.0; n=37; n=39; P=.55 two-tailed), and hallucination (U=859.0; n=37; n=39; P=.01 two-tailed). A statistically significant difference was noted for the Coleman-Liau Index (U=307.5; n=20; n=16; P<.001 two-tailed).
CONCLUSIONS
Our RAG framework can effectively answer medical questions about targeted topics and can be deployed in resource-constrained settings.
背景
社交媒体越来越多地被用于分享物质使用的生活经历，这为获取有关新型精神活性物质的副作用、使用模式和观点的信息提供了独特的机会。然而，由于数据量庞大，通过诸如大语言模型等自然语言处理技术获得有用的见解具有挑战性。
目的
本文旨在开发一种检索增强生成（RAG）架构，用于医学问答，以回答临床医生关于与健康相关主题的新出现问题的查询，使用社交媒体上用户生成的医学信息。
方法
我们提出了一个用于聚焦查询答案生成的两层RAG框架，并在从社交媒体论坛生成聚焦查询摘要的背景下评估了该框架的概念验证，重点关注新出现的药物相关信息。我们的模块化框架先生成单独的摘要，然后生成汇总摘要，以高效地回答来自大量用户生成的社交媒体数据的医学查询。我们将可在低资源环境中部署的量化大语言模型（Nous-Hermes-2-7B-DPO）的性能与GPT-4进行了比较。对于这项概念验证研究，我们使用来自Reddit的用户生成数据来回答临床医生关于赛拉嗪和氯胺酮使用的问题。
结果
当使用GPT-4和Nous-Hermes-2-7B-DPO进行评估时，我们的框架在相关性、长度、幻觉、覆盖范围和连贯性方面取得了可比的中位数分数，对20个查询和76个样本进行了评估。GPT-4和Nous-Hermes-2-7B-DPO在覆盖范围（曼-惠特尼U = 733.0；n = 37；n = 39；P = 0.89，双侧）、连贯性（U = 670.0；n = 37；n = 39；P = 0.49，双侧）、相关性（U = 662.0；n = 37；n = 39；P = 0.15，双侧）、长度（U = 672.0；n = 37；n = 39；P = 0.55，双侧）和幻觉（U = 859.0；n = 37；n = 39；P = 0.01，双侧）方面没有统计学上的显著差异。在科尔曼-廖指数方面观察到统计学上的显著差异（U = 307.5；n = 20；n = 16；P < 0.001，双侧）。
结论
我们的RAG框架可以有效地回答关于目标主题的医学问题，并且可以部署在资源受限的环境中。
Zotero 插件