Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, TN 37203, United States.
Department of Pediatrics, Indiana University School of Medicine, Indianapolis, IN 46202, United States.
J Am Med Inform Assoc. 2024 Oct 1;31(10):2294-2303. doi: 10.1093/jamia/ocae186.
Returning aggregate study results is an important ethical responsibility to promote trust and inform decision making, but the practice of providing results to a lay audience is not widely adopted. Barriers include significant cost and time required to develop lay summaries and scarce infrastructure necessary for returning them to the public. Our study aims to generate, evaluate, and implement ChatGPT 4 lay summaries of scientific abstracts on a national clinical study recruitment platform, ResearchMatch, to facilitate timely and cost-effective return of study results at scale.
We engineered prompts to summarize abstracts at a literacy level accessible to the public, prioritizing succinctness, clarity, and practical relevance. Researchers and volunteers assessed ChatGPT-generated lay summaries across five dimensions: accuracy, relevance, accessibility, transparency, and harmfulness. We used precision analysis and adaptive random sampling to determine the optimal number of summaries for evaluation, ensuring high statistical precision.
ChatGPT achieved 95.9% (95% CI, 92.1-97.9) accuracy and 96.2% (92.4-98.1) relevance across 192 summary sentences from 33 abstracts based on researcher review. 85.3% (69.9-93.6) of 34 volunteers perceived ChatGPT-generated summaries as more accessible and 73.5% (56.9-85.4) more transparent than the original abstract. None of the summaries were deemed harmful. We expanded ResearchMatch's technical infrastructure to automatically generate and display lay summaries for over 750 published studies that resulted from the platform's recruitment mechanism.
Implementing AI-generated lay summaries on ResearchMatch demonstrates the potential of a scalable framework generalizable to broader platforms for enhancing research accessibility and transparency.
汇总研究结果是促进信任和为决策提供信息的一项重要道德责任，但向非专业人士提供结果的做法并未得到广泛采用。这其中的障碍包括：开发通俗易懂的摘要需要大量的成本和时间，并且缺乏将结果反馈给公众所需的基础设施。我们的研究旨在生成、评估和实施 ChatGPT4 对科学摘要的通俗摘要，以便在全国性的临床研究招募平台 ResearchMatch 上大规模及时且经济有效地返还研究结果。
我们设计了提示词，以便将摘要总结为公众易于理解的水平，重点是简洁、清晰和实际相关性。研究人员和志愿者从五个维度评估 ChatGPT 生成的通俗摘要：准确性、相关性、可及性、透明度和危害性。我们使用精度分析和自适应随机抽样来确定评估所需的最佳摘要数量，以确保高统计精度。
根据研究人员的评估，ChatGPT 在 33 个摘要中的 192 个摘要句子中达到了 95.9%（95%置信区间，92.1-97.9）的准确性和 96.2%（92.4-98.1）的相关性。在 34 名志愿者中，有 85.3%（69.9-93.6）的人认为 ChatGPT 生成的摘要比原始摘要更容易理解，73.5%（56.9-85.4）的人认为它们更透明。没有一份摘要被认为是有害的。我们扩展了 ResearchMatch 的技术基础设施，以便为平台的招募机制产生的 750 多项已发表研究自动生成和显示通俗摘要。
在 ResearchMatch 上实施 AI 生成的通俗摘要展示了一种可扩展的框架的潜力，该框架可推广到更广泛的平台，以增强研究的可及性和透明度。
