检索问题
AI降幻觉律，有什么最新研究吗
AI降幻觉（即减少人工智能编造虚假信息的现象）的最新研究证实：幻觉是大模型与生俱来的先天限制，无法被彻底消除，但可以通过检索增强生成（RAG）、自我验证机制、智能体工作流以及人类专家介入等手段大幅压制和管控。
下面我们把这些高深的学术研究，掰开揉碎了，用大白话讲给你听。
一、 为什么AI一定会“幻觉”？这是它的“先天基因”
大模型的“幻觉”，说白了就是AI在一本正经地胡说八道。它生成的句子读起来通顺连贯，但事实却是错的、逻辑是乱的，甚至完全是凭空捏造的。你可能会问，咱们使劲改，能不能把它彻底改掉？
最新研究给出的答案很残酷：不能，幻觉是大模型的宿命。
为什么会这样？这得从大模型的“出身”说起。大模型本质上是一个概率预测机器，它不是在数据库里“查资料”，而是根据上文猜下一个最可能出现的字。研究从数学的“可计算性理论”上证明了大模型不可能学会世界上所有的可计算函数，当它遇到没学过或者学得不牢固的知识时，它不会老老实实说“我不知道”，而是会顺着概率瞎编一个看起来最像正确答案的东西。因为现实世界比数学模型还要复杂得多，所以在真实应用中，大模型的幻觉是不可避免的。这就好比一个人只背了半本字典，却硬要装作认识所有字，遇到不认识的就只能靠发音瞎蒙。
二、 降幻觉的四大“法宝”
既然除不了根，咱们怎么把幻觉压制在安全范围内？目前学术界和工业界有四大主流法宝。
法宝一：给AI配个“外脑”——检索增强生成（RAG）
既然AI自己的记忆不靠谱，那最直接的办法就是让它开卷考试。这就是目前最火的RAG（Retrieval-Augmented Generation，检索增强生成）技术。
RAG的原理很简单：在AI回答问题之前，先去外部权威资料库里把相关的真实文档搜出来，然后让AI看着这些文档再回答。这就像给一个容易忘事的老人配了个随身笔记本，或者让律师开庭前先翻卷宗。
在极其容易出人命的医疗领域，RAG已经被广泛采用。因为医学知识更新极快，AI脑子里的旧知识很容易耽误事，而RAG能提供最新的医学指南。比如，有研究专门针对肾脏病学，把大模型和包含最新慢性肾病指南（KDIGO 2023）的RAG系统结合，结果生成的医疗建议既专业又准确。在大规模的医学问答测试中，加了RAG的系统能把不同大模型的准确率最高提升18%，甚至让一些中等水平的AI达到了顶级模型（如GPT-4）的水平。
但是，RAG也不是万能药。最新测试发现，AI用RAG时有几个致命弱点：
- 耳根子软（抗噪性差）：如果搜回来的资料里夹杂了没用的噪音，AI容易被带偏。
- 不会拒答（负向排斥弱）：如果搜回来的资料其实都不包含答案，AI硬要强行总结，就会出错。
- 不会拼图（信息整合难）：当答案散落在多篇文档里时，AI往往拼凑不出完整的真相。
- 中间遗忘（迷失在中间）：研究发现，AI非常像人，对资料开头和结尾的信息记忆深刻，但对中间部分的信息经常视而不见，这被称为“迷失在中间效应”。
法宝二：让AI自己“复查作业”——自我验证与纠错
除了给资料，研究者还在训练AI的自我反思能力。
有一种很管用的提示词技巧叫思维链（Chain-of-Thought, CoT），就是让AI不要直接给答案，而是一步一步写出推导过程。研究发现，在那些因为提示词没给好而引发幻觉的场景里，让AI用思维链一步步想，能大幅降低胡编的概率。
更进一步的最新技术叫验证链。这就好比AI写完卷子，再给自己出一份检查题：
- 先写个初稿；
- 针对初稿里的关键点，自己出几道验证题（比如初稿说“爱因斯坦1921年获奖”，验证题就是“爱因斯坦哪年获奖？”）；
- 独立回答这些验证题，确保不受初稿干扰；
- 根据验证结果，修改初稿，给出最终答案。 这套自己给自己挑错的方法，在各种任务中都有效减少了幻觉。
学术界还梳理了自动纠错的三大流派：在训练时纠错、在生成时纠错，以及生成后的纠错，核心都是用自动化反馈来让AI“知错就改”。
法宝三：给AI配个“安检员”和“专家团”——智能体架构
现在的降幻觉前沿，已经从单打独斗走向了团队作战，也就是智能体架构。
一种方法是在系统里设一个“安检员”。比如LangGraph系统把AI的回答分为三类：
- 知识正面幻觉（HK+）：AI懂这个知识，但嘴瓢说错了。这时候触发“人工介入”机制，让真人专家来纠正。
- 知识负面幻觉（HK-）：AI根本不懂，瞎蒙的。这时候自动触发RAG，去外网查资料补充知识。
- 正常回答：准确无误，直接放行。
在医疗场景中，这种团队作战更成熟。比如最新的“Polaris”医疗星座架构系统，包含一万亿个参数，它由一个主聊AI（负责跟患者拉家常、建立信任）和几个专科AI（负责核对药品、医学常识等）组成。主聊AI一旦涉及专业问题，专科AI就会在后台默默审核，一旦发现主聊AI要胡编，立刻纠正。这套系统招募了1100多名美国注册护士和130多名医生进行测试，结果显示它在医疗安全性和临床准备度上，已经达到了人类护士的水平，而且幻觉率远低于单体的大模型（如GPT-4）。
法宝四：关键时刻请真人出山——人类专家介入
在危机自救等高风险场景下，AI幻觉真的会害死人。实验模拟了火灾自救场景，发现当AI给出高幻觉的错误指导时，用户会感到极大的认知负担（脑子发懵）和负面情绪，对AI的评价直线下降。
怎么化解？研究发现，如果在AI给出建议的同时，配上人类专家的指导意见，情况就会大逆转。专家的意见就像一颗定心丸，能大幅降低用户因为AI胡说八道而产生的认知负担，缓解焦虑，也弥补了AI幻觉带来的负面影响。这就好比导航虽然有时导错路，但旁边坐着个老司机指路，心里就踏实了。
三、 降幻觉也有“安全度量衡”
降幻觉不能光凭感觉，还得有科学的评估方法。特别是在医疗文书的生成上，幻觉可能关乎人命。最新的研究提出了一个严密的临床安全评估框架，包括对AI输出错误进行分类、进行迭代对比，以及临床安全评估。在一项包含近13000句医生标注的实验中，原始的AI写病历幻觉率是1.47%，遗漏率是3.45%。通过不断优化提示词和工作流，研究人员成功把重大错误率降到了比人类手写病历还要低的水平。这说明，只要方法对，AI是可以比人类更细心的。
同时，研究人员也在呼吁建立更全面的“护栏”机制。这就像给汽车装上安全带和防撞梁，从数据源头把关，到模型训练约束，再到生成后的拦截，全方位防止AI的恶意攻击、偏见和毒性输出，而幻觉就是其中最需要防的风险之一。
总结
总而言之，最新的研究告诉我们，AI幻觉就像人的基因缺陷一样，是永远存在的。指望AI绝对不撒谎是不现实的。但我们现在有了很多聪明的对策：给它外挂知识库（RAG），教它自我复查，给它组个专家团，并在关键时刻让人类专家兜底。在医疗等高风险领域，这些方法已经能把幻觉压制到比人类出错率还低的水平。未来，用好AI的关键，就是接受它会胡说八道的事实，并熟练使用这些“降幻觉法宝”来系好安全带。
References
1🧜Siren’s Song in the AI Ocean: A Survey on Hallucination in Large Language ModelsOpenAlex
Yue Zhang, Yafu Li, Leyang Cui, et al.
Abstract While large language models (LLMs) have demonstrated remarkable capabilities across a range of downstream tasks, a significant concern revolves around their propensity to exhibit hallucinations: LLMs occasionally generate content that diverges from the user input, contradicts previously generated context, or misaligns with established world knowledge. This phenomenon poses a substantial challenge to the reliability of LLMs in real-world scenarios. In this article, we survey recent efforts on the detection, explanation, and mitigation of hallucination, with an emphasis on the unique challenges posed by LLMs. We present taxonomies of the LLM hallucination phenomena and evaluation benchmarks, analyze existing approaches aiming at mitigating LLM hallucination, and discuss potential directions for future research.
2A Survey of Large Language ModelsOpenAlex
Wayne Xin Zhao, Kun Zhou, Junyi Li, et al.
Abstract The rapid evolution of large language models (LLMs) has driven a transformative shift in artificial intelligence (AI), reshaping both research paradigms and practical applications. Distinguished from their predecessors by unprecedented scale and advanced capabilities, LLMs necessitate new frameworks for understanding their development, behavior, and societal impact. This survey systematically reviews recent advancements in LLM techniques across four key dimensions: (1) pre-training methodologies, which establish core model capabilities through large-scale self-supervised training, architectural innovations, and data curation strategies; (2) post-training techniques, including supervised fine-tuning and reinforcement learning, which adapt foundational models to downstream tasks and enhance their alignment and safety; (3) utilization strategies, such as in-context learning, prompt engineering, and agentic reasoning, that optimize real-world deployment and enable effective interaction with external environments; and (4) evaluation methods, encompassing benchmarks for key ability dimensions such as core language capabilities, reasoning, and safety, which support comprehensive and reliable assessment of model performance. Additionally, we identify critical research issues, including those concerning theoretical foundations, efficient scaling, alignment, and agentic capability, and highlight the open challenges they present. By synthesizing state-of-the-art insights and emerging trends, this survey aims to provide a systematic and comprehensive framework for understanding the trajectory, current limitations, and future directions of LLM progress.
3AI Hallucination in Crisis Self-Rescue Scenarios: The Impact on AI Service Evaluation and the Mitigating Effect of Human Expert AdviceOpenAlex
Yan Song, Mingjia Cui, Fei Wan, et al.
Generative artificial intelligence (GAI) technology is rapidly spreading across various fields. Although AI can generate creative ideas and useful information, it often produces misleading content, known as AI hallucination. AI hallucination may affect users’ evaluation of AI service in high-risk areas such as crisis self-rescue, and even pose a threat to their lives. How to determine the impact of AI hallucination on users AI service evaluation in crisis self-rescue scenarios and reduce the negative effects of AI hallucination has become an urgent research question. To address this issue, we designed a between-subject experiment using a fire self-rescue scenario as an example, with 2 (AI hallucination: high/low) × 2 (human expert advice: Yes/No), to explore the impact of AI hallucination on individual’s AI service evaluation. Based on the theory of resource conservation, we analyzed the impact mechanism of AI hallucination on AI service evaluation from both cognitive and emotional perspectives, and further proposed the moderating role of human expert advice in this impact and mechanism. The results indicate that AI hallucination has a significant negative impact on AI service evaluation. The degree of emotional relief and cognitive load mediate the impact between AI hallucination and AI service evaluation. Human expert advice significantly reduces the cognitive load caused by AI hallucination, alleviates users’ negative emotions, and mitigates the negative impact of AI hallucination on AI service evaluation. Our research contributes to the literature on AI hallucination and provides practical recommendation for effectively mitigating the negative impacts of AI hallucination.
4Hallucination is Inevitable: An Innate Limitation of Large Language ModelsOpenAlex
Ziwei Xu, Sanjay Jain, Mohan Kankanhalli
Hallucination has been widely recognized to be a significant drawback for large language models (LLMs). There have been many works that attempt to reduce the extent of hallucination. These efforts have mostly been empirical so far, which cannot answer the fundamental question whether it can be completely eliminated. In this paper, we formalize the problem and show that it is impossible to eliminate hallucination in LLMs. Specifically, we define a formal world where hallucination is defined as inconsistencies between a computable LLM and a computable ground truth function. By employing results from learning theory, we show that LLMs cannot learn all the computable functions and will therefore inevitably hallucinate if used as general problem solvers. Since the formal world is a part of the real world which is much more complicated, hallucinations are also inevitable for real world LLMs. Furthermore, for real world LLMs constrained by provable time complexity, we describe the hallucination-prone tasks and empirically validate our claims. Finally, using the formal world framework, we discuss the possible mechanisms and efficacies of existing hallucination mitigators as well as the practical implications on the safe deployment of LLMs.
5Benchmarking Large Language Models in Retrieval-Augmented GenerationOpenAlex
Jiawei Chen, Hongyu Lin, Xianpei Han, et al.
Retrieval-Augmented Generation (RAG) is a promising approach for mitigating the hallucination of large language models (LLMs). However, existing research lacks rigorous evaluation of the impact of retrieval-augmented generation on different large language models, which make it challenging to identify the potential bottlenecks in the capabilities of RAG for different LLMs. In this paper, we systematically investigate the impact of Retrieval-Augmented Generation on large language models. We analyze the performance of different large language models in 4 fundamental abilities required for RAG, including noise robustness, negative rejection, information integration, and counterfactual robustness. To this end, we establish Retrieval-Augmented Generation Benchmark (RGB), a new corpus for RAG evaluation in both English and Chinese. RGB divides the instances within the benchmark into 4 separate testbeds based on the aforementioned fundamental abilities required to resolve the case. Then we evaluate 6 representative LLMs on RGB to diagnose the challenges of current LLMs when applying RAG. Evaluation reveals that while LLMs exhibit a certain degree of noise robustness, they still struggle significantly in terms of negative rejection, information integration, and dealing with false information. The aforementioned assessment outcomes indicate that there is still a considerable journey ahead to effectively apply RAG to LLMs.
6Survey and analysis of hallucinations in large language models: attribution to prompting strategies or model behaviorOpenAlex
Dang Anh-Hoang, Vu Tran, Le-Minh Nguyen
Hallucination in Large Language Models (LLMs) refers to outputs that appear fluent and coherent but are factually incorrect, logically inconsistent, or entirely fabricated. As LLMs are increasingly deployed in education, healthcare, law, and scientific research, understanding and mitigating hallucinations has become critical. In this work, we present a comprehensive survey and empirical analysis of hallucination attribution in LLMs. Introducing a novel framework to determine whether a given hallucination stems from not optimize prompting or the model's intrinsic behavior. We evaluate state-of-the-art LLMs—including GPT-4, LLaMA 2, DeepSeek, and others—under various controlled prompting conditions, using established benchmarks (TruthfulQA, HallucinationEval) to judge factuality. Our attribution framework defines metrics for Prompt Sensitivity (PS) and Model Variability (MV) , which together quantify the contribution of prompts vs. model-internal factors to hallucinations. Through extensive experiments and comparative analyses, we identify distinct patterns in hallucination occurrence, severity, and mitigation across models. Notably, structured prompt strategies such as chain-of-thought (CoT) prompting significantly reduce hallucinations in prompt-sensitive scenarios, though intrinsic model limitations persist in some cases. These findings contribute to a deeper understanding of LLM reliability and provide insights for prompt engineers, model developers, and AI practitioners. We further propose best practices and future directions to reduce hallucinations in both prompt design and model development pipelines.
8Controlling Large Language Model Hallucination Based on Agent AI with LangGraphOpenAlex
Jialin Wang, Zhihua Duan
As Large Language Models (LLMs) become integral to Natural Language Processing (NLP) applications, their tendency to generate hallucinations — outputs misaligned with input data or real-world facts — presents a significant challenge. This paper introduces a novel approach leveraging Agent AI within the LangGraph system to systematically detect and mitigate LLM hallucinations. The framework categorizes hallucinations into three distinct types: Hallucination Knowledge Positive (HK+), Hallucination Knowledge Negative (HK-), and normal responses. HK+ represents errors where the model has relevant knowledge but provides incorrect answers, while HK- denotes hallucinations arising from insufficient knowledge. Normal responses are accurate outputs requiring no intervention. The proposed method employs corrective Retrieval-Augmented Generation (RAG) to address HK- cases by supplementing missing knowledge. For HK+, a human-in-the-loop mechanism is activated to rectify errors and ensure accuracy. Normal responses proceed without additional processing. This structured approach improves the precision and reliability of LLMs across diverse application scenarios. By integrating Agent AI for dynamic classification and targeted intervention, the LangGraph system significantly reduces hallucination rates, offering a robust solution for enhancing LLM performance in real-world deployments. This research provides new perspectives and methodologies for advancing LLM stability and dependability.
9Chain-of-Verification Reduces Hallucination in Large Language ModelsOpenAlex
Shehzaad Dhuliawala, Mojtaba Komeili, Jing Xu, et al.
Generation of plausible yet incorrect factual information, termed hallucination, is an unsolved issue in large language models.We study the ability of language models to deliberate on the responses they give in order to correct their mistakes.We develop the Chain-of-Verification (COVE) method whereby the model first (i) drafts an initial response; then (ii) plans verification questions to fact-check its draft; (iii) answers those questions independently so the answers are not biased by other responses; and (iv) generates its final verified response.In experiments, we show COVE decreases hallucinations across a variety of tasks, from list-based questions from Wikidata, closed book Multi-SpanQA and longform text generation.
10Benchmarking Retrieval-Augmented Generation for MedicineOpenAlex
Guangzhi Xiong, Qiao Jin, Zhiyong Lu, et al.
While large language models (LLMs) have achieved state-of-the-art performance on a wide range of medical question answering (QA) tasks, they still face challenges with hallucinations and outdated knowledge.Retrievalaugmented generation (RAG) is a promising solution and has been widely adopted.However, a RAG system can involve multiple flexible components, and there is a lack of best practices regarding the optimal RAG setting for various medical purposes.To systematically evaluate such systems, we propose the Medical Information Retrieval-Augmented Generation Evaluation (MIRAGE), a first-of-its-kind benchmark including 7,663 questions from five medical QA datasets.Using MIRAGE, we conducted large-scale experiments with over 1.8 trillion prompt tokens on 41 combinations of different corpora, retrievers, and backbone LLMs through the MEDRAG toolkit introduced in this work.Overall, MEDRAG improves the accuracy of six different LLMs by up to 18% over chain-of-thought prompting, elevating the performance of GPT-3.5 and Mixtral to GPT-4level.Our results show that the combination of various medical corpora and retrievers achieves the best performance.In addition, we discovered a log-linear scaling property and the "lostin-the-middle" effects in medical RAG.We believe our comprehensive evaluations can serve as practical guidelines for implementing RAG systems for medicine 1 .
11The Clinicians’ Guide to Large Language Models: A General Perspective With a Focus on HallucinationsOpenAlex
Dimitri Roustan, François Bastardot
Large language models (LLMs) are artificial intelligence tools that have the prospect of profoundly changing how we practice all aspects of medicine. Considering the incredible potential of LLMs in medicine and the interest of many health care stakeholders for implementation into routine practice, it is therefore essential that clinicians be aware of the basic risks associated with the use of these models. Namely, a significant risk associated with the use of LLMs is their potential to create hallucinations. Hallucinations (false information) generated by LLMs arise from a multitude of causes, including both factors related to the training dataset as well as their auto-regressive nature. The implications for clinical practice range from the generation of inaccurate diagnostic and therapeutic information to the reinforcement of flawed diagnostic reasoning pathways, as well as a lack of reliability if not used properly. To reduce this risk, we developed a general technical framework for approaching LLMs in general clinical practice, as well as for implementation on a larger institutional scale.
12Red teaming ChatGPT via Jailbreaking: Bias, Robustness, Reliability and ToxicityOpenAlex
Terry Yue Zhuo, Yujin Huang, Chunyang Chen, et al.
Recent breakthroughs in natural language processing (NLP) have permitted the synthesis and comprehension of coherent text in an open-ended way, therefore translating the theoretical algorithms into practical applications. The large language models (LLMs) have significantly impacted businesses such as report summarization software and copywriters. Observations indicate, however, that LLMs may exhibit social prejudice and toxicity, posing ethical and societal dangers of consequences resulting from irresponsibility. Large-scale benchmarks for accountable LLMs should consequently be developed. Although several empirical investigations reveal the existence of a few ethical difficulties in advanced LLMs, there is little systematic examination and user study of the risks and harmful behaviors of current LLM usage. To further educate future efforts on constructing ethical LLMs responsibly, we perform a qualitative research method called ``red teaming'' on OpenAI's ChatGPT\footnote{In this paper, ChatGPT refers to the version released on Dec 15th.} to better understand the practical features of ethical dangers in recent LLMs. We analyze ChatGPT comprehensively from four perspectives: 1) \textit{Bias} 2) \textit{Reliability} 3) \textit{Robustness} 4) \textit{Toxicity}. In accordance with our stated viewpoints, we empirically benchmark ChatGPT on multiple sample datasets. We find that a significant number of ethical risks cannot be addressed by existing benchmarks, and hence illustrate them via additional case studies. In addition, we examine the implications of our findings on AI ethics and harmal behaviors of ChatGPT, as well as future problems and practical design considerations for responsible LLMs. We believe that our findings may give light on future efforts to determine and mitigate the ethical hazards posed by machines in LLM applications.
13Comprehensive Review of AI Hallucinations: Impacts and Mitigation Strategies for Financial and Business ApplicationsOpenAlex
Joshi, Satyadhar
This paper investigates the causes, implications, and mitigation strategies of AI hallucinations, with a focus on generative AI systems.This paper examines the phenomenon of AI hallucinations in large language models, analyzing root causes and evaluating mitigation strategies.We synthesize insights from recent academic research and industry findings to explain how hallucinations often arise due to problems in the data used to train language models, limitations in model architecture, and the way large language models (LLMs) generate text.Through a systematic review of current literature, we identify key patterns in how hallucinations emerge and examine the growing concern about their impact as AI becomes more embedded in decision-making systems.We identify core contributors such as data quality issues, model complexity, lack of grounding, and limitations inherent in the generative process.The risks are examined in various domains, including legal, business, and user-facing applications, highlighting consequences like misinformation, trust erosion, and productivity loss.To address these challenges, we survey mitigation techniques including data curation, retrieval-augmented generation (RAG), prompt engineering, fine-tuning, multi-model systems, and human-in-the-loop oversight.Our analysis draws from a wide range of academic and industry sources, offering both theoretical understanding and practical insights for AI practitioners.This is a pure review paper and all results are from the cited references.
14Automatically Correcting Large Language Models: <i>Surveying the Landscape of Diverse Automated Correction Strategies</i>OpenAlex
Liangming Pan, Michael Saxon, Wenda Xu, et al.
Abstract While large language models (LLMs) have shown remarkable effectiveness in various NLP tasks, they are still prone to issues such as hallucination, unfaithful reasoning, and toxicity. A promising approach to rectify these flaws is correcting LLMs with feedback, where the LLM itself is prompted or guided with feedback to fix problems in its own output. Techniques leveraging automated feedback—either produced by the LLM itself (self-correction) or some external system—are of particular interest as they make LLM-based solutions more practical and deployable with minimal human intervention. This paper provides an exhaustive review of the recent advances in correcting LLMs with automated feedback, categorizing them into training-time, generation-time, and post-hoc approaches. We also identify potential challenges and future directions in this emerging field.
15A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language ModelsOpenAlex
Wenqi Fan, Yujuan Ding, Liangbo Ning, et al.
As one of the most advanced techniques in AI, Retrieval-Augmented Generation (RAG) can offer reliable and up-to-date external knowledge, providing huge convenience for numerous tasks. Particularly in the era of AI-Generated Content (AIGC), the powerful capacity of retrieval in providing additional knowledge enables RAG to assist existing generative AI in producing high-quality outputs. Recently, Large Language Models (LLMs) have demonstrated revolutionary abilities in language understanding and generation, while still facing inherent limitations such as hallucinations and out-of-date internal knowledge. Given the powerful abilities of RAG in providing the latest and helpful auxiliary information, Retrieval-Augmented Large Language Models (RA-LLMs) have emerged to harness external and authoritative knowledge bases, rather than solely relying on the model's internal knowledge, to augment the quality of the generated content of LLMs. In this survey, we comprehensively review existing research studies in RA-LLMs, covering three primary technical perspectives: Furthermore, to deliver deeper insights, we discuss current limitations and several promising directions for future research. Updated information about this survey can be found at: https://advanced-recommender-systems.github.io/RAG-Meets-LLMs/
16Safeguarding large language models: a surveyOpenAlex
Yi Dong, Ronghui Mu, Yanghao Zhang, et al.
In the burgeoning field of Large Language Models (LLMs), developing a robust safety mechanism, colloquially known as "safeguards" or "guardrails", has become imperative to ensure the ethical use of LLMs within prescribed boundaries. This article provides a systematic literature review on the current status of this critical mechanism. It discusses its major challenges and how it can be enhanced into a comprehensive mechanism dealing with ethical issues in various contexts. First, the paper elucidates the current landscape of safeguarding mechanisms that major LLM service providers and the open-source community employ. This is followed by the techniques to evaluate, analyze, and enhance some (un)desirable properties that a guardrail might want to enforce, such as hallucinations, fairness, privacy, and so on. Based on them, we review techniques to circumvent these controls (i.e., attacks), to defend the attacks, and to reinforce the guardrails. While the techniques mentioned above represent the current status and the active research trends, we also discuss several challenges that cannot be easily dealt with by the methods and present our vision on how to implement a comprehensive guardrail through the full consideration of multi-disciplinary approach, neural-symbolic method, and systems development lifecycle.
17Polaris: A Safety-focused LLM Constellation Architecture for HealthcareOpenAlex
Subhabrata Mukherjee, Paul Gamble, Markel Sanz Ausin, et al.
We develop Polaris, the first safety-focused LLM constellation for real-time patient-AI healthcare conversations. Unlike prior LLM works in healthcare focusing on tasks like question answering, our work specifically focuses on long multi-turn voice conversations. Our one-trillion parameter constellation system is composed of several multibillion parameter LLMs as co-operative agents: a stateful primary agent that focuses on driving an engaging conversation and several specialist support agents focused on healthcare tasks performed by nurses to increase safety and reduce hallucinations. We develop a sophisticated training protocol for iterative co-training of the agents that optimize for diverse objectives. We train our models on proprietary data, clinical care plans, healthcare regulatory documents, medical manuals, and other medical reasoning documents. We align our models to speak like medical professionals, using organic healthcare conversations and simulated ones between patient actors and experienced nurses. This allows our system to express unique capabilities such as rapport building, trust building, empathy and bedside manner. Finally, we present the first comprehensive clinician evaluation of an LLM system for healthcare. We recruited over 1100 U.S. licensed nurses and over 130 U.S. licensed physicians to perform end-to-end conversational evaluations of our system by posing as patients and rating the system on several measures. We demonstrate Polaris performs on par with human nurses on aggregate across dimensions such as medical safety, clinical readiness, conversational quality, and bedside manner. Additionally, we conduct a challenging task-based evaluation of the individual specialist support agents, where we demonstrate our LLM agents significantly outperform a much larger general-purpose LLM (GPT-4) as well as from its own medium-size class (LLaMA-2 70B).
19Integrating Retrieval-Augmented Generation with Large Language Models in Nephrology: Advancing Practical ApplicationsOpenAlex
Jing Miao, Charat Thongprayoon, Supawadee Suppadungsuk, et al.
The integration of large language models (LLMs) into healthcare, particularly in nephrology, represents a significant advancement in applying advanced technology to patient care, medical research, and education. These advanced models have progressed from simple text processors to tools capable of deep language understanding, offering innovative ways to handle health-related data, thus improving medical practice efficiency and effectiveness. A significant challenge in medical applications of LLMs is their imperfect accuracy and/or tendency to produce hallucinations-outputs that are factually incorrect or irrelevant. This issue is particularly critical in healthcare, where precision is essential, as inaccuracies can undermine the reliability of these models in crucial decision-making processes. To overcome these challenges, various strategies have been developed. One such strategy is prompt engineering, like the chain-of-thought approach, which directs LLMs towards more accurate responses by breaking down the problem into intermediate steps or reasoning sequences. Another one is the retrieval-augmented generation (RAG) strategy, which helps address hallucinations by integrating external data, enhancing output accuracy and relevance. Hence, RAG is favored for tasks requiring up-to-date, comprehensive information, such as in clinical decision making or educational applications. In this article, we showcase the creation of a specialized ChatGPT model integrated with a RAG system, tailored to align with the KDIGO 2023 guidelines for chronic kidney disease. This example demonstrates its potential in providing specialized, accurate medical advice, marking a step towards more reliable and efficient nephrology practices.
20A framework to assess clinical safety and hallucination rates of LLMs for medical text summarisationOpenAlex
Elham Asgari, Nina Montaña-Brown, Magda Dubois, et al.
Integrating large language models (LLMs) into healthcare can enhance workflow efficiency and patient care by automating tasks such as summarising consultations. However, the fidelity between LLM outputs and ground truth information is vital to prevent miscommunication that could lead to compromise in patient safety. We propose a framework comprising (1) an error taxonomy for classifying LLM outputs, (2) an experimental structure for iterative comparisons in our LLM document generation pipeline, (3) a clinical safety framework to evaluate the harms of errors, and (4) a graphical user interface, CREOLA, to facilitate these processes. Our clinical error metrics were derived from 18 experimental configurations involving LLMs for clinical note generation, consisting of 12,999 clinician-annotated sentences. We observed a 1.47% hallucination rate and a 3.45% omission rate. By refining prompts and workflows, we successfully reduced major errors below previously reported human note-taking rates, highlighting the framework's potential for safer clinical documentation.