Department of Pathology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA.
Department of Pathology, Massachusetts General Hospital, Harvard Medical School, Boston, MA, USA.
Nature. 2024 Oct;634(8033):466-473. doi: 10.1038/s41586-024-07618-3. Epub 2024 Jun 12.
Computational pathology has witnessed considerable progress in the development of both task-specific predictive models and task-agnostic self-supervised vision encoders. However, despite the explosive growth of generative artificial intelligence (AI), there have been few studies on building general-purpose multimodal AI assistants and copilots tailored to pathology. Here we present PathChat, a vision-language generalist AI assistant for human pathology. We built PathChat by adapting a foundational vision encoder for pathology, combining it with a pretrained large language model and fine-tuning the whole system on over 456,000 diverse visual-language instructions consisting of 999,202 question and answer turns. We compare PathChat with several multimodal vision-language AI assistants and GPT-4V, which powers the commercially available multimodal general-purpose AI assistant ChatGPT-4 (ref. ). PathChat achieved state-of-the-art performance on multiple-choice diagnostic questions from cases with diverse tissue origins and disease models. Furthermore, using open-ended questions and human expert evaluation, we found that overall PathChat produced more accurate and pathologist-preferable responses to diverse queries related to pathology. As an interactive vision-language AI copilot that can flexibly handle both visual and natural language inputs, PathChat may potentially find impactful applications in pathology education, research and human-in-the-loop clinical decision-making.
计算病理学在任务特定的预测模型和任务不可知的自监督视觉编码器的发展方面取得了相当大的进展。然而,尽管生成式人工智能 (AI) 呈爆炸式增长,但针对病理学构建通用多模态 AI 助手和副驾的研究却很少。在这里,我们提出了 PathChat,这是一种用于人体病理学的视觉语言通才 AI 助手。我们通过适应用于病理学的基础视觉编码器来构建 PathChat,将其与预训练的大型语言模型相结合,并在由 999,202 个问答轮次组成的超过 456,000 个不同的视觉语言指令上对整个系统进行微调。我们将 PathChat 与几个多模态视觉语言 AI 助手和 GPT-4V 进行了比较,后者为商业化的多模态通用 AI 助手 ChatGPT-4 提供支持(参考文献)。PathChat 在来自具有不同组织起源和疾病模型的病例的多项多项选择诊断问题上取得了最先进的性能。此外,通过使用开放式问题和人类专家评估,我们发现总体而言,PathChat 对与病理学相关的各种查询产生了更准确和病理学家更喜欢的响应。作为一种灵活处理视觉和自然语言输入的交互式视觉语言 AI 副驾,PathChat 可能在病理学教育、研究和人机交互临床决策方面具有重要的应用价值。
Nature. 2024-10
J Med Internet Res. 2025-6-23
J Med Internet Res. 2025-2-7
J Med Internet Res. 2024-11-15
Cochrane Database Syst Rev. 2024-10-17
Cochrane Database Syst Rev. 2023-11-15
Front Pharmacol. 2025-8-20
Precis Clin Med. 2025-7-17
Nature. 2023-8
Cancer Cell. 2022-10-10