Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, Tennessee.
Department of Medicine, Vanderbilt University Medical Center, Nashville, Tennessee.
JAMA Netw Open. 2024 Aug 1;7(8):e2428276. doi: 10.1001/jamanetworkopen.2024.28276.
The Sentinel System is a key component of the US Food and Drug Administration (FDA) postmarketing safety surveillance commitment and uses clinical health care data to conduct analyses to inform drug labeling and safety communications, FDA advisory committee meetings, and other regulatory decisions. However, observational data are frequently deemed insufficient for reliable evaluation of safety concerns owing to limitations in underlying data or methodology. Advances in large language models (LLMs) provide new opportunities to address some of these limitations. However, careful consideration is necessary for how and where LLMs can be effectively deployed for these purposes.
LLMs may provide new avenues to support signal-identification activities to identify novel adverse event signals from narrative text of electronic health records. These algorithms may be used to support epidemiologic investigations examining the causal relationship between exposure to a medical product and an adverse event through development of probabilistic phenotyping of health outcomes of interest and extraction of information related to important confounding factors. LLMs may perform like traditional natural language processing tools by annotating text with controlled vocabularies with additional tailored training activities. LLMs offer opportunities for enhancing information extraction from adverse event reports, medical literature, and other biomedical knowledge sources. There are several challenges that must be considered when leveraging LLMs for postmarket surveillance. Prompt engineering is needed to ensure that LLM-extracted associations are accurate and specific. LLMs require extensive infrastructure to use, which many health care systems lack, and this can impact diversity, equity, and inclusion, and result in obscuring significant adverse event patterns in some populations. LLMs are known to generate nonfactual statements, which could lead to false positive signals and downstream evaluation activities by the FDA and other entities, incurring substantial cost.
LLMs represent a novel paradigm that may facilitate generation of information to support medical product postmarket surveillance activities that have not been possible. However, additional work is required to ensure LLMs can be used in a fair and equitable manner, minimize false positive findings, and support the necessary rigor of signal detection needed for regulatory activities.
Sentinel 系统是美国食品和药物管理局 (FDA) 上市后安全性监测承诺的关键组成部分，它使用临床医疗保健数据进行分析，以告知药物标签和安全性通信、FDA 顾问委员会会议和其他监管决策。然而，由于基础数据或方法的限制，观察性数据通常被认为不足以可靠地评估安全性问题。大型语言模型 (LLM) 的进步为解决其中的一些限制提供了新的机会。然而，对于如何以及在何处可以有效地将 LLM 用于这些目的，需要进行仔细考虑。
LLM 可能为从电子健康记录的叙述性文本中识别新的不良事件信号的信号识别活动提供新途径。这些算法可用于通过开发感兴趣的健康结果的概率表型和提取与重要混杂因素相关的信息，来支持检查暴露于医疗产品与不良事件之间因果关系的流行病学调查。LLM 可以通过使用带有附加定制培训活动的受控词汇表来标注文本，从而像传统的自然语言处理工具一样执行。LLM 为从不良事件报告、医学文献和其他生物医学知识来源中提取信息提供了机会。在利用 LLM 进行上市后监测时，必须考虑几个挑战。需要进行提示工程，以确保 LLM 提取的关联是准确和具体的。LLM 需要广泛的基础设施才能使用，而许多医疗保健系统都缺乏这种基础设施，这会影响多样性、公平性和包容性，并导致某些人群中某些重要不良事件模式不明显。LLM 已知会生成非事实陈述，这可能导致虚假的阳性信号，并导致 FDA 和其他实体进行下游评估活动，从而产生巨大的成本。
LLM 代表了一种新的范例，它可以促进生成支持医疗产品上市后监测活动的信息，这些活动以前是不可能的。然而，需要做更多的工作来确保 LLM 可以以公平和公正的方式使用，最小化假阳性发现，并支持监管活动所需的必要信号检测严谨性。
