Savage Thomas, Wang John, Shieh Lisa
Division of Hospital Medicine, Department of Medicine, Stanford University, Palo Alto, CA, United States.
Divison of Gastroenterology and Hepatology, Department of Medicine, Stanford University, Palo Alto, CA, United States.
JMIR Med Inform. 2023 Nov 27;11:e49886. doi: 10.2196/49886.
Best Practice Alerts (BPAs) are alert messages to physicians in the electronic health record that are used to encourage appropriate use of health care resources. While these alerts are helpful in both improving care and reducing costs, BPAs are often broadly applied nonselectively across entire patient populations. The development of large language models (LLMs) provides an opportunity to selectively identify patients for BPAs.
In this paper, we present an example case where an LLM screening tool is used to select patients appropriate for a BPA encouraging the prescription of deep vein thrombosis (DVT) anticoagulation prophylaxis. The artificial intelligence (AI) screening tool was developed to identify patients experiencing acute bleeding and exclude them from receiving a DVT prophylaxis BPA.
Our AI screening tool used a BioMed-RoBERTa (Robustly Optimized Bidirectional Encoder Representations from Transformers Pretraining Approach; AllenAI) model to perform classification of physician notes, identifying patients without active bleeding and thus appropriate for a thromboembolism prophylaxis BPA. The BioMed-RoBERTa model was fine-tuned using 500 history and physical notes of patients from the MIMIC-III (Medical Information Mart for Intensive Care) database who were not prescribed anticoagulation. A development set of 300 MIMIC patient notes was used to determine the model's hyperparameters, and a separate test set of 300 patient notes was used to evaluate the screening tool.
Our MIMIC-III test set population of 300 patients included 72 patients with bleeding (ie, were not appropriate for a DVT prophylaxis BPA) and 228 without bleeding who were appropriate for a DVT prophylaxis BPA. The AI screening tool achieved impressive accuracy with a precision-recall area under the curve of 0.82 (95% CI 0.75-0.89) and a receiver operator curve area under the curve of 0.89 (95% CI 0.84-0.94). The screening tool reduced the number of patients who would trigger an alert by 20% (240 instead of 300 alerts) and increased alert applicability by 14.8% (218 [90.8%] positive alerts from 240 total alerts instead of 228 [76%] positive alerts from 300 total alerts), compared to nonselectively sending alerts for all patients.
These results show a proof of concept on how language models can be used as a screening tool for BPAs. We provide an example AI screening tool that uses a HIPAA (Health Insurance Portability and Accountability Act)-compliant BioMed-RoBERTa model deployed with minimal computing power. Larger models (eg, Generative Pre-trained Transformers-3, Generative Pre-trained Transformers-4, and Pathways Language Model) will exhibit superior performance but require data use agreements to be HIPAA compliant. We anticipate LLMs to revolutionize quality improvement in hospital medicine.
最佳实践警报（BPA）是电子健康记录中发送给医生的警报信息，用于鼓励合理使用医疗资源。虽然这些警报有助于改善医疗服务并降低成本，但BPA通常在整个患者群体中不加选择地广泛应用。大语言模型（LLM）的发展为有选择地识别适合BPA的患者提供了机会。
在本文中，我们展示了一个案例，其中使用了一个LLM筛选工具来选择适合BPA的患者，该BPA鼓励开具深静脉血栓形成（DVT）抗凝预防药物。开发人工智能（AI）筛选工具是为了识别正在经历急性出血的患者，并将他们排除在接受DVT预防BPA之外。
我们的AI筛选工具使用了BioMed-RoBERTa（来自Transformer预训练方法的稳健优化双向编码器表示；AllenAI）模型对医生记录进行分类，识别没有活动性出血的患者，因此适合接受血栓栓塞预防BPA。使用来自MIMIC-III（重症监护医学信息库）数据库的500份未开具抗凝药物的患者病史和体格检查记录对BioMed-RoBERTa模型进行了微调。使用300份MIMIC患者记录的开发集来确定模型的超参数，并使用300份患者记录的单独测试集来评估筛选工具。
我们的300名患者的MIMIC-III测试集包括72名有出血情况的患者（即不适合DVT预防BPA）和228名无出血情况且适合DVT预防BPA的患者。AI筛选工具取得了令人印象深刻的准确率，曲线下精确召回面积为0.82（95%CI 0.75-0.89），曲线下接收者操作曲线面积为0.89（95%CI 0.84-0.94）。与对所有患者不加选择地发送警报相比，筛选工具将触发警报的患者数量减少了20%（240次警报而不是300次警报），并将警报适用性提高了14.8%（240次总警报中有218次[90.8%]阳性警报，而不是300次总警报中有228次[76%]阳性警报）。
这些结果证明了语言模型如何用作BPA的筛选工具这一概念。我们提供了一个示例AI筛选工具，该工具使用符合《健康保险流通与责任法案》（HIPAA）的BioMed-RoBERTa模型，以最小的计算能力进行部署。更大的模型（例如，生成式预训练变换器-3、生成式预训练变换器-4和路径语言模型）将表现出卓越的性能，但需要数据使用协议符合HIPAA标准。我们预计LLM将彻底改变医院医学的质量改进。
