Lester Corey, Rowell Brigid, Zheng Yifan, Co Zoe, Marshall Vincent, Kim Jin Yong, Chen Qiyuan, Kontar Raed, Yang X Jessie
Department of Clinical Pharmacy, College of Pharmacy, University of Michigan, 428 Church Street, Ann Arbor, MI, 48109, United States, 1 734-647-8849.
Department of Learning Health Sciences, University of Michigan School of Medicine, Ann Arbor, MI, United States.
JMIR Med Inform. 2025 Apr 18;13:e64902. doi: 10.2196/64902.
Artificial intelligence (AI)-based clinical decision support systems are increasingly used in health care. Uncertainty-aware AI presents the model's confidence in its decision alongside its prediction, whereas black-box AI only provides a prediction. Little is known about how this type of AI affects health care providers' work performance and reaction time.
This study aimed to determine the effects of black-box and uncertainty-aware AI advice on pharmacist decision-making and reaction time.
Recruitment emails were sent to pharmacists through professional listservs describing a web-based, crossover, randomized controlled trial. Participants were randomized to the black-box AI or uncertainty-aware AI condition in a 1:1 manner. Participants completed 100 mock verification tasks with AI help and 100 without AI help. The order of no help and AI help was randomized. Participants were exposed to correct and incorrect prescription fills, where the correct decision was to "accept" or "reject," respectively. AI help provided correct (79%) or incorrect (21%) advice. Reaction times, participant decisions, AI advice, and AI help type were recorded for each verification. Likelihood ratio tests compared means across the three categories of AI type for each level of AI correctness.
A total of 30 participants provided complete datasets. An equal number of participants were in each AI condition. Participants' decision-making performance and reaction times differed across the 3 conditions. Accurate AI recommendations resulted in the rejection of the incorrect drug 96.1% and 91.8% of the time for uncertainty-aware AI and black-box AI respectively, compared with 81.2% without AI help. Correctly dispensed medications were accepted at rates of 99.2% with black-box help, 94.1% with uncertainty-aware AI help, and 94.6% without AI help. Uncertainty-aware AI protected against bad AI advice to approve an incorrectly filled medication compared with black-box AI (83.3% vs 76.7%). When the AI recommended rejecting a correctly filled medication, pharmacists without AI help had a higher rate of correctly accepting the medication (94.6%) compared with uncertainty-aware AI help (86.2%) and black-box AI help (81.2%). Uncertainty-aware AI resulted in shorter reaction times than black-box AI and no AI help except in the scenario where "AI rejects the correct drug." Black-box AI did not lead to reduced reaction times compared with pharmacists acting alone.
Pharmacists' performance and reaction times varied by AI type and AI accuracy. Overall, uncertainty-aware AI resulted in faster decision-making and acted as a safeguard against bad AI advice to approve a misfilled medication. Conversely, black-box AI had the longest reaction times, and user performance degraded in the presence of bad AI advice. However, uncertainty-aware AI could result in unnecessary double-checks, but it is preferred over false negative advice, where patients receive the wrong medication. These results highlight the importance of well-designed AI that addresses users' needs, enhances performance, and avoids overreliance on AI.
基于人工智能（AI）的临床决策支持系统在医疗保健领域的应用日益广泛。不确定性感知型人工智能在给出预测的同时，还会呈现模型对其决策的置信度，而黑箱人工智能仅提供预测结果。对于这类人工智能如何影响医疗保健提供者的工作绩效和反应时间，人们了解甚少。
本研究旨在确定黑箱和不确定性感知型人工智能建议对药剂师决策和反应时间的影响。
通过专业邮件列表向药剂师发送招募邮件，介绍一项基于网络的交叉随机对照试验。参与者以1:1的方式随机分配到黑箱人工智能组或不确定性感知型人工智能组。参与者在有AI帮助的情况下完成100项模拟核查任务，在没有AI帮助的情况下完成100项模拟核查任务。无帮助和有AI帮助的顺序是随机的。参与者会遇到正确和错误的处方配药情况，正确的决策分别是“接受”或“拒绝”。AI帮助提供正确（79%）或错误（21%）的建议。记录每次核查的反应时间、参与者的决策、AI建议和AI帮助类型。似然比检验比较了每种AI正确性水平下三种AI类型的均值。
共有30名参与者提供了完整的数据集。每个AI组的参与者数量相等。参与者的决策表现和反应时间在三种情况下有所不同。对于不确定性感知型人工智能和黑箱人工智能，准确的AI建议分别在96.1%和91.8%的时间内导致对错误药物的拒绝，而在没有AI帮助的情况下这一比例为81.2%。在黑箱AI帮助下，正确配发的药物接受率为99.2%，在不确定性感知型AI帮助下为94.1%，在没有AI帮助的情况下为94.6%。与黑箱人工智能相比，不确定性感知型人工智能能防止因错误的AI建议而批准错误配药的情况（83.3%对76.7%）。当AI建议拒绝正确配发的药物时，没有AI帮助的药剂师正确接受药物的比例（94.6%）高于不确定性感知型AI帮助（86.2%）和黑箱AI帮助（81.2%）。除了“AI拒绝正确药物”的情况外，不确定性感知型人工智能导致的反应时间比黑箱人工智能和没有AI帮助的情况更短。与药剂师单独操作相比，黑箱人工智能并没有缩短反应时间。
药剂师的表现和反应时间因AI类型和AI准确性而异。总体而言，不确定性感知型人工智能能实现更快的决策，并能防止因错误的AI建议而批准错误配药的情况。相反，黑箱人工智能的反应时间最长，在错误的AI建议下用户表现会下降。然而，不确定性感知型人工智能可能会导致不必要的双重检查，但比起患者收到错误药物的假阴性建议，它更可取。这些结果凸显了设计良好的AI的重要性，这种AI应满足用户需求、提高性能并避免过度依赖AI。
