Morey Dane A, Rayo Michael F, Woods David D
The Ohio State University, Columbus, OH, USA.
NPJ Digit Med. 2025 Jun 18;8(1):374. doi: 10.1038/s41746-025-01784-y.
Processes to assure the safe, effective, and responsible deployment of artificial intelligence (AI) in safety-critical settings are urgently needed. Here we show a procedure to empirically evaluate the impacts of AI augmentation as a basis for responsible deployment. We evaluated three augmentative AI technologies nurses used to recognize imminent patient emergencies, including combinations of AI recommendations and explanations. The evaluation involved 450 nursing students and 12 licensed nurses assessing 10 historical patient cases. With each technology, nurses' performance was both improved and degraded when the AI algorithm was most correct and misleading, respectively. Our findings caution that AI capabilities alone do not guarantee a safe and effective joint human-AI system. We propose two minimum requirements for evaluating AI in safety-critical settings: (1) empirically measure the performance of people and AI together and (2) examine a range of challenging cases which produce a range of strong, mediocre, and poor AI performance.
迫切需要制定相关流程，以确保人工智能（AI）在安全关键环境中的安全、有效和负责任的部署。在此，我们展示了一种通过实证评估AI增强影响的程序，作为负责任部署的基础。我们评估了护士用于识别即将发生的患者紧急情况的三种增强型AI技术，包括AI建议和解释的组合。该评估涉及450名护理专业学生和12名执业护士，他们对10个历史患者病例进行了评估。对于每种技术，当AI算法最正确和最具误导性时，护士的表现分别得到了改善和下降。我们的研究结果提醒人们，仅靠AI能力并不能保证一个安全有效的人机联合系统。我们提出了在安全关键环境中评估AI的两个最低要求：（1）通过实证测量人与AI的共同表现；（2）检查一系列具有挑战性的案例，这些案例会产生一系列优秀、中等和较差的AI表现。
