Centre for Health Innovation, Leadership and Learning, Nottingham University Business School, University of Nottingham, Nottingham, UK.
Risk Anal. 2022 Sep;42(9):1999-2025. doi: 10.1111/risa.13850. Epub 2021 Nov 23.
Efforts to develop autonomous and intelligent systems (AIS) have exploded across a range of settings in recent years, from self-driving cars to medical diagnostic chatbots. These have the potential to bring enormous benefits to society but also have the potential to introduce new-or amplify existing-risks. As these emerging technologies become more widespread, one of the most critical risk management challenges is to ensure that failures of AIS can be rigorously analyzed and understood so that the safety of these systems can be effectively governed and improved. AIS are necessarily developed and deployed within complex human, social, and organizational systems, but to date there has been little systematic examination of the sociotechnical sources of risk and failure in AIS. Accordingly, this article develops a conceptual framework that characterizes key sociotechnical sources of risk in AIS by reanalyzing one of the most publicly reported failures to date: the 2018 fatal crash of Uber's self-driving car. Publicly available investigative reports were systematically analyzed using constant comparative analysis to identify key sources and patterns of sociotechnical risk. Five fundamental domains of sociotechnical risk were conceptualized-structural, organizational, technological, epistemic, and cultural-each indicated by particular patterns of sociotechnical failure. The resulting SOTEC framework of sociotechnical risk in AIS extends existing theories of risk in complex systems and highlights important practical and theoretical implications for managing risk and developing infrastructures of learning in AIS.
近年来，从自动驾驶汽车到医疗诊断聊天机器人，自主和智能系统（AIS）的开发在各种环境中迅速发展。这些系统有可能给社会带来巨大的好处，但也有可能引入新的风险，或放大现有的风险。随着这些新兴技术的普及，最关键的风险管理挑战之一是确保能够严格分析和理解 AIS 的故障，从而有效地管理和改进这些系统的安全性。AIS 必须在复杂的人为、社会和组织系统中进行开发和部署，但迄今为止，对 AIS 中的风险和故障的社会技术源几乎没有进行系统的研究。因此，本文通过重新分析迄今为止最公开报道的故障之一——2018 年优步自动驾驶汽车致命撞车事故，开发了一个概念框架，该框架通过重新分析迄今为止最公开报道的故障之一——2018 年优步自动驾驶汽车致命撞车事故，来描述 AIS 中的关键社会技术风险源。使用恒定性比较分析对公开的调查报告进行了系统分析，以确定社会技术风险的关键来源和模式。构想了五个基本的社会技术风险领域——结构、组织、技术、认知和文化——每个领域都有特定的社会技术故障模式。由此产生的 AIS 中的社会技术风险 SOTEC 框架扩展了复杂系统中风险的现有理论，并突出了在 AIS 中管理风险和开发学习基础设施的重要实践和理论意义。
