IEEE Trans Neural Netw Learn Syst. 2021 May;32(5):1821-1830. doi: 10.1109/TNNLS.2020.2991090. Epub 2021 May 3.
The vulnerability of artificial intelligence (AI) and machine learning (ML) against adversarial disturbances and attacks significantly restricts their applicability in safety-critical systems including cyber-physical systems (CPS) equipped with neural network components at various stages of sensing and control. This article addresses the reachable set estimation and safety verification problems for dynamical systems embedded with neural network components serving as feedback controllers. The closed-loop system can be abstracted in the form of a continuous-time sampled-data system under the control of a neural network controller. First, a novel reachable set computation method in adaptation to simulations generated out of neural networks is developed. The reachability analysis of a class of feedforward neural networks called multilayer perceptrons (MLPs) with general activation functions is performed in the framework of interval arithmetic. Then, in combination with reachability methods developed for various dynamical system classes modeled by ordinary differential equations, a recursive algorithm is developed for over-approximating the reachable set of the closed-loop system. The safety verification for neural network control systems can be performed by examining the emptiness of the intersection between the over-approximation of reachable sets and unsafe sets. The effectiveness of the proposed approach has been validated with evaluations on a robotic arm model and an adaptive cruise control system.
人工智能 (AI) 和机器学习 (ML) 对对抗性干扰和攻击的脆弱性极大地限制了它们在包括配备神经网络组件的网络物理系统 (CPS) 在内的安全关键系统中的适用性，这些系统在传感和控制的各个阶段都配备了神经网络组件。本文针对作为反馈控制器的神经网络组件嵌入的动态系统的可达集估计和安全验证问题进行了研究。在神经网络控制器的控制下，闭环系统可以抽象为连续时间采样数据系统。首先，开发了一种新的可达集计算方法，以适应神经网络生成的仿真。在区间运算框架内，对具有通用激活函数的前馈神经网络（多层感知器，MLP）进行可达性分析。然后，结合针对由常微分方程建模的各种动态系统类开发的可达性方法，为闭环系统的可达集开发了一个递归算法。通过检查可达集的上近似和不安全集之间的交集是否为空，可以对神经网络控制系统进行安全验证。该方法在机器人手臂模型和自适应巡航控制系统上的评估中得到了验证。
