Nath Utkarsh, Wang Yancheng, Turaga Pavan, Yang Yingzhen
School of Computing and Augmented Intelligence, Arizona State University, 699 S Mill Ave, Tempe, AZ 85281, USA.
School of Electrical, Computer and Energy Engineering, Arizona State University, 950 S Forest Mall, Tempe, AZ 85281, USA.
Int J Comput Vis. 2024 Jun 24;132(12):5698-5717. doi: 10.1007/s11263-024-02133-4.
Deep Neural Networks are often vulnerable to adversarial attacks. Neural Architecture Search (NAS), one of the tools for developing novel deep neural architectures, demonstrates superior performance in prediction accuracy in various machine learning applications. However, the performance of a neural architecture discovered by NAS against adversarial attacks has not been sufficiently studied, especially under the regime of knowledge distillation. Given the presence of a robust teacher, we investigate if NAS would produce a robust neural architecture by inheriting robustness from the teacher. In this paper, we propose Robust Neural Architecture Search by Cross-Layer knowledge distillation (RNAS-CL), a novel NAS algorithm that improves the robustness of NAS by learning from a robust teacher through cross-layer knowledge distillation. Unlike previous knowledge distillation methods that encourage close student-teacher output only in the last layer, RNAS-CL automatically searches for the best teacher layer to supervise each student layer. Experimental results demonstrate the effectiveness of RNAS-CL and show that RNAS-CL produces compact and adversarially robust neural architectures. Our results point to new approaches for finding compact and robust neural architecture for many applications. The code of RNAS-CL is available at https://github.com/Statistical-Deep-Learning/RNAS-CL.
深度神经网络通常容易受到对抗攻击。神经架构搜索（NAS）作为开发新型深度神经架构的工具之一，在各种机器学习应用中的预测准确性方面表现出卓越性能。然而，由NAS发现的神经架构在对抗攻击方面的性能尚未得到充分研究，尤其是在知识蒸馏的情况下。鉴于存在一个强大的教师模型，我们研究NAS是否会通过从教师模型继承鲁棒性来产生一个鲁棒的神经架构。在本文中，我们提出了通过跨层知识蒸馏的鲁棒神经架构搜索（RNAS-CL），这是一种新颖的NAS算法，通过跨层知识蒸馏从一个鲁棒的教师模型学习来提高NAS的鲁棒性。与以往仅在最后一层鼓励学生-教师输出接近的知识蒸馏方法不同，RNAS-CL自动搜索最佳教师层来监督每个学生层。实验结果证明了RNAS-CL的有效性，并表明RNAS-CL产生了紧凑且对抗鲁棒的神经架构。我们的结果为许多应用寻找紧凑且鲁棒的神经架构指出了新的方法。RNAS-CL的代码可在https://github.com/Statistical-Deep-Learning/RNAS-CL获取。
