Khosla Deepak, Chen Yang, Kim Kyungnam
HRL Laboratories, LLC Malibu, CA, USA.
Front Comput Neurosci. 2014 Nov 28;8:147. doi: 10.3389/fncom.2014.00147. eCollection 2014.
Automated video object recognition is a topic of emerging importance in both defense and civilian applications. This work describes an accurate and low-power neuromorphic architecture and system for real-time automated video object recognition. Our system, Neuormorphic Visual Understanding of Scenes (NEOVUS), is inspired by computational neuroscience models of feed-forward object detection and classification pipelines for processing visual data. The NEOVUS architecture is inspired by the ventral (what) and dorsal (where) streams of the mammalian visual pathway and integrates retinal processing, object detection based on form and motion modeling, and object classification based on convolutional neural networks. The object recognition performance and energy use of the NEOVUS was evaluated by the Defense Advanced Research Projects Agency (DARPA) under the Neovision2 program using three urban area video datasets collected from a mix of stationary and moving platforms. These datasets are challenging and include a large number of objects of different types in cluttered scenes, with varying illumination and occlusion conditions. In a systematic evaluation of five different teams by DARPA on these datasets, the NEOVUS demonstrated the best performance with high object recognition accuracy and the lowest energy consumption. Its energy use was three orders of magnitude lower than two independent state of the art baseline computer vision systems. The dynamic power requirement for the complete system mapped to commercial off-the-shelf (COTS) hardware that includes a 5.6 Megapixel color camera processed by object detection and classification algorithms at 30 frames per second was measured at 21.7 Watts (W), for an effective energy consumption of 5.45 nanoJoules (nJ) per bit of incoming video. These unprecedented results show that the NEOVUS has the potential to revolutionize automated video object recognition toward enabling practical low-power and mobile video processing applications.
自动视频目标识别在国防和民用应用中都是一个日益重要的话题。这项工作描述了一种用于实时自动视频目标识别的精确且低功耗的神经形态架构和系统。我们的系统，即神经形态场景视觉理解（NEOVUS），受到用于处理视觉数据的前馈目标检测和分类管道的计算神经科学模型的启发。NEOVUS架构受到哺乳动物视觉通路的腹侧（“什么”）和背侧（“哪里”）信息流的启发，并整合了视网膜处理、基于形状和运动建模的目标检测以及基于卷积神经网络的目标分类。国防高级研究计划局（DARPA）在Neovision2计划下，使用从固定和移动平台混合收集的三个市区视频数据集，对NEOVUS的目标识别性能和能源使用情况进行了评估。这些数据集具有挑战性，包括杂乱场景中大量不同类型的目标，光照和遮挡条件各不相同。在DARPA对这些数据集的五个不同团队进行的系统评估中，NEOVUS表现出了最佳性能，具有高目标识别准确率和最低能耗。其能源使用比两个独立的先进基线计算机视觉系统低三个数量级。映射到商用现货（COTS）硬件的完整系统的动态功率需求，该硬件包括一个560万像素的彩色相机，由目标检测和分类算法以每秒30帧的速度处理，测量结果为21.7瓦（W），对于每比特输入视频的有效能耗为5.45纳焦（nJ）。这些前所未有的结果表明，NEOVUS有潜力彻底改变自动视频目标识别，以实现实用的低功耗和移动视频处理应用。
