Center for Brain-Inspired Computing Research (CBICR), Optical Memory National Engineering Research Center, and Department of Precision Instrument, Tsinghua University, Beijing 100084, China.
Institute of Theoretical Computer Science, Graz University of Technology, Graz, Austria.
Sci Robot. 2023 May 17;8(78):eabm6996. doi: 10.1126/scirobotics.abm6996. Epub 2023 May 10.
Place recognition is an essential spatial intelligence capability for robots to understand and navigate the world. However, recognizing places in natural environments remains a challenging task for robots because of resource limitations and changing environments. In contrast, humans and animals can robustly and efficiently recognize hundreds of thousands of places in different conditions. Here, we report a brain-inspired general place recognition system, dubbed NeuroGPR, that enables robots to recognize places by mimicking the neural mechanism of multimodal sensing, encoding, and computing through a continuum of space and time. Our system consists of a multimodal hybrid neural network (MHNN) that encodes and integrates multimodal cues from both conventional and neuromorphic sensors. Specifically, to encode different sensory cues, we built various neural networks of spatial view cells, place cells, head direction cells, and time cells. To integrate these cues, we designed a multiscale liquid state machine that can process and fuse multimodal information effectively and asynchronously using diverse neuronal dynamics and bioinspired inhibitory circuits. We deployed the MHNN on Tianjic, a hybrid neuromorphic chip, and integrated it into a quadruped robot. Our results show that NeuroGPR achieves better performance compared with conventional and existing biologically inspired approaches, exhibiting robustness to diverse environmental uncertainty, including perceptual aliasing, motion blur, light, or weather changes. Running NeuroGPR as an overall multi-neural network workload on Tianjic showcases its advantages with 10.5 times lower latency and 43.6% lower power consumption than the commonly used mobile robot processor Jetson Xavier NX.
位置识别是机器人理解和导航世界的基本空间智能能力。然而，由于资源限制和不断变化的环境，机器人识别自然环境中的位置仍然是一项具有挑战性的任务。相比之下，人类和动物可以在不同的条件下稳健高效地识别数十万的地点。在这里，我们报告了一个受大脑启发的通用位置识别系统，称为 NeuroGPR，它通过连续的空间和时间来模拟多模态传感、编码和计算的神经机制，使机器人能够识别位置。我们的系统由一个多模态混合神经网络 (MHNN) 组成，该网络对来自传统和神经形态传感器的多模态线索进行编码和整合。具体来说，为了编码不同的感觉线索，我们构建了各种空间视图细胞、位置细胞、头部方向细胞和时间细胞的神经网络。为了整合这些线索，我们设计了一个多尺度液体状态机，它可以使用不同的神经元动力学和受生物启发的抑制电路有效地异步地处理和融合多模态信息。我们将 MHNN 部署在 Tianjic 混合神经形态芯片上，并将其集成到四足机器人中。我们的结果表明，NeuroGPR 与传统和现有的生物启发方法相比具有更好的性能，表现出对各种环境不确定性的鲁棒性，包括感知混淆、运动模糊、光照或天气变化。在 Tianjic 上作为一个整体的多神经网络工作负载运行 NeuroGPR 具有 10.5 倍的延迟降低和 43.6%的功率降低，比常用的移动机器人处理器 Jetson Xavier NX 更有优势。
