Oess Timo, Löhr Maximilian P R, Schmid Daniel, Ernst Marc O, Neumann Heiko
Applied Cognitive Psychology, Institute of Psychology and Education, Ulm University, Ulm, Germany.
Vision and Perception Science Lab, Institute of Neural Information Processing, Ulm University, Ulm, Germany.
Front Neurorobot. 2020 May 15;14:29. doi: 10.3389/fnbot.2020.00029. eCollection 2020.
While interacting with the world our senses and nervous system are constantly challenged to identify the origin and coherence of sensory input signals of various intensities. This problem becomes apparent when stimuli from different modalities need to be combined, e.g., to find out whether an auditory stimulus and a visual stimulus belong to the same object. To cope with this problem, humans and most other animal species are equipped with complex neural circuits to enable fast and reliable combination of signals from various sensory organs. This multisensory integration starts in the brain stem to facilitate unconscious reflexes and continues on ascending pathways to cortical areas for further processing. To investigate the underlying mechanisms in detail, we developed a canonical neural network model for multisensory integration that resembles neurophysiological findings. For example, the model comprises multisensory integration neurons that receive excitatory and inhibitory inputs from unimodal auditory and visual neurons, respectively, as well as feedback from cortex. Such feedback projections facilitate multisensory response enhancement and lead to the commonly observed inverse effectiveness of neural activity in multisensory neurons. Two versions of the model are implemented, a rate-based neural network model for qualitative analysis and a variant that employs spiking neurons for deployment on a neuromorphic processing. This dual approach allows to create an evaluation environment with the ability to test model performances with real world inputs. As a platform for deployment we chose IBM's neurosynaptic chip TrueNorth. Behavioral studies in humans indicate that temporal and spatial offsets as well as reliability of stimuli are critical parameters for integrating signals from different modalities. The model reproduces such behavior in experiments with different sets of stimuli. In particular, model performance for stimuli with varying spatial offset is tested. In addition, we demonstrate that due to the emergent properties of network dynamics model performance is close to optimal Bayesian inference for integration of multimodal sensory signals. Furthermore, the implementation of the model on a neuromorphic processing chip enables a complete neuromorphic processing cascade from sensory perception to multisensory integration and the evaluation of model performance for real world inputs.
在与外界互动时，我们的感官和神经系统不断面临挑战，需要识别各种强度的感官输入信号的来源和连贯性。当需要组合来自不同模态的刺激时，这个问题就变得很明显，例如，要确定一个听觉刺激和一个视觉刺激是否属于同一个物体。为了应对这个问题，人类和大多数其他动物物种都配备了复杂的神经回路，以便能够快速可靠地组合来自各种感觉器官的信号。这种多感官整合始于脑干，以促进无意识反射，并在向上的通路中继续传递到皮层区域进行进一步处理。为了详细研究其潜在机制，我们开发了一个用于多感官整合的典型神经网络模型，该模型类似于神经生理学发现。例如，该模型包括多感官整合神经元，它们分别从单模态听觉和视觉神经元接收兴奋性和抑制性输入，以及来自皮层的反馈。这种反馈投射促进了多感官反应增强，并导致多感官神经元中神经活动普遍观察到的反向有效性。该模型实现了两个版本，一个用于定性分析的基于速率的神经网络模型，以及一个采用脉冲神经元以便在神经形态处理上进行部署的变体。这种双重方法允许创建一个评估环境，能够用现实世界的输入来测试模型性能。作为部署平台，我们选择了IBM的神经突触芯片TrueNorth。人类的行为研究表明，时间和空间偏移以及刺激的可靠性是整合来自不同模态信号的关键参数。该模型在不同组刺激的实验中再现了这种行为。特别是，测试了具有不同空间偏移的刺激的模型性能。此外，我们证明，由于网络动力学的涌现特性，模型性能接近用于整合多模态感官信号的最优贝叶斯推理。此外，该模型在神经形态处理芯片上的实现使得从感官感知到多感官整合的完整神经形态处理级联成为可能，并能够评估针对现实世界输入的模型性能。
