Geng Mengyue, Wang Lizhi, Zhu Lin, Zhang Wei, Xiong Ruiqin, Tian Yonghong
IEEE Trans Pattern Anal Mach Intell. 2025 Jan;47(1):206-223. doi: 10.1109/TPAMI.2024.3465455. Epub 2024 Dec 4.
Snapshot Mosaic Hyperspectral Cameras (SMHCs) are popular hyperspectral imaging devices for acquiring both color and motion details of scenes. However, the narrow-band spectral filters in SMHCs may negatively impact their motion perception ability, resulting in blurry SMHC frames. In this paper, we propose a hardware-software collaborative approach to address the blurring issue of SMHCs. Our approach involves integrating SMHCs with neuromorphic event cameras for efficient event-enhanced SMHC frame deblurring. To achieve spectral information recovery guided by event signals, we formulate a spectral-aware Event-based Double Integral (sEDI) model that links SMHC frames and events from a spectral perspective, providing principled model design insights. Then, we develop a Diffusion-guided Noise Awareness (DNA) training framework that utilizes diffusion models to learn noise-aware features and promote model robustness towards camera noise. Furthermore, we design an Event-enhanced Hyperspectral frame Deblurring Network (EvHDNet) based on sEDI, which is trained with DNA and features improved spatial-spectral learning and modality interaction for reliable SMHC frame deblurring. Experiments on both synthetic data and real data show that the proposed DNA + EvHDNet outperforms state-of-the-art methods on both spatial and spectral fidelity. The code and dataset will be made publicly available.
快照马赛克高光谱相机（SMHC）是用于获取场景颜色和运动细节的流行高光谱成像设备。然而，SMHC中的窄带光谱滤波器可能会对其运动感知能力产生负面影响，导致SMHC帧模糊。在本文中，我们提出了一种硬件-软件协作方法来解决SMHC的模糊问题。我们的方法包括将SMHC与神经形态事件相机集成，以实现高效的事件增强SMHC帧去模糊。为了在事件信号的引导下恢复光谱信息，我们制定了一种光谱感知的基于事件的双重积分（sEDI）模型，该模型从光谱角度将SMHC帧和事件联系起来，提供了有原则的模型设计见解。然后，我们开发了一种扩散引导的噪声感知（DNA）训练框架，该框架利用扩散模型来学习噪声感知特征，并提高模型对相机噪声的鲁棒性。此外，我们基于sEDI设计了一个事件增强的高光谱帧去模糊网络（EvHDNet），该网络使用DNA进行训练，并具有改进的空间-光谱学习和模态交互功能，以实现可靠的SMHC帧去模糊。在合成数据和真实数据上的实验表明，所提出的DNA + EvHDNet在空间和光谱保真度方面均优于现有方法。代码和数据集将公开提供。
