Zou Yunhao, Fu Ying, Takatani Tsuyoshi, Zheng Yinqiang
IEEE Trans Pattern Anal Mach Intell. 2025 Jan;47(1):32-50. doi: 10.1109/TPAMI.2024.3469571. Epub 2024 Dec 4.
Event cameras are innovative neuromorphic sensors that asynchronously capture the scene dynamics. Due to the event-triggering mechanism, such cameras record event streams with much shorter response latency and higher intensity sensitivity compared to conventional cameras. On the basis of these features, previous works have attempted to reconstruct high dynamic range (HDR) videos from events, but have either suffered from unrealistic artifacts or failed to provide sufficiently high frame rates. In this paper, we present a recurrent convolutional neural network that reconstruct high-speed HDR videos from event sequences, with a key frame guidance to prevent potential error accumulation caused by the sparse event data. Additionally, to address the problem of severely limited real dataset, we develop a new optical system to collect a real-world dataset with paired high-speed HDR videos and event streams, facilitating future research in this field. Our dataset provides the first real paired dataset for event-to-HDR reconstruction, avoiding potential inaccuracies from simulation strategies. Experimental results demonstrate that our method can generate high-quality, high-speed HDR videos. We further explore the potential of our work in cross-camera reconstruction and downstream computer vision tasks, including object detection, panoramic segmentation, optical flow estimation, and monocular depth estimation under HDR scenarios.
事件相机是一种创新的神经形态传感器，能够异步捕捉场景动态。由于其事件触发机制，与传统相机相比，此类相机记录事件流的响应延迟更短，强度敏感度更高。基于这些特性，先前的研究尝试从事件中重建高动态范围（HDR）视频，但要么存在不真实的伪影，要么未能提供足够高的帧率。在本文中，我们提出了一种循环卷积神经网络，该网络可从事件序列中重建高速HDR视频，并通过关键帧引导来防止由稀疏事件数据导致的潜在误差积累。此外，为了解决真实数据集严重受限的问题，我们开发了一种新的光学系统，以收集包含配对高速HDR视频和事件流的真实世界数据集，促进该领域的未来研究。我们的数据集为事件到HDR重建提供了首个真实配对数据集，避免了模拟策略可能产生的不准确问题。实验结果表明，我们的方法能够生成高质量、高速的HDR视频。我们还进一步探索了我们的工作在跨相机重建以及下游计算机视觉任务中的潜力，包括目标检测、全景分割、光流估计以及HDR场景下的单目深度估计。
