Department of Artificial Intelligence, Korea University, Seoul, 02841, South Korea.
Reality Labs Research, Meta, USA.
Neural Netw. 2023 Oct;167:141-158. doi: 10.1016/j.neunet.2023.08.009. Epub 2023 Aug 9.
Photometric stereo methods typically rely on RGB cameras and are usually performed in a dark room to avoid ambient illumination. Ambient illumination poses a great challenge in photometric stereo due to the restricted dynamic range of the RGB cameras. To address this limitation, we present a novel method, namely Event Fusion Photometric Stereo Network (EFPS-Net), which estimates the surface normals of an object in an ambient light environment by utilizing a deep fusion of RGB and event cameras. The high dynamic range of event cameras provides a broader perspective of light representations that RGB cameras cannot provide. Specifically, we propose an event interpolation method to obtain ample light information, which enables precise estimation of the surface normals of an object. By using RGB-event fused observation maps, our EFPS-Net outperforms previous state-of-the-art methods that depend only on RGB frames, resulting in a 7.94% reduction in mean average error. In addition, we curate a novel photometric stereo dataset by capturing objects with RGB and event cameras under numerous ambient light environments.
光度立体视觉方法通常依赖于 RGB 相机，并且通常在暗室中进行，以避免环境光照。由于 RGB 相机的动态范围有限，环境光照给光度立体视觉带来了巨大的挑战。为了解决这个限制，我们提出了一种新的方法，即事件融合光度立体视觉网络（EFPS-Net），它通过深度融合 RGB 和事件相机来估计环境光条件下物体的表面法线。事件相机的高动态范围提供了 RGB 相机无法提供的更广泛的光照表示视角。具体来说，我们提出了一种事件插值方法来获取充足的光照信息，从而能够精确估计物体的表面法线。通过使用 RGB-事件融合观测图，我们的 EFPS-Net 优于以前仅依赖于 RGB 帧的最先进方法，平均误差降低了 7.94%。此外，我们通过在多种环境光照条件下使用 RGB 和事件相机捕获物体，创建了一个新颖的光度立体视觉数据集。
