Tran Khanh Bao, Carballo Alexander, Takeda Kazuya
Graduate School of Informatics, Nagoya University, Furo-cho, Chikusa-ku, Nagoya 464-8601, Japan.
Faculty of Engineering and Graduate School of Engineering, Gifu University, 1-1 Yanagido, Gifu 501-1193, Japan.
Sensors (Basel). 2025 Aug 8;25(16):4902. doi: 10.3390/s25164902.
Object detection plays a significant role in various industrial and scientific domains, particularly in autonomous driving. It enables vehicles to detect surrounding objects, construct spatial maps, and facilitate safe navigation. To accomplish these tasks, a variety of sensors have been employed, including LiDAR, radar, RGB cameras, and ultrasonic sensors. Among these, LiDAR and RGB cameras are frequently utilized due to their advantages. RGB cameras offer high-resolution images with rich color and texture information but tend to underperform in low light or adverse weather conditions. In contrast, LiDAR provides precise 3D geometric data irrespective of lighting conditions, although it lacks the high spatial resolution of cameras. Recently, thermal cameras have gained significant attention in both standalone applications and in combination with RGB cameras. They offer strong perception capabilities under low-visibility conditions or adverse weather conditions. Multimodal sensor fusion effectively overcomes individual sensor limitations. In this paper, we propose a novel multimodal fusion method that integrates LiDAR, a 360 RGB camera, and a 360 thermal camera to fully leverage the strengths of each modality. Our method employs a feature-level fusion strategy that temporally accumulates and synchronizes multiple LiDAR frames. This design not only improves the detection accuracy but also enhances the spatial coverage and robustness. The use of 360 images significantly reduces blind spots and provides comprehensive environmental awareness, which is especially beneficial in complex or dynamic scenes.
目标检测在各种工业和科学领域中都发挥着重要作用，尤其是在自动驾驶领域。它使车辆能够检测周围物体，构建空间地图，并促进安全导航。为了完成这些任务，人们采用了多种传感器，包括激光雷达、雷达、RGB摄像头和超声波传感器。其中，激光雷达和RGB摄像头由于其优势而经常被使用。RGB摄像头提供具有丰富颜色和纹理信息的高分辨率图像，但在低光照或恶劣天气条件下往往表现不佳。相比之下，激光雷达无论光照条件如何都能提供精确的3D几何数据，尽管它缺乏摄像头的高空间分辨率。最近，热成像摄像头在独立应用以及与RGB摄像头结合使用方面都受到了广泛关注。它们在低能见度条件或恶劣天气条件下具有强大的感知能力。多模态传感器融合有效地克服了单个传感器的局限性。在本文中，我们提出了一种新颖的多模态融合方法，该方法集成了激光雷达、360度RGB摄像头和360度热成像摄像头，以充分利用每种模态的优势。我们的方法采用特征级融合策略，对多个激光雷达帧进行时间上的累积和同步。这种设计不仅提高了检测精度，还增强了空间覆盖范围和鲁棒性。使用360度图像显著减少了盲点，并提供了全面的环境感知，这在复杂或动态场景中尤其有益。
