Usmani Kashif, O'Connor Timothy, Wani Pranav, Javidi Bahram
Opt Express. 2023 Jan 2;31(1):479-491. doi: 10.1364/OE.478125.
In this paper, we address the problem of object recognition in degraded environments including fog and partial occlusion. Both long wave infrared (LWIR) imaging systems and LiDAR (time of flight) imaging systems using Azure Kinect, which combine conventional visible and lidar sensing information, have been previously demonstrated for object recognition in ideal conditions. However, the object detection performance of Azure Kinect depth imaging systems may decrease significantly in adverse weather conditions such as fog, rain, and snow. The concentration of fog degrades the depth images of Azure Kinect camera, and the overall visibility of RGBD images (fused RGB and depth image), which can make object recognition tasks challenging. LWIR imaging may avoid these issues of lidar-based imaging systems. However, due to poor spatial resolution of LWIR cameras, thermal imaging provides limited textural information within a scene and hence may fail to provide adequate discriminatory information to identify between objects of similar texture, shape and size. To improve the object detection task in fog and occlusion, we use three-dimensional (3D) integral imaging (InIm) system with a visible range camera. 3D InIm provides depth information, mitigates the occlusion and fog in front of the object, and improves the object recognition capabilities. For object recognition, the YOLOv3 neural network is used for each of the tested imaging systems. Since the concentration of fog affects the images from different sensors (visible, LWIR, and Azure Kinect depth cameras) in different ways, we compared the performance of the network on these images in terms of average precision and average miss rate. For the experiments we conducted, the results indicate that in degraded environment 3D InIm using visible range cameras can provide better image reconstruction as compared to the LWIR camera and Azure Kinect RGBD camera, and therefore it may improve the detection accuracy of the network. To the best of our knowledge, this is the first report comparing the performance of object detection between passive integral imaging system vs active (LiDAR) sensing in degraded environments such as fog and partial occlusion.
在本文中，我们研究了在包括雾和部分遮挡在内的退化环境中的目标识别问题。长波红外（LWIR）成像系统和使用Azure Kinect的激光雷达（飞行时间）成像系统，它们结合了传统的可见光和激光雷达传感信息，此前已被证明可用于理想条件下的目标识别。然而，Azure Kinect深度成像系统在雾、雨、雪等恶劣天气条件下的目标检测性能可能会显著下降。雾的浓度会降低Azure Kinect相机的深度图像以及RGB-D图像（融合的RGB和深度图像）的整体能见度，这会使目标识别任务具有挑战性。LWIR成像可以避免基于激光雷达的成像系统的这些问题。然而，由于LWIR相机的空间分辨率较差，热成像在场景中提供的纹理信息有限，因此可能无法提供足够的鉴别信息来区分纹理、形状和大小相似的物体。为了改进在雾和遮挡情况下的目标检测任务，我们使用了带有可见光相机的三维（3D）积分成像（InIm）系统。3D InIm提供深度信息，减轻物体前方的遮挡和雾，并提高目标识别能力。对于目标识别，YOLOv3神经网络被用于每个测试的成像系统。由于雾的浓度以不同方式影响来自不同传感器（可见光、LWIR和Azure Kinect深度相机）的图像，我们根据平均精度和平均漏检率比较了网络在这些图像上的性能。对于我们进行的实验，结果表明，在退化环境中，与LWIR相机和Azure Kinect RGB-D相机相比使用可见光相机的3D InIm可以提供更好的图像重建，因此它可能提高网络的检测精度。据我们所知，这是第一份比较在雾和部分遮挡等退化环境中被动积分成像系统与主动（激光雷达）传感的目标检测性能的报告。
