Junger Christina, Buch Benjamin, Notni Gunther
Group for Quality Assurance and Industrial Image Processing, Technische Universität Ilmenau, 98693 Ilmenau, Germany.
Fraunhofer Institute for Applied Optics and Precision Engineering IOF Jena, 07745 Jena, Germany.
Sensors (Basel). 2023 Aug 8;23(16):7030. doi: 10.3390/s23167030.
The projection of a point cloud onto a 2D camera image is relevant in the case of various image analysis and enhancement tasks, e.g., (i) in multimodal image processing for data fusion, (ii) in robotic applications and in scene analysis, and (iii) for deep neural networks to generate real datasets with ground truth. The challenges of the current single-shot projection methods, such as simple state-of-the-art projection, conventional, polygon, and deep learning-based upsampling methods or closed source SDK functions of low-cost depth cameras, have been identified. We developed a new way to project point clouds onto a dense, accurate 2D raster image, called Triangle-Mesh-Rasterization-Projection (TMRP). The only gaps that the 2D image still contains with our method are valid gaps that result from the physical limits of the capturing cameras. Dense accuracy is achieved by simultaneously using the 2D neighborhood information (rx,ry) of the 3D coordinates in addition to the points P(X,Y,V). In this way, a fast triangulation interpolation can be performed. The interpolation weights are determined using sub-triangles. Compared to single-shot methods, our algorithm is able to solve the following challenges. This means that: (1) no false gaps or false neighborhoods are generated, (2) the density is XYZ independent, and (3) ambiguities are eliminated. Our TMRP method is also open source, freely available on GitHub, and can be applied to almost any sensor or modality. We also demonstrate the usefulness of our method with four use cases by using the KITTI-2012 dataset or sensors with different modalities. Our goal is to improve recognition tasks and processing optimization in the perception of transparent objects for robotic manufacturing processes.
在各种图像分析和增强任务中，例如（i）在用于数据融合的多模态图像处理中，（ii）在机器人应用和场景分析中，以及（iii）用于深度神经网络以生成带有真实标注的真实数据集时，点云到二维相机图像的投影都具有重要意义。当前单次投影方法存在一些挑战，例如简单的最先进投影、传统的多边形投影、基于深度学习的上采样方法或低成本深度相机的闭源软件开发工具包函数等问题已被识别出来。我们开发了一种将点云投影到密集、精确的二维光栅图像上的新方法，称为三角网格光栅化投影（TMRP）。使用我们的方法，二维图像中仍然存在的唯一间隙是由捕获相机的物理限制导致的有效间隙。通过除了点P(X,Y,V)之外同时使用三维坐标的二维邻域信息(rx,ry)来实现密集精度。通过这种方式，可以执行快速三角剖分插值。插值权重使用子三角形来确定。与单次投影方法相比，我们的算法能够解决以下挑战。这意味着：（1）不会产生虚假间隙或虚假邻域，（2）密度与XYZ无关，（3）消除了模糊性。我们的TMRP方法也是开源的，可在GitHub上免费获取，并且几乎可以应用于任何传感器或模态。我们还通过使用KITTI - 2012数据集或不同模态的传感器，用四个用例展示了我们方法的实用性。我们的目标是改进机器人制造过程中透明物体感知方面的识别任务和处理优化。
