Xu Jian, Han DeWei, Li Kang, Li JunJie, Ma ZhaoYuan
School of System Design and Intelligent Manufacturing, Southern University of Science and Technology, 1088 Xueyuan Blvd, Nan Shan District, Shen Zhen, 518055, Guang Dong, China.
University College, Korea University, Seoul, Korea.
Sci Rep. 2025 Jul 1;15(1):21458. doi: 10.1038/s41598-025-02487-w.
Keypoint detection and descriptor matching are essential in tasks like feature matching, object tracking, and 3D reconstruction. While CNN-based methods have advanced these areas, most focus on perspective projection cameras, with limited consideration of fisheye cameras, which introduce significant distortion. Conventional keypoint methods have limitations on fisheye images, causing camera models to underperform in hybrid camera systems. This paper proposes a robust keypoint detection and description method under a hybrid camera model, addressing challenges in mixed camera systems. Since fisheye datasets are scarce, we used viewpoint and projection transformations to augment our training data. We propose a method for generating fisheye data, and inspired by SuperPoint, we modify the network architecture for feature extraction and descriptor generation. By employing nearest neighbor (NN) matching, our proposed dataset generation method improves the performance of the original SuperPoint network, while the network architecture introduced in this paper further improves overall performance.
关键点检测和描述符匹配在特征匹配、目标跟踪和三维重建等任务中至关重要。虽然基于卷积神经网络（CNN）的方法推动了这些领域的发展，但大多数方法都集中在透视投影相机上，对鱼眼相机的考虑有限，而鱼眼相机存在显著的畸变。传统的关键点方法在鱼眼图像上存在局限性，导致相机模型在混合相机系统中的性能不佳。本文提出了一种在混合相机模型下的鲁棒关键点检测和描述方法，以应对混合相机系统中的挑战。由于鱼眼数据集稀缺，我们使用视点和投影变换来扩充训练数据。我们提出了一种生成鱼眼数据的方法，并受SuperPoint启发，修改了用于特征提取和描述符生成的网络架构。通过采用最近邻（NN）匹配，我们提出的数据集生成方法提高了原始SuperPoint网络的性能，而本文引入的网络架构进一步提升了整体性能。
