Hou Chaojun, Zhang Xiaodi, Tang Yu, Zhuang Jiajun, Tan Zhiping, Huang Huasheng, Chen Weilin, Wei Sheng, He Yong, Luo Shaoming
Academy of Contemporary Agriculture Engineering Innovations, Zhongkai University of Agriculture and Engineering, Guangzhou, China.
Academy of Interdisciplinary Studies, Guangdong Polytechnic Normal University, Guangzhou, China.
Front Plant Sci. 2022 Jul 29;13:972445. doi: 10.3389/fpls.2022.972445. eCollection 2022.
Intelligent detection and localization of mature citrus fruits is a critical challenge in developing an automatic harvesting robot. Variable illumination conditions and different occlusion states are some of the essential issues that must be addressed for the accurate detection and localization of citrus in the orchard environment. In this paper, a novel method for the detection and localization of mature citrus using improved You Only Look Once (YOLO) v5s with binocular vision is proposed. First, a new loss function (polarity binary cross-entropy with logit loss) for YOLO v5s is designed to calculate the loss value of class probability and objectness score, so that a large penalty for false and missing detection is applied during the training process. Second, to recover the missing depth information caused by randomly overlapping background participants, Cr-Cb chromatic mapping, the Otsu thresholding algorithm, and morphological processing are successively used to extract the complete shape of the citrus, and the kriging method is applied to obtain the best linear unbiased estimator for the missing depth value. Finally, the citrus spatial position and posture information are obtained according to the camera imaging model and the geometric features of the citrus. The experimental results show that the recall rates of citrus detection under non-uniform illumination conditions, weak illumination, and well illumination are 99.55%, 98.47%, and 98.48%, respectively, approximately 2-9% higher than those of the original YOLO v5s network. The average error of the distance between the citrus fruit and the camera is 3.98 mm, and the average errors of the citrus diameters in the 3D direction are less than 2.75 mm. The average detection time per frame is 78.96 ms. The results indicate that our method can detect and localize citrus fruits in the complex environment of orchards with high accuracy and speed. Our dataset and codes are available at https://github.com/AshesBen/citrus-detection-localization.
成熟柑橘果实的智能检测与定位是开发自动采摘机器人的一项关键挑战。可变光照条件和不同的遮挡状态是果园环境中柑橘准确检测与定位必须解决的一些重要问题。本文提出了一种使用改进的单阶段多框检测器（YOLO）v5s结合双目视觉进行成熟柑橘检测与定位的新方法。首先，为YOLO v5s设计了一种新的损失函数（带逻辑损失的极性二元交叉熵）来计算类别概率和目标得分的损失值，以便在训练过程中对误检和漏检施加较大惩罚。其次，为恢复由随机重叠的背景对象导致的缺失深度信息，依次使用Cr-Cb色度映射、大津阈值算法和形态学处理来提取柑橘的完整形状，并应用克里金法获得缺失深度值的最佳线性无偏估计。最后，根据相机成像模型和柑橘的几何特征获取柑橘的空间位置和姿态信息。实验结果表明，在非均匀光照条件、弱光照和良好光照下柑橘检测的召回率分别为99.55%、98.47%和98.48%，比原始YOLO v5s网络高出约2%-9%。柑橘果实与相机之间距离的平均误差为3.98毫米，柑橘在三维方向上直径的平均误差小于2.75毫米。每帧的平均检测时间为78.96毫秒。结果表明，我们的方法能够在果园复杂环境中高精度、快速地检测和定位柑橘果实。我们的数据集和代码可在https://github.com/AshesBen/citrus-detection-localization获取。
