The kidney bean is an important cash crop whose growth and yield are severely affected by brown spot disease. Traditional target detection models cannot effectively screen out key features, resulting in model overfitting and weak generalization ability. In this study, a Bi-Directional Feature Pyramid Network (BiFPN) and Squeeze and Excitation (SE) module were added to a YOLOv5 model to improve the multi-scale feature fusion and key feature extraction abilities of the improved model. The results show that the BiFPN and SE modules show higher heat in the target location region and pay less attention to irrelevant environmental information in the non-target region. The detection Precision, Recall, and mean average Precision (mAP@0.5) of the improved YOLOv5 model are 94.7%, 88.2%, and 92.5%, respectively, which are 4.9% higher in Precision, 0.5% higher in Recall, and 25.6% higher in the mean average Precision compared to the original YOLOv5 model. Compared with the YOLOv5-SE, YOLOv5-BiFPN, FasterR-CNN, and EfficientDet models, detection Precision improved by 1.8%, 3.0%, 9.4%, and 9.5%, respectively. Moreover, the rate of missed and wrong detection in the improved YOLOv5 model is only 8.16%. Therefore, the YOLOv5-SE-BiFPN model can more effectively detect the brown spot area of kidney beans.