Microalgae play a major role in the invasion of alien organisms with ballast water as a carrier, and traditional ballast water detection methods have many limitations in identifying microalgae species. Therefore, this paper proposes a method to identify microalgae in ballast water based on an Improved YOLOv3 model. The method first used a lightweight network MobileNet instead of the Darknet-53 network as the backbone network of feature extraction in the original YOLOv3 model. Secondly, improved spatial pyramid pooling (SPP) is introduced to pool and concatenate the multi-scale regional features so as to reduce the position error when detecting small objects. Then, by considering the overlap area of the bounding box, central point distance and aspect ratio, the Complete IoU (CIoU) algorithm is used to optimize the loss function of the YOLOv3 model. Finally, the proposed method is experimentally compared with other latest methods on the established dataset. The experimental results demonstrated that under the same conditions, this Improved YOLOv3 model achieves an average accuracy of 98.90%, and the detection efficiency is 8.59% higher than that of the original YOLOv3 model and is better than the existing methods. The average time of this method to identify a single image is 0.086 s, and it has a good detection effect on the identification of microalgae species.