Deep learning is currently being used to automate surface defect detection in aluminum. The common target detection models based on neural networks often have a large number of parameters and a slow detection speed, which is not conducive to real-time detection. Therefore, this paper proposes a lightweight aluminum surface defect detection model, M2-BL-YOLOv4, based on the YOLOv4 algorithm. First, in the YOLOv4 model, the complex CSPDarkNet53 backbone network was modified into an inverted residual structure, which greatly reduced the number of parameters in the model and increased the detection speed. Second, a new feature fusion network, BiFPN-Lite, is designed to improve the fusion ability of the network and further improve its detection accuracy. The final results show that the mean average precision of the improved lightweight YOLOv4 algorithm in the aluminum surface defect test set reaches 93.5%, the number of model parameters is reduced to 60% of the original, and the number of frames per second (FPS) detected is 52.99, which increases the detection speed by 30%. The efficient detection of aluminum surface defects is realized.