The purpose of this study is to realize the automatic identification and classification of fouls in football matches and improve the overall identification accuracy. Therefore, a Deep Learning-Based Saliency Prediction Model (DLSPM) is proposed. DLSPM combines the improved DeepPlaBV 3+architecture for salient region detection, Graph Convolutional Networks (GCN) for feature extraction and Deep Neural Network (DNN) for classification. By automatically identifying the key action areas in the image, the model reduces the dependence on traditional image processing technology and manual feature extraction, and improves the accuracy and robustness of foul behavior identification. The experimental results show that DLSPM performs significantly better than the existing methods on multiple video motion recognition data sets, especially when dealing with complex scenes and dynamic changes. The research results not only provide a new perspective and method for the field of video motion recognition, but also lay a foundation for the application in intelligent monitoring and human-computer interaction.