The videofluoroscopic swallowing study (VFSS) is the gold standard for diagnosing dysphagia, but its interpretation is time-consuming and requires expertise. This study developed a deep learning model for automatically detecting penetration and aspiration in VFSS and assessed its diagnostic accuracy. Images corresponding to the highest and lowest positions of the hyoid bone -representing the moment of upper esophageal sphincter opening during swallow and the pre-swallow and post-swallow phases, respectively- were automatically extracted from VFSS videos, resulting in a total of 18,145 images from 1,467 patients. The model was trained with a convolutional neural network architecture, incorporating techniques to address class imbalance and optimize performance. The model achieved high diagnostic accuracy at the patient level, with the area under the receiver operating characteristic curve values of 0.935 (normal swallowing), 0.889 (penetration), and 0.845 (aspiration). However, despite strong performance in identifying normal swallowing, the model exhibited low sensitivity for detecting penetration and aspiration. The findings suggest that the proposed model may reduce interpretation time by minimizing the need for repeated video review to identify penetration or aspiration, enabling clinicians to focus on other clinically relevant VFSS findings. Future studies should address its limitations by analyzing full-frame VFSS data and incorporating multicenter datasets.