BACKGROUND: Healthy aging is crucial to the quality of life of older adults, of which mental health is an essential part. Physical exercise strongly affects their mental health and can alleviate psychological problems to a certain extent. Nevertheless, the correlation between physical exercise and the mental health of older adults individuals, as well as the underlying mechanism by which physical exercise impacts mental health, remains rather ambiguous. METHODS: We utilized multiple linear regression models to investigate the relationship between physical activity and mental health in 3,240 persons aged 60 and up. The propensity score matching (PSM) method was used to assess the robustness of the regression results. In addition, sequential recursive models were employed to investigate the mediating role of social competence in the link between physical activity and mental health. RESULTS AND DISCUSSION: We discovered a strong favorable association between physical exercise and mental health, which was mediated by social competence. Furthermore, the effect of physical exercise on mental health differed across older persons from various socioeconomic backgrounds. CONCLUSION: Older adults should strengthen their understanding of the role of physical exercise. Sports organizations that serve older adults by providing resources and services to help them maintain physical fitness and by hosting sports activities and competitions according to their needs should be established.