Due to inconsistent results in earlier investigations regarding the relationship between vitamin D and prostate-specific antigen (PSA), this study was conducted to gain a deeper understanding of the association between vitamin D and PSA. A total of 7174 male samples with 25(OH)D, PSA, and other variables were obtained from the National Health and Nutrition Examination Survey (NHANES) database. Three models, created through stepwise logistic regression, were employed to examine the dose-response association between PSA and 25(OH)D. Subsequently, restricted cubic spline analysis (RCS) was used to explore the nonlinear association between 25(OH)D and PSA. The study also compared the performance of four machine learning models in predicting PSA levels. The dose-response relationship indicated a negative impact of high 25(OH)D levels on PSA (p for trend 0.05). The odds ratio (OR) of Q4 (7.73 with 95% CI (0.26, 15.76)) was significantly higher than Q1 (6.23 with 95% CI (0.24, 12.57)). OR values in Q2 and Q3 were less than 1 (Q2= 0.57 with 95% CI (-6.37, 8.04) and Q3= 0.26 with 95% CI (-5.94, 6.86)), suggesting a potential protective effect of 25(OH)D on PSA. RCS analysis revealed a U-shaped relationship between blood 25(OH)D levels and PSA, with serum 25(OH)D in the range of 20-134 ng/ml showing a potential decrease in PSA levels. Above this range, an increase in 25(OH)D might elevate PSA levels. Age (2.67 with 95% CI 2.24 to 3.1) and BMI (17.52 with 95% CI 7.65 to 26.32), along with the OR of obesity (10.36 with 95% CI 0.68 to 20.18), were identified as potential PSA risk factors. Among the machine learning models, the random forest algorithm performed the best in predicting PSA levels. This study revealed a U-shaped relationship between 25(OH)D and PSA, with PSA potentially declining when 25(OH)D is between 20 and 134 ng/mL and possibly rising above this range. The random forest method proved effective in both predicting PSA levels and guiding vitamin D dosage.