Undisturbed reproduction is key for successful breeding of beef and dairy cattle. Improving reproductive ability can be difficult because of antagonistic relationships with other economically relevant traits. In cattle, thorough investigation of female fertility revealed unfavorable genetic correlations with various production phenotypes. However, the correlation between male reproductive ability and production traits remains poorly understood. Here, we investigated the genetic relationships among and between male fertility characteristics and economically relevant traits in a population of Brown Swiss cattle. We performed GWAS with imputed genotypes at nearly 12 million sequence variants for semen quality (sperm head and tail anomalies, motility, concentration, and volume), male fertility, and 57 production phenotypes. Allele substitution effects were then correlated on a trait-by-trait basis to estimate genetic correlations. Correlations between male reproductive characteristics and traits of economic value were small and ranged from -0.0681 to 0.0787. Among the semen quality parameters, sperm motility was negatively correlated with anomalies (head: r = -0.7083 ± 0.0002; tail: r = -0.7739 ± 0.0002) and volume (r = -0.1266 ± 0.0003), whereas volume was negatively correlated with concentration (r = -0.3503 ± 0.0002). Sire nonreturn rate was negatively correlated with sperm anomalies (head: r = -0.1640 ± 0.0002; tail: r = -0.1580 ± 0.0002) and positively correlated with motility (r = 0.1598 ± 0.0002). A meta-analysis of male reproductive traits identified 2 quantitative trait loci: a previously described region on chromosome 6 showed pleiotropic effects and a novel region on chromosome 11 was associated with sperm head anomalies. In conclusion, our results suggest that selection for economically important dairy and production phenotypes has little impact on semen quality and fertility of Brown Swiss bulls.