Reduced sperm motility and/or count are among the major causes of reduced fertility in men, and sperm membranes play an important role in the spermatogenesis and fertilization processes. However, the impact of sperm lipid composition on male fertility remains under-investigated. The aim of the present study was to perform a lipidomic analysis of human sperm membranes: we performed an untargeted analysis of membrane lipid composition in fertile (N = 33) and infertile subjects (N = 29). In parallel, we evaluated their serum lipid levels. Twenty-one lipids were identified by their mass/charge ratio and post-source decay spectra. Sulfogalactosylglycerolipid (SGG, seminolipid) was the most abundant lipid component in the membranes. In addition, we observed a significant proportion of PUFAs. Important differences have emerged between the fertile and infertile groups, leading to the identification of a lipid cluster that was associated with semen parameters. Among these, cholesterol sulfate, SGG, and PUFAs represented the most important predictors of semen quality. No association was found between the serum and sperm lipids. Dietary PUFAs and SGG have acknowledged antioxidant functions and could, therefore, represent sensitive markers of sperm quality and testicular function. Altogether, these results underline the important role of sperm membrane lipids, which act independently of serum lipids levels and may rather represent an independent marker of reproductive function.