Spermatogenesis is precisely controlled by sophisticated gene expression programs and is driven by epigenetic reprogramming, including histone modification alterations and histone-to-protamine transition. Nuclear receptor binding SET domain protein 2 (Nsd2) is the predominant histone methyltransferase catalyzing H3K36me2 and its role in male germ cell development remains elusive. Here, we report that NSD2 protein is abundant in spermatogenic cells. Conditional loss of Nsd2 in postnatal germ cells impaired fertility owing to apoptosis of spermatocytes and aberrant spermiogenesis. Nsd2 deficiency results in dysregulation of thousands of genes and remarkable reduction of both H3K36me2 and H3K36me3 in spermatogenic cells, with H3K36me2 occupancy correlating positively with expression of germline genes. Nsd2 deficiency leads to H4K16ac elevation in spermatogenic cells, probably through interaction between NSD2 and PSMA8, which regulates acetylated histone degradation. We further reveal that Nsd2 deficiency impairs EP300-induced H4K5/8ac, recognized by BRDT to mediate the eviction of histones. Accordingly, histones are largely retained in Nsd2-deficient spermatozoa. In addition, Nsd2 deficiency enhances expression of protamine genes, leading to increased protamine proteins in Nsd2-deficient spermatozoa. Our findings thus reveal a previously unappreciated role of the Nsd2-dependent chromatin remodeling during spermatogenesis and provide clues to the molecular mechanisms in epigenetic abnormalities impacting male reproductive health.