Patients with neurodegenerative diseases exhibit diminished basal forebrain (BF) volume compared to healthy individuals. However, it's uncertain whether this difference is consistent between sexes. It has been reported that BF volume moderately atrophies during aging, but the effect of sex on BF volume changes during the normal aging process remains unclear. In the cross-sectional study, we observed a significant reduction in BF volume in patients with mild cognitive impairment (MCI) and Alzheimer's disease (AD) compared to Healthy Controls (HCs), especially in the Ch4 subregion. Notably, significant differences in BF volume between MCI and HCs were observed solely in the female group. Additionally, we identified asymmetrical atrophy in the left and right Ch4 subregions in female patients with AD. In the longitudinal analysis, we found that aging seemed to have a minimal impact on BF volume in males. Our study highlights the importance of considering sex as a research variable in brain science.