We evaluated sex differences in MRI-based volume loss and differences in predictors of this neurodegeneration in cognitively healthy older adults. Mixed-effects regression was used to compare regional brain volume trajectories of 295 male and 328 female cognitively healthy Baltimore Longitudinal Study of Aging participants, aged 55-92 years, with up to 20 years of follow-up and to assess sex differences in the associations of age, hypertension, obesity, APOE e4 carrier status, and high-density lipoprotein cholesterol with regional brain volume trajectories. For both sexes, older age was associated with steeper volumetric declines in many brain regions, with sex differences in volume loss observed in frontal, temporal, and parietal regions. In males, hypertension and higher high-density lipoprotein cholesterol were protective against volume loss in the hippocampus, entorhinal cortex, and parahippocampal gyrus. In females, hypertension was associated with steeper volumetric decline in gray matter, and obesity was protective against volume loss in temporal gray matter. Predictors of volume change may affect annual rates of volume change differently between men and women.