Midlife vascular disease increases risk for dementia and effects of vascular dysfunction on brain health differ between men and women. Elevated pulse pressure, a surrogate for arterial stiffness, contributes to cerebrovascular pathology and white matter damage that may advance cognitive aging; however, it remains unclear how associations between pulse pressure and neural integrity differ by sex and age. This study used restriction spectrum imaging to examine associations between pulse pressure and brain microstructure in community-dwelling women (N=88) and men (N=55), aged 56 to 97 (mean, 76.3) years. Restricted isotropic (presumed intracellular), hindered isotropic (presumed extracellular), neurite density, and free water diffusion were computed in white matter tracts and subcortical regions. After adjustment for age and sex, higher pulse pressure correlated with lower restricted isotropic diffusion in global white matter, with more pronounced associations in parahippocampal cingulum, as well as in thalamus and hippocampus. Subgroup analyses demonstrated stronger correlations between pulse pressure and restricted isotropic diffusion in association fibers for participants ≤75 years than for older participants, with stronger effects for women than men of this age group. Microstructure in parahippocampal cingulum and thalamus differed by pulse pressure level regardless of antihypertensive treatment. Increased pulse pressure may lead to widespread injury to white matter and subcortical structures, with greatest vulnerability for women in late middle to early older age. Restriction spectrum imaging could be useful for monitoring microstructural changes indicative of neuronal loss or shrinkage, demyelination, or inflammation that accompany age-related cerebrovascular dysfunction.