Anxiety has been associated with a greater risk of Alzheimer's disease (AD). Existing research has identified structural differences in regional brain tissue in participants with anxiety, but results have been inconsistent. We sought to determine the association between anxiety and regional brain volumes, and the moderation effect of APOE ε4. Using data from participants in the National Alzheimer's Coordinating Center (NACC) Uniform Data Set, with complete imaging (MRI) and biomarker data ( = 1533), multiple linear regression estimated the adjusted effect of anxiety on 30 structural MRI regions. The moderation effect of APOE ε4 on the relation between structural MRI regions and anxiety was assessed as was the moderation effect of cognitive status. False discovery rate was used to adjust for multiple comparisons. After controlling for intracranial volume, age, sex, years of education, race, Hispanic ethnicity, and cognitive status, seven MRI regions demonstrated lower volumes among participants with anxiety: total cerebrum gray matter volume, right hippocampus volume, hippocampal volume (total), right and left frontal lobe cortical gray matter volume, and right and total temporal lobe cortical gray matter volume. Findings suggest that anxiety is associated with significant atrophy in multiple brain regions, with corresponding ventricular enlargement. Future research should investigate if anxiety-related changes to brain morphology contribute to greater AD risk.