Data on 2,045 non-demented individuals with memory complaints were drawn from the Memento cohort study to examine the association between Apolipoprotein E ε4 allele (APOE4) and regional brain gray matter volumes. Linear regression was used to examine the association of APOE4 and measures of regional gray matter volumes in cross-sectional analysis and change therein using longitudinal analyses based on two brain MRI performed at baseline and at two-year follow-up. Overall, in analyses adjusted for age, sex, and intracranial volume, the presence of APOE4 was associated with lower total gray matter volume at baseline and with a higher atrophy rate over the follow-up. The hippocampus and entorhinal cortex were the two gray matter regions most associated with APOE4. Further adjustment for cardiovascular risk factors had little impact on these associations. There was an interaction between age, APOE4 status and total brain volume atrophy rate, with evidence of an earlier age at onset of atrophy in hippocampal volume in APOE4 carriers compared to non-carriers. Those results are in accordance with the role of medial temporal structures in the greater risk of dementia observed in people carrying the APOE4 allele.