Deviations of brain age from chronologic age, known as the brain age gap (BAG), have been linked to neurodegenerative diseases such as Alzheimer disease (AD). Here, we compare the associations of MRI-derived (atrophy) or F-FDG PET-derived (brain metabolism) BAG with cognitive performance, neuropathologic burden, and disease progression in cognitively normal individuals (CNs) and individuals with subjective cognitive decline (SCD) or mild cognitive impairment (MCI). Machine learning pipelines were trained to estimate brain age from 185 matched T1-weighted MRI or F-FDG PET scans of CN from the Alzheimer's Disease Neuroimaging Initiative and validated in external test sets from the Open Access of Imaging and German Center for Neurodegenerative Diseases-Longitudinal Cognitive Impairment and Dementia studies. BAG was correlated with measures of cognitive performance and AD neuropathology in CNs, SCD subjects, and MCI subjects. Finally, BAG was compared between cognitively stable and declining individuals and subsequently used to predict disease progression. MRI (mean absolute error, 2.49 y) and F-FDG PET (mean absolute error, 2.60 y) both estimated chronologic age well. At the SCD stage, MRI-based BAG correlated significantly with beta-amyloid (Aβ) in cerebrospinal fluid, whereas F-FDG PET BAG correlated with memory performance. At the MCI stage, both BAGs were associated with memory and executive function performance and cerebrospinal fluid Aβ, but only MRI-derived BAG correlated with phosphorylated-tau/Aβ Lastly, MRI-estimated BAG predicted MCI-to-AD progression better than F-FDG PET-estimated BAG (areas under the curve, 0.73 and 0.60, respectively). Age was reliably estimated from MRI or F-FDG PET. MRI BAG reflected cognitive and pathologic markers of AD in SCD and MCI, whereas F-FDG PET BAG was sensitive mainly to early cognitive impairment, possibly constituting an independent biomarker of brain age-related changes.