Epidemiological studies show a reciprocal inverse association between cancer and Alzheimer's disease (AD). The common mechanistic theory for this effect posits that cells have an innate tendency toward apoptotic or survival pathways, translating to increased risk for either neurodegeneration or cancer. However, it has been shown that cancer patients experience cognitive dysfunction pre- and post-treatment as well as alterations in cerebral gray matter density (GMD) on MRI. To further investigate these issues, we analyzed the association between cancer history (CA±) and age of AD onset, and the relationship between GMD and CA± status across diagnostic groups in the Alzheimer's Disease Neuroimaging Initiative (ADNI) cohort study. Data was analyzed from 1609 participants with information on baseline cancer history and AD diagnosis, age of AD onset, and baseline MRI scans. Participants were CA+ (N = 503) and CA- (N = 1106) diagnosed with AD, mild cognitive impairment (MCI), significant memory concerns (SMC), and cognitively normal older adults. As in previous studies, CA+ was inversely associated with AD at baseline (P = 0.025); interestingly, this effect appears to be driven by non-melanoma skin cancer (NMSC), the largest cancer category in this study (P = 0.001). CA+ was also associated with later age of AD onset (P < 0.001), independent of apolipoprotein E (APOE) ε4 allele status, and individuals with two prior cancers had later mean age of AD onset than those with one or no prior cancer (P < 0.001), suggesting an additive effect. Voxel-based morphometric analysis of GMD showed CA+ had lower GMD in the right superior frontal gyrus compared to CA- across diagnostic groups (P crit < 0.001, uncorrected); this cluster of lower GMD appeared to be driven by history of invasive cancer types, rather than skin cancer. Thus, while cancer history is associated with a measurable delay in AD onset independent of APOE ε4, the underlying mechanism does not appear to be cancer-related preservation of GMD.