In the present study, we investigated the association of genetic predisposition with specific dimensions of dementia pathophysiology for global and domain-specific cognitive decline in older adults. The sample was drawn from the Hellenic Longitudinal Investigation of Aging and Diet (HELIAD) study, comprising 512 cognitively normal individuals over 64 years of age, with a mean follow-up of 2.9 years. Cognitive function was evaluated through a neuropsychological test battery, while genetic predisposition was assessed based on two distinct Polygenic Risk Scores (PRS) for amyloid-beta 42 (Aβ) and white matter hyperintensities (WMH). The association of each PRS with the cognitive decline rate was examined using generalized estimating equation models. In the whole sample, higher PRSs Aβ (β = -0.042) and WMH (β =-0.029) were associated with a higher rate of global cognitive decline per year, an association which remained significant in age, sex, and education subgroups. Moreover, higher PRSs Aβ and WMH were related to significant memory decline only in females, older, and highly educated participants. Thus, while the association of both PRSs with global cognitive decline over time was independent of age, sex, or education, the relationship of the specific PRSs with the memory decline rate appeared to vary depending on these factors.