Functional neuroimaging studies have consistently reported age-related changes in prefrontal cortex (PFC) activity during a variety of cognitive tasks, including episodic memory. These changes are often interpreted within the context of one of the following three neural models of age-related changes in brain function: dedifferentiation, neural inefficiency, and neural plasticity and compensation models. Distinguishing between these competing models has proven difficult when interpreting results using functional imaging data alone. In this paper we suggest that a more accurate interpretation of age-related changes in PFC activity requires consideration of age-related differences in gray matter volume (GMv) in PFC and the medial temporal lobes (MTL). We review fMRI studies of cognitive aging that have directly examined the relationship between PFC activity and both local (PFC) and distal (MTL) GMv in older versus younger adults. We also considered how structure-function relationships may be further modified in pathological aging (i.e. mild cognitive impairment (MCI) and Alzheimer's disease (AD)). We found that when task performance was matched between age-groups there was a negative association between regional PFC volume and activity in older adults. However, when older adults performed worse than young adults we observed a positive association between volume and activity in right lateral PFC. Additionally during memory tasks, several studies revealed that PFC activity is positively related to GM volume in MTL in healthy older adults, but negatively related in MCI and AD patients. We conclude that PFC activity is related to age-related changes in local and distal GM volume reductions and that consideration of these structural measures aids the interpretation of fMRI results. Furthermore, the study of structure-function relationships may provide important insights into the biological mechanisms underlying healthy versus pathological aging.