Aging is often associated with cognitive and neural decline, but how these factors interact is still not fully understood. Recently, functional connectivity, or the degree to which brain regions are concurrently active, has provided insight into age-related differences. However, functional connectivities during task and rest differ and few studies have examined how these relate to a broad range of cognitive functions. The present study investigated the effect of age on cognition, whole-brain functional connectivity during resting-state and task, and their relationships across the adult lifespan. Cognition was broadly assessed using a battery of cognitive assessments and mean network characteristics were calculated across the whole brain. Behaviorally, increased age was associated with worse recall, executive function, and verbal working memory abilities but better language performance. Neurally, increased age was associated with lower overall within- and between-network functional connectivities during both rest and task, and these age-connectivity relationships were stronger during task performance. Connectivity was also related to cognition, and for all participants, these relationships were strongest during rest. Specifically, higher resting-state between-network functional connectivity was associated with poorer cognition for all adults and poorer language ability among older adults. Collectively, these findings demonstrate that while age effects were strongest during the task, resting-state functional connectivity was most closely tied to cognition. Moreover, these results are theoretically consistent with dedifferentiation accounts of cognition and aging and show that less differentiated functional connectivities are associated with cognitive costs for both older and younger adults.