We investigated sex differences in the association between a measure of physical health, cardiorespiratory fitness (CRF), and brain function using resting-state functional connectivity fMRI. We examined these sex differences in the default, frontoparietal control, and cingulo-opercular networks, assemblies of functionally connected brain regions known to be impacted by both age and fitness level. Healthy older adults ( n = 49; 29 women) were scanned to obtain measures of intrinsic connectivity within and across these 3 networks. We calculated global efficiency (a measure of network integration) and local efficiency (a measure of network specialization) using graph theoretical methods. Across all three networks combined, local efficiency was positively associated with CRF, and this was more robust in male versus female older adults. Furthermore, global efficiency was negatively associated with CRF, but only in males. Our findings suggest that in older adults, associations between brain network integrity and physical health are sex-dependent. These results underscore the importance of considering sex differences when examining associations between fitness and brain function in older adulthood. NEW & NOTEWORTHY We examined the association between cardiorespiratory fitness and resting state functional connectivity in several brain networks known to be impacted by age and fitness level. We found significant associations between fitness and measures of network integration and network specialization, but in a sex-dependent manner, highlighting the interplay between sex differences, fitness, and aging brain health. Our findings underscore the importance of considering sex differences when examining associations between fitness and brain function in older adulthood.