Although previous studies have reported the sex differences in behavior/cognition and the brain, the sex difference in the relationship between memory abilities and the underlying neural basis in the aging process remains unclear. In this study, we used a machine learning model to estimate the association between cortical thickness and verbal/visuospatial memory in females and males and then explored the sex difference of these associations based on a community-elderly cohort (n = 1153, age ranged from 50.42 to 86.67 years). We validated that females outperformed males in verbal memory, while males outperformed females in visuospatial memory. The key regions related to verbal memory in females include the medial temporal cortex, orbitofrontal cortex, and some regions around the insula. Further, those regions are more located in limbic, dorsal attention, and default-model networks, and are associated with face recognition and perception. The key regions related to visuospatial memory include the lateral prefrontal cortex, anterior cingulate gyrus, and some occipital regions. They overlapped more with dorsal attention, frontoparietal and visual networks, and were associated with object recognition. These findings imply the memory performance advantage of females and males might be related to the different memory processing tendencies and their associated network.