Studies have shown differences in specific cognitive ability domains and risk of Alzheimer's disease between the men and women at later age. However it is important to know that sex differences in cognitive function during adulthood may have their basis in both organizational effects, i.e., occurring as early as during the neuronal development period, as well as in activational effects, where the influence of the sex steroids influence brain function in adulthood. Further, the rate of cognitive decline with aging is also different between the sexes. Understanding the biology of sex differences in cognitive function will not only provide insight into Alzheimer's disease prevention, but also is integral to the development of personalized, gender-specific medicine. This review draws on epidemiological, translational, clinical, and basic science studies to assess the impact of sex differences in cognitive function from young to old, and examines the effects of sex hormone treatments on Alzheimer's disease in men and women.