Selection varies between categories of individuals, with far-reaching ramifications: Sex-specific selection can impede or accelerate adaptation, and differences in selection between young and old individuals are ultimately responsible for senescence. Here, we measure early- and late-life fitness in adults of both sexes from the Drosophila genetic reference panel and perform quantitative genetic and transcriptomic analyses. Fitness was heritable, showed positive pleiotropy across sexes and age classes, and appeared to be influenced by very large numbers of loci with small effects plus a smaller number with moderate effects. Most loci affected male and female fitness in the same direction; relatively few candidate sexually antagonistic loci were found, though these were enriched on the X chromosome as predicted by theory. The expression level of many genes showed an opposite correlation with fitness in males and females, consistent with unresolved sexual conflict over transcription. The load of deleterious mutations correlated negatively with fitness across genotypes, and we found some evidence for the mutation accumulation (but not the antagonistic pleiotropy) theory of aging.