Competition for mates can be a major source of selection, not just on secondary sexual traits but across the genome. Mate competition strengthens selection on males via sexual selection, which typically favors healthy, vigorous individuals and, thus, all genetic variants that increase overall quality. However, recent studies suggest another major effect of mate competition that could influence genome-wide selection: Sexual harassment by males can drastically weaken selection on quality in females. Because of these conflicting effects, the net effect of mate competition is uncertain, although perhaps not entirely unpredictable. We propose that the environment in which mate competition occurs mediates the importance of sexual selection relative to sexual conflict and, hence, the net effect of mate competition on nonsexual fitness. To test this, we performed experimental evolution with 63 fruit fly populations adapting to novel larval conditions where each population was maintained with or without mate competition. In half the populations with mate competition, adults interacted in simple, high-density environments. In the remainder, adults interacted in more spatially complex environments in which male-induced harm is reduced. Populations evolving with mate competition in the complex environment adapted faster to novel larval environments than did populations evolving without mate competition or with mate competition in the simple environment. Moreover, mate competition in the complex environment caused a substantial reduction in inbreeding depression for egg-to-adult viability relative to the other two mating treatments. These results demonstrate that the mating environment has a substantial and predictable effect on nonsexual fitness through adaptation and purging.