Nonadditive genetic effects in animal behavior.

Heritabilities, commonly used to predict evolutionary potential, are notoriously low for behaviors. Apart from strong contributions of environmental variance in reducing heritabilities, the additive genetic components can be very low, especially when they are camouflaged by nonadditive genetic effects. We first report the heritabilities of courtship traits in founder-flush and control populations of the housefly (Musca domestica L.). We estimated the heritability of each male and female display through the regression of the courtships involving daughters and sons (with randomly selected mates) onto the "midparental" courtship values of their parents. Overall, the average heritability was significantly (P = .012) higher for the parent-daughter assays than for the parent-son assays. We attributed the low (even negative) heritabilities to genotype-by-environment interactions whereby the male's behavior is influenced by the "environment" of his mating partner's preferences for the display, generating epistasis through indirect genetic effects. Moreover, bottlenecked lines had up to 800% of the heritability of the controls, suggesting "conversion" of additive genetic variance from nonadditive components. Second, we used line-cross assays on separate populations that had been selected for divergence in mating behavior to identify dominance and epistasis through heterosis and outbreeding depression in courtship. Finally, our literature review confirms the prevalence of such low heritabilities (i.e., a conservative mean of 0.38) and nonadditive genetics in other behavioral repertoires (64% of the studies). We conclude that animal behavior is especially prone to the gamut of quantitative genetic complexities that can result in negative heritabilities, negative selection responses, inbreeding depression, conversion, heterosis, and outbreeding depression.