Selection, generalized transmission and the evolution of modifier genes. I. The reduction principle.

Modifier gene models are used to explore the evolution of features of organisms, such as the genetic system, that are not directly involved in the determination of fitness. Recent work has shown that a general "reduction principle" holds in models of selectively neutral modifiers of recombination, mutation, and migration. Here we present a framework for models of modifier genes that shows these reduction results to be part of a more general theory, for which recombination and mutation are special cases. The deterministic forces that affect the genetic composition of a population can be partitioned into two categories: selection and transmission. Selection includes differential viabilities, fertilities, and mating success. Imperfect transmission occurs as a result of such phenomena as recombination, mutation and migration, meiosis, gene conversion, and meiotic drive. Selectively neutral modifier genes affect transmission, and a neutral modifier gene can evolve only by generating association with selected genes whose transmission it affects. We show that, in randomly mating populations at equilibrium, imperfect transmission of selected genes allows a variance in their marginal fitnesses to be maintained. This variance in the marginal fitnesses of selected genes is what drives the evolution of neutral modifier genes. Populations with a variance in marginal fitnesses at equilibrium are always subject to invasion by modifier genes that bring about perfect transmission of the selected genes. It is also found, within certain constraints, that for modifier genes producing what we call "linear variation" in the transmission processes, a new modifier allele can invade a population at equilibrium if it reduces the level of imperfect transmission acting on the selected genes, and will be expelled if it increases the level of imperfect transmission. Moreover, the strength of the induced selection on the modifier gene is shown to range up to the order of the departure of the genetic system from perfect transmission.