Reciprocal sign epistasis and truncation selection: When is recombination favorable in a pre-breeding program with a selfing species?

Since the dawn of agriculture, humans have applied artificial selection on traits of interest, regardless of their genetic architecture. Yet, still today, most models used to study and streamline this process overlook genetic interactions. In this study, we determined the conditions in which a target genotype can be fixed when truncation selection is applied on an epistatic trait. Previous studies have shown that reciprocal sign epistasis with two fitness peaks of unequal height involves multiple equilibrium states, i.e. below one critical parameter value, such as a critical recombination rate, one genotype may be fixed, and above it, another one may be fixed. Using a haploid bi-locus model, we identified which genotype would be fixed, and how quickly, in an infinite population selected for a phenotypic trait subject to reciprocal sign epistasis with unequal peak heights, depending on two criteria: the recombination rate and percentage of selected individuals. The critical parameter values at which bistability sets in, were also calculated. These results were complemented by stochastic simulations in finite populations. Our results confirmed that, in the case of fitness under reciprocal sign epistasis, high recombination rates induce blockage at the local optimum or attainment of an equilibrium state between the two peaks. However, if linkage disequilibrium is negative in the initial population, recombination is necessary to create the most favorable genotype. Therefore, in this case, reciprocal sign epistasis favors non-null recombination rates, particularly if selection is intense.