Mating system as a barrier to gene flow.

Understanding mating system as one of reproductive isolating barriers remains important although this barrier is classified in a different sense from behavioral, ecological, and mechanical isolating barriers. Selfing enhances incipient speciation while outcrossing facilitates species integrity. Here, I study how mating system affects gene exchanges between genetically diverging species in a hybrid zone. Results show that a predominant selfing species has a greater barrier to selective gene flow than does a predominant outcrossing species. Barrier to neutral gene flow convexly changes with the selfing rate due to linkage disequilibrium, with a maximum at around intermediate selfing rate. Asymmetric transient or steady-state barriers to neutral gene flow occur between two sides of a hybrid zone when the neutral gene is affected by its linked selective gene whose alternative alleles are adaptive to heterogeneous habitats. Selfing interacts with both a physical barrier and a density-dependent ecological regulation (a logarithmic model) to strengthen the barriers to neutral and selective gene flow. This theory helps to interpret incipient speciation driven by selfing or to explain the asymmetric gene flow or unequal genomic mixtures between closely related species caused by their asymmetric mating systems in natural hybrid zones.