In search of the Goldilocks zone for hybrid speciation II: hard times for hybrid speciation?

Hybridization opens a unique window for observing speciation mechanisms and is a potential engine of speciation. One controversially discussed outcome of hybridization is homoploid hybrid speciation by reciprocal sorting, where a hybrid population maintains a mixed combination of the parental genetic incompatibilities, preventing further gene exchange between the newly formed population and the two parental sources. Previous work showed that, for specific linkage architectures (i.e., the genomic location and order of hybrid incompatibilities), reciprocal sorting could reliably result in hybrid speciation. Yet, the sorting of incompatibilities creates a risk of population extinction. To understand how the demographic consequences of the purging of incompatibilities interact with the formation of a hybrid species, we model an isolated hybrid population resulting from a single admixture event. We study how population size, linkage architecture, and the strength of the incompatibility affect survival of the hybrid population, resolution/purging of the genetic incompatibilities and the probability of observing hybrid speciation. We demonstrate that the extinction risk is highest for intermediately strong hybrid incompatibilities. In addition, the linkage architecture displaying the highest hybrid speciation probabilities changes drastically with population size. Overall, this indicates that population dynamics can strongly affect the outcome of hybridization and the hybrid speciation probability.