Partial genotyping at polymorphic markers can improve heritability estimates in sibling groups.

Accurate estimates of heritability (h2) are necessary to assess adaptive responses of populations and evolution of fitness-related traits in changing environments. For plants, h2 estimates generally rely on maternal progeny designs, assuming that offspring are either half-sibs or unrelated. However, plant mating systems often depart from half-sib assumptions, this can bias h2 estimates. Here, we investigate how to accurately estimate h2 in nonmodel species through the analysis of sibling designs with a moderate genotyping effort. We performed simulations to investigate how microsatellite marker information available for only a subset of offspring can improve h2 estimates based on maternal progeny designs in the presence of nonrandom mating, inbreeding in the parental population or maternal effects. We compared the basic family method, considering or not adjustments based on average relatedness coefficients, and methods based on the animal model. The animal model was used with average relatedness information, or with hybrid relatedness information: associating one-generation pedigree and family assumptions, or associating one-generation pedigree and average relatedness coefficients. Our results highlighted that methods using marker-based relatedness coefficients performed as well as pedigree-based methods in the presence of nonrandom mating (i.e. unequal male reproductive contributions, selfing), offering promising prospects to investigate in situ heritabilities in natural populations. In the presence of maternal effects, only the use of pairwise relatednesses through pedigree information improved the accuracy of h2 estimates. In that case, the amount of father-related offspring in the sibling design is the most critical. Overall, we showed that the method using both one-generation pedigree and average relatedness coefficients was the most robust to various ecological scenarios.