Does GST underestimate genetic differentiation from marker data?

The widely applied genetic differentiation statistics F(ST) and G(ST) have recently been criticized for underestimating differentiation when applied to highly polymorphic markers such as microsatellites. New statistics claimed to be unaffected by marker polymorphisms have been proposed and advocated to replace the traditional F(ST) and G(ST). This study shows that G(ST) gives accurate estimates and underestimates of differentiation when demographic factors are more and less important than mutations, respectively. In the former case, all markers, regardless of diversity (H(S)), have the same G(ST) value in expectation and thus give replicated estimates of differentiation. In the latter case, markers of higher H(S) have lower G(ST) values, resulting in a negative, roughly linear correlation between G(ST) and H(S) across loci. I propose that the correlation coefficient between G(ST) and H(S) across loci, r(GH), can be used to distinguish the two cases and to detect mutational effects on G(ST). A highly negative and significant r(GH), when coupled with highly variable G(ST) values among loci, would reveal that marker G(ST) values are affected substantially by mutations and marker diversity, underestimate population differentiation, and are not comparable among studies, species and markers. Simulated and empirical data sets are used to check the power and statistical behaviour, and to demonstrate the usefulness of the correlation analysis.