Assessing population allele frequency differences using low-depth sequencing data.

Sequencing-based methods are increasingly being used for genetic studies. Genotypes derived from the sequence data are sometimes assigned incorrectly, due to having no reads for one of the alleles. This is particularly the case when the data has low sequencing depth. Here, we develop a correction that can be applied to both a measure and a test for the population differentiation metric when considering populations as a fixed effect, to account for this incomplete genotyping. It is shown that the measure of differentiation is not influenced much by this correction in reasonably sized studies but that significance testing is too liberal without the correction. This correction will allow appropriate inference in population studies.