Large sample approximations of probabilities of correct evolutionary tree estimation and biases of maximum likelihood estimation.

Simulation studies have been the main way in which properties of maximum likelihood estimation of evolutionary trees from aligned sequence data have been studied. Because trees are unusual parameters and because fitting is computationally intensive, such studies have a heavy computational cost. We develop an asymptotic framework that can be used to obtain probabilities of correct topological reconstruction and study other properties of likelihood methods when a single split is poorly resolved. Simulations suggest that while approximations to log likelihood differences are better for less well-resolved topologies, approximations to probabilities of correct reconstruction are generally good. We used the approximations to investigate biases in estimation and found that maximum likelihood estimation has a long-branch-repels bias. This differs from the long-branch-attracts bias often reported in the literature because it is a different form of bias. For maximum likelihood estimation, usually long-branch-attracts bias results arise in the presence of model misspecification and are a form of statistical inconsistency where the estimated tree converges upon an incorrect tree with long edges together. Here, by bias we mean a tendency to favour a particular topology when data are generated from a four-taxon star tree. While we find a tendency to favour the tree with long branches apart, with more extreme long edges, a strong small sequence-length long-branch-attracts bias overwhelms the long-branch-repels bias. The long-branch-repels bias generalizes to five and six taxa in the sense that subtrees containing taxa that are all distant from the poorly resolved split repel each other.