We investigate yeast sex chromosome evolution by comparing genome sequences from 16 species in the family Saccharomycetaceae, including data from genera Tetrapisispora, Kazachstania, Naumovozyma, and Torulaspora. We show that although most yeast species contain a mating-type (MAT) locus and silent HML and HMR loci structurally analogous to those of Saccharomyces cerevisiae, their detailed organization is highly variable and indicates that the MAT locus is a deletion hotspot. Over evolutionary time, chromosomal genes located immediately beside MAT have continually been deleted, truncated, or transposed to other places in the genome in a process that is gradually shortening the distance between MAT and HML. Each time a gene beside MAT is removed by deletion or transposition, the next gene on the chromosome is brought into proximity with MAT and is in turn put at risk for removal. This process has also continually replaced the triplicated sequence regions, called Z and X, that allow HML and HMR to be used as templates for DNA repair at MAT during mating-type switching. We propose that the deletion and transposition events are caused by evolutionary accidents during mating-type switching, combined with natural selection to keep MAT and HML on the same chromosome. The rate of deletion accelerated greatly after whole-genome duplication, probably because genes were redundant and could be deleted without requiring transposition. We suggest that, despite its mutational cost, switching confers an evolutionary benefit by providing a way for an isolated germinating spore to reform spores if the environment is too poor.