The Colletotrichum gloeosporioides species complex (CGSC) is one of the most devastating fungal phytopathogens, and is composed of three main clades: Kahawae, Musae, and Theobromicola. Despite the diversity of CGSC, there is limited understanding on their evolutionary mechanisms. By analysing 49 newly assembled genomes, we found that the expansion of transposable elements, especially long terminal repeat retrotransposons, facilitates the expansion of genome size and genetic variation. In-depth analyses suggested that an intra-chromosomal inversion may have been the driving force behind the divergence of Kahawae clade from its ancestor. Within the Kahawae clade, the narrow-hosted quarantine species C. kahawae has undergone extensive chromosomal rearrangements mediated by repetitive sequences, generating highly dynamic lineage-specific genomic regions compared to the closely related broad-hosted species C. cigarro. The findings of this study highlight the role of chromosomal rearrangements in promoting genetic diversification and host adaptation, and provide new perspectives for understanding the evolution of phytopathogenic fungi.