Recombination is critical both for accelerating adaptation and purging deleterious mutations. Chromosomal inversions can act as recombination modifiers that suppress local recombination in heterozygotes and thus, under some conditions, are predicted to accumulate such mutations. In this study, we investigated patterns of recombination, transposable element abundance, and coding sequence evolution across the genomes of 1,445 individuals from three sunflower species, as well as within nine inversions segregating within species. We also analyzed the effects of inversion genotypes on 87 phenotypic traits to test for overdominance. We found significant negative correlations of long terminal repeat retrotransposon abundance and deleterious mutations with recombination rates across the genome in all three species. However, we failed to detect an increase in these features in the inversions, except for a modest increase in the proportion of stop codon mutations in several very large or rare inversions. Consistent with this finding, there was little evidence of overdominance of inversions in phenotypes that may relate to fitness. On the other hand, significantly greater load was observed for inversions in populations polymorphic for a given inversion compared to populations monomorphic for one of the arrangements, suggesting that the local state of inversion polymorphism affects deleterious load. These seemingly contradictory results can be explained by the low frequency of inversion heterozygotes in wild sunflower populations, apparently due to divergent selection and associated geographic structure. Inversions contributing to local adaptation represent ideal recombination modifiers, acting to facilitate adaptive divergence with gene flow, while largely escaping the accumulation of deleterious mutations.