Chromosomal inversions are ubiquitous in genomes and often coordinate complex phenotypes, such as the covariation of behavior and morphology in many birds, fishes, insects or mammals. However, why and how inversions become associated with polymorphic traits remains obscure. Here we show that despite a strong selective advantage when they form, inversions accumulate recessive deleterious mutations that generate frequency-dependent selection and promote their maintenance at intermediate frequency. Combining genomics and in vivo fitness analyses in a model butterfly for wing-pattern polymorphism, Heliconius numata, we reveal that three ecologically advantageous inversions have built up a heavy mutational load from the sequential accumulation of deleterious mutations and transposable elements. Inversions associate with sharply reduced viability when homozygous, which prevents them from replacing ancestral chromosome arrangements. Our results suggest that other complex polymorphisms, rather than representing adaptations to competing ecological optima, could evolve because chromosomal rearrangements are intrinsically prone to carrying recessive harmful mutations.