Why the DNA-containing organelles, chloroplasts, and mitochondria, are inherited maternally is a long standing and unsolved question. However, recent years have seen a paradigm shift, in that the absoluteness of uniparental inheritance is increasingly questioned. Here, we review the field and propose a unifying model for organelle inheritance. We argue that the predominance of the maternal mode is a result of higher mutational load in the paternal gamete. Uniparental inheritance evolved from relaxed organelle inheritance patterns because it avoids the spread of selfish cytoplasmic elements. However, on evolutionary timescales, uniparentally inherited organelles are susceptible to mutational meltdown (Muller's ratchet). To prevent this, fall-back to relaxed inheritance patterns occurs, allowing low levels of sexual organelle recombination. Since sexual organelle recombination is insufficient to mitigate the effects of selfish cytoplasmic elements, various mechanisms for uniparental inheritance then evolve again independently. Organelle inheritance must therefore be seen as an evolutionary unstable trait, with a strong general bias to the uniparental, maternal, mode.