Lipid compositions vary greatly among organelles, and specific sorting mechanisms are required to establish and maintain these distinct compositions. In this review, we discuss how the biophysical properties of the membrane bilayer and the chemistry of individual lipid molecules play a role in the intracellular trafficking of the lipids themselves, as well as influencing the trafficking of transmembrane proteins. The large diversity of lipid head groups and acyl chains lead to a variety of weak interactions, such as ionic and hydrogen bonding at the lipid/water interfacial region, hydrophobic interactions, and van-der-Waals interactions based on packing density. In simple model bilayers, these weak interactions can lead to large-scale phase separations, but in more complex mixtures, which mimic cell membranes, such phase separations are not observed. Nevertheless, there is growing evidence that domains (i.e., localized regions with non-random lipid compositions) exist in biological membranes, and it is likely that the formation of these domains are based on interactions similar to those that lead to phase separations in model systems. Sorting of lipids appears to be based in part on the inclusion or exclusion of certain types of lipids in vesicles or tubules as they bud from membrane organelles.