There is emerging evidence that the mammary epithelium in both mice and humans is arranged as a hierarchy that spans from stem cells to differentiated hormone-sensing, milk-producing and myoepithelial cells. It is well established that estrogen is an important mediator of mammary gland morphogenesis and exposure to this hormone is associated with increased breast cancer risk. Yet surprisingly, the primitive cells of the mammary epithelium do not express the estrogen receptor-α (ERα) or the progesterone receptor. This article will review the mammary epithelial cell hierarchy, possible cells of origin of different types of breast tumors, and the potential mechanisms on how estrogen and progesterone may influence the different subcomponents in normal development and in cancer. Also presented are some hypothetical scenarios on how this underlying biology may be reflected in the behavior of ERα(+) and ERα(-) breast tumors.