During pregnancy, the maternal body undergoes a considerable transformation regarding the anatomy, metabolism, and immune profile that, after delivery, allows for protection and nourishment of the offspring via lactation. Pregnancy hormones are responsible for the development and functionality of the mammary gland for breast milk production, but little is known about how hormones control its immune properties. Breast milk composition is highly dynamic, adapting to the nutritional and immunological needs that the infant requires in the first months of life and is responsible for the main immune modeling of breastfed newborns. Therefore, alterations in the mechanisms that control the endocrinology of mammary gland adaptation for lactation could disturb the properties of breast milk that prepare the neonatal immune system to respond to the first immunologic challenges. In modern life, humans are chronically exposed to endocrine disruptors (EDs), which alter the endocrine physiology of mammals, affecting the composition of breast milk and hence the neonatal immune response. In this review, we provide a landscape of the possible role of hormones in the control of passive immunity transferred by breast milk and the possible effect of maternal exposure to EDs on lactation, as well as their impacts on the development of neonatal immunity.