The central nervous system (CNS) harbors its own immune system composed of microglia in the parenchyma and CNS-associated macrophages (CAMs) in the perivascular space, leptomeninges, dura mater, and choroid plexus. Recent advances in understanding the CNS resident immune cells gave new insights into development, maturation and function of its immune guard. Microglia and CAMs undergo essential steps of differentiation and maturation triggered by environmental factors as well as intrinsic transcriptional programs throughout embryonic and postnatal development. These shaping steps allow the macrophages to adapt to their specific physiological function as first line of defense of the CNS and its interfaces. During infancy, the CNS might be targeted by a plethora of different pathogens which can cause severe tissue damage with potentially long reaching defects. Therefore, an efficient immune response of infant CNS macrophages is required even at these early stages to clear the infections but may also lead to detrimental consequences for the developing CNS. Here, we highlight the recent knowledge of the infant CNS immune system during embryonic and postnatal infections and the consequences for the developing CNS.