The intrauterine environment is an important determinant of immunity later in life of the offspring. An altered prenatal immune development can result in a high postnatal risk for infections, chronic immune diseases, and autoimmunity. Many of these immune diseases show a strong sex bias, such as a high incidence of autoimmune diseases and allergies in adult females or a high risk for infections in males. Here, we comprehensively review established pathways and propose novel concepts modulating the risk for such poor immunity during childhood and throughout life. Moreover, we highlight how an adverse fetal environment may affect or aggravate the risk for poor immunity in a sex-specific manner. An improved understanding of a sex-specific susceptibility to poor immunity along with insights on how such risk can be modulated before or around birth will allow the development of tailored prevention strategies.