Sex hormones are key determinants of gender-related differences and regulate growth and development during puberty. They also exert a broad range modulation of immune cell functions, and a dichotomy exists in the immune response between the sexes. Both clinical and animal models have demonstrated that androgens, estrogens, and progestogens mediate many of the gender-specific differences in immune responses, from the susceptibility to infectious diseases to the prevalence of autoimmune disorders. Androgens and progestogens mainly promote immunosuppressive or immunomodulatory effects, whereas estrogens enhance humoral immunity both in men and in women. This study summarizes the available evidence regarding the physiological effects of sex hormones on human immune cell function and the underlying biological mechanisms, focusing on gender differences triggered by different amounts of androgens between males and females.