Significant sex disparities have been observed in cancer incidence, treatment response to immunotherapy, and susceptibility to adverse effects, affecting both reproductive and non-reproductive organ cancers. While lifestyle factors, carcinogenic exposure, and healthcare access contribute to these disparities, they do not fully explain the observed male-female variation in anti-tumor immunity. Despite the preferential expression of sex hormone receptors in immune cells, X chromosome also contains numerous genes involved in immune function, and its incomplete inactivation may enhance anti-tumor immune responses in females. In contrast, loss or downregulation of Y-linked genes in males has been associated with an increased cancer risk. Additionally, estrogen, progesterone and androgen signaling pathways influence both innate and adaptive immune responses, contributing to sex-specific outcomes in cancer progression and therapy. Sex-biased differences are also evident in the epigenetic regulation of gene expression, cellular senescence, microbiota composition, metabolism, and DNA damage response, all of which impact anti-tumor immunity and immunotherapy treatment efficacy. In general, the combination of sex chromosomes, sex hormones, and hormone receptors orchestrates the phenotype and function of various immune cells involved in tumor immunity. However, sex disparity in each specific immune cell are context and environment dependent, considering the preferential expression of hormone receptor in immune cell and sex hormone levels fluctuate significantly across different life stages. This review aims to outline the molecular, cellular, and epigenetic changes in T cells, B cells, NK cells, DCs, neutrophils, and macrophages driven by sex chromosomes and sex hormone signaling. These insights may inform the design of sex-specific targeted therapies and leading to more individualized cancer treatment strategies.