The processes of cancer initiation, progression, and response to therapy are affected by the sex of cancer patients. Immunotherapy responses largely depend on the tumor microenvironment (TME), but how sex may shape some TME features, remains unknown. Here, we analyzed immune infiltration signatures across 19 cancer types from 1771 male and 1137 female patients in The Cancer Genome Atlas to evaluate how sex may affect the tumor mutational burden (TMB), immune scores, stromal scores, tumor purity, immune cells, immune checkpoint genes, and functional pathways in the TME. Pan-cancer analyses showed higher TMB and tumor purity scores, as well as lower immune and stromal scores in male patients as compared to female patients. Lung adenocarcinoma, lung squamous carcinoma, kidney papillary carcinoma, and head and neck squamous carcinoma showed the most significant sex biases in terms of infiltrating immune cells, immune checkpoint gene expression, and functional pathways. We further focused on lung adenocarcinoma samples in order to identify and validate sex-specific immune cell biomarkers with prognostic potential. Overall, sex may affect the tumor microenvironment, and sex-specific TME biomarkers may help tailor cancer immunotherapy in certain cancer types.