Immunotherapy, particularly immune checkpoint inhibitors, have become widely used in various settings across many different cancer types in recent years. Whilst patients are often treated on the basis of the primary cancer type and clinical stage, recent studies have highlighted disparity in response to immune checkpoint inhibitors at different sites of metastasis, and their impact on overall response and survival. Studies exploring the tumor immune microenvironment at different organ sites have provided insights into the immune-related mechanisms behind organ-specific patterns of response to immunotherapy. In this review, we aimed to highlight the key learnings from clinical studies across various cancers including melanoma, lung cancer, renal cell carcinoma, colorectal cancer, breast cancer and others, assessing the association of site of metastasis and response to immune checkpoint inhibitors. We also summarize the key clinical and pre-clinical findings from studies exploring the immune microenvironment of specific sites of metastasis. Ultimately, further characterization of the tumor immune microenvironment at different metastatic sites, and understanding the biological drivers of these differences, may identify organ-specific mechanisms of resistance, which will lead to more personalized treatment approaches for patients with innate or acquired resistance to immunotherapy.