Clear cell renal cell carcinoma (ccRCC) is a highly immunogenic tumor with variable responses to immune checkpoint therapy. The significance of the immune cell infiltrate in distant metastases, their association with the immune infiltrate in the primary tumors and their impact on prognosis are poorly described. We hypothesized that specific subtypes of immune cells may be involved in the control of metastases and may have an impact on the prognosis of ccRCC. We analyzed the immune microenvironment in ccRCC primary tumors with distant metastases, paired distant metastases and non-metastasized ccRCC (n = 25 each group) by immunohistochemistry. Confirmatory analyses for CD8+ and CD103+ cells were performed in a large ccRCC cohort (n = 241) using a TCGA-KIRC data set (ITGAE/CD103). High immune cell infiltration in primary ccRCC tumors was significantly correlated with the development of distant tumor metastasis (p < 0.05). A high density of CD103+ cells in ccRCC was more frequent in poorly differentiated tumors (p < 0.001). ccRCCs showed high levels of ITGAE/CD103 compared with adjacent non-neoplastic tissue. A higher density of CD103+ cells and a higher ITGAE/CD103 expression were significantly correlated with poor overall survival in ccRCC (log rank p < 0.05). Our results show a major prognostic value of the immune pattern, in particular CD103+ cell infiltration in ccRCC, and highlight the importance of the tumor immune microenvironment.