The complex interactions between tumors and their microenvironment remain to be elucidated. Combining large-scale approaches, we examined the spatio-temporal dynamics of 28 different immune cell types (immunome) infiltrating tumors. We found that the immune infiltrate composition changed at each tumor stage and that particular cells had a major impact on survival. Densities of T follicular helper (Tfh) cells and innate cells increased, whereas most T cell densities decreased along with tumor progression. The number of B cells, which are key players in the core immune network and are associated with prolonged survival, increased at a late stage and showed a dual effect on recurrence and tumor progression. The immune control relevance was demonstrated in three endoscopic orthotopic colon-cancer mouse models. Genomic instability of the chemokine CXCL13 was a mechanism associated with Tfh and B cell infiltration. CXCL13 and IL21 were pivotal factors for the Tfh/B cell axis correlating with survival. This integrative study reveals the immune landscape in human colorectal cancer and the major hallmarks of the microenvironment associated with tumor progression and recurrence.