The tumor microenvironment (TME) considerably influences colorectal cancer (CRC) progression, therapeutic response and clinical outcome, but studies of interindividual heterogeneities of the TME in CRC are lacking. Here, by integrating human colorectal single-cell transcriptomic data from approximately 200 donors, we comprehensively characterized transcriptional remodeling in the TME compared to noncancer tissues and identified a rare tumor-specific subset of endothelial cells with T cell recruitment potential. The large sample size enabled us to stratify patients based on their TME heterogeneity, revealing divergent TME subtypes in which cancer cells exploit different immune evasion mechanisms. Additionally, by associating single-cell transcriptional profiling with risk genes identified by genome-wide association studies, we determined that stromal cells are major effector cell types in CRC genetic susceptibility. In summary, our results provide valuable insights into CRC pathogenesis and might help with the development of personalized immune therapies.