Colorectal cancer (CRC) is a leading global cancer with high mortality, especially in metastatic cases, with limited therapeutic options. The tumor microenvironment (TME), a network comprising various immune cells, stromal cells and extracellular (ECM) components plays a crucial role in influencing tumor progression and therapy outcome. The genetic heterogeneity of CRC and the complex TME complicates the development of effective, personalized treatment strategies. The prognosis has slowly improved during the past decades, but metastatic CRC (mCRC) is common among patients and is still associated with low survival. The therapeutic options for CRC differ from those for mCRC and include surgery (mostly for CRC), chemotherapy, growth factor receptor signaling pathway targeting, as well as immunotherapy. Malignant CRC cells are established in the TME, which varies depending on the primary or metastatic site. Herein, we review the role and interactions of several ECM components in 3D models of CRC and mCRC tumor cells, with an emphasis on how the TME affects tumor growth and treatment. This comprehensive summary provides support for the development of 3D models that mimic the interactions within the TME, which will be essential for the development of novel anticancer therapies.