Stromal components of the tumour microenvironment, such as cancer-associated fibroblasts (CAFs) and the extracellular matrix (ECM), are actively involved in tumorigenesis. CAFs and the ECM co-evolve with resultant molecular and mechanical pressure on tumour cells mediated by CAFs via the ECM. Meanwhile, ECM fibers determine CAF differentiation and activity, establishing a protumorigenic feed-forward loop. Oesophageal cancer carries a high morbidity and mortality, and curative surgical resection is only an option for a limited number of patients while early lymphatic spread and poor therapeutic responses are common. Although studies report marked heterogeneity in investigation of the stromal density of gastrointestinal cancers, it is generally accepted that oesophageal cancer is highly fibrotic, and stromal components like CAFs may outnumber cancer cells. Therefore, a comprehensive understanding of the reciprocal interaction between CAFs and the ECM in oesophageal cancer is essential to improving diagnostics and prognostication, as well as designing innovative anti-cancer strategies. Here, we summarise current understanding of oesophageal cancer from a stromal perspective. Then, we discuss that CAFs and the ECM in oesophageal cancer can independently and synergistically contribute to tumour progression and therapeutic resistance. We also summarise potential stromal targets that have been described in transcriptomic analyses, highlighting those validated in downstream experimental studies. Importantly, clinical translation of stromal-targeting strategies in oesophageal cancer is still in its infancy but holds significant promise for future therapeutic combinations.