The tumor microenvironment (TME) has been identified as one of the driving factors of tumor progression and invasion. Within this microenvironment, cancer-associated fibroblasts (CAF) have multiple tumor-promoting functions and play key roles in drug resistance, through multiple mechanisms, including extracellular matrix (ECM) remodeling, production of growth factors, cytokines, and chemokines, and modulation of metabolism and angiogenesis. More recently, a growing body of evidence has shown that CAF also modulate immune cell activity and suppress anti-tumor immune response. In this review, we describe the current knowledge on CAF heterogeneity in terms of identity and functions. Moreover, we analyze how distinct CAF subpopulations differentially interact with immune cells, with a particular focus on T lymphocytes. We address how specific CAF subsets contribute to cancer progression through induction of an immunosuppressive microenvironment. Finally, we highlight potential therapeutic strategies for targeting CAF subpopulations in cancer.