Pancreatic tumors are known to harbor an abundant and highly desmoplastic stroma. Among the various cell types that reside within tumor stroma, cancer-associated fibroblasts (CAFs) have gained a lot of attention in the cancer field due to their contributions to carcinogenesis and tumor architecture. These cells are not a homogeneous population, but have been shown to have different origins, phenotypes, and contributions. In pancreatic tumors, CAFs generally emerge through the activation and/or recruitment of various cell types, most notably resident fibroblasts, pancreatic stellate cells (PSCs), and tumor-infiltrating mesenchymal stem cells (MSCs). In recent years, single cell transcriptomic studies allowed the identification of distinct CAF populations in pancreatic tumors. Nonetheless, the exact sources and functions of those different CAF phenotypes remain to be fully understood. Considering the importance of stromal cells in pancreatic cancer, many novel approaches have aimed at targeting the stroma but current stroma-targeting therapies have yielded subpar results, which may be attributed to heterogeneity in the fibroblast population. Thus, fully understanding the roles of different subsets of CAFs within the stroma, and the cellular dynamics at play that contribute to heterogeneity in CAF subsets may be essential for the design of novel therapies and improving clinical outcomes. Fortunately, recent advances in technologies such as microfluidics and bio-printing have made it possible to establish more advanced models that will likely prove useful. In this review, we will present the different roles of stromal cells in pancreatic cancer, focusing on CAF origin as a source of heterogeneity, and the role this may play in therapy failure. We will discuss preclinical models that could be of benefit to the field and that may contribute to further clinical development.