The tumor microenvironment (TME) is emerging as one of the primary barriers in cancer therapy. Cancer-associated fibroblasts (CAF) are a common inhabitant of the TME in several tumor types and play a critical role in tumor progression and drug resistance via different mechanisms such as desmoplasia, angiogenesis, immune modulation, and cancer metabolism. Due to their abundance and significance in pro-tumorigenic mechanisms, CAF are gaining attention as a diagnostic target as well as to improve the efficacy of cancer therapy by their modulation. In this review, we highlight existing imaging techniques that are used for the visualization of CAF and CAF-induced fibrosis and provide an overview of compounds that are known to modulate CAF activity. Subsequently, we also discuss CAF-targeted and CAF-modulating nanocarriers. Finally, our review addresses ongoing challenges and provides a glimpse into the prospects that can spearhead the transition of CAF-targeted therapies from opportunity to reality.