The tumor microenvironment is proposed to contribute substantially to the progression of cancers, including breast cancer. Cancer-associated fibroblasts (CAFs) are the most abundant components of the tumor microenvironment. Studies have revealed that CAFs in breast cancer originate from several types of cells and promote breast cancer malignancy by secreting factors, generating exosomes, releasing nutrients, reshaping the extracellular matrix, and suppressing the function of immune cells. CAFs are also becoming therapeutic targets for breast cancer due to their specific distribution in tumors and their unique biomarkers. Agents interrupting the effect of CAFs on surrounding cells have been developed and applied in clinical trials. Here, we reviewed studies examining the heterogeneity of CAFs in breast cancer and expression patterns of CAF markers in different subtypes of breast cancer. We hope that summarizing CAF-related studies from a historical perspective will help to accelerate the development of CAF-targeted therapeutic strategies for breast cancer.