Cancer-associated fibroblasts (CAFs) are generally associated with poor clinical outcome. CAFs support tumor growth in a variety of ways and can suppress antitumor immunity and response to immunotherapy. However, a precise understanding of CAF contributions to tumor growth and therapeutic response is lacking. Discrepancies in this field of study may stem from heterogeneity in the composition and function of fibroblasts in the tumor microenvironment. Furthermore, it remains unclear whether CAFs directly interact with and suppress T cells. Here, mouse and human breast tumors were used to examine stromal cells expressing fibroblast activation protein (FAP), a surface marker for CAFs. Two discrete populations of FAP mesenchymal cells were identified on the basis of podoplanin (PDPN) expression: a FAPPDPN population of CAFs and a FAPPDPN population of cancer-associated pericytes (CAPs). Although both subsets expressed extracellular matrix molecules, the CAF transcriptome was enriched in genes associated with TGFβ signaling and fibrosis compared with CAPs. In addition, CAFs were enriched at the outer edge of the tumor, in close contact with T cells, whereas CAPs were localized around vessels. Finally, FAPPDPN CAFs suppressed the proliferation of T cells in a nitric oxide-dependent manner, whereas FAPPDPN pericytes were not immunosuppressive. Collectively, these findings demonstrate that breast tumors contain multiple populations of FAP-expressing stromal cells of dichotomous function, phenotype, and location.