In chronic infection, inflammation and cancer, the tissue microenvironment controls how local immune cells behave, with tissue-resident fibroblasts emerging as a key cell type in regulating activation or suppression of an immune response. Fibroblasts are heterogeneous cells, encompassing functionally distinct populations, the phenotypes of which differ according to their tissue of origin and type of inciting disease. Their immunological properties are also diverse, ranging from the maintenance of a potent inflammatory environment in chronic inflammation to promoting immunosuppression in malignancy, and encapsulating and incarcerating infectious agents within tissues. In this Review, we compare the mechanisms by which fibroblasts control local immune responses, as well as the factors regulating their inflammatory and suppressive profiles, in different tissues and pathological settings. This cross-disease perspective highlights the importance of tissue context in determining fibroblast-immune cell interactions, as well as potential therapeutic avenues to exploit this knowledge for the benefit of patients with chronic infection, inflammation and cancer.