Prostate cancer (PC) is an age-related disease and represents, after lung cancer, the second cause of cancer death in males worldwide. Mortality is due to the metastatic disease, which mainly involves the bones, lungs, and liver. In the last 20 years, the incidence of metastatic PC has increased in Western Countries, and a further increase is expected in the near future, due to the population ageing. Current treatment options, including state of the art cancer immunotherapy, need to be more effective to achieve long-term disease control. The most significant anatomical barrier to overcome to improve the effectiveness of current and newly designed drug strategies consists of the prostatic stroma, in particular the fibroblasts and the extracellular matrix, which are the most abundant components of both the normal and tumor prostatic microenvironment. By weaving a complex communication network with the glandular epithelium, the immune cells, the microbiota, the endothelium, and the nerves, in the healthy prostatic microenvironment, the fibroblasts and the extracellular matrix support organ development and homeostasis. However, during inflammation, ageing and prostate tumorigenesis, they undergo dramatic phenotypic and genotypic changes, which impact on tumor growth and progression and on the development of therapy resistance. Here, we focus on the characteristics and functions of the prostate associated fibroblasts and of the extracellular matrix in health and cancer. We emphasize their roles in shaping tumor behavior and the feasibility of manipulating and/or targeting these stromal components to overcome the limitations of current treatments and to improve precision medicine's chances of success.