Colorectal carcinoma (CRC) is a malignant tumor of the digestive system. Cancer-associated fibroblasts (CAFs) are important cellular elements in the tumor microenvironment of CRC, which contribute to CRC progression and immune escape. To predict the survival outcome and therapeutic responses of CRC patients, we identified genes connected with stromal CAF and generated a risk model. In this study, we used multiple algorithms to reveal CAF-related genes in the Gene Expression Omnibus and The Cancer Genome Atlas datasets and construct a risk model composed by prognostic CAF-associated genes. Then, we evaluated whether the risk score could predict CAF infiltrations and immunotherapy in CRC and confirmed the expression of the risk model in CAFs. Our results showed that CRC patients with high CAF infiltrations and stromal score had worse prognosis than those with low-CAF infiltrations and stromal score. We obtained 88 stromal CAF-associated hub-genes and generated a CAF risk model consisting of ZNF532 and COLEC12. Compared with low-risk group, the overall survival in high-risk group was shorter. The relationship between risk score, ZNF532 and COLEC12, and stromal CAF infiltrations and CAF markers was positive. In addition, the effect of immunotherapy in the high-risk group was not as good as that in the low-risk group. Patients with the high-risk group were enriched in chemokine signaling pathway, cytokine-cytokine receptor interaction, and focal adhesion. Finally, we confirmed that the expressions of ZNF532 and COLEC12 in risk model were widely distributed in fibroblasts of CRC, and the expression levels were higher in fibroblasts than CRC cells. In conclusion, the prognostic CAF signature of ZNF532 and COLEC12 can be applied not only to predict the prognosis of CRC patients but also to evaluate the immunotherapy response in CRC patients, and these findings provide the possibility for further development of individualized treatment for CRC.