As one of the most common cancers of the digestive system, colon cancer is a predominant cause of cancer-related deaths worldwide. To investigate prognostic genes in the tumor microenvironment of colon cancer, we collected 461 colon adenocarcinoma (COAD) and 172 rectal adenocarcinoma (READ) samples from The Cancer Genome Atlas (TCGA) database, and calculated the stromal and immune scores of each sample. We demonstrated that stromal and immune scores were significantly associated with colon cancer stages. By analyzing differentially expressed genes (DEGs) between two stromal and immune score groups, we identified 952 common DEGs. The significantly enriched Gene Ontology (GO) and Kyoto Encyclopedia of Genes and Genomes (KEGG) terms for these DEGs were associated with T-cell activation, immune receptor activity, and cytokine-cytokine receptor interaction. Through univariate Cox regression analysis, we identified 22 prognostic genes. Furthermore, nine key prognostic genes, namely, , , 2, , , , , , and , were identified using the LASSO Cox regression analysis. The risk score of each sample was calculated using the gene expression of the nine genes. Patients with high-risk scores had a poorer prognosis than those with low-risk scores. The prognostic model established with the nine-gene signature was able to effectively predict the outcome of colon cancer patients. Our findings may help in the clinical decisions and improve the prognosis for colon cancer.