INTRODUCTION: The conventional tumor-node-metastasis (TNM) classification system remains limited in accurately forecasting prognosis and guiding adjuvant chemotherapy decisions for patients with colorectal cancer (CRC). To address this gap, we introduced and validated a novel pathomics signature (PS) derived from hematoxylin and eosin-stained whole slide images, leveraging a deep learning framework. METHODS: This retrospective study analyzed 883 slides from two independent cohorts. An interpretable multi-instance learning model was developed to construct PS, with SHapley Additive exPlanations (SHAP) and gradient-weighted class activation mapping (Grad-CAM) for the improvement of model interpretability and the identification of critical histopathological features, respectively. The transcriptomic data was provided by The Cancer Genome Atlas (TCGA) and integrated to investigate the biological mechanisms underpinning PS. RESULTS: The results demonstrated that PS was proven to be an independent prognostic indicator for both overall and disease-free survival. It significantly enhanced the prognostic performance alongside TNM staging, as shown by improvements in net reclassification and integrated discrimination indices. Furthermore, patients in stages II and III with low PS levels were more likely to benefit from chemotherapy. Morphologically, PS reflected features such as tumor infiltration, adipocyte presence, fibrotic stroma, and immune cell engagement. Transcriptome analysis further revealed links between PS and pathways involved in tumor progression and immune evasion. DISCUSSION: Our findings suggested that the application of deep learning to histopathological images could be an efficient method to improve the prognostic accuracy and evaluate the treatment responses in CRC. The PS offers a promising aid for clinical decision-making by shedding light on key pathogenic processes. Nevertheless, further validation through prospective studies remains essential.