BACKGROUND: Despite recent advancements in the diagnosis and prognosis of Esophageal cancer (EC), it remains among the leading causes of cancer-related mortality. Timely and cost-effective diagnosis, particularly in predicting the risk of metastasis and identifying the deregulation of oncogenic signaling pathways, could open new frontiers towards precision medicine and targeted therapy of EC. However, current diagnostic practices in identifying metastasis and deregulated oncogenic pathways involve molecular testing, which is time-consuming and costly. Advances in deep learning analysis of digital pathological imagery data offer promising avenues for automating and enhancing cancer diagnosis and risk stratification. METHODS: High-resolution H&E-stained diagnostic whole slide images were obtained from the open repository of The Cancer Genome Atlas (TCGA). The WSIs underwent several pre-processing steps, including patching, color normalization and augmentation. A deep learning model was designed and trained on WSI data and tissue-level labels to generate image feature representations for predicting metastatic potential and identifying the deregulation of four major oncogenic signaling pathways, viz. mTOR, PTEN, p53, and PI3K/AKT. RESULTS: The proposed model achieved an AUC of 0.92 for predicting metastatic risk and AUCs ranging from 0.64 to 0.92 for the identification of deregulated oncogenic pathways. In a first, we were able to operate the model without the need for exhaustive patch-level annotations, relying instead on slide-level annotations only. CONCLUSION: In this work, we highlighted the transformative potential of deep learning in accurately detecting metastasis and identifying deregulated oncogenic pathways from H&E slides using slide-level annotation, thus opening new doors in precision medicine and targeted therapy.