Pathological sections hold rich diagnostic information, yet their prognostic potential is underutilized. This study leverages deep learning to predict outcomes, advancing precision oncology of pathological sections with focus on pancreatic cancer. We analyzed H&E-stained whole section images of 125 cases from public databases as well as 28 real-world patients with pancreatic cancer and precancerous lesions. After image preprocessing, we identified and selected representative patches for subsequent analysis. We develop a modified visual transformer (ViT) model with spatial attention and fine-tuned on ImageNet2012, which was subsequently used to predict the survival times of the corresponding patients and to calculate risk scores. The modified ViT model demonstrated strong predictive accuracy for patient prognosis, with C-indices of 0.79 and 0.82 for Overall Survival (OS) and Disease Free Survival (DFS) in the test set and 0.62 in the validation set. Risk scores correlated well with patient survival, showing clustering between 0.17 and 0.95, aligning with a median survival of 24 months. Higher risk scores were associated with worse clinical prognosis, including shorter survival times and increased tumor recurrence risk, validated across all datasets. The model's AUCs for OS and DFS prediction reached 0.847/0.849 in the training set and 0.813/0.834 in the test set, confirming its high accuracy and potential for clinical application in risk stratification and prognosis prediction. ViT network can discover the prognostic value of pancreatic cancer pathology sections via interpretable risk scores, providing a new insight for prognosis evaluation as well as opens new technology building on existing clinical diagnostics.