Pancreatic ductal adenocarcinoma (PDAC) is a highly aggressive primary malignancy of the pancreas, with a dismal prognosis and limited treatment options. It possesses a unique tumor microenvironment (TME), generating dense stroma with complex elements cross-talking with each other to promote tumor growth and progression. Diversified neural components makes for not having a full understanding of their influence on its aggressive behavior. The aim of the study was to summarize and integrate the role of nerves in the pancreatic tumor microenvironment. The role of autonomic nerve fibers on PDAC development has been recently studied, which resulted in considering the targeting of sympathetic and parasympathetic pathways as a novel treatment opportunity. Perineural invasion (PNI) is commonly found in PDAC. As the severity of the PNI correlates with a poorer prognosis, new quantification of this phenomenon, distinguishing between perineural and endoneural invasion, could feature in routine pathological examination. The concepts of cancer-related neurogenesis and axonogenesis in PDAC are understudied; so, further research in this field may be warranted. A better understanding of the interdependence between the neural component and cancer cells in the PDAC microenvironment could bring new nerve-oriented treatment options into clinical practice and improve outcomes in patients with pancreatic cancer. In this review, we aim to summarize and integrate the current state of knowledge and future challenges concerning nerve-cancer interactions in PDAC.