The incidence of well-differentiated non-functioning pancreatic neuroendocrine tumors (NF-PanNET) increased during the last decades. The risk of relapse after curative surgery, albeit low, is not negligible; moreover, adjuvant treatment is currently not an option and a reliable predictive model based on prognostic characteristics is urgently needed for tailoring a follow-up strategy. The histological classification of PanNET now relies only on the proliferative activity (mitosis and Ki67) and staging. In contrast to other endocrine neoplasms, the role of infiltrative growth pattern in NF-PanNET is not taken into consideration at present. In the current study, 247 consecutive patients who underwent surgical resection for a NF-PanNET were examined for the histological growth pattern of the tumor. Two distinct patterns (non-infiltrative vs. infiltrative) were described with the latter being further subclassified according to the type of structures invaded by the tumor (non-infiltrative: pattern 1; infiltration of adjacent pancreatic parenchyma and/or peripancreatic soft tissue: pattern 2; invasion of nearby organs and/or major vessels: pattern 3). The infiltrative growth resulted to be strongly associated with a poorer survival compared to a non-infiltrative growth (p < 0.001). In particular, the distinction between pancreatic parenchyma and/or peripancreatic soft tissue invasion versus adjacent organs and/or major vessels invasion was the most powerful predictor of recurrence after surgery at multivariate analysis (pattern 2 vs. pattern 1: HR 10.136, p = 0.028; pattern 3 vs. pattern 1: HR 15.775, p = 0.015). The infiltrative growth pattern could therefore provide additional prognostic information implementing the current grading and staging system.