Papillary thyroid carcinoma (PTC) is the most prevalent form of thyroid cancer, with a disease recurrence rate of around 20%. Lymphoid formations, which occur in nonlymphoid tissues during chronic inflammatory, infectious, and immune responses, have been linked with tumor suppression. Lymphoid aggregates potentially enhance the body's antitumor response, offering an avenue for attracting tumor-infiltrating lymphocytes and fostering their coordination. Increasing evidence highlights the role of lymphoid aggregate density in managing tumor invasion and metastasis, with a favorable impact noted on overall and disease-free survival (DFS) across various cancer types. In this study, we present a machine vision model to predict recurrence in different histologic subtypes of PTC using measurements related to peritumoral lymphoid aggregate density. We demonstrated that quantifying peritumoral lymphocytic presence not only is associated with better prognosis but also, along with tumor-infiltrating lymphocytes within the tumor, adds additional prognostic value in the absence of well-known second mutations including TERT. Annotations of peritumoral lymphoid aggregates on 171 well-differentiated PTCs in the Cancer Genome Atlas Thyroid Carcinoma (TCGA-THCA) data set were used to train a deep-learning model to predict regions of lymphoid aggregates across the entire tissue. The fractional area of the tissue regions covered by these lymphocytes was dichotomized to determine the following 2 risk groups: a significant and low density of peritumoral lymphocytes. DFS prognosticated using these risk groups via the Kaplan-Meier analysis revealed a hazard ratio (HR) of 2.51 (95% CI: 2.36, 2.66), tested on 170 new patients also from the TCGA-THCA data set. The prognostic performance of peritumoral lymphocyte aggregate density was compared against the univariate Kaplan-Meier analysis of DFS using the fractional area of intratumoral lymphocytes within the primary tumor with an HR of 2.04 (95% CI: 1.89, 2.19). Combining the lymphocyte features in and around the tumor yielded a statistically significant improvement in prognostic performance (HR, 3.17 [95% CI: 3.02, 3.32]) on training and were independently evaluated against 62 patients outside TCGA-THCA with an HR of 2.44 (95% CI: 2.19, 2.69). Multivariable Cox regression analysis on the validation set revealed that the density of peritumoral and intratumoral lymphocytes was prognostic independent of histologic subtype with a concordance index of 0.815.