Papillary thyroid carcinoma (PTC) is the most prevalent form of thyroid cancer, with a disease recurrence rate of around 20%. Lymphoid formations which occur in non-lymphoid tissues during chronic inflammatory, infectious, immune responses have been linked with tumor suppression. Lymphoid aggregates potentially enhance the body's anti-tumor response, offering an avenue for attracting tumor-infiltrating lymphocytes (TILs) and fostering their coordination. Increasing evidence highlights the role of lymphoid aggregate density in managing tumor invasion and metastasis, with a favorable impact noted on overall and disease-free survival (DFS) across various cancer types. In this work we present a machine vision model to predict recurrence in different histologic subtypes of PTC using measurements related to peritumoral lymphoid aggregate density. We demonstrate that quantifying the peritumoral lymphocytic presence is not only associated with associated with better prognosis, but along with TILs within the tumor, adds additional prognostic value in the absence of well-known second mutations including TERT. Annotations of peritumoral lymphoid aggregates on 171 well-differentiated PTCs in the TCGA-THCA dataset were used to train a deep-learning model to predict regions of lymphoid aggregates across the entire tissue. The fractional area of the tissue regions covered by these lymphocytes was dichotomized to determine two risk groups: significant and low density of peritumoral lymphocytes. DFS prognosticated using these risk groups via Kaplan-Meier (KM) analysis revealed a hazard ratio (HR)=2.51 (95% confidence interval (CI): 2.36, 2.66), tested on 170 new patients also from the TCGA-THCA dataset. The prognostic performance of peritumoral lymphocyte aggregate density were compared against univariate KM analysis of DFS using the fractional area of intratumoral lymphocytes within the primary tumor with HR = 2.04 (95% CI: 1.89, 2.19). Combining the lymphocyte features in and around the tumor yielded a statistically significantly improvement in prognostic performance (HR = 3.17 (95% CI: 3.02, 3.32)) on training and were independently evaluated against 62 patients outside TCGA-THCA with HR = 2.44 (95% CI: 2.19, 2.69). Multivariable Cox-regression analysis on the validation set revealed that the density of peritumoral and intratumoral lymphocytes were prognostic independent of histological subtype with a C-index = 0.815.