Adenocarcinoma of the esophagogastric junction (AEGJ) with a specific pathological profile and poor prognosis has limited therapeutic options. Previous studies have found that TILs exhibit distinct characteristics in different tumors and are correlated with tumor prognosis. We established cellular training sets to obtain auto-quantified TILs in pathological images. And we compared the characteristics of TILs in AEGJ with those in esophageal squamous cell carcinoma (ESCC) and gastric adenocarcinoma (GAC) to gain insight into the unique immune environments of these three tumors and investigate the prognostic value of TILs in these three tumors. Utilizing a case-control study design, we analyzed 214 AEGJ, 256 GAC, and 752 ESCC cases. The TCGA dataset was used to validate prognostic value of auto-quantified TILs. The specific cellular training sets were established by experienced pathologists to determine TILs counts. Kruskal-Wallis test and multi-variable linear regression were conducted to explore TILs characteristics. Survival analyses were performed with Kaplan-Meier method and Cox proportional hazards model. The three cellular training sets of these cancers achieved a classification accuracy of 87.55 at least. The median auto-quantified TILs of AEGJ, GAC, and ESCC cases were 4.82%, 1.92%, and 0.12%, respectively. The TILs demonstrated varied characteristics under distinctive clinicopathological traits. The higher TILs proportion was associated with better prognosis in esophagogastric cancers (all P < 0.05) and was an independent prognostic biomarker on AEGJ in both datasets (Taixing: HR = 0.965, 95% CI = 0.938-0.994; TCGA: HR = 0.811, 95% CI = 0.712-0.925). We found variations in TILs across ESCC, GAC, and AEGJ, as assessed by image processing algorithms. Additionally, TILs in these three cancers are an independent prognostic factor. This enhances our understanding of the unique immune characteristics of the three tumors, improving patient outcomes.