(1) Background: Comparable prognoses after definitive chemoradiation therapy (CRT) to surgery alone for esophageal squamous cell carcinoma (ESCC) have been previously reported; however, no robust prognostic markers have been established. The clonality of tumor-infiltrating lymphocytes (TILs) and tumor microenvironments (TMEs) in ESCC relapsed after CRT were examined to explore prognostic markers. (2) Methods: Clonality of TIL and TME were examined in ESCC with and without preceding CRT, as well as oral squamous cell carcinoma (OSCC) and healthy volunteers as controls. The clonality of TIL was assessed by T-cell receptor (TCR) α and β repertoire analyses and evaluated by diversity indices. The TME was assessed by quantitative polymerase chain reaction evaluating PD-L1 and CD8B. (3) Results: The clonal expansion of TIL was significantly induced within ESCCs and OSCCs, when compared to healthy volunteers, and was mostly induced within ESCCs after definitive CRT. Diversity indices of TIL were not associated with the prognosis, but the ratio of PD-L1 mRNA to CD8B mRNA in TME was significantly associated with a poor prognosis after salvage surgery (p = 0.007). (4) Conclusions: The clonal expansion of TIL is induced after definitive CRT for ESCC, and the ratio of PD-L1 mRNA to CD8B mRNA within tumor tissues is a prognostic marker candidate for salvage esophagectomy after CRT.