Indoleamine 2,3-dioxygenase (IDO1) plays an important role in tumor immune evasion. In this study, we investigated the changes of tumor IDO1 expression and CD8+ tumor-infiltrating lymphocytes (TILs) status in tumor microenvironment (TME) after neoadjuvant chemoradiotherapy (NCRT) or neoadjuvant chemotherapy (NCT) in esophageal squamous cell carcinoma (ESCC), respectively. Moreover, the potential predictive value of the changes of tumor IDO1 expression and CD8+TILs status on pathologic response and clinical outcome was further evaluated. By matching propensity scores in 295 patients, a total of 85 ESCC patients with neoadjuvant therapy followed by surgery were recruited, including 17 patients with NCRT and 68 patients with NCT. Tumor IDO1 expression and CD8+TILs within TME in paired specimens were evaluated by immunohistochemistry, and the changes of tumor IDO1 expression and CD8+TILs between the paired specimens were estimated. Tumor IDO1 expression significantly increased from baseline to postoperative tumor tissue after NCT (p = 0.002), whereas no significant difference was detected after NCRT (p = 0.44). The density of CD8+TILs in the tumor-invasive margin increased significantly after neoadjuvant therapy, and there was no significant difference in density changes of CD8+TILs between the NCRT and NCT groups (p = 0.118). Upregulation of tumor IDO1 expression after neoadjuvant therapy was associated with poor pathologic response (p = 0.002). Lastly, multivariate Cox analysis showed that IDO1-rise patients after neoadjuvant therapy were related to poor prognosis (p = 0.047). These results indicated that chemotherapy could promote tumor IDO1 expression, and the increased tumor IDO1 expression after neoadjuvant therapy predicted poor pathologic response and prognosis in ESCC.