Pathological lymph node metastasis (LNM) following multimodal therapy is an important indicator of poor prognosis in patients with esophageal cancer. However, a significant number of patients without LNM are still at high risk for recurrence. We assessed prognostic factors in 143 patients without pathological LNM who were diagnosed with locally advanced esophageal squamous cell carcinoma (ESCC) and underwent neoadjuvant chemotherapy (NAC) or chemoradiotherapy (NACRT), followed by surgery. Using univariate and multivariate analyses of recurrence-free survival, carcinoembryonic antigen (CEA) levels (hazard ratio [HR]: 2.17, 95% confidence interval [CI]: 1.12-4.23, and p=0.02) and neutrophil-to-lymphocyte ratio (NLR) (HR: 1.22, 95% CI: 1.04-1.43, and p=0.02) were significant independent covariates. Furthermore, pretherapeutic LNM (HR: 1.94, 95% CI: 1.003-3.76, and p=0.049), NACRT (HR: 3.29, 95% CI: 1.30-8.33, and p=0.01), poorly differentiated tumors (HR: 2.52, 95% CI: 1.28-4.98, and p=0.01), and lymphovascular invasion (LVI) (HR: 2.78, 95% CI: 1.27-6.09, and p=0.01) were also significant independent covariates. The recurrence rates among patients with 0/1, 2, 3, and 4/5 poor prognostic factors were significantly different (5.0%, 25.0%, 35.7%, and 53.8%, respectively; p=0.001); the survival rates were stratified among these prognostic groups. Pretherapeutic CEA and NLR levels, pretherapeutic LNM, NACRT, poorly differentiated tumors, and LVI were significantly correlated with survivals in patients without pathological LNM after neoadjuvant therapy and surgery. Postoperative therapy should be considered in patients with ESCC with several indicators of recurrence, even in those without pathological LNM who underwent surgery following neoadjuvant therapy.