Lateral node metastasis confers a poor prognosis in rectal cancer. Several multidisciplinary treatments have been proposed with favorable outcomes. However, appropriate neoadjuvant/adjuvant treatments or follow-up plans based on information about the probable recurrence site have not been specified. We aimed to clarify the distinctive features of recurrence patterns for lateral node-positive low rectal cancer according to the lateral and mesorectal lymph node status. We retrospectively analyzed 508 patients with stage III low rectal cancer who underwent lateral node dissection. We investigated the impact of lateral and mesorectal lymph node status on site-specific recurrence rates and patient survival. Analyses for relapse-free survival revealed the prognostic impact of lateral node positivity in stage III low rectal cancer (p < 0.0001). Lateral node-positive patients exhibited higher risk of overall recurrence, local recurrence, and recurrence in extra-regional nodes than lateral node-negative patients (p < 0.0001, p = 0.001, and p < 0.0001, respectively). However, lateral node positivity was not statistically associated with a hematogenous recurrence rate. In lateral node-positive patients, both tumor-node-metastasis (TNM)-N status and number of lateral nodes involved were revealed as significant prognostic factors (p < 0.0001, both). In addition, the number of lateral nodes involved could be a discriminatory indicator of probabilities of local recurrence and recurrence in extra-regional nodes (p = 0.02, and p < 0.0001, respectively). Lateral node-positive low rectal cancer exhibits higher local recurrence and extra-regional node recurrence rates that correlate with the number of lateral nodes involved.