Background: Early-stage tongue cancer has a good prognosis in general; however, high-risk patients with late cervical lymph node and distant metastases have a poor prognosis. Elective neck dissection and postoperative chemoradiotherapy are considered for these patients, although no clear criteria have been identified for their evaluation. Methods: This retrospective observational study aimed to determine the predictive factors for late cervical lymph node and distant metastases in 102 patients with cT1-2N0 tongue cancer. The data regarding the demographic characteristics, as well as the depth of invasion, tumor budding, histological grade, and tumor-stromal ratio, among other things, were extracted from medical records. Results: We found that the potential lymph node metastasis rate was 27.5%. The significant clinical predictors of late cervical lymph node metastasis were the tumor thickness and endophytic growth pattern and the significant histopathological factors were poorly and moderately differentiated tumors and ≥3 tumor buds. In addition, the prognostic factors for distant metastasis included ≥4 lymph node metastases, ≥7 tumor budding, and moderate and poor tumor differentiation. Conclusions: The usefulness of tumor budding as a predictor of metastasis for tongue cancer was suggested. The findings of this study can help establish the criteria for evaluating the metastasis risk and prognosis of patients with tongue cancers.