Binge drinking is a deadly pattern of excessive alcohol use that is associated with multiple diseases in the United States. To date, little is known about the associations between the early onset of substance use and other factors with the severity of adult binge drinking. The 2018 National Survey on Drug Use and Health data was used to identify binge drinking (binary and in number of days in the past month). Age at onset was categorized into four groups as 1–12, 13–14, 15–17, or beyond 18. Weighted multivariate logistic regression and Poisson regression analyses were performed to examine the associations between early onset of alcohol, smokeless tobacco, and marijuana use with binge drinking. The severity of binge drinking was statistically significantly associated with substance use (4.15 days in a month), early onset of alcohol, smokeless tobacco, and marijuana use (2.15–4.93 days, all p-values < 0.0001), after accounting for the covariates. Past year substance use disorder is strongly associated with binge drinking. The severity of adult binge drinking is significantly associated with early onset of substance use including alcohol, smokeless tobacco, and marijuana. Continued efforts are warranted to improve substance use prevention and treatment tailored for adolescents and youths to prevent development of adult binge drinking.