Background: To investigate the associations of weight change patterns across adulthood with the risk of non-alcoholic fatty liver disease (NAFLD). Methods: Using data from the National Health and Nutrition Examination Survey (NHANES) 2017–2018 cycle, we performed a retrospective cohort study with 2212 non-obese participants aged 36 years old over. Weight change patterns were categorized as “stable non-obese”, “early adulthood weight gain”, “middle and late adulthood weight gain” and “revert to non-obese” according to the body mass index (BMI) at age 25, 10 years prior and at baseline. Vibration-controlled transient elastography (VCTE) was performed to diagnose NAFLD. Modified Poisson regression was used to quantify the associations of weight change patterns with NAFLD. Results: Compared with participants in the “stable non-obese” group, those who gained weight at early or middle and late adulthood had an increased risk of NAFLD, with an adjusted rate ratio (RR) of 2.19 (95% CI 1.64–2.91) and 1.92 (95% CI 1.40–2.62), respectively. The risk of NAFLD in “revert to the non-obese” group showed no significant difference with the stable non-obese group. If the association of weight change and NAFLD was causal, we estimated that 73.09% (95% CI 55.62–82.93%) of incident NAFLD would be prevented if the total population had a normal BMI across adulthood. Conclusions: Weight gain to obese at early or middle and late adulthood was associated with an evaluated risk of NAFLD. A large proportion would have been prevented with effective weight intervention.