Objective: To analyze the factors associated with weaning in infants with cow's milk allergy (CMA) treated at a food allergy reference center in a state in the northeast of Brazil. Method: A prospective cohort study, with a case group (children with CMA) and two control groups (symptomatic nonallergic children [SC] and asymptomatic [AC]). At the beginning of the study, 30 children comprised the CMA group, 84 the SC group, and 52 the AC group. Survival analysis was performed to compare breastfeeding time between the three groups and an adjusted linear regression model to verify the factors associated with breastfeeding time. Results: At the beginning of the study, 33.3% of children in the CMA group, 17.1% in the SC group, and 69.6% in the AC group were exclusively breastfeeding (p = 0.005). The most common factors for weaning in children with CMA were the cow's milk elimination diet (30%), allergic symptoms in the child (20%), and breast engorgement (20%). Children who used a cup as a means of offering infant formula spent 281 more days breastfeeding compared with those who used a baby bottle (Bstd = 1.39; p = 0.031). Conclusion: Children with CMA and nonallergic gastrointestinal complaints weaned earlier compared with asymptomatic children. The main causes of weaning in CMA children were maternal difficulty adhering to the elimination diet, breast engorgement, and allergic symptoms in the child. Using a cup was the main factor associated with longer breastfeeding duration, regardless of gastrointestinal symptoms and socioeconomic factors.