Background: The effects of the COVID-19 pandemic on mental health have been understudied among vulnerable and marginalized populations, particularly in fragile and conflict-affected settings. We analyzed changes in mental health and parenting stress among caregivers, many of whom are Internally Displaced Persons (IDPs), in a conflict-affected setting in Colombia.Method: We used longitudinal data from a psychosocial support program in which 1,374 caregivers with children aged 2-5 were randomly assigned to treatment and control groups across four sequential cohorts. At end-line, two cohorts were assessed prior to the pandemic, while two cohorts were assessed two to five weeks into the national lockdown. We estimated the impact of COVID-19 by comparing end-line outcomes between pre and post-pandemic cohorts using lagged-dependent variable models.Findings: Results showed that the likelihood of reporting symptoms above the risk threshold increased by 14 percentage points for anxiety (95% CI 0·100 – 0·174), five percentage points for depression (0·005 – 0·091), and ten percentage points for parental stress (0·046 – 0·146). Impact was stronger for Internally Displaced Persons (IDPs), and participants with lower educationor with pre-existing mental health conditions. Impact was also higher for those reporting food insecurity, job loss, or illness of a relative.Interpretation: Mental health significantly worsened due to COVID-19. Considering the vulnerability and preexisting mental health conditions of this population, the estimated effects are substantial. Policies in fragile and conflict-affected settings targeting IDPs and other vulnerable people will be important to mitigate further mental health and socioeconomic problems.Trial Registration: The trial was registered at the Clinical Trials Registry #NCT03502252 and at the Social Science Registry #2868, while the amendment for the phone survey was registered in the latter registry on April 20, 2020.Funding Statement: Saving Brains – Grand Challenges Canada; Fundación Éxito; Fundación FEMSA; United Way Colombia.Declaration of Interests: Dr. Moya reports grants from Saving Brains - Grand Challenges Canada, Fundación Exito, Fundación Femsa, and United Way Colombia, which supported implementation and impact evaluation of the Semillas de Apego program in Tumaco, Colombia. All other authors have nothing to declare. Ethics Approval Statement: Study procedures were approved by the Universidad de los Andes’s Ethics Committee (Protocol 786, 2017).