Congenital diaphragmatic hernia (CDH) is a congenital defect associated with significant mortality and morbidity. We sought to assess the health-related quality of life (HRQoL) for caregivers of infants/children with CDH and determine risk factors for poorer HRQoL. Families were recruited from a CDH-specific outpatient clinic and HRQoL was assessed by a validated HRQoL instrument (PedsQLTM Family Impact Module) at several time points. Mixed models were used to identify demographic and clinical factors associated with worse HRQoL for caregivers. A total of 29 subjects were recruited at a mean age of 2.4 ± 2.3 years. In terms of defect size, 6.9% had a Type A, 37.9% a Type B, 31.0% a Type C, and 24.1% a Type D. The mean HRQoL score at the first encounter was 67.6 ± 18.3; scores are reported from 0 to 100 with higher scores representing the higher reported quality of life. Lower median household incomes (p = .021) and use of extracorporeal membrane oxygenation (p = .013) were associated with poorer HRQoL scores. The presence of respiratory symptoms decreased HRQoL for caregivers, including daytime symptoms (p < .001) and nighttime symptoms (p < .001). While emergency department visits were not associated with a decrease in HRQoL, hospital admissions (p = .002), and reoperations for CDH (p < .001) were. Our study found a reduced quality of life associated with socioeconomic factors and severity of ongoing disease. Further study is needed to confirm these findings and identify strategies for aiding families cope with the chronicity of this congenital disease.