CHARGE syndrome (CS) is a rare congenital syndrome characterized by coloboma, heart anomaly, choanal atresia, retardation of growth and development, and genital and ear anomalies. While several neuroimaging studies have revealed abnormalities such as hypoplasia of the semicircular canal, olfactory nerve, cerebellum, and brainstem, no quantitative analysis of brain morphology in CS has been reported. We quantitatively investigated brain morphology in CS participants using structural magnetic resonance imaging (MRI) (N = 10, mean age 14.7 years old) and high-angular resolution diffusion MRI (HARDI) tractography (N = 8, mean age 19.4 years old) comparing with gender- and age-matched controls. Voxel-based analyses revealed decreased volume of the bilateral globus pallidus (left and right; p = 0.021 and 0.029), bilateral putamen (p = 0.016 and 0.011), left subthalamic nucleus (p = 0.012), bilateral cerebellum (p = 1.5 × 10−6 and 1.2 × 10−6), and brainstem (p = 0.031), and the enlargement of the lateral ventricles (p = 0.011 and 0.0031) bilaterally in CS. Surface-based analysis revealed asymmetrically increased cortical thickness in the right hemisphere (p = 0.013). The group-wise differences observed in global cortical volume, gyrification index, and left cortical thickness were not statistically significant. HARDI tractography revealed reduced volume, elongation, and higher ADC values in multiple fiber tracts in patients in CS compared to the controls, but FA values were not statistically significantly different between the two groups. Facial features are known to be asymmetric in CS, which has been recognized as an important symptom in CS. Our results revealed that the cortex in CS has an asymmetric appearance similar to the facial features. In addition, the signal pattern of high ADC with statistically unchanged FA values of tractography pathways indicated the presence of other pathogenesis than vasogenic edema or myelination dysfunction in developmental delay in CS.