The absence of olfactory input causes structural brain remodelling in humans. Mainly, the olfactory bulb and cortical olfactory areas are involved in this process. The aim of our study was to investigate volume changes of the gray and white matter in a group of subjects with an impaired but not complete loss of olfaction (hyposmia). Magnetic resonance images of hyposmic subjects and an age- and sex-matched control group were acquired on a 3 T scanner. Voxel-based morphometry (VBM) was performed using VBM8 toolbox and SPM8 in a Matlab environment. The analysis revealed significant gray matter volume loss in the insular cortex, anterior cingulate cortex, orbitofrontal cortex, cerebellum, fusiform gyrus, precuneus, middle temporal gyrus and piriform cortex. In the VBM white matter analysis areas of volume loss were found underneath the insular cortex, in the cerebellum and middle frontal gyrus. All areas of white matter atrophy were spatially connected to areas of gray matter volume loss except the middle frontal gyrus alterations. No significant gray or white matter volume increases could be observed. The pattern of gray matter alterations was similar to that known from anosmic subjects with a lower extent. To our knowledge, we report here for the first time on white matter volume alterations in patients with olfactory deficit.