Structural brain abnormalities associated with congenital nystagmus (CN) are still unknown. In some patients with CN additional sensory, metabolic, or gross structural alterations can be detected. In the present study voxel-based morphometry was used to compare the gray matter (GM) brain volumes of 14 individuals with CN without associated sensory, metabolic, or obvious structural alterations (i.e., idiopathic CN) to those of a group of controls. Further, GM brain volumes were correlated with nystagmus severity as measured by sway path. Intergroup comparison exhibited significant volume increases in the human motion sensitive complex V5/MT+, the fusiform gyrus, and the middle occipital gyrus bilaterally in CN. These volume increases may be associated with excess visual motion stimulation due to involuntary retinal slip of the visual scene. A positive correlation (linear model) of nystagmus sway path with cerebellar GM volume was seen in the following areas: vermal parts VIII-X as well as hemisphere lobule II, hemisphere VI, crus I, crus II, and lobule VII-IX bilaterally. There is evidence that the reported GM volume changes in the vestibulo-cerebellum, which correlated with nystagmus sway path, might be related to the subjects‘ attempt to maintain fixation, rather than be due to the generation of nystagmus.