Previous studies have shown that white matter in the deaf brain changes due to hearing loss. However, how white-matter development is influenced by early hearing experience of deaf people is still unknown. Using diffusion tensor imaging and tract-based spatial statistics, we compared white-matter structures among three groups of subjects including 60 congenitally deaf individuals, 36 acquired deaf (AD) individuals, and 38 sex- and age-matched hearing controls (HC). The result showed that the deaf individuals had significantly reduced fractional anisotropy (FA) values in bilateral superior temporal cortex and the splenium of corpus callosum compared to HC. The reduction of FA values in acquired deafness correlated with onset age of deafness, but not the duration of deafness. To explore the underlying mechanism of FA changes in the deaf groups, we further analyzed radial and axial diffusivities and found that (1) the reduced FA values in deaf individuals compared to HC is primarily driven by higher radial diffusivity values and (2) in the AD, higher radial diffusivity was correlated with earlier onset age of deafness, but not the duration of deafness. These findings imply that early sensory experience is critical for the growth of fiber myelination, and anatomical reorganization following auditory deprivation is sensitive to early plasticity in the brain.