Early childhood neglect can impact brain development across the lifespan. Using voxel-based approaches we recently reported that severe and time-limited institutional deprivation in early childhood was linked to substantial reductions in total brain volume in adulthood, >20 years later. Here, we extend this analysis to explore deprivation-related regional white matter volume and microstructural organization using diffusion-based techniques. A combination of tensor-based morphometry (TBM) analysis and tractography was conducted on diffusion-weighted imaging (DWI) data from 59 young adults who spent between 3 and 41 months in the severely depriving Romanian institutions of the 1980s before being adopted into United Kingdom families, and 20 nondeprived age-matched United Kingdom controls. Independent of total volume, institutional deprivation was associated with smaller volumes in localized regions across a range of white matter tracts including (1) long-ranging association fibers such as bilateral inferior longitudinal fasciculus (ILF), bilateral inferior fronto-occipital fasciculus (IFOF), left superior longitudinal fasciculi (SLFs), and left arcuate fasciculus; (2) tracts of the limbic circuitry including fornix and cingulum; and (3) projection fibers with the corticospinal tract particularly affected. Tractographic analysis found no evidence of altered microstructural organization of any tract in terms of hindrance modulated orientational anisotropy (HMOA), fractional anisotropy (FA), or mean diffusivity (MD). We provide further evidence for the effects of early neglect on brain development and their persistence in adulthood despite many years of environmental enrichment associated with successful adoption. Localized white matter effects appear limited to volumetric changes with microstructural organization unaffected.