Although altered large-scale brain network organization in patients with temporal lobe epilepsy (TLE) has been shown using morphologic measurements such as cortical thickness, these studies, have not included critical subcortical structures (such as hippocampus and amygdala) and have had relatively small sample sizes. Here, we investigated differences in topological organization of the brain volumetric networks between patients with right TLE (RTLE) and left TLE (LTLE) with unilateral hippocampal atrophy. We performed a cross-sectional analysis of 86 LTLE patients, 70 RTLE patients, and 116 controls. RTLE and LTLE groups were balanced for gender (p = 0.64), seizure frequency (Mann-Whitney U test, p = 0.94), age (p = 0.39), age of seizure onset (p = 0.21), and duration of disease (p = 0.69). Brain networks were constructed by thresholding correlation matrices of volumes from 80 cortical/subcortical regions (parcellated with Freesurfer v5.3 https://surfer.nmr.mgh.harvard.edu/) that were then analyzed using graph theoretical approaches. We identified reduced cortical/subcortical connectivity including bilateral hippocampus in both TLE groups, with the most significant interregional correlation increases occurring within the limbic system in LTLE and contralateral hemisphere in RTLE. Both TLE groups demonstrated less optimal topological organization, with decreased global efficiency and increased local efficiency and clustering coefficient. LTLE also displayed a more pronounced network disruption. Contrary to controls, hub nodes in both TLE groups were not distributed across whole brain, but rather found primarily in the paralimbic/limbic and temporal association cortices. Regions with increased centrality were concentrated in occipital lobes for LTLE and contralateral limbic/temporal areas for RTLE. These findings provide first evidence of altered topological organization of the whole brain volumetric network in TLE, with disruption of the coordinated patterns of cortical/subcortical morphology.