Panic disorder (PD) is an anxiety disorder that impairs life quality and social function and is associated with distributed brain regions. However, the alteration of the structural network remains unclear in PD patients. This study explored the specific characteristics of the structural brain network in patients with PD by graph theory analysis of diffusion tensor images (DTI). A total of 81 PD patients and 48 matched healthy controls were recruited for this study. The structural networks were constructed, and the network topological properties for individuals were estimated. At the global level, the network efficiency was higher, while the shortest path length and clustering coefficient were lower in the PD group compared to the healthy control (HC) group. At the nodal level, the PD group showed a widespread higher nodal efficiency and lower average shortest path length in the prefrontal, sensorimotor, limbic, insula, and cerebellum regions. Overall, the current results showed that the alteration of information processing in the fear network might play a role in the pathophysiology of PD.