A framework using peridynamic theory is developed and demonstrated for deformation and failure analysis of carbon nanotube (CNT) yarn-based structural composites. Experimental work involved tension testing of a CNT yarn/polymer composite resulting in stress–strain response up to and including failure. The as-prepared specimen was characterized using x-ray micro computed tomography (CT), which was then converted into voxel-based data with CNT yarn, polymer, and void phases as well as surface undulation. A Density-Based Spatial Clustering of Applications with Noise algorithm was applied to detect and quantify the clusters of voids, of CNT-rich, and of resin-rich regions. The voxel data, with all microstructural details, were used in peridynamic simulations. These demonstrate the critical roles of resin and void clusters and surface undulation in fracture initiation and propagation. Additional analysis was performed to construct probability density functions (PDFs) of different phases (yarn, resin, and void) with the goal of constructing synthetic virtual composite specimens. The synthetically reconstructed peridynamic models correctly captured the experimental stress–strain response. The similarities and differences between the failure (initiation and propagation) behaviors predicted by x-ray CT-based and PDF-based peridynamic model simulations are presented in detail and discussed.