By utilizing computed tomography (CT) technology, we can gain a comprehensive understanding of the specific details within the material. When combined with computational mechanics, this approach allows us to predict the structural response through numerical simulation, thereby avoiding the high experimental costs. In this study, the tensile cracking behavior of carbon–silicon carbide (C/SiC) composites is numerically simulated using the bond-based peridynamics model (BB-PD), which is based on geometric models derived from segmented images of three-dimensional (3D) CT data. To obtain results efficiently and accurately, we adopted a deep learning-based image recognition model to identify the kinds of material and then the pixel type that corresponds to the material point, which can be modeled by BB-PD for failure simulation. The numerical simulations of the composites indicate that the proposed image-based peridynamics (IB-PD) model can accurately reconstruct the actual composite microstructure. It can effectively simulate various fracture phenomena such as interfacial debonding, crack propagation affected by defects, and damage to the matrix.