Semi-flexible pavement (SFP) is designed for heavy traffic but its susceptibility to cracking hinders its development and application. This study aims to investigate the cracking characteristics and failure modes of SFP materials through macroscopic tests and image processing, thereby guiding anti-cracking technology. First, the effects of gradation and bitumen content on cracking process curves were analyzed using three-point bending tests across a wide temperature range. Second, the variations in maximum tensile strain, tensile strength, and fracture energy were comparatively examined. Finally, the morphology of specimens at crack initiation and after complete fracture was analyzed using image processing techniques to quantify the failure modes. The results indicate that the SFP material exhibits elastic development and brittle damage at the low temperature, and viscoelastic development and ductile damage at medium and the high temperature. At the low temperature, the changes in maximum tensile strain and fracture energy are consistent. The primary weak points at crack initiation include internal cement failure, cement-bitumen interface failure, and bitumen-aggregate interface failure. After complete fracture, internal cement damage and bitumen-bitumen/aggregate interface damage are dominant. To improve low-temperature cracking resistance of SFP, it is recommended to increase the maximum aggregate size, add fine aggregates, and increase the bitumen content. Additionally, enhancing the internal cohesion and external bonding properties of bitumen can mitigate the negative effects of increased bitumen consumption. This study provides guidance on designing the SFP material to enhance its crack resistance.