Pavement cracking is a primary cause of early damage in asphalt pavements, and fiber-reinforcement technology is an effective method for enhancing the anti-cracking performance of pavement mixtures. However, due to the multi-scale dispersed structure of pavement mixtures, it is challenging to address cracking and damage with a single fiber type or fibers of the same scale. To investigate the toughening mechanisms and damage behavior of hybrid-fiber-modified mixtures, we analyzed the fracture process and damage behavior of these mixtures using a combination of basalt fiber and calcium sulfate whisker hybrid fiber modification, along with semicircular bending tests. Additionally, digital imaging was employed to examine the fracture interface characteristics, revealing the toughening mechanisms at play. The results demonstrated that basalt fibers effectively broaden the toughness range of the modified mixture at the same temperature, reduce mixture stiffness, increase residual load at the same displacement, and improve crack resistance in the mixture matrix. While calcium sulfate whiskers enhanced the peak load of the mixture, their high stiffness modulus was found to be detrimental to the mixture’s crack toughness. The fracture interface analysis indicated that the three-dimensionally distributed fibers form a spatial network within the mixture, restricting the relative movement of cement and aggregate, delaying crack propagation, and significantly improving the overall crack resistance of the mixture.