Landslide dams, as a particular type of secondary geological disaster, can cause serious flood disasters. Therefore, accurately predicting potential dam failure processes is crucial for developing reasonable emergency response plans. Currently, several landslide dam failure models have been proposed, but most of these models do not appropriately consider the wide gradation of landslide dam materials, which is essential for accurate erosion calculations. Therefore, this research proposes a relative exposure formula under three-dimensional conditions to account for the concealment and exposure effects between large and small particles in wide-graded soil. Based on this, the incipient velocity of wide-graded soil is derived, and a mathematical model of overtopping failure of landslide dams is established. As part of the model, a numerical solution process is also developed to ensure convergence. To evaluate the performance of this new mathematical model, the discharge process of Tangjiashan landslide dam was used as a real-world case for dam failure calculation. The calculated results of key dam failure outcomes were compared with actual measurements, showing that the results of this model are largely consistent with actual measurements, with the error controlled within 10%. Additionally, the mathematical model proposed in this article was compared with two other existing dam failure models, revealing that the model proposed here has advantages in controlling the error of key dam failure outcomes, mainly due to the comprehensive consideration of erosion calculation in wide-graded soil. To further evaluate the stability of the model, the breach processes of the “11.3” Baige landslide dam and the Yigong landslide dam were also calculated, and the results showed reasonable consistency. Based on the model presented in this article, the influence of the grading width of landslide dam materials on key dam failure outcomes was discussed. It was found that grading width significantly affects the development process of breach flow. Specifically, wider grading width leads to smaller peak values of breach flow and later peak times. Additionally, the final size of the breach is affected by grading width, as it has a significant impact on erosion strength, although the sensitivity of this parameter is relatively weak. Given the significant impact of grading width on the dam failure process, especially for landslide dams containing wide-graded soil, it is essential to fully consider the wide grading of materials for accurate dam failure calculations.