Open-pit mining generates significant amounts of waste material, leading to the formation of large waste dumps that pose environmental risks such as land degradation and potential slope failures. The paper presents a stability analysis of waste dump slopes in open-pit mining, focusing on the Motru coalfield in Romania. To assess the stability of these dumps, the study employs the Shear Strength Reduction Method (SSRM) implemented in the COMSOL Multiphysics version 6 software, considering both associative and non-associative plasticity models. (1) Various slope angles were analyzed, and the Factor of Safety (FoS) was calculated, showing that the FoS decreases as the slope angle increases. (2) The study also demonstrates that the use of non-associative plasticity leads to lower FoS values compared to associative plasticity. (3) The results are visualized through 2D and 3D models, highlighting failure surfaces and displacement patterns, which offer insight into the rock mass behavior prior to failure. (4) The research also emphasizes the effectiveness of numerical modeling in geotechnical assessments of stability. (5) The results suggest that a non-associative flow rule should be adopted for slope stability analysis. (7) Quantitative results are obtained, with small variations compared to those obtained by LEM. (6) Dilatation angle, soil moduli, or domain changes cause differences of just a few percent and are not critical for the use of the SSRM in engineering.