The mesh stiffness plays a crucial role in influencing the performance and behavior of straight bevel gear (SBG) systems. Precisely determining the mesh stiffness enables to assess the SBG system’s dynamic behavior more accurately and anticipate potential concerns such as crack identification and noise reduction. A novel experimental method is developed, employing experimental modal analysis associated with metaheuristic algorithms and an innovative setup. This method effectively determines the mesh stiffness in healthy and cracked systems. Additionally, an analytical method based on the potential energy associated with crack modeling is proposed. Both methods are implemented on SBG systems with varying crack depths. The results obtained from the experimental method are compared with those from the analytical method, revealing good agreement between them. This demonstrates that the newly proposed experimental method effectively considers all parts of mesh stiffness and is appropriate for determining the mesh stiffness in healthy and cracked SBG systems.