Rock strength parameters are essential to understanding the rock failure mechanism and safely constructing rock excavation. It is a challenging problem for determining the rock failure criterion and its parameters due to the complexity of rock media. This study adopts an artificial bee colony (ABC) algorithm to determine the Hoek-Brown failure criterion, widely used in rock engineering practice, based on experimental data. The ABC-based approach is presented in detail and applied to a collection of experimental data collected from the literature. The ABC-based approach successfully determines the Hoek-Brown failure criterion, and the determined failure envelope is in excellent agreement with the measured curve. The maximum relative error obtained by ABC is only 2.15% and is far less than the 12.24% obtained by the traditional method. Then, the developed approach is applied to the Goupitan Hydropower Station, China, and determines the rheological parameters of soft rock based on the Burgers model. The deformation of an experiment located in the Goupitan Hydropower Station is evaluated based on obtained parameters by the developed approach. The predicted deformation matches the monitored displacement in the field. The obtained parameters of the failure criterion characterize the mechanical behavior of rock mass well. Thus, the method used provides a reliable and robust approach to determining the mechanical parameters of the failure criterion.