Small variation in shear strength parameters results in remarkable changes in the safety factor (SF) of a rock slope. In this regard, rock mass strength of a failed slope can be estimated with reasonable accuracy using back analysis approach. The conventional back analysis (BA) procedure is based on trial and error approach on the mb and s parameters of Hoek–Brown failure criterion in conjunction with the Geological Strength Index (GSI) classification system. A new BA procedure presented in this paper, a functional relationship, that indicted all possible combinations of mb and s constants, was derived using stability analysis of a slope assuming that the SF of the slope is equal to unity. Then, another relationship is established between mb and s based on the magnitude of GSI. Intersection of the two relationships when plotting on a graph, defines a unique value for mb and s. Finally, effect of the constant a in the generalized Hoek–Brown failure criterion on the GSI value obtained from the method BA is discussed and its effect is considered in the BA approach. The most advantage of the proposed method is that a reliable GSI is obtained in a relatively short time. For better explanation of the new BA procedure, this method is used for the back analysis of a failed slope along the national railway of Iran. Comparing results obtained from the new BA approach and the traditional back analysis method well-specified which the new proposed BA approach increases the accuracy of parameter estimation.