The shear strength parameters of a sliding surface are important parameters for the stability evaluation and engineering design of a landslide. To consider the randomness, uncertainty and the nonlinear characteristics of the geotechnical, the reliability back analysis method of determining shear strength parameters of a landslide based on nonlinear Mohr-Coulomb failure criteria is proposed in this study. By analyzing a slope with linear failure surface, the feasibility and validity of the proposed method are discussed in an engineering case. The proposed method provides a new choice and reference for the determination of the shear strength parameters of the sliding surface of a landslide.