Geomechanical properties of rock mass are of vital importance for design optimization and stability evaluation in geotechnical engineering. Traditionally, they are estimated by laboratory experiments, in situ tests, and empirical classification, which ignore the scale effect, and may be not representative, leading to uncertainties of parameters. Back analysis method based on the monitoring displacement data provides the state-of-the-art tool for the estimation of geomechanical parameters. This paper proposed a novel back analysis method, which combined the optimized particle swarm optimization (OPSO) algorithm with the support vector machine (SVM) algorithm. And then, we cooperate the method with finite element method (FEM) forming the comprehensive method named OPSO-SVM-ABAQUS. This method has developed the standard PSO algorithm from two aspects for more powerful optimization capability and higher optimization precision. The OPSO algorithm is then used for the estimation of the SVM hyperparameters and the optimal values of geomechanical parameters. Furthermore, the performance of the proposed algorithm is compared with other six algorithms. Finally, the geomechanical parameters of the interlayered soft and hard rocks are calculated accurately, with the average error between the back-analyzed and assumed actual parameters to be 0.91 %. And the parametric investigations are implemented with the trained model. It dominates that the proposed algorithm is feasible for reasonable parameters in geotechnical engineering under complex geological conditions.