In order to efficiently and reasonably determine the permeability coefficient of engineering rock mass, the two key problems of complex seepage field simulation and objective function solution in permeability coefficient inversion are studied and improved. Firstly, a 3D refined seepage numerical model is established considering the main fault fracture zones and dam seepage control system. Secondly, combined with the support vector machine (SVM) algorithm, the complex mapping relationship between permeability coefficient of each rock layer and impervious structure and piezometer head is constructed. Finally, the particle swarm optimization (PSO) algorithm is introduced, and then the efficient optimization of permeability coefficient inversion of engineering rock mass and impervious structure is realized based on PSO-SVM. The case study of LJH arch dam shows that the inversion calculation framework based on PSO-SVM not only completes the rapid proxy calculation of the numerical model of complex seepage field, but also realizes the accurate optimization of the inversion objective function. The simulation results under the final inversion parameters reproduce the variation characteristics of the piezometric tube water level of each typical measuring point under the change of reservoir water level. In addition, with the correlation analysis between the permeability coefficient and the measured value of the corresponding piezometer, the parameter inversion method is endowed with new application value. This study is of great significance for clarifying the seepage characteristics of complex rock mass and improving the inversion efficiency of dam seepage monitoring data.