The finite element numerical simulation results of deep pit deformation are greatly influenced by soil layer parameters, which are crucial in determining the accuracy of deformation prediction results. This study employs the orthogonal experimental design to determine the combinations of various soil layer parameters in deep pits. Displacement values at specific measurement points were calculated using PLAXIS 3D under these varying parameter combinations to generate training samples. The nonlinear mapping ability of the Back Propagation (BP) neural network and Particle Swarm Optimization (PSO) were used for sample global optimization. Combining these with actual onsite measurements, we inversely calculate soil layer parameter values to update the input parameters for PLAXIS 3D. This allows us to conduct dynamic deformation prediction studies throughout the entire excavation process of deep pits. The results indicate that the use of the PSO-BP neural network for inverting soil layer parameters effectively enhances the convergence speed of the BP neural network model and avoids the issue of easily falling into local optimal solutions. The use of PLAXIS 3D to simulate the excavation process of the pit accurately reflects the dynamic changes in the displacement of the retaining structure, and the numerical simulation results show good agreement with the measured values. By updating the model parameters in real-time and calculating the pile displacement under different working conditions, the absolute errors between the measured and simulated values of pile top vertical displacement and pile body maximum horizontal displacement can be effectively reduced. This suggests that inverting soil layer parameters using measured values from working conditions is a feasible method for dynamically predicting the excavation process of the pit. The research results have some reference value for the selection of soil layer parameters in similar areas.