This paper presents a method for extracting the digital elevation model (DEM) of forested areas from polarimetric interferometric synthetic aperture radar (PolInSAR) data. The method models the ground phase as a Von Mises distribution, with a mean of the topographic phase computed from an external DEM. By combining the prior distribution of the ground phase with the complex Wishart distribution of the observation covariance matrix, we derive the maximum a posterior (MAP) inversion method based on the RVoG model and analyze its Cramer–Rao Lower Bound (CRLB). Furthermore, considering the characteristics of the objective function, this paper introduces a Four-Step Optimization (FSO) method based on gradient optimization, which solves the inefficiency problem caused by exhaustive search in solving ground phase using the MAP method. The method is validated using spaceborne L-band repeat-pass SAOCOM data from a test forest area. The test results for FSO indicate that it is approximately 5.6 times faster than traditional methods without compromising accuracy. Simultaneously, the experimental results demonstrate that the method effectively solves the problem of elevation jumps in DEM inversion when modeling the ground phase with the Gaussian distribution. ICESAT-2 data are used to evaluate the accuracy of the inverted DEM, revealing that our method improves the root mean square error (RMSE) by about 23.6% compared to the traditional methods.