In the digital holographic microscopy, accurate phase restoration is essential to quantitatively test microstructure, especially in the case of dislocations and coherent noise. In this paper, a new phase restoration method with an adaptive reliability mask for phase unwrapping is proposed. By using horizontal and vertical projections to segment the unreliability map and performing the adaptive iterative thresholding on each segmented region with a global expected unreliability value, the adaptive reliability mask for phase unwrapping can be obtained. The advantage of our method is that it can distinguish the true high unreliable pixels (dislocations, spikes and noises) and structure edge pixels. Ten simulated phase maps of partial USAF 1951 resolution target with dislocations, progressive aberration coefficients and noise standard deviations are generated to evaluate the performance of the proposed method. Experimental phase data of MEMS chip is adopted to verify the effectiveness of the proposed method. Simulation and experimental results demonstrated that the proposed method can restore a more accurate and reliable sample profile, which indicates that our method is suitable for 3D topography measurement of the complicated dense microstructure.