Atomic force microscope (AFM) is one of the most versatile and powerful devices capable of producing high-resolution images of nanomaterial. Many researchers are widely investigating to improve the scanning speed and image quality of AFM by proposing different techniques. Here, we aim to present a novel approach based on the smooth orthogonal decomposition for the estimation of the surface topography in AFM. The technique proposed in this research not only eliminates the need for a closed-loop controller but also acquires the surface three-dimensional shape (topography) very quickly and accurately. The proposed technique relies on the fact that in the tapping mode of atomic force microscopy, the tip displacements are very fast compared to the topography changes, and the surface topography as a slowly varying parameter can be estimated using the smooth orthogonal decomposition algorithm. To this aim, the state space is reconstructed based on Takens’ theorem and used only the tip displacement measurement data. According to Takens’ theorem, using the delay time and embedding dimension parameters, we are able to create a system in which dynamical behaviors are similar to the original system. The results demonstrate that the proposed estimation approach is robust to noise and does not require large data or computational resources to be implemented. Also, the performance of the proposed method is appropriate for any type of force interactions between the tip and sample.