Graph denoising is becoming a promising solution for robust graph embedding, which aims to construct an ideal network by removing noisy edges. Currently, the mainstream denoising approaches often build upon the low-rank assumption by removing high-rank harmful edges with singular value decomposition. Although this approach often allows yielding good graph embedding, the high time complexity limits its application to large-scale networks. In this paper, we propose an effective and efficient algorithm for robust graph embedding: Latent Feature-driven Graph Denoising (LFGD). The basic idea is to leverage node latent features to construct an ideal low-rank network by exploiting the relationship between topology and feature information. To this end, the original features are first mapped into a latent space, and then a low-rank network is reconstructed by imposing the semantic preservation loss and structure loss (including regression loss and sparsity constraint). In addition, we propose a sampling-based strategy to further speed up the proposed method, which finally results in linear time complexity. LFGD works independently of downstream task, which makes the denoised structure more general and reliable. Extensive experiments have demonstrated the superiority of the LFGD to many state-of-the-art algorithms under various attacks in terms of node classification, robustness and running time. Our code is available at: https://github.com/hanwangme/LFGD.