For smart cities, location-based services (LBSs) are indispensable; however, the urban environment is typically a multipath channel and achieving high accuracy localization is challenging, especially in GNSS-denied environments. It is already known that there are two key factors that constrain IoT localization performance, namely, the presence of outliers in the inter-node ranging data and the difficulty to guarantee that ranging is carried out between all nodes, which means that we are dealing with an incomplete Euclidean distance matrix (EDM) contaminated with outliers. In this paper, we propose a robust localization framework, termed low-rank approximation-based localization (LRAbL). LRAbL enables network localization in a stepwise manner using partially observed EDMs that contain coarse noise (outliers). Specifically, the working process of LRAbL can be divided into three stages: the preprocessing of the observed EDM, aiming to eliminate outliers with large values, the use of low-rank approximation means to obtain the complete EDM, and finally the application of non-classical MDS to calculate the coordinates of the nodes in the network. To confirm the applicability of the proposed framework, extensive numerical experiments were conducted, which indicated that LRAbL was still able to achieve satisfactory localization results when the network links were sparse (only about 1/4 of the entries can be observed) and contained a certain percentage of large value outliers. In summary, our work provides a solution worthy of consideration for location-based services in the future Internet of Things.