In the area of pattern recognition, Low Rank Representation (LRR) is an efficient method in recovering the subspace structure of the dataset. However, LRR is unsupervised. Without any label information, LRR constructs an informative graph which is then combined with the mature graph-based semi-supervised learning (GSSL) framework to complete the classification task. In this paper, we propose a new low rank learning method which constructs the low rank representation matrix utilizing label information to obtain a more informative graph. This method integrates the low rank graph construction and the label information propagation processes together. Thus the optimization of the low rank representation and the soft label prediction function are calculated iteratively at the same time. We name this method as Semi-Supervised Low Rank Learning (SSLRL). It enhanced the classification performance of traditional LRR-Graph based SSL by 5–30% and the running time is reduced from hundreds to less than ten seconds. Based on this method, a new outlier detection strategy is presented. This strategy succeeds with an AUC of at least 93% even if the detection condition of LRR is not satisfied. The effectiveness of SSLRL is demonstrated in semi-supervised classification, outlier detection, and salient detection tasks. These extensive experimental results highlight the outperforming of our method over state-of-the-art methods.