Abstract Image classification is one important task in image processing and pattern recognition. Traditional image classification methods commonly transform the image into a vector. However, in essence, image is a matrix data and using vector instead of image loses the correlations of the matrix data. To address this problem, we propose a graph-based multiple rank regression model (GMRR), which employs multiple-rank left and right projecting vectors to regress each matrix data to its label for each category. To exploit the discriminating structure of the data space, a class compactness graph is constructed to constrain these left and right projecting vectors. The extensive experimental results on image classification have demonstrated the effectiveness of our proposed method.